# Terraform & Infrastructure as Code — Part 5: Providers, Data Sources & Provisioners Deep Dive

> **Series:** Terraform & Infrastructure as Code (5 of 9)
> **Part 1:** `01-fundamentals-and-workflow.md` — Fundamentals, HCL & the Plan/Apply Workflow
> **Part 2:** `02-state-management-and-remote-backends.md` — State Management & Remote Backends
> **Part 3:** `03-modules-and-reusable-design.md` — Modules & Reusable Infrastructure Design
> **Part 4:** `04-workspaces-and-environments.md` — Workspaces, Environments & Real-World Repository Structure
> **Part 5:** This file — Providers, Data Sources & Provisioners Deep Dive
> **Part 6:** `06-drift-detection-import-and-refactoring.md` — Drift Detection, Import & Refactoring Existing Infrastructure
> **Part 7:** `07-testing-terraform.md` — Testing Terraform: Static Analysis, the Native Test Framework & Terratest
> **Part 8:** `08-cicd-for-terraform.md` — CI/CD for Terraform: Pipelines, Gates & GitOps for Infrastructure
> **Part 9:** `09-governance-cost-and-multi-cloud-at-scale.md` — Terraform at Team Scale: Governance, Cost & Multi-Cloud Patterns
> **Questions:** `questions.md`

Assumes you're comfortable with Part 1's provider basics and Part 4's cross-account `assume_role` pattern —
this chapter goes deeper on provider aliasing, adds the two mechanisms for reading and (narrowly) acting on
infrastructure Terraform doesn't fully own, and is honest about provisioners' real, narrow, correctly-scoped
place in a mature configuration.

## Table of Contents

1. [Multiple Provider Configurations, Revisited](#multiple-provider-configurations-revisited)
2. [The Provider-Depends-on-a-Resource Trap](#the-provider-depends-on-a-resource-trap)
3. [Data Sources — Reading What Terraform Doesn't Manage](#data-sources--reading-what-terraform-doesnt-manage)
4. [Data Source Gotchas: Zero Results, Multiple Results](#data-source-gotchas-zero-results-multiple-results)
5. [Choosing the Right Input Mechanism](#choosing-the-right-input-mechanism)
6. [Filtering Data Sources for Precision, Not Convenience](#filtering-data-sources-for-precision-not-convenience)
7. [The local-exec Provisioner](#the-local-exec-provisioner)
8. [The remote-exec Provisioner and Connection Blocks](#the-remote-exec-provisioner-and-connection-blocks)
9. [The file Provisioner](#the-file-provisioner)
10. [Creation-Time vs. Destroy-Time Provisioners](#creation-time-vs-destroy-time-provisioners)
11. [on_failure — Controlling What Happens When a Provisioner Fails](#on_failure--controlling-what-happens-when-a-provisioner-fails)
12. [Why Provisioners Are a Last Resort — the Alternatives, in Order](#why-provisioners-are-a-last-resort--the-alternatives-in-order)
13. [terraform_data — the Modern Resourceless Anchor](#terraform_data--the-modern-resourceless-anchor)
14. [The External Data Source — a Careful Escape Hatch](#the-external-data-source--a-careful-escape-hatch)
15. [Generating Config Files With templatefile()](#generating-config-files-with-templatefile)
16. [Worked Scenario: a Flaky remote-exec That Passed in Dev, Failed in Prod](#worked-scenario-a-flaky-remote-exec-that-passed-in-dev-failed-in-prod)
17. [Worked Scenario: Replacing a Provisioner With Packer and user_data](#worked-scenario-replacing-a-provisioner-with-packer-and-user_data)
18. [Worked Scenario: a Data Source Silently Matching the Wrong Resource](#worked-scenario-a-data-source-silently-matching-the-wrong-resource)
19. [Part 5 CLI Cheat Sheet](#part-5-cli-cheat-sheet)
20. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
21. [Worked Practice Problems](#worked-practice-problems)
22. [Summary and What's Next](#summary-and-whats-next)

---

## Multiple Provider Configurations, Revisited

**Part 1 introduced provider aliasing briefly; at real scale — the multi-account, multi-region reality Part
4 built — a root module routinely juggles half a dozen or more distinct provider configurations, and keeping
them legible is its own discipline.**

```hcl
provider "aws" {
  region = "us-east-1"
}

provider "aws" {
  alias  = "west"
  region = "us-west-2"
}

provider "aws" {
  alias  = "prod"
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::444455556666:role/terraform-prod-deployer"
  }
}
```

```hcl
resource "aws_s3_bucket" "replica" {
  provider = aws.west
  bucket   = "checkout-service-replica"
}
```

Every resource and data source defaults to the unaliased provider unless a `provider = aws.<alias>` argument
says otherwise — an easy detail to miss when skimming a large configuration, since the absence of a
`provider` argument is itself meaningful (it means "the default"), not just an omission.

> [!TIP]
> **Best practice**: name aliases for what they *mean*, not just where they point — `aws.prod` or
> `aws.dr_region` reads clearly at every call site; `aws.provider2` or `aws.b` forces a reader to go find the
> `provider` block just to understand what a resource is actually targeting. Part 4's naming-convention
> discipline applies to provider aliases too, not just resource names.

## The Provider-Depends-on-a-Resource Trap

**A sharp, frequently-hit gotcha: configuring one provider (commonly `kubernetes` or `helm`) using
attributes computed from a resource created by a *different* provider in the same apply — an EKS cluster's
endpoint feeding the `kubernetes` provider's `host` argument, for instance — looks like it should work, and
often does, right up until it doesn't.**

```hcl
# The tempting, fragile pattern — DON'T structure it this way
resource "aws_eks_cluster" "main" {
  # ...
}

provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

resource "kubernetes_namespace" "checkout" {
  metadata { name = "checkout" }
}
```

The problem is structural, not a syntax mistake: **Terraform's provider configurations are not nodes in the
dependency graph** the way resources are. Terraform can't guarantee the EKS cluster is fully ready — DNS
propagated, API server actually reachable — at the exact moment it evaluates the `kubernetes` provider block,
because providers are configured once, early, outside the normal resource-ordering guarantees. The practical
symptom is intermittent: it works reliably on a fresh cluster's *second* apply (state already exists, timing
no longer matters), and fails unpredictably on the *first* apply that creates the cluster and tries to use it
in the same run — exactly the kind of "works in my testing, fails for someone else" bug this chapter keeps
returning to.

```mermaid
flowchart TD
    Apply["Single apply,<br/>two providers"] --> EKS["aws_eks_cluster.main<br/>created"]
    Apply --> K8sProvider["kubernetes provider<br/>configured — NOT graph-ordered<br/>relative to EKS cluster creation"]
    K8sProvider --> Race{"Is the cluster's API<br/>actually reachable yet?"}
    Race -->|"Sometimes yes"| Works["kubernetes_namespace<br/>succeeds"]
    Race -->|"Sometimes no"| Fails["Connection refused —<br/>non-deterministic, timing-dependent"]

    classDef risk fill:#fbe8e6,stroke:#b3261e,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    class Race,Fails risk
    class Works ok
```

**The fix, confirmed as the standard resolution**: split cluster infrastructure and in-cluster resources into
**two separate states** — exactly Part 2's state-splitting discipline, applied to this specific case. A
`cluster-infrastructure` state creates the EKS cluster with no `kubernetes`/`helm` provider anywhere in it;
a separate `cluster-services` state configures the `kubernetes` provider using `terraform_remote_state` (Part
2) to read the already-fully-created cluster's endpoint and credentials, applied strictly afterward.

```hcl
# cluster-services/main.tf — a SEPARATE state from cluster-infrastructure
data "terraform_remote_state" "cluster" {
  backend = "s3"
  config = {
    bucket = "meridian-platform-tfstate"
    key    = "shared/eks-cluster/terraform.tfstate"
    region = "us-east-1"
  }
}

provider "kubernetes" {
  host                   = data.terraform_remote_state.cluster.outputs.endpoint
  cluster_ca_certificate = data.terraform_remote_state.cluster.outputs.ca_certificate
  token                  = data.aws_eks_cluster_auth.main.token
}
```

> [!IMPORTANT]
> This is the single most common way the `kubernetes`/`helm` providers bite a team standing up their first
> EKS cluster with Terraform, and it's worth committing to memory as a rule, not just this one example: never
> configure a provider using attributes computed from a resource created in the *same* apply, when that
> provider's own resources are also applied in that run. Split into separate states (Part 2) whenever this
> shape appears, regardless of which two providers are involved.

## Data Sources — Reading What Terraform Doesn't Manage

**A `data` block queries an existing object through a provider — read-only, no create/update/delete — and
is the correct mechanism for referencing anything that already exists but isn't (and shouldn't be) tracked
in this configuration's own state.**

```hcl
data "aws_vpc" "default" {
  filter {
    name   = "tag:Name"
    values = ["meridian-prod-vpc"]
  }
}

data "aws_ami" "app" {
  most_recent = true
  owners      = ["self"]
  filter {
    name   = "name"
    values = ["checkout-app-*"]
  }
}

resource "aws_instance" "worker" {
  ami           = data.aws_ami.app.id
  subnet_id     = data.aws_vpc.default.id
  instance_type = "t3.micro"
}
```

Data sources are re-evaluated on every `plan` (part of the refresh step from Part 1) — they always reflect
current reality, not a cached snapshot from whenever the configuration was first written, which is exactly
why `most_recent = true` above means "give me whatever the newest matching AMI is *right now*," re-checked
every single run.

| Question | Points toward |
|---|---|
| Does Terraform need to create/modify/delete this object? | `resource` |
| Does Terraform only need to *read* something that already exists, managed elsewhere (or by a human)? | `data` |
| Is the value a human-supplied parameter with no external system to look up? | `variable` |

> [!WARNING]
> A data source can return **sensitive values** exactly like a resource can (a database's endpoint, a secret
> stored in a secrets manager and looked up via `data "aws_secretsmanager_secret_version"`), and Part 1's
> caveat about `sensitive = true` applies identically: it redacts CLI output, not the state file's plain-JSON
> contents. Treat a data source that reads a genuine secret with the same state-encryption and access-control
> discipline Part 2 established for resources.

## Data Source Gotchas: Zero Results, Multiple Results

**Unlike a resource (which Terraform created and therefore knows exists), a data source's filter can
legitimately match zero results or more than one — and the failure mode for each is different, and both are
worth handling deliberately rather than discovering at a bad moment.**

```hcl
data "aws_ami" "app" {
  most_recent = true          # Resolves ambiguity: pick the newest of several matches
  owners      = ["self"]
  filter {
    name   = "name"
    values = ["checkout-app-*"]
  }
}
```

Without `most_recent = true`, a filter matching two or more AMIs fails `plan` outright with a "your query
returned more than one result" error — which is often exactly the right behavior (an ambiguous match
shouldn't silently pick one), but only if the configuration is actually prepared to handle it. A filter
matching **zero** results fails differently — a "no matching AMI found" error — which typically indicates
either a genuine typo in the filter or (more insidiously) an environment where the expected object simply
hasn't been created yet.

```hcl
data "aws_ami" "app" {
  most_recent = true
  owners      = ["self"]
  filter {
    name   = "name"
    values = ["checkout-app-*"]
  }
}

locals {
  # try() catches a data source lookup failure and falls back gracefully,
  # rather than letting the whole plan fail outright.
  app_ami_id = try(data.aws_ami.app.id, var.fallback_ami_id)
}
```

> [!TIP]
> **Best practice**: reach for `most_recent = true` (or an equivalently explicit tie-breaker) whenever a
> data source's filter *could* plausibly match more than one real object — an unhandled "multiple results"
> error at `plan` time, in the middle of a CI run, is a worse place to discover an ambiguous filter than
> writing the tie-breaker up front. Reserve `try()` for genuinely optional lookups where a sensible fallback
> exists; using it to silently swallow a lookup that *should* have found something just delays the same
> problem to wherever the fallback value causes a less obvious failure downstream.

## Choosing the Right Input Mechanism

**By this point in the series, four distinct ways of getting a value into a configuration all exist — data
sources, `terraform_remote_state` (Part 2), variables, and module outputs (Part 3) — and picking the wrong
one for a given situation creates real, if subtle, coupling problems.**

| Mechanism | Reads from | Right for |
|---|---|---|
| `variable` | Explicit caller input | A value with no external source of truth — an environment name, an instance size choice |
| `data` source | A live query against a provider's API | An object that exists in the real platform but isn't tracked in *any* Terraform state (a pre-existing AMI, a DNS zone created outside Terraform) |
| `terraform_remote_state` | Another Terraform configuration's own state | An object Terraform-managed elsewhere, when that other configuration's outputs are the authoritative source |
| Module `output` | The calling module's resource attributes | An object this same configuration/call graph creates |

**The subtle trap**: using a `data` source to look up something that's actually managed by *another
Terraform configuration* (querying `aws_vpc` by tag, instead of consuming that VPC's own
`terraform_remote_state` output) works, but it creates an invisible coupling — nothing in either
configuration declares the dependency explicitly, so a rename of that tag, or a change to which VPC bears
it, silently breaks the data-source lookup with no warning at either config's own review time. Preferring
`terraform_remote_state` for anything genuinely Terraform-managed elsewhere keeps that dependency explicit
and versioned, per Part 2's outputs-as-contract discipline.

## Filtering Data Sources for Precision, Not Convenience

A loosely-written filter is a common source of the "wrong environment" class of incident this series has
returned to repeatedly (Part 2's destroy scenario, Part 4's credential scenario) — a data source that's
*supposed* to find `checkout-service`'s security group but is filtered loosely enough to also match
`catalog-service`'s, in some environment where naming happens to collide, fails silently in the worst way:
it returns exactly one result, so none of the previous section's error handling ever triggers.

```hcl
# Loose — plausible collision risk across services/environments
data "aws_security_group" "app" {
  filter {
    name   = "tag:Team"
    values = ["platform"]
  }
}

# Precise — scoped to exactly the intended resource
data "aws_security_group" "app" {
  filter {
    name   = "tag:Name"
    values = ["${local.name_prefix}-checkout-sg"]
  }
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.this.id]
  }
}
```

> [!IMPORTANT]
> A data source that matches "successfully" (exactly one result) is not proof it matched the *intended*
> object — a filter loose enough to admit ambiguity can still resolve to exactly one wrong match under
> different real-world conditions than whatever was true when it was first written and tested. Filter on the
> most specific identifying attributes available (a fully-qualified name including the environment prefix, a
> resource ID, a combination of tag plus VPC scope) rather than a single loosely-shared tag.

## The local-exec Provisioner

**`local-exec` runs a command on the machine executing Terraform itself — not on any remote resource — and
needs no connection configuration, which makes it the least risky of the provisioner family, though still
subject to every caveat this chapter builds toward.**

```hcl
resource "aws_instance" "worker" {
  # ...
  provisioner "local-exec" {
    command = "echo ${self.private_ip} >> inventory.txt"
  }
}
```

A common, more defensible `local-exec` use: triggering a genuinely external, non-Terraform-native action
after a resource is created — notifying a webhook, updating an external inventory system that has no
Terraform provider of its own. Even here, `local-exec` output isn't tracked in state and doesn't participate
in drift detection (Part 6) — Terraform has no way to know if the command's effect was later undone outside
of Terraform, which is the core reason this entire chapter treats provisioners as a narrow tool, not a
general-purpose escape hatch.

## The remote-exec Provisioner and Connection Blocks

**`remote-exec` connects into a freshly-created resource (via SSH for Linux, WinRM for Windows) and runs
commands directly on it — the provisioner most often reached for to "finish configuring" a VM right after
Terraform creates it, and the one Part 1's original guidance called out as needing the strongest
justification.**

```hcl
resource "aws_instance" "worker" {
  ami           = data.aws_ami.app.id
  instance_type = "t3.micro"
  key_name      = aws_key_pair.deployer.key_name

  connection {
    type        = "ssh"
    user        = "ec2-user"
    host        = self.public_ip
    private_key = file("~/.ssh/id_rsa")
  }

  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install -y nginx",
      "sudo systemctl start nginx",
    ]
  }
}
```

Note `self.public_ip`, not `aws_instance.worker.public_ip` — referencing the resource's own address directly
from inside its own block would create a dependency cycle (the resource depending on its own completed
creation to compute its own configuration); `self` refers to the current resource's own attributes without
that cycle. A `connection` block can also be set at the provisioner level instead of the resource level, with
provisioner-level settings taking precedence when both exist — useful when a resource has multiple
provisioners needing different connection details.

> [!WARNING]
> A `remote-exec` provisioner's success is entirely dependent on network reachability and timing that
> `terraform plan` cannot foresee — a security group not yet propagated, an SSH daemon not yet started, a
> transient network blip — none of which show up as a problem until `apply` is actually running, often
> minutes into what looked like a clean plan. This unpredictability, more than any other single factor, is
> why HashiCorp's own guidance treats `remote-exec` as a genuine last resort.

## The file Provisioner

**`file` copies a local file or directory to a remote resource over the same connection mechanism as
`remote-exec` — narrower in scope, but carrying the identical connection-reliability risk.**

```hcl
resource "aws_instance" "worker" {
  # ...
  connection {
    type        = "ssh"
    user        = "ec2-user"
    host        = self.public_ip
    private_key = file("~/.ssh/id_rsa")
  }

  provisioner "file" {
    source      = "config/app.conf"
    destination = "/etc/app/app.conf"
  }
}
```

The far more common, more reliable alternative for exactly this use case is baking the file into a
pre-built image (Part 5's alternatives section, and this chapter's Packer scenario, cover this in depth) or
passing its content through `user_data`/cloud-init instead — both avoid the runtime SSH dependency entirely,
since the content is present the moment the instance boots rather than requiring a separate, timing-sensitive
connection after creation.

## Creation-Time vs. Destroy-Time Provisioners

**By default, a provisioner runs at resource *creation* — adding `when = destroy` flips it to run
immediately before the resource is destroyed instead, for cleanup actions that genuinely need to happen on
the way out.**

```hcl
resource "aws_instance" "worker" {
  # ...
  provisioner "local-exec" {
    when    = destroy
    command = "curl -X POST https://inventory.internal/deregister -d 'host=${self.private_ip}'"
  }
}
```

A destroy-time provisioner runs using the resource's **last-known state**, not any updated configuration —
which matters because if the resource was already partially modified or its state is stale, the provisioner
executes against whatever attributes state still holds, not necessarily current reality. Destroy-time
provisioners also only run on an explicit `destroy` of that specific resource — they do **not** run if the
resource is removed from state via `terraform state rm` (Part 2), since that command deliberately bypasses
the normal destroy lifecycle entirely.

> [!NOTE]
> A destroy-time `local-exec` deregistering a host from an external inventory system (as shown above) is one
> of the more defensible provisioner use cases in this entire chapter — there's often no cleaner mechanism
> for "notify something outside Terraform's control exactly when this resource is going away," and the
> action itself (a webhook call) carries little of `remote-exec`'s connection-reliability risk.

## on_failure — Controlling What Happens When a Provisioner Fails

**By default, a failed provisioner marks the resource as `tainted` (Part 6 covers tainting in depth) and
fails the entire `apply` — `on_failure = continue` changes that, letting the apply proceed despite the
provisioner's failure, appropriate only for genuinely non-critical provisioner actions.**

```hcl
provisioner "local-exec" {
  command    = "curl -sf https://monitoring.internal/notify -d 'event=instance_created'"
  on_failure = continue
}
```

| Setting | Behavior on provisioner failure | Use for |
|---|---|---|
| `fail` (default) | Apply stops, resource marked tainted | Anything the resource genuinely isn't correctly configured without |
| `continue` | Apply proceeds, failure logged but non-fatal | A best-effort side action (a notification) where failure shouldn't block real infrastructure from existing |

> [!CAUTION]
> Reaching for `on_failure = continue` to make an annoying, flaky provisioner failure stop blocking applies
> — without fixing *why* it's flaky — just hides the underlying reliability problem instead of solving it.
> If a provisioner fails often enough that `continue` feels necessary to keep applies unblocked, that's a
> strong signal the action belongs in this chapter's alternatives section instead, not evidence that
> `continue` is the right permanent setting.

## Why Provisioners Are a Last Resort — the Alternatives, in Order

**Every provisioner type covered so far shares the same structural weakness: Terraform has no visibility
into what a provisioner script actually does, can't show its effect in a `plan`, doesn't track its result in
state, and can't detect if that effect is later undone — which is exactly why HashiCorp's own documentation,
and this chapter, treat provisioners as a last resort rather than a normal tool.**

```mermaid
flowchart TD
    Need["Need: configure something<br/>on/around a resource<br/>Terraform creates"] --> Native{"Does the provider have<br/>a native resource for this?"}
    Native -->|"Yes"| UseNative["Use the native resource —<br/>tracked in state, visible in<br/>plan, no provisioner needed"]
    Native -->|"No"| CloudInit{"Can cloud-init/user_data<br/>express it at boot time?"}
    CloudInit -->|"Yes"| UseCloudInit["user_data / cloud-init —<br/>no SSH dependency, runs<br/>at first boot"]
    CloudInit -->|"No"| Image{"Would a pre-built image<br/>(Packer) solve it once,<br/>for every future instance?"}
    Image -->|"Yes"| UsePacker["Bake it into an AMI/image<br/>with Packer — zero runtime<br/>dependency at all"]
    Image -->|"No"| ConfigMgmt{"Is this ongoing config<br/>management, not one-time setup?"}
    ConfigMgmt -->|"Yes"| UseAnsible["Hand off to a config-<br/>management tool (Ansible) —<br/>the right tool for this job"]
    ConfigMgmt -->|"No"| Provisioner["Only now: a provisioner,<br/>as a genuine last resort"]

    classDef good fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef last fill:#fbe8e6,stroke:#b3261e,color:#10161c
    class UseNative,UseCloudInit,UsePacker,UseAnsible good
    class Provisioner last
```

**This chapter's caption**: the decision tree deliberately exhausts four better options before ever reaching
a provisioner — each earlier branch trades away exactly the invisibility problem provisioners carry, in
exchange for its own tradeoff (an image rebuild pipeline, a second config-management tool) that's usually
still cheaper than the reliability and visibility cost of a provisioner in production.

| Alternative | Solves | Tradeoff |
|---|---|---|
| A provider-native resource | Anything the provider already models as a first-class resource | None — always prefer this when it exists |
| `user_data` / cloud-init | One-time boot-time setup, expressed declaratively | Limited to what cloud-init's own directives support |
| Packer-built image | Anything that should be baked in once, reused across many instances | A separate build pipeline to maintain |
| Ansible (or equivalent) | Ongoing configuration management, not one-time setup | A second tool with its own inventory/execution model |
| A provisioner | Whatever's left after genuinely exhausting the above | No plan visibility, no state tracking, no drift detection |

## terraform_data — the Modern Resourceless Anchor

**`terraform_data` (replacing the older `null_resource` pattern) is a provider-agnostic, built-in resource
type that exists purely to hold a provisioner, a `triggers_replace` value, or computed values — without
needing any real cloud resource to attach to.**

```hcl
resource "terraform_data" "cluster_bootstrap" {
  triggers_replace = [aws_eks_cluster.main.id]

  provisioner "local-exec" {
    command = "aws eks update-kubeconfig --name ${aws_eks_cluster.main.name}"
  }
}
```

`triggers_replace` explicitly controls when this resource (and therefore its provisioner) re-runs — here,
any change to the EKS cluster's ID forces `terraform_data` to be replaced, re-running the provisioner. This
is the correct modern home for "I need a provisioner but there's no real resource to attach it to" — exactly
the same job `null_resource` used to do, with a clearer, purpose-built name and no confusing implication
that it represents an actual "null" provider resource.

> [!NOTE]
> Reaching for `terraform_data` doesn't change any of this chapter's caution about provisioners themselves —
> it's the correct *container* for a provisioner that has no natural resource to live on, not an exception to
> "exhaust the alternatives first."

## The External Data Source — a Careful Escape Hatch

**`data "external"` executes an external program and reads its stdout (JSON) as the data source's result —
the read-only counterpart to a provisioner, for querying a system with no Terraform provider of its own.**

```hcl
data "external" "vault_secret" {
  program = ["python3", "${path.module}/scripts/fetch-secret.py"]
  query = {
    secret_path = "secret/checkout/db-password"
  }
}

resource "aws_db_instance" "checkout" {
  # ...
  password = data.external.vault_secret.result.password
}
```

The external program must output a single flat JSON object of string values on stdout — no nested
structures, no other output types — which is a real constraint worth knowing before reaching for this.
Like any data source, it re-runs on every `plan`, meaning the external program executes on every single
plan/apply, not just once — a slow or unreliable external script becomes a slow or unreliable Terraform
workflow, exactly the same reliability transfer risk `remote-exec` carries, just on the read side instead of
the write side.

> [!WARNING]
> `data "external"` is a genuine escape hatch, not a substitute for a real provider. Before reaching for it,
> check whether a purpose-built provider already exists for the system in question (HashiCorp publishes a
> `vault` provider, for instance, which is the correct choice over a hand-rolled external script for exactly
> the secret-lookup example above) — a real provider gets proper schema validation, better error messages,
> and none of the flat-JSON-only constraint.

## Generating Config Files With templatefile()

**`templatefile()` renders a local template file with variables substituted in — the mechanism behind the
`user_data` example in this chapter's Packer scenario, and the standard way to generate any config file
content (a `user_data` script, an application config, a systemd unit) from within Terraform without a
provisioner at all.**

```hcl
# templates/user-data.sh.tpl
#!/bin/bash
echo "ENVIRONMENT=${environment}" >> /etc/app/env
echo "CONSUL_ADDR=${consul_addr}" >> /etc/app/env
systemctl restart checkout-app
```

```hcl
resource "aws_instance" "worker" {
  ami           = data.aws_ami.app.id
  instance_type = "t3.micro"
  user_data = templatefile("${path.module}/templates/user-data.sh.tpl", {
    environment = var.environment
    consul_addr = var.consul_addr
  })
}
```

Because `templatefile()`'s result is a plain string argument, it's fully visible in `terraform plan` output
— any change to the rendered content shows as a diff, exactly like any other argument, with none of a
provisioner's plan-invisibility. Templates support full Terraform expression syntax inside `${...}`,
including conditionals and `for` loops, which covers the overwhelming majority of what a `remote-exec`
inline script would otherwise have tried to construct imperatively.

| Mechanism | Visible in plan? | Tracked in state? | Runs |
|---|---|---|---|
| `templatefile()` + `user_data` | Yes — a diffable string | Yes — as a resource argument | At instance boot, no SSH needed |
| `provisioner "remote-exec"` | No | No | After creation, over SSH, timing-dependent |

> [!TIP]
> **Best practice**: default to `templatefile()` feeding `user_data` (or an equivalent provider-native
> mechanism) for any boot-time configuration that can be expressed as a script or config file — reserve
> `remote-exec` for the narrow residue of cases that genuinely need to run *after* boot, against a
> long-running instance's already-established state, which `user_data` (a first-boot-only mechanism) can't
> reach.

## Worked Scenario: a Flaky remote-exec That Passed in Dev, Failed in Prod

```mermaid
sequenceDiagram
    participant TF as Terraform
    participant API as Cloud provider API
    participant SG as Security group
    participant Inst as New instance

    TF->>API: Create instance
    API-->>TF: Instance created, public_ip known
    TF->>API: Create security group rule (port 22)
    API-->>TF: Rule creation API call returns success
    Note over SG: Rule enforcement can lag<br/>the API's "success" response<br/>by a few seconds — the actual<br/>propagation gap this scenario hits
    TF->>Inst: remote-exec attempts SSH connection
    alt Rule already enforced
        Inst-->>TF: Connection succeeds
    else Rule not yet enforced
        Inst-->>TF: Connection refused / timeout
    end
```

**This chapter's caption**: the API call for the security group rule returning "success" and the rule
actually being *enforced* are two different moments in time — `remote-exec`'s SSH attempt races against
that gap with no way for Terraform to know which side of it any given apply will land on.

An engineer added a `remote-exec` provisioner to install and configure a monitoring agent immediately after
instance creation, tested repeatedly in `dev` with consistent success. In `prod`, the same configuration
failed intermittently — roughly one apply in five — with an SSH connection timeout.

The immediate cause: `prod`'s security group (correctly, per Part 4's tighter production hardening) took
slightly longer to fully propagate than `dev`'s more permissive one, occasionally still blocking port 22
in the brief window right after instance creation when Terraform's `remote-exec` first attempted to connect.
The underlying condition: nothing in the configuration accounted for the real-world timing gap between "the
API call creating the security group rule returned success" and "the rule is actually enforced and
reachable" — a gap that happened to be small enough to never manifest in `dev`'s laxer setup, and large
enough to intermittently bite in `prod`'s tighter one. The team's fix followed this chapter's own decision
tree directly: the monitoring agent install moved into the base AMI via Packer (this chapter's next
scenario), eliminating the runtime SSH dependency — and therefore the timing race — entirely.

## Worked Scenario: Replacing a Provisioner With Packer and user_data

Following the previous scenario's fix, the platform team built a Packer pipeline producing a
`checkout-app-*`-named AMI (the exact AMI this chapter's data-source examples query) with the monitoring
agent, base packages, and application runtime already baked in — reducing what Terraform's own
`aws_instance` resource needs to do at boot time to a small `user_data` script handling only genuinely
per-instance, runtime-dependent configuration (registering with the current environment's specific service
discovery endpoint):

```hcl
resource "aws_instance" "worker" {
  ami           = data.aws_ami.app.id   # Packer-built, monitoring agent pre-installed
  instance_type = "t3.micro"

  user_data = templatefile("${path.module}/templates/user-data.sh.tpl", {
    environment = var.environment
    consul_addr = var.consul_addr
  })
}
```

The net effect matched this chapter's decision-tree prediction closely: zero SSH-timing-related apply
failures since the migration, faster instance boot-to-ready time (no waiting on a runtime SSH connection and
package installation), and a `terraform plan` that now shows exactly what `user_data` will run (it's a
visible, diffable string argument) instead of a provisioner's opaque, unplannable side effect.

## Worked Scenario: a Data Source Silently Matching the Wrong Resource

Following an internal reorg, `catalog-service` briefly shared a transitional security group tagged
`Team = "platform"` with `checkout-service` during a migration window — a temporary, deliberate choice by
the networking team. An unrelated `checkout-service` module's data source, filtered only on
`tag:Team = "platform"` (exactly the "loose filter" example earlier in this chapter), silently began
resolving to the shared transitional security group instead of `checkout-service`'s own dedicated one,
because both now matched the filter and the data source's own tie-breaking logic picked one without any
error — there was no "multiple results" failure to catch, because at the moment this ran, the filter's
loose scope hadn't yet produced an actual ambiguous match, only a wrong single one.

The mistake surfaced days later as unexpected network reachability behavior on `checkout-service`, not as
any Terraform error — exactly the "successfully matched, but matched the wrong thing" failure mode this
chapter's filtering-precision section warns about directly. The fix tightened the filter to the exact,
environment-and-service-prefixed name (`${local.name_prefix}-checkout-sg`) established by Part 4's naming
convention, which by construction can never collide with another service's resource — a concrete,
retrospective justification for treating that naming discipline as load-bearing, not merely tidy.

## Part 5 CLI Cheat Sheet

| Command | Purpose |
|---|---|
| `terraform providers` | List every provider (and alias) a configuration requires |
| `terraform console` | Test a data source's resolved value interactively (Part 1) |
| `terraform plan` | The only real way to see a data source's *current* resolved result before apply |
| `terraform taint <addr>` | Force a resource (and any creation-time provisioner) to be replaced on the next apply |
| `terraform apply -replace=<addr>` | The modern equivalent of `taint`, scoped to one apply |

## Common Mistakes and Interview Traps

| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Filtering a data source loosely, on a single shared tag | Can silently resolve to the wrong object without any error, if more than one real object happens to match | Filter on the most specific identifying attributes available — a fully-qualified name, a resource ID, a combined tag+scope filter |
| Using a data source to look up something another Terraform configuration manages | Creates an invisible, undeclared coupling — a rename elsewhere breaks this lookup with no warning at either end | Prefer `terraform_remote_state` for anything genuinely Terraform-managed elsewhere |
| Reaching for `remote-exec` before checking for a native resource, `user_data`, or a pre-built image | Carries real connection-timing risk, and is invisible to `plan`/state/drift-detection | Exhaust the alternatives in this chapter's decision tree first |
| Setting `on_failure = continue` to silence a flaky provisioner | Hides the underlying reliability problem instead of fixing it | Treat frequent failure as a signal to remove the provisioner in favor of an alternative, not to suppress the failure |
| Referencing a resource's own attribute directly instead of `self` inside its own provisioner/connection block | Creates a dependency cycle error | Use `self.<attribute>` inside a resource's own provisioner/connection block |
| Assuming a `terraform state rm`'d resource still runs its destroy-time provisioner | `state rm` bypasses the normal destroy lifecycle entirely — no provisioner runs | Only an actual `destroy` (or a replace) triggers a `when = destroy` provisioner |
| Configuring a provider from a resource created in the same apply | Provider configuration isn't graph-ordered — intermittent, timing-dependent connection failures on the first apply | Split cluster/dependency creation and its dependent provider's resources into two separate states |

## Worked Practice Problems

**Problem 1**: A `data "aws_subnet"` block filters only on `values = ["private"]` against a `tag:Tier`
value, in an account with three VPCs, each tagged with a `private` subnet. What happens, and how should the
filter be fixed?

*Answer*: If more than one subnet across the account matches `tag:Tier = "private"`, `terraform plan` fails
with a "multiple results" error (assuming no `most_recent`-style tie-breaker exists for this data source,
which `aws_subnet` doesn't have) — this is actually the *safer* failure mode, since it surfaces the ambiguity
immediately rather than silently picking one. The fix is scoping the filter further — adding a `vpc-id`
filter (from a `data "aws_vpc"` already scoped to the correct VPC by name) alongside the tier tag — so the
combination can only ever match within the intended VPC.

**Problem 2**: A team adds a `remote-exec` provisioner to install a security agent immediately after EC2
instance creation. Three months later, they notice `terraform plan` never reports drift even after the
agent is manually uninstalled from several instances during an incident. Why not, and what's the actual gap?

*Answer*: `remote-exec`'s effect (the installed agent) is never recorded in Terraform state at all — the
provisioner ran once, at creation time, and Terraform has no ongoing record of "this instance should have
this agent installed" to compare against reality on future plans. This is the structural gap this chapter
opened with: provisioners aren't tracked, so they can't participate in drift detection (Part 6) the way a
real resource attribute would. The fix is exactly this chapter's decision tree — bake the agent into the
base image via Packer (so its presence is part of the AMI, verifiable and rebuildable) or manage it via a
proper configuration-management tool that has its own drift-detection story, not a one-time provisioner.

**Problem 3**: A `terraform_data` resource has `triggers_replace = [aws_eks_cluster.main.id]` and holds a
`local-exec` provisioner. The EKS cluster is later replaced (a genuine destroy-and-recreate, not an
in-place update) due to an immutable-argument change. What happens to `terraform_data`'s provisioner, and
why was `triggers_replace` set up this way?

*Answer*: Because the cluster's `id` changes when it's replaced, and `triggers_replace` explicitly watches
that value, `terraform_data` itself gets replaced too — which re-runs its `local-exec` provisioner (in this
chapter's earlier example, re-running `aws eks update-kubeconfig` against the new cluster). This is exactly
the intended design: without `triggers_replace` pointing at something that actually changes when the
underlying dependency changes, `terraform_data` would have no reason to ever re-run its provisioner after
the cluster it depends on is replaced, silently leaving a stale kubeconfig pointing at a cluster that no
longer exists.

**Problem 4**: A single `apply` creates an EKS cluster with `aws_eks_cluster` and, in the same configuration,
configures the `kubernetes` provider using that cluster's own computed endpoint, then creates a
`kubernetes_namespace`. The first apply against a brand-new AWS account fails intermittently with a
connection error; re-running the exact same apply immediately afterward succeeds. What's actually happening,
and what's the correct structural fix?

*Answer*: This is the provider-depends-on-a-resource trap — provider configurations aren't graph-ordered
nodes the way resources are, so Terraform has no guarantee the EKS cluster's API is actually reachable at the
moment it configures the `kubernetes` provider, even though the cluster resource itself reports as created.
The retry "succeeding" is a red herring, not a fix: on the second run, the cluster has had time to fully
stabilize, which is exactly the timing dependency, not a genuine resolution. The correct fix is splitting
into two separate Terraform states — one creating the EKS cluster with no `kubernetes`/`helm` provider at
all, a second consuming that cluster's outputs via `terraform_remote_state` and applied strictly afterward —
removing the same-apply race entirely rather than hoping timing works out.

## Summary and What's Next

Providers scale from Part 1's single-region default into a real, multi-aliased configuration once Part 4's
account/region structure demands it — named clearly, never left to a bare `aws.b`, and never configured from
a resource created in the same apply, per this chapter's most structurally sharp gotcha. Data sources are the
correct mechanism for reading anything genuinely outside this configuration's own management, but only when
filtered precisely enough that "matched successfully" actually means "matched the intended object," not just
"matched something." Provisioners remain real, occasionally necessary tools — but this chapter's decision
tree, and its two worked scenarios showing what goes wrong when it's skipped, should be the default reflex
before reaching for one: a native resource, `user_data`, a Packer image, or a proper configuration-management
tool almost always beats a provisioner's invisible, untracked, timing-dependent side effect.

Everything so far has assumed Terraform's own state faithfully reflects reality. Part 6 is about the moment
that assumption breaks: drift (someone or something changed real infrastructure outside Terraform), how to
detect it before it surprises a plan, how `import` (both the CLI command and the newer, reviewable
configuration-driven `import` block) brings an unmanaged resource under Terraform's control safely, and the
`moved`/`removed` block mechanics this series has referenced repeatedly without yet showing in full.