Providers, Data Sources & Provisioners Deep Dive

Assumes you're comfortable with Part 1's provider basics and Part 4's cross-account assume_role pattern — this chapter goes deeper on provider aliasing, adds the two mechanisms for reading and (narrowly) acting on infrastructure Terraform doesn't fully own, and is honest about provisioners' real, narrow, correctly-scoped place in a mature configuration.

Table of Contents#

1. Multiple Provider Configurations, Revisited
2. The Provider-Depends-on-a-Resource Trap
3. Data Sources — Reading What Terraform Doesn't Manage
4. Data Source Gotchas: Zero Results, Multiple Results
5. Choosing the Right Input Mechanism
6. Filtering Data Sources for Precision, Not Convenience
7. The local-exec Provisioner
8. The remote-exec Provisioner and Connection Blocks
9. The file Provisioner
10. Creation-Time vs. Destroy-Time Provisioners
11. on_failure — Controlling What Happens When a Provisioner Fails
12. Why Provisioners Are a Last Resort — the Alternatives, in Order
13. terraform_data — the Modern Resourceless Anchor
14. The External Data Source — a Careful Escape Hatch
15. Generating Config Files With templatefile()
16. Worked Scenario: a Flaky remote-exec That Passed in Dev, Failed in Prod
17. Worked Scenario: Replacing a Provisioner With Packer and user_data
18. Worked Scenario: a Data Source Silently Matching the Wrong Resource
19. Part 5 CLI Cheat Sheet
20. Common Mistakes and Interview Traps
21. Worked Practice Problems
22. Summary and What's Next

Multiple Provider Configurations, Revisited#

Part 1 introduced provider aliasing briefly; at real scale — the multi-account, multi-region reality Part 4 built — a root module routinely juggles half a dozen or more distinct provider configurations, and keeping them legible is its own discipline.

provider "aws" {
  region = "us-east-1"
}

provider "aws" {
  alias  = "west"
  region = "us-west-2"
}

provider "aws" {
  alias  = "prod"
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::444455556666:role/terraform-prod-deployer"
  }
}

resource "aws_s3_bucket" "replica" {
  provider = aws.west
  bucket   = "checkout-service-replica"
}

Every resource and data source defaults to the unaliased provider unless a provider = aws.<alias> argument says otherwise — an easy detail to miss when skimming a large configuration, since the absence of a provider argument is itself meaningful (it means "the default"), not just an omission.

The Provider-Depends-on-a-Resource Trap#

A sharp, frequently-hit gotcha: configuring one provider (commonly kubernetes or helm) using attributes computed from a resource created by a different provider in the same apply — an EKS cluster's endpoint feeding the kubernetes provider's host argument, for instance — looks like it should work, and often does, right up until it doesn't.

# The tempting, fragile pattern — DON'T structure it this way
resource "aws_eks_cluster" "main" {
  # ...
}

provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

resource "kubernetes_namespace" "checkout" {
  metadata { name = "checkout" }
}

The problem is structural, not a syntax mistake: Terraform's provider configurations are not nodes in the dependency graph the way resources are. Terraform can't guarantee the EKS cluster is fully ready — DNS propagated, API server actually reachable — at the exact moment it evaluates the kubernetes provider block, because providers are configured once, early, outside the normal resource-ordering guarantees. The practical symptom is intermittent: it works reliably on a fresh cluster's second apply (state already exists, timing no longer matters), and fails unpredictably on the first apply that creates the cluster and tries to use it in the same run — exactly the kind of "works in my testing, fails for someone else" bug this chapter keeps returning to.

The fix, confirmed as the standard resolution: split cluster infrastructure and in-cluster resources into two separate states — exactly Part 2's state-splitting discipline, applied to this specific case. A cluster-infrastructure state creates the EKS cluster with no kubernetes/helm provider anywhere in it; a separate cluster-services state configures the kubernetes provider using terraform_remote_state (Part 2) to read the already-fully-created cluster's endpoint and credentials, applied strictly afterward.

# cluster-services/main.tf — a SEPARATE state from cluster-infrastructure
data "terraform_remote_state" "cluster" {
  backend = "s3"
  config = {
    bucket = "meridian-platform-tfstate"
    key    = "shared/eks-cluster/terraform.tfstate"
    region = "us-east-1"
  }
}

provider "kubernetes" {
  host                   = data.terraform_remote_state.cluster.outputs.endpoint
  cluster_ca_certificate = data.terraform_remote_state.cluster.outputs.ca_certificate
  token                  = data.aws_eks_cluster_auth.main.token
}

Data Sources — Reading What Terraform Doesn't Manage#

A data block queries an existing object through a provider — read-only, no create/update/delete — and is the correct mechanism for referencing anything that already exists but isn't (and shouldn't be) tracked in this configuration's own state.

data "aws_vpc" "default" {
  filter {
    name   = "tag:Name"
    values = ["meridian-prod-vpc"]
  }
}

data "aws_ami" "app" {
  most_recent = true
  owners      = ["self"]
  filter {
    name   = "name"
    values = ["checkout-app-*"]
  }
}

resource "aws_instance" "worker" {
  ami           = data.aws_ami.app.id
  subnet_id     = data.aws_vpc.default.id
  instance_type = "t3.micro"
}

Data sources are re-evaluated on every plan (part of the refresh step from Part 1) — they always reflect current reality, not a cached snapshot from whenever the configuration was first written, which is exactly why most_recent = true above means "give me whatever the newest matching AMI is right now," re-checked every single run.

Data Source Gotchas: Zero Results, Multiple Results#

Unlike a resource (which Terraform created and therefore knows exists), a data source's filter can legitimately match zero results or more than one — and the failure mode for each is different, and both are worth handling deliberately rather than discovering at a bad moment.

data "aws_ami" "app" {
  most_recent = true          # Resolves ambiguity: pick the newest of several matches
  owners      = ["self"]
  filter {
    name   = "name"
    values = ["checkout-app-*"]
  }
}

Without most_recent = true, a filter matching two or more AMIs fails plan outright with a "your query returned more than one result" error — which is often exactly the right behavior (an ambiguous match shouldn't silently pick one), but only if the configuration is actually prepared to handle it. A filter matching zero results fails differently — a "no matching AMI found" error — which typically indicates either a genuine typo in the filter or (more insidiously) an environment where the expected object simply hasn't been created yet.

data "aws_ami" "app" {
  most_recent = true
  owners      = ["self"]
  filter {
    name   = "name"
    values = ["checkout-app-*"]
  }
}

locals {
  # try() catches a data source lookup failure and falls back gracefully,
  # rather than letting the whole plan fail outright.
  app_ami_id = try(data.aws_ami.app.id, var.fallback_ami_id)
}

Choosing the Right Input Mechanism#

By this point in the series, four distinct ways of getting a value into a configuration all exist — data sources, terraform_remote_state (Part 2), variables, and module outputs (Part 3) — and picking the wrong one for a given situation creates real, if subtle, coupling problems.

The subtle trap: using a data source to look up something that's actually managed by another Terraform configuration (querying aws_vpc by tag, instead of consuming that VPC's own terraform_remote_state output) works, but it creates an invisible coupling — nothing in either configuration declares the dependency explicitly, so a rename of that tag, or a change to which VPC bears it, silently breaks the data-source lookup with no warning at either config's own review time. Preferring terraform_remote_state for anything genuinely Terraform-managed elsewhere keeps that dependency explicit and versioned, per Part 2's outputs-as-contract discipline.

Filtering Data Sources for Precision, Not Convenience#

A loosely-written filter is a common source of the "wrong environment" class of incident this series has returned to repeatedly (Part 2's destroy scenario, Part 4's credential scenario) — a data source that's supposed to find checkout-service's security group but is filtered loosely enough to also match catalog-service's, in some environment where naming happens to collide, fails silently in the worst way: it returns exactly one result, so none of the previous section's error handling ever triggers.

# Loose — plausible collision risk across services/environments
data "aws_security_group" "app" {
  filter {
    name   = "tag:Team"
    values = ["platform"]
  }
}

# Precise — scoped to exactly the intended resource
data "aws_security_group" "app" {
  filter {
    name   = "tag:Name"
    values = ["${local.name_prefix}-checkout-sg"]
  }
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.this.id]
  }
}

The local-exec Provisioner#

local-exec runs a command on the machine executing Terraform itself — not on any remote resource — and needs no connection configuration, which makes it the least risky of the provisioner family, though still subject to every caveat this chapter builds toward.

resource "aws_instance" "worker" {
  # ...
  provisioner "local-exec" {
    command = "echo ${self.private_ip} >> inventory.txt"
  }
}

A common, more defensible local-exec use: triggering a genuinely external, non-Terraform-native action after a resource is created — notifying a webhook, updating an external inventory system that has no Terraform provider of its own. Even here, local-exec output isn't tracked in state and doesn't participate in drift detection (Part 6) — Terraform has no way to know if the command's effect was later undone outside of Terraform, which is the core reason this entire chapter treats provisioners as a narrow tool, not a general-purpose escape hatch.

The remote-exec Provisioner and Connection Blocks#

remote-exec connects into a freshly-created resource (via SSH for Linux, WinRM for Windows) and runs commands directly on it — the provisioner most often reached for to "finish configuring" a VM right after Terraform creates it, and the one Part 1's original guidance called out as needing the strongest justification.

resource "aws_instance" "worker" {
  ami           = data.aws_ami.app.id
  instance_type = "t3.micro"
  key_name      = aws_key_pair.deployer.key_name

  connection {
    type        = "ssh"
    user        = "ec2-user"
    host        = self.public_ip
    private_key = file("~/.ssh/id_rsa")
  }

  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install -y nginx",
      "sudo systemctl start nginx",
    ]
  }
}

Note self.public_ip, not aws_instance.worker.public_ip — referencing the resource's own address directly from inside its own block would create a dependency cycle (the resource depending on its own completed creation to compute its own configuration); self refers to the current resource's own attributes without that cycle. A connection block can also be set at the provisioner level instead of the resource level, with provisioner-level settings taking precedence when both exist — useful when a resource has multiple provisioners needing different connection details.

The file Provisioner#

file copies a local file or directory to a remote resource over the same connection mechanism as remote-exec — narrower in scope, but carrying the identical connection-reliability risk.

resource "aws_instance" "worker" {
  # ...
  connection {
    type        = "ssh"
    user        = "ec2-user"
    host        = self.public_ip
    private_key = file("~/.ssh/id_rsa")
  }

  provisioner "file" {
    source      = "config/app.conf"
    destination = "/etc/app/app.conf"
  }
}

The far more common, more reliable alternative for exactly this use case is baking the file into a pre-built image (Part 5's alternatives section, and this chapter's Packer scenario, cover this in depth) or passing its content through user_data/cloud-init instead — both avoid the runtime SSH dependency entirely, since the content is present the moment the instance boots rather than requiring a separate, timing-sensitive connection after creation.

Creation-Time vs. Destroy-Time Provisioners#

By default, a provisioner runs at resource creation — adding when = destroy flips it to run immediately before the resource is destroyed instead, for cleanup actions that genuinely need to happen on the way out.

resource "aws_instance" "worker" {
  # ...
  provisioner "local-exec" {
    when    = destroy
    command = "curl -X POST https://inventory.internal/deregister -d 'host=${self.private_ip}'"
  }
}

A destroy-time provisioner runs using the resource's last-known state, not any updated configuration — which matters because if the resource was already partially modified or its state is stale, the provisioner executes against whatever attributes state still holds, not necessarily current reality. Destroy-time provisioners also only run on an explicit destroy of that specific resource — they do not run if the resource is removed from state via terraform state rm (Part 2), since that command deliberately bypasses the normal destroy lifecycle entirely.

on_failure — Controlling What Happens When a Provisioner Fails#

By default, a failed provisioner marks the resource as tainted (Part 6 covers tainting in depth) and fails the entire apply — on_failure = continue changes that, letting the apply proceed despite the provisioner's failure, appropriate only for genuinely non-critical provisioner actions.

provisioner "local-exec" {
  command    = "curl -sf https://monitoring.internal/notify -d 'event=instance_created'"
  on_failure = continue
}

Why Provisioners Are a Last Resort — the Alternatives, in Order#

Every provisioner type covered so far shares the same structural weakness: Terraform has no visibility into what a provisioner script actually does, can't show its effect in a plan, doesn't track its result in state, and can't detect if that effect is later undone — which is exactly why HashiCorp's own documentation, and this chapter, treat provisioners as a last resort rather than a normal tool.

This chapter's caption: the decision tree deliberately exhausts four better options before ever reaching a provisioner — each earlier branch trades away exactly the invisibility problem provisioners carry, in exchange for its own tradeoff (an image rebuild pipeline, a second config-management tool) that's usually still cheaper than the reliability and visibility cost of a provisioner in production.

terraform_data — the Modern Resourceless Anchor#

terraform_data (replacing the older null_resource pattern) is a provider-agnostic, built-in resource type that exists purely to hold a provisioner, a triggers_replace value, or computed values — without needing any real cloud resource to attach to.

resource "terraform_data" "cluster_bootstrap" {
  triggers_replace = [aws_eks_cluster.main.id]

  provisioner "local-exec" {
    command = "aws eks update-kubeconfig --name ${aws_eks_cluster.main.name}"
  }
}

triggers_replace explicitly controls when this resource (and therefore its provisioner) re-runs — here, any change to the EKS cluster's ID forces terraform_data to be replaced, re-running the provisioner. This is the correct modern home for "I need a provisioner but there's no real resource to attach it to" — exactly the same job null_resource used to do, with a clearer, purpose-built name and no confusing implication that it represents an actual "null" provider resource.

The External Data Source — a Careful Escape Hatch#

data "external" executes an external program and reads its stdout (JSON) as the data source's result — the read-only counterpart to a provisioner, for querying a system with no Terraform provider of its own.

data "external" "vault_secret" {
  program = ["python3", "${path.module}/scripts/fetch-secret.py"]
  query = {
    secret_path = "secret/checkout/db-password"
  }
}

resource "aws_db_instance" "checkout" {
  # ...
  password = data.external.vault_secret.result.password
}

The external program must output a single flat JSON object of string values on stdout — no nested structures, no other output types — which is a real constraint worth knowing before reaching for this. Like any data source, it re-runs on every plan, meaning the external program executes on every single plan/apply, not just once — a slow or unreliable external script becomes a slow or unreliable Terraform workflow, exactly the same reliability transfer risk remote-exec carries, just on the read side instead of the write side.

Generating Config Files With templatefile()#

templatefile() renders a local template file with variables substituted in — the mechanism behind the user_data example in this chapter's Packer scenario, and the standard way to generate any config file content (a user_data script, an application config, a systemd unit) from within Terraform without a provisioner at all.

# templates/user-data.sh.tpl
#!/bin/bash
echo "ENVIRONMENT=${environment}" >> /etc/app/env
echo "CONSUL_ADDR=${consul_addr}" >> /etc/app/env
systemctl restart checkout-app

resource "aws_instance" "worker" {
  ami           = data.aws_ami.app.id
  instance_type = "t3.micro"
  user_data = templatefile("${path.module}/templates/user-data.sh.tpl", {
    environment = var.environment
    consul_addr = var.consul_addr
  })
}

Because templatefile()'s result is a plain string argument, it's fully visible in terraform plan output — any change to the rendered content shows as a diff, exactly like any other argument, with none of a provisioner's plan-invisibility. Templates support full Terraform expression syntax inside ${...}, including conditionals and for loops, which covers the overwhelming majority of what a remote-exec inline script would otherwise have tried to construct imperatively.

Worked Scenario: a Flaky remote-exec That Passed in Dev, Failed in Prod#

This chapter's caption: the API call for the security group rule returning "success" and the rule actually being enforced are two different moments in time — remote-exec's SSH attempt races against that gap with no way for Terraform to know which side of it any given apply will land on.

An engineer added a remote-exec provisioner to install and configure a monitoring agent immediately after instance creation, tested repeatedly in dev with consistent success. In prod, the same configuration failed intermittently — roughly one apply in five — with an SSH connection timeout.

The immediate cause: prod's security group (correctly, per Part 4's tighter production hardening) took slightly longer to fully propagate than dev's more permissive one, occasionally still blocking port 22 in the brief window right after instance creation when Terraform's remote-exec first attempted to connect. The underlying condition: nothing in the configuration accounted for the real-world timing gap between "the API call creating the security group rule returned success" and "the rule is actually enforced and reachable" — a gap that happened to be small enough to never manifest in dev's laxer setup, and large enough to intermittently bite in prod's tighter one. The team's fix followed this chapter's own decision tree directly: the monitoring agent install moved into the base AMI via Packer (this chapter's next scenario), eliminating the runtime SSH dependency — and therefore the timing race — entirely.

Worked Scenario: Replacing a Provisioner With Packer and user_data#

Following the previous scenario's fix, the platform team built a Packer pipeline producing a checkout-app-*-named AMI (the exact AMI this chapter's data-source examples query) with the monitoring agent, base packages, and application runtime already baked in — reducing what Terraform's own aws_instance resource needs to do at boot time to a small user_data script handling only genuinely per-instance, runtime-dependent configuration (registering with the current environment's specific service discovery endpoint):

resource "aws_instance" "worker" {
  ami           = data.aws_ami.app.id   # Packer-built, monitoring agent pre-installed
  instance_type = "t3.micro"

  user_data = templatefile("${path.module}/templates/user-data.sh.tpl", {
    environment = var.environment
    consul_addr = var.consul_addr
  })
}

The net effect matched this chapter's decision-tree prediction closely: zero SSH-timing-related apply failures since the migration, faster instance boot-to-ready time (no waiting on a runtime SSH connection and package installation), and a terraform plan that now shows exactly what user_data will run (it's a visible, diffable string argument) instead of a provisioner's opaque, unplannable side effect.

Worked Scenario: a Data Source Silently Matching the Wrong Resource#

Following an internal reorg, catalog-service briefly shared a transitional security group tagged Team = "platform" with checkout-service during a migration window — a temporary, deliberate choice by the networking team. An unrelated checkout-service module's data source, filtered only on tag:Team = "platform" (exactly the "loose filter" example earlier in this chapter), silently began resolving to the shared transitional security group instead of checkout-service's own dedicated one, because both now matched the filter and the data source's own tie-breaking logic picked one without any error — there was no "multiple results" failure to catch, because at the moment this ran, the filter's loose scope hadn't yet produced an actual ambiguous match, only a wrong single one.

The mistake surfaced days later as unexpected network reachability behavior on checkout-service, not as any Terraform error — exactly the "successfully matched, but matched the wrong thing" failure mode this chapter's filtering-precision section warns about directly. The fix tightened the filter to the exact, environment-and-service-prefixed name (${local.name_prefix}-checkout-sg) established by Part 4's naming convention, which by construction can never collide with another service's resource — a concrete, retrospective justification for treating that naming discipline as load-bearing, not merely tidy.

Part 5 CLI Cheat Sheet#

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A data "aws_subnet" block filters only on values = ["private"] against a tag:Tier value, in an account with three VPCs, each tagged with a private subnet. What happens, and how should the filter be fixed?

Answer: If more than one subnet across the account matches tag:Tier = "private", terraform plan fails with a "multiple results" error (assuming no most_recent-style tie-breaker exists for this data source, which aws_subnet doesn't have) — this is actually the safer failure mode, since it surfaces the ambiguity immediately rather than silently picking one. The fix is scoping the filter further — adding a vpc-id filter (from a data "aws_vpc" already scoped to the correct VPC by name) alongside the tier tag — so the combination can only ever match within the intended VPC.

Problem 2: A team adds a remote-exec provisioner to install a security agent immediately after EC2 instance creation. Three months later, they notice terraform plan never reports drift even after the agent is manually uninstalled from several instances during an incident. Why not, and what's the actual gap?

Answer: remote-exec's effect (the installed agent) is never recorded in Terraform state at all — the provisioner ran once, at creation time, and Terraform has no ongoing record of "this instance should have this agent installed" to compare against reality on future plans. This is the structural gap this chapter opened with: provisioners aren't tracked, so they can't participate in drift detection (Part 6) the way a real resource attribute would. The fix is exactly this chapter's decision tree — bake the agent into the base image via Packer (so its presence is part of the AMI, verifiable and rebuildable) or manage it via a proper configuration-management tool that has its own drift-detection story, not a one-time provisioner.

Problem 3: A terraform_data resource has triggers_replace = [aws_eks_cluster.main.id] and holds a local-exec provisioner. The EKS cluster is later replaced (a genuine destroy-and-recreate, not an in-place update) due to an immutable-argument change. What happens to terraform_data's provisioner, and why was triggers_replace set up this way?

Answer: Because the cluster's id changes when it's replaced, and triggers_replace explicitly watches that value, terraform_data itself gets replaced too — which re-runs its local-exec provisioner (in this chapter's earlier example, re-running aws eks update-kubeconfig against the new cluster). This is exactly the intended design: without triggers_replace pointing at something that actually changes when the underlying dependency changes, terraform_data would have no reason to ever re-run its provisioner after the cluster it depends on is replaced, silently leaving a stale kubeconfig pointing at a cluster that no longer exists.

Problem 4: A single apply creates an EKS cluster with aws_eks_cluster and, in the same configuration, configures the kubernetes provider using that cluster's own computed endpoint, then creates a kubernetes_namespace. The first apply against a brand-new AWS account fails intermittently with a connection error; re-running the exact same apply immediately afterward succeeds. What's actually happening, and what's the correct structural fix?

Answer: This is the provider-depends-on-a-resource trap — provider configurations aren't graph-ordered nodes the way resources are, so Terraform has no guarantee the EKS cluster's API is actually reachable at the moment it configures the kubernetes provider, even though the cluster resource itself reports as created. The retry "succeeding" is a red herring, not a fix: on the second run, the cluster has had time to fully stabilize, which is exactly the timing dependency, not a genuine resolution. The correct fix is splitting into two separate Terraform states — one creating the EKS cluster with no kubernetes/helm provider at all, a second consuming that cluster's outputs via terraform_remote_state and applied strictly afterward — removing the same-apply race entirely rather than hoping timing works out.

Summary and What's Next#

Providers scale from Part 1's single-region default into a real, multi-aliased configuration once Part 4's account/region structure demands it — named clearly, never left to a bare aws.b, and never configured from a resource created in the same apply, per this chapter's most structurally sharp gotcha. Data sources are the correct mechanism for reading anything genuinely outside this configuration's own management, but only when filtered precisely enough that "matched successfully" actually means "matched the intended object," not just "matched something." Provisioners remain real, occasionally necessary tools — but this chapter's decision tree, and its two worked scenarios showing what goes wrong when it's skipped, should be the default reflex before reaching for one: a native resource, user_data, a Packer image, or a proper configuration-management tool almost always beats a provisioner's invisible, untracked, timing-dependent side effect.

Everything so far has assumed Terraform's own state faithfully reflects reality. Part 6 is about the moment that assumption breaks: drift (someone or something changed real infrastructure outside Terraform), how to detect it before it surprises a plan, how import (both the CLI command and the newer, reviewable configuration-driven import block) brings an unmanaged resource under Terraform's control safely, and the moved/removed block mechanics this series has referenced repeatedly without yet showing in full.