State Management & Remote Backends

Assumes you're comfortable with Part 1's resource addresses, the plan/apply cycle, and the refresh step — this chapter is about the file that makes all of that possible in the first place.

Table of Contents#

1. Why State Is the Most Dangerous File in a Terraform Project
2. What's Actually Inside a State File
3. Local State, and Why It Fails at Any Team Size
4. Remote Backends — Shared, Locked, Versioned
5. Configuring the S3 Backend, With Native Locking
6. State Locking — What It Actually Prevents
7. Partial Backend Configuration for CI/CD
8. HCP Terraform as a Managed Backend
9. Encrypting State and Restricting Who Can Read It
10. State Versioning and Recovering From a Bad Apply
11. Splitting State — Why One Giant State File Is an Anti-Pattern
12. terraform_remote_state — Reading Outputs Across State Files
13. State Manipulation Commands, and Their Blast Radius
14. Performance at Scale: -refresh=false and -target
15. Migrating From Local to Remote State Without Downtime
16. Inspecting State Programmatically with terraform show -json
17. Worked Scenario: the destroy Run Against the Wrong Backend
18. Worked Scenario: Splitting the Platform Team's Monolithic State
19. Worked Scenario: Recovering From a Corrupted State After a Failed Apply
20. Choosing a Backend — a Decision Framework
21. Part 2 CLI Cheat Sheet
22. Common Mistakes and Interview Traps
23. Worked Practice Problems
24. Summary and What's Next

Why State Is the Most Dangerous File in a Terraform Project#

Terraform state is the only record that maps your .tf configuration's resource addresses to the actual, real-world objects they represent — lose it, corrupt it, or let two people write to it at once, and Terraform's entire "declarative, idempotent" promise collapses into guesswork. Without state, Terraform would have no way to know that aws_db_instance.checkout in your configuration corresponds to a specific RDS instance with a specific ARN somewhere in AWS — it would have to either re-create everything from scratch on every apply, or ask you to manually confirm every mapping. State is what makes "declare what you want, and Terraform figures out the delta" actually work.

That same centrality is exactly what makes it dangerous. A state file is simultaneously the thing that lets Terraform delete infrastructure confidently ("this address is gone from config, and state confirms it's the same object that exists live — destroy it") and the single point of failure for that same confidence. This chapter is about the practices that keep state trustworthy at team scale: where it lives, who can write to it, how concurrent writes are prevented, how to recover when something goes wrong, and when one state file has become too large for its own good.

What's Actually Inside a State File#

A state file is plain JSON, versioned by a version field (currently 4), containing every tracked resource's full attribute set, plus metadata that makes concurrent-write detection possible.

{
  "version": 4,
  "terraform_version": "1.15.8",
  "serial": 47,
  "lineage": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "outputs": {
    "vpc_id": { "value": "vpc-0a1b2c3d", "type": "string" }
  },
  "resources": [
    {
      "mode": "managed",
      "type": "aws_db_instance",
      "name": "checkout",
      "instances": [
        {
          "attributes": {
            "id": "checkout-db",
            "engine": "postgres",
            "endpoint": "checkout-db.abc123.us-east-1.rds.amazonaws.com:5432"
          }
        }
      ]
    }
  ]
}

Two fields matter more than the rest of the schema combined:

• lineage — a UUID generated once, when a state file is first created for a given configuration. Two state files with different lineages are, by definition, unrelated histories — Terraform refuses to treat them as compatible, which is exactly the safety check that catches "someone pointed this workspace at the wrong backend."
• serial — a monotonically incrementing counter, bumped on every write. This is the actual mechanism behind conflict detection: if a client's local view of serial is behind what the backend currently holds, something else wrote a newer state since this client last read it, and Terraform refuses to overwrite it blindly.

Every resource's attributes block holds the entire provider-returned object — not just the arguments you wrote in HCL, but every computed attribute the API returned (ARNs, generated IDs, default values the provider filled in). This is why state files routinely contain values you never typed anywhere, including, depending on the resource type, values that should be treated as secrets (an RDS instance's initial master password if it was ever set via a plain password argument, a generated API key, a certificate's private key material).

Local State, and Why It Fails at Any Team Size#

By default, with no backend block configured, Terraform writes state to a plain terraform.tfstate file in the working directory — fine for a five-minute experiment, actively dangerous for anything a second person will ever touch.

Local state fails in three specific, concrete ways beyond the obvious "it's on one laptop": it isn't versioned (a bad apply overwrites the only record of the previous good state, with no history), it isn't locked (two people running apply concurrently can corrupt it or silently apply against a stale view), and it isn't accessible to CI (a pipeline runner has no laptop to read from). Any one of these is disqualifying for a team of more than one; all three together make local state a genuine liability the moment a second contributor or an automated pipeline is involved.

Remote Backends — Shared, Locked, Versioned#

A remote backend moves state storage out of the local filesystem into a shared service that natively supports locking and, ideally, versioning — S3, Azure Blob Storage, Google Cloud Storage, or HCP Terraform's own managed state storage are the common choices.

terraform {
  backend "s3" {
    bucket       = "meridian-platform-tfstate"
    key          = "checkout-service/network/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Every backend, regardless of which one you pick, needs to answer the same three questions: where does state live (a bucket, a managed service), who can lock it during a write (preventing concurrent writers), and what happens on conflict (a stale write is rejected, not silently merged). The key argument above is worth pausing on — it's the actual mechanism for splitting state by service/component (covered later in this chapter), since each distinct key within the same bucket is a fully independent state file.

Configuring the S3 Backend, With Native Locking#

Terraform 1.10 introduced S3-native state locking via use_lockfile, eliminating the DynamoDB table that used to be mandatory for safe concurrent S3-backed state — a real simplification worth adopting on any new setup and migrating existing ones toward.

The old pattern required a companion DynamoDB table purely to hold a lock record:

# The old pattern — still works, but adds an extra piece of infrastructure
# purely to support locking.
terraform {
  backend "s3" {
    bucket         = "meridian-platform-tfstate"
    key            = "checkout-service/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

The new pattern uses S3's own conditional-write support (If-None-Match) to create a lock object directly in the same bucket, with no second service to provision, monitor, or pay for:

terraform {
  backend "s3" {
    bucket       = "meridian-platform-tfstate"
    key          = "checkout-service/network/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Migrating an existing setup is a two-step change: add use_lockfile = true, remove dynamodb_table, then run terraform init -reconfigure to have Terraform pick up the backend change (this does not touch your actual managed resources — only where and how state itself is stored and locked).

State Locking — What It Actually Prevents#

A lock is acquired before any operation that might write state (plan acquires a read lock momentarily, apply holds a write lock for the operation's duration) and released after — its entire job is making the second scenario below impossible.

This chapter's caption: the second apply doesn't corrupt anything or silently queue — it fails loudly and immediately, which is the correct behavior; a lock that failed silently would be far more dangerous than one that blocks.

A lock that appears stuck (the holding process crashed mid-apply, or a CI job was killed without cleanup) can be force-released with terraform force-unlock <lock-id> — a command that exists specifically for this recovery case and should never be reached for casually. Force-unlocking while a genuine apply is actually still in progress reopens exactly the corruption window locking exists to prevent.

Partial Backend Configuration for CI/CD#

A backend block's arguments don't all have to be hardcoded in .tf files — Terraform supports "partial configuration," where some or all backend settings are supplied at init time instead, which is exactly what lets the same configuration target a different bucket/key per environment without editing code.

# backend.tf — deliberately incomplete
terraform {
  backend "s3" {}
}

# CI supplies the actual values, per environment, at init time
terraform init \
  -backend-config="bucket=meridian-platform-tfstate" \
  -backend-config="key=checkout-service/${ENVIRONMENT}/terraform.tfstate" \
  -backend-config="region=us-east-1" \
  -backend-config="use_lockfile=true"

The same pattern works from a file instead of repeated flags — -backend-config=prod.backend.hcl pointing at a small .hcl file holding just the backend arguments for that environment. This is the mechanism that makes Part 4's per-environment directory structure and Part 8's CI pipeline actually work together: one shared configuration, a different backend target selected per pipeline run, with no environment-specific values baked into version-controlled .tf files at all.

HCP Terraform as a Managed Backend#

HCP Terraform (the current name for what was Terraform Cloud) provides state storage, locking, and versioning as a managed service, plus a run pipeline (Part 8 goes deep on this) — using it as just a state backend, with your own CI driving plan/apply, is a legitimate and common middle-ground choice.

terraform {
  cloud {
    organization = "meridian-platform"
    workspaces {
      name = "checkout-service-network"
    }
  }
}

Every state write goes through HCP Terraform's API instead of directly to a storage bucket you manage — locking, versioning, and a browsable state history UI come for free, with no bucket, IAM policy, or lockfile mechanism to maintain yourself. The tradeoff, covered fully with real numbers in Part 9, is the resource-based pricing that replaced the legacy free tier: for a team running the numbers, self-managed S3 state plus a self-hosted run pipeline can be materially cheaper at scale, at the cost of maintaining that infrastructure yourselves.

Encrypting State and Restricting Who Can Read It#

Encryption at rest and access control are the two controls that actually address the secrets-in-state problem Part 1's "Marking Sensitive Values" section flagged but didn't solve.

For an S3 backend, encryption at rest is a bucket-level setting (encrypt = true in the backend block enables SSE, and pairing it with a customer-managed KMS key is the stronger option for anything holding genuinely sensitive attributes), and access control is ordinary IAM: a bucket policy scoping read/write to the specific roles that legitimately run Terraform against this state, with no broader read access than that.

# Bucket policy fragment: only the platform team's CI role and the
# platform team's own IAM role can read this specific state prefix.
{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/platform-ci" },
  "Action": ["s3:GetObject", "s3:PutObject"],
  "Resource": "arn:aws:s3:::meridian-platform-tfstate/checkout-service/*"
}

Terraform itself also ships a native state-encryption feature (stable in OpenTofu since 1.7, available in Terraform CLI in newer releases) that encrypts the state file's contents before it's ever written to the backend, independent of what encryption the backend storage itself provides — a defense-in-depth layer worth adopting for state holding anything a leaked-but-still-encrypted-at-rest bucket wouldn't fully protect against (an insider with valid IAM read access, for instance).

State Versioning and Recovering From a Bad Apply#

Object versioning on the state bucket (S3 bucket versioning, GCS object versioning, Azure blob versioning) turns every state write into a recoverable point in time — the single most useful safety net for a bad apply that state-file corruption or an unwanted destroy can otherwise turn into a genuine incident.

# List all versions of the state object
aws s3api list-object-versions \
  --bucket meridian-platform-tfstate \
  --prefix checkout-service/network/terraform.tfstate

# Restore a specific prior version as the current object
aws s3api copy-object \
  --bucket meridian-platform-tfstate \
  --copy-source "meridian-platform-tfstate/checkout-service/network/terraform.tfstate?versionId=<prior-version-id>" \
  --key checkout-service/network/terraform.tfstate

HCP Terraform's equivalent is built directly into its UI — a browsable state version history with a one-click rollback, no manual S3 API calls needed, which is one of the concrete conveniences that pricing tradeoff in the previous section is actually buying.

Splitting State — Why One Giant State File Is an Anti-Pattern#

A single state file covering every environment, every service, and every piece of shared infrastructure is the most common structural mistake a growing Terraform codebase makes — and the fix (splitting by service or lifecycle boundary) is a deliberate design decision, not something to back into accidentally.

The right boundary to split along is almost always a real ownership or lifecycle boundary, not an arbitrary file-count target: infrastructure with a genuinely different rate of change (a VPC that changes rarely vs. an autoscaling group's launch template that changes weekly), infrastructure owned by a different team, and infrastructure with a genuinely different blast-radius tolerance (a shared EKS cluster vs. one service's own Lambda functions) are all legitimate split points. checkout-service, catalog-service, and inventory-service each getting their own state, with a shared network and platform state underneath them, is exactly this pattern applied to the throughline system.

terraform_remote_state — Reading Outputs Across State Files#

Once state is split, the terraform_remote_state data source is how one configuration reads another's outputs — the mechanism that lets checkout-service's state reference the shared network state's VPC ID without owning or duplicating that resource itself.

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "meridian-platform-tfstate"
    key    = "shared/network/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_db_instance" "checkout" {
  # ...
  db_subnet_group_name = data.terraform_remote_state.network.outputs.private_subnet_group
}

This is a read-only, one-way dependency — checkout-service's configuration can consume network's outputs, but it never writes to network's state, and network's own apply has no awareness checkout-service exists. This loose coupling is the point: the network team can add resources or refactor internals freely, as long as the exposed outputs' shape stays stable, exactly the same contract a well- designed module's outputs provide (Part 3).

State Manipulation Commands, and Their Blast Radius#

A small family of terraform state subcommands let you directly inspect or edit what's tracked — genuinely necessary tools, and also some of the easiest ways to hurt yourself in the entire CLI, because they bypass the normal plan/review safety net entirely.

terraform state mv is the imperative counterpart to the moved block covered in Part 6 — both solve "this resource's address needs to change without destroying and recreating it," but moved is declarative, reviewable in a PR, and repeatable in CI, while state mv is a one-time, local, unreviewed edit run directly against the backend. Prefer moved blocks for anything that will be applied through your normal pipeline; reserve direct state mv for genuinely one-off, interactive recovery situations.

Performance at Scale: -refresh=false and -target#

Two flags exist specifically for state files large enough that a full plan cycle becomes slow or risky to run in full — both are deliberate escape hatches, not everyday defaults, and both trade completeness for speed in ways worth understanding before reaching for them under pressure.

# Skip the refresh step — trust current state as-is, don't re-query real infrastructure
terraform plan -refresh=false

# Limit planning/apply to one resource (and its dependencies) instead of the whole state
terraform apply -target=aws_db_instance.checkout

-refresh=false skips exactly the step Part 1 identified as how Terraform detects drift — using it means the plan is computed purely from what state already believes is true, which is faster (no round-trip to every provider API) but blind to any change made outside Terraform since the last real refresh. It's a reasonable choice for a routine, low-risk change against a large state file where you've refreshed recently and trust nothing has drifted; it is not a substitute for an occasional full, refreshed plan.

-target narrows a plan/apply to one resource address and whatever it depends on, skipping evaluation of everything else. HashiCorp's own guidance treats this as a break-glass tool for a genuine emergency (a single resource needs an urgent fix and a full-state plan would take too long or touch too much), not a routine workflow — a -targeted apply can leave the configuration and the full state subtly out of sync with each other in ways a subsequent untargeted plan is needed to reconcile.

Migrating From Local to Remote State Without Downtime#

Adding a backend block to an existing local-state configuration and running terraform init triggers Terraform's own guided migration — it detects the backend change and offers to copy existing state into the new location, with no manual JSON surgery required for the common case.

# 1. Add the backend block to your configuration.
# 2. Re-initialize — Terraform detects the backend change automatically.
terraform init

# Terraform prompts:
#   Initializing the backend...
#   Do you want to copy existing state to the new backend?
#     Pre-existing state was found while migrating the previous "local" backend to the
#     newly configured "s3" backend. ... Enter "yes" to copy... "no" to start with an empty state.

Confirming the copy migrates the local terraform.tfstate into the new backend intact — every resource stays tracked, no destroy/recreate, and no downtime for the real infrastructure, because nothing about the actual resources changes, only where their tracking record lives.

Inspecting State Programmatically with terraform show -json#

terraform show -json dumps the current state (or a saved plan file) as structured JSON — the mechanism every serious CI cost-estimation, policy, or drift-reporting tool (Infracost and OPA/Conftest in Part 7 and Part 9 both consume this) actually parses, rather than scraping human-readable CLI output.

terraform show -json terraform.tfstate | jq '.values.root_module.resources[] | {address, type}'

{"address": "aws_db_instance.checkout", "type": "aws_db_instance"}
{"address": "aws_vpc.main", "type": "aws_vpc"}

The same flag against a saved plan file (terraform show -json tfplan) produces a richer structure — every planned change's actions array (["create"], ["update"], ["delete", "create"] for a replace), which is exactly what a CI policy check parses to answer "does this plan destroy anything in production" programmatically, without a human having to read prose plan output for every single run.

This chapter's caption: terraform_remote_state (bottom right) is the only edge that crosses a state boundary — everything else in this diagram lives inside one state file, which is exactly the split point the earlier section on splitting state draws the line at.

Worked Scenario: the destroy Run Against the Wrong Backend#

Early in the platform team's history, before per-service state splitting, an engineer ran terraform destroy intending to tear down a disposable load-testing environment. The working directory's backend configuration, however, still pointed at the shared production key from a copy-pasted backend block nobody had updated for the new environment — the .tf files looked like a fresh load-test setup, but terraform init had silently connected to production's actual state.

The engineer, trusting the directory name over the actual plan output, typed yes. The immediate cause was the stale key value in a copy-pasted backend block; the underlying condition was that nothing in the workflow forced a human to look at which state was about to be destroyed before confirming — the directory name and the backend's actual target had silently diverged, and nothing surfaced that gap. The team's fix, adopted immediately afterward and now standard across every environment: every backend key is generated from the same local.name_prefix the rest of the configuration uses (never hand-typed independently), and every destroy — anywhere, for any reason — requires the same CI-gated plan review as an apply (Part 8), with no local, unreviewed destroy permitted against anything but a genuinely disposable sandbox with its own throwaway backend.

Worked Scenario: Splitting the Platform Team's Monolithic State#

By the time checkout-service, catalog-service, and inventory-service were all live, the team's single state file tracked 140+ resources, and a routine plan for a one-line tag change on catalog-service took over 90 seconds purely refreshing unrelated resources. The team split it into four states — network, platform (the shared EKS cluster and IAM roles), and one state per service — using terraform state mv for the one-time migration:

# Pull current (monolithic) state as a backup first.
terraform state pull > backup-before-split.tfstate

# For each resource that belongs in the new checkout-service state,
# move it out of the monolith and into the new backend/key.
terraform state mv \
  -state-out=checkout-service.tfstate \
  aws_db_instance.checkout aws_db_instance.checkout

Post-split, catalog-service's plans dropped to under 10 seconds, and — the real point — a mistake in catalog-service's configuration could no longer produce a plan that even mentioned checkout-service's resources, since they no longer shared a state file at all. The migration itself was done resource-by-resource during a scheduled low-traffic window, verified with a plan showing zero changes against each new state before considering that piece of the split complete.

Worked Scenario: Recovering From a Corrupted State After a Failed Apply#

An apply against inventory-service was killed mid-run (a CI runner's spot instance was reclaimed) after successfully creating two new resources but before writing the updated state back to the backend — the in-flight lock was never cleanly released, and the backend's recorded state didn't reflect the two resources that now genuinely existed in AWS.

The recovery, in order: first, terraform force-unlock (after confirming, per the earlier warning, that the CI job was actually dead, not just slow), then terraform plan, which showed the two orphaned resources as 0 to add for the ones state already knew about and — critically — a terraform import (Part 6) needed for the two resources that existed in AWS but nowhere in state, since Terraform had no record of them at all. The team confirmed each imported resource's plan showed zero further changes before considering the recovery complete, and used the bucket-versioning history covered earlier to confirm the pre-crash state as a reference point throughout.

Choosing a Backend — a Decision Framework#

With four legitimate backend options on the table across this chapter, the actual decision usually comes down to three questions, not a feature checklist.

Part 2 CLI Cheat Sheet#

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: Two engineers both run terraform apply against the same S3-backed state within seconds of each other, with use_lockfile = true configured. What happens, and why is this safer than local state in the same situation?

Answer: The second apply fails immediately with a lock-held error rather than proceeding — S3's conditional-write-based lock (an If-None-Match PUT that only succeeds if no lock object already exists) means only one client can hold the lock at a time, and the backend rejects the second attempt outright. With local state, no such coordination exists at all — both applies could proceed independently against separate copies of the state file, with the second write silently clobbering the first with no error and no warning.

Problem 2: A team's state file lineage doesn't match what a newly cloned CI runner expects, and terraform plan fails with a lineage-mismatch error. What does this most likely indicate, and what's the safe way to investigate before doing anything else?

Answer: It most likely indicates the backend configuration points at a different state file than the one this configuration's history was built against — a stale or wrong key, a copy-pasted backend block pointed at the wrong bucket/prefix, or a genuinely unrelated state was somehow written to this location. The safe first step is terraform state pull on both the expected and the actual target to compare their resources lists directly, rather than forcing past the mismatch (there's no safe "override lineage" operation) — this is exactly the class of problem the destroy-scenario in this chapter shows going wrong when skipped.

Problem 3: A platform team splits one monolithic state into four, using terraform_remote_state to connect them. Six months later, the network team wants to rename an output from subnet_group to private_subnet_group for clarity. What's the risk in doing this without checking further, and what should happen first?

Answer: Any other state's configuration reading data.terraform_remote_state.network.outputs.subnet_group has no compile-time or plan-time warning that this output is about to disappear — the rename would only surface as a runtime error the next time a dependent configuration's plan runs and finds the referenced output gone. Before renaming, the network team should search (or ask) which other states actually consume this output, coordinate the rename with those teams' own next apply, and — if backward compatibility during a transition period matters — consider exposing both the old and new output names temporarily rather than a single atomic rename.

Problem 4: A CI pipeline runs terraform plan -refresh=false on every pull request for speed, and a full, refreshed plan only on a nightly schedule. A production incident occurs where a manually-deleted security group rule (changed directly in the AWS console during an emergency) wasn't caught by any PR's plan for six hours. Was the pipeline's design flawed, and what's the actual tradeoff being made?

Answer: The design wasn't flawed so much as making a real, known tradeoff explicit — -refresh=false on PR-time plans trades drift visibility for speed, which is a defensible choice for routine review, but it does mean any out-of-band change genuinely isn't visible until the next refreshed plan runs. The nightly full-refresh schedule was the safety net for exactly this gap, and a six-hour visibility window matches what a nightly cadence should be expected to produce. The real fix isn't abandoning -refresh=false on PRs (that would slow down every single review for a rare event) — it's shortening the refreshed-plan interval (hourly instead of nightly) or, better, triggering an immediate refreshed plan on any out-of-band change detected by a separate drift-monitoring signal (CloudTrail-based alerting on manual console changes to Terraform-managed resources), which several teams layer on top of exactly this kind of scheduled-refresh setup.

Summary and What's Next#

State is Terraform's single source of truth for what it manages, and every practice in this chapter follows from taking that seriously: a remote backend with native locking prevents concurrent-write corruption, versioning and encryption make a bad write recoverable and a leaked bucket less catastrophic, splitting state along real ownership boundaries keeps blast radius proportional to the actual change, and the direct state- manipulation commands are powerful enough to deserve real caution, not casual use. terraform_remote_state is what makes splitting state a genuine architectural choice rather than a loss of the ability to share values across boundaries.

Part 3 moves from where state lives to how configuration itself is structured for reuse: writing modules that the checkout-service, catalog-service, and inventory-service state files from this chapter's split can all consume without duplicating the same VPC, database, or IAM logic three times over — including exactly how a module's own state and this chapter's splitting strategy interact.