# Terraform & Infrastructure as Code — Part 2: State Management & Remote Backends > **Series:** Terraform & Infrastructure as Code (2 of 9) > **Part 1:** `01-fundamentals-and-workflow.md` — Fundamentals, HCL & the Plan/Apply Workflow > **Part 2:** This file — State Management & Remote Backends > **Part 3:** `03-modules-and-reusable-design.md` — Modules & Reusable Infrastructure Design > **Part 4:** `04-workspaces-and-environments.md` — Workspaces, Environments & Multi-Account Patterns > **Part 5:** `05-providers-data-sources-and-provisioners.md` — Providers, Data Sources & Provisioners Deep Dive > **Part 6:** `06-drift-detection-import-and-refactoring.md` — Drift Detection, Import & Refactoring Existing Infrastructure > **Part 7:** `07-testing-terraform.md` — Testing Terraform: Static Analysis, the Native Test Framework & Terratest > **Part 8:** `08-cicd-for-terraform.md` — CI/CD for Terraform: Pipelines, Gates & GitOps for Infrastructure > **Part 9:** `09-governance-cost-and-multi-cloud-at-scale.md` — Terraform at Team Scale: Governance, Cost & Multi-Cloud Patterns > **Questions:** `questions.md` Assumes you're comfortable with Part 1's resource addresses, the plan/apply cycle, and the refresh step — this chapter is about the file that makes all of that possible in the first place. ## Table of Contents 1. [Why State Is the Most Dangerous File in a Terraform Project](#why-state-is-the-most-dangerous-file-in-a-terraform-project) 2. [What's Actually Inside a State File](#whats-actually-inside-a-state-file) 3. [Local State, and Why It Fails at Any Team Size](#local-state-and-why-it-fails-at-any-team-size) 4. [Remote Backends — Shared, Locked, Versioned](#remote-backends--shared-locked-versioned) 5. [Configuring the S3 Backend, With Native Locking](#configuring-the-s3-backend-with-native-locking) 6. [State Locking — What It Actually Prevents](#state-locking--what-it-actually-prevents) 7. [Partial Backend Configuration for CI/CD](#partial-backend-configuration-for-cicd) 8. [HCP Terraform as a Managed Backend](#hcp-terraform-as-a-managed-backend) 9. [Encrypting State and Restricting Who Can Read It](#encrypting-state-and-restricting-who-can-read-it) 10. [State Versioning and Recovering From a Bad Apply](#state-versioning-and-recovering-from-a-bad-apply) 11. [Splitting State — Why One Giant State File Is an Anti-Pattern](#splitting-state--why-one-giant-state-file-is-an-anti-pattern) 12. [terraform_remote_state — Reading Outputs Across State Files](#terraform_remote_state--reading-outputs-across-state-files) 13. [State Manipulation Commands, and Their Blast Radius](#state-manipulation-commands-and-their-blast-radius) 14. [Performance at Scale: -refresh=false and -target](#performance-at-scale--refreshfalse-and--target) 15. [Migrating From Local to Remote State Without Downtime](#migrating-from-local-to-remote-state-without-downtime) 16. [Inspecting State Programmatically with terraform show -json](#inspecting-state-programmatically-with-terraform-show--json) 17. [Worked Scenario: the destroy Run Against the Wrong Backend](#worked-scenario-the-destroy-run-against-the-wrong-backend) 18. [Worked Scenario: Splitting the Platform Team's Monolithic State](#worked-scenario-splitting-the-platform-teams-monolithic-state) 19. [Worked Scenario: Recovering From a Corrupted State After a Failed Apply](#worked-scenario-recovering-from-a-corrupted-state-after-a-failed-apply) 20. [Choosing a Backend — a Decision Framework](#choosing-a-backend--a-decision-framework) 21. [Part 2 CLI Cheat Sheet](#part-2-cli-cheat-sheet) 22. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps) 23. [Worked Practice Problems](#worked-practice-problems) 24. [Summary and What's Next](#summary-and-whats-next) --- ## Why State Is the Most Dangerous File in a Terraform Project **Terraform state is the only record that maps your `.tf` configuration's resource addresses to the actual, real-world objects they represent — lose it, corrupt it, or let two people write to it at once, and Terraform's entire "declarative, idempotent" promise collapses into guesswork.** Without state, Terraform would have no way to know that `aws_db_instance.checkout` in your configuration corresponds to a specific RDS instance with a specific ARN somewhere in AWS — it would have to either re-create everything from scratch on every apply, or ask you to manually confirm every mapping. State is what makes "declare what you want, and Terraform figures out the delta" actually work. That same centrality is exactly what makes it dangerous. A state file is simultaneously the thing that lets Terraform delete infrastructure confidently ("this address is gone from config, and state confirms it's the same object that exists live — destroy it") and the single point of failure for that same confidence. This chapter is about the practices that keep state trustworthy at team scale: where it lives, who can write to it, how concurrent writes are prevented, how to recover when something goes wrong, and when one state file has become too large for its own good. > [!WARNING] > Every practice in this chapter exists because of a real, repeated failure mode. State mismanagement — not > a misconfigured resource — is the single most common cause of a Terraform incident serious enough to page > someone, according to nearly every platform team's own postmortem history. Treat this chapter as > production-hardening, not background reading. ## What's Actually Inside a State File **A state file is plain JSON, versioned by a `version` field (currently `4`), containing every tracked resource's full attribute set, plus metadata that makes concurrent-write detection possible.** ```json { "version": 4, "terraform_version": "1.15.8", "serial": 47, "lineage": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "outputs": { "vpc_id": { "value": "vpc-0a1b2c3d", "type": "string" } }, "resources": [ { "mode": "managed", "type": "aws_db_instance", "name": "checkout", "instances": [ { "attributes": { "id": "checkout-db", "engine": "postgres", "endpoint": "checkout-db.abc123.us-east-1.rds.amazonaws.com:5432" } } ] } ] } ``` Two fields matter more than the rest of the schema combined: - **`lineage`** — a UUID generated once, when a state file is first created for a given configuration. Two state files with different lineages are, by definition, unrelated histories — Terraform refuses to treat them as compatible, which is exactly the safety check that catches "someone pointed this workspace at the wrong backend." - **`serial`** — a monotonically incrementing counter, bumped on every write. This is the actual mechanism behind conflict detection: if a client's local view of `serial` is behind what the backend currently holds, something else wrote a newer state since this client last read it, and Terraform refuses to overwrite it blindly. Every resource's `attributes` block holds the **entire** provider-returned object — not just the arguments you wrote in HCL, but every computed attribute the API returned (ARNs, generated IDs, default values the provider filled in). This is why state files routinely contain values you never typed anywhere, including, depending on the resource type, values that should be treated as secrets (an RDS instance's initial master password if it was ever set via a plain `password` argument, a generated API key, a certificate's private key material). ## Local State, and Why It Fails at Any Team Size **By default, with no `backend` block configured, Terraform writes state to a plain `terraform.tfstate` file in the working directory — fine for a five-minute experiment, actively dangerous for anything a second person will ever touch.** ```mermaid flowchart TD Dev1["Engineer A:
terraform apply"] --> Local1["Local terraform.tfstate,
on Engineer A's laptop"] Dev2["Engineer B:
terraform apply,
same config, same time"] --> Local2["Local terraform.tfstate,
on Engineer B's laptop —
a DIFFERENT file"] Local1 --> Conflict["Both applies succeed
independently — no lock,
no shared source of truth"] Local2 --> Conflict Conflict --> Drift["Next apply from either
machine sees a state that
doesn't match what the
OTHER engineer's apply
actually created"] classDef danger fill:#fbe8e6,stroke:#b3261e,color:#10161c classDef normal fill:#e5f0fa,stroke:#1d6fb8,color:#10161c class Conflict,Drift danger class Dev1,Dev2,Local1,Local2 normal ``` Local state fails in three specific, concrete ways beyond the obvious "it's on one laptop": it isn't versioned (a bad apply overwrites the only record of the previous good state, with no history), it isn't locked (two people running `apply` concurrently can corrupt it or silently apply against a stale view), and it isn't accessible to CI (a pipeline runner has no laptop to read from). Any one of these is disqualifying for a team of more than one; all three together make local state a genuine liability the moment a second contributor or an automated pipeline is involved. > [!NOTE] > Local state is legitimately fine for a solo learning environment, a fully disposable sandbox, or a module's > own example/test fixtures that get destroyed at the end of every run — the concern here is specifically > shared, persistent, team-owned infrastructure. ## Remote Backends — Shared, Locked, Versioned **A remote backend moves state storage out of the local filesystem into a shared service that natively supports locking and, ideally, versioning — S3, Azure Blob Storage, Google Cloud Storage, or HCP Terraform's own managed state storage are the common choices.** ```hcl terraform { backend "s3" { bucket = "meridian-platform-tfstate" key = "checkout-service/network/terraform.tfstate" region = "us-east-1" encrypt = true use_lockfile = true } } ``` Every backend, regardless of which one you pick, needs to answer the same three questions: **where** does state live (a bucket, a managed service), **who** can lock it during a write (preventing concurrent writers), and **what happens on conflict** (a stale write is rejected, not silently merged). The `key` argument above is worth pausing on — it's the actual mechanism for splitting state by service/component (covered later in this chapter), since each distinct `key` within the same bucket is a fully independent state file. | Backend | Locking mechanism | Versioning | Best fit | |---|---|---|---| | S3 (`use_lockfile`) | Native S3 conditional writes (Terraform 1.10+) | S3 bucket versioning (separate, opt-in setting) | AWS-native teams not using HCP Terraform | | Azure Blob Storage | Native blob lease | Blob versioning (opt-in) | Azure-native teams | | Google Cloud Storage | Native GCS object generation | Object versioning (opt-in) | GCP-native teams | | HCP Terraform | Built-in, automatic | Built-in, automatic, with a UI history browser | Teams wanting a fully managed backend + run pipeline together | ## Configuring the S3 Backend, With Native Locking **Terraform 1.10 introduced S3-native state locking via `use_lockfile`, eliminating the DynamoDB table that used to be mandatory for safe concurrent S3-backed state — a real simplification worth adopting on any new setup and migrating existing ones toward.** The old pattern required a companion DynamoDB table purely to hold a lock record: ```hcl # The old pattern — still works, but adds an extra piece of infrastructure # purely to support locking. terraform { backend "s3" { bucket = "meridian-platform-tfstate" key = "checkout-service/network/terraform.tfstate" region = "us-east-1" dynamodb_table = "terraform-locks" encrypt = true } } ``` The new pattern uses S3's own conditional-write support (`If-None-Match`) to create a lock object directly in the same bucket, with no second service to provision, monitor, or pay for: ```hcl terraform { backend "s3" { bucket = "meridian-platform-tfstate" key = "checkout-service/network/terraform.tfstate" region = "us-east-1" encrypt = true use_lockfile = true } } ``` Migrating an existing setup is a two-step change: add `use_lockfile = true`, remove `dynamodb_table`, then run `terraform init -reconfigure` to have Terraform pick up the backend change (this does not touch your actual managed resources — only where and how state itself is stored and locked). > [!TIP] > **Best practice**: enable S3 bucket versioning on the state bucket regardless of which locking mechanism > you use — `use_lockfile` solves concurrent-write safety, not accidental-overwrite recovery. Versioning is > what lets you restore the prior state object if a bad apply (or a manual `state push`, covered later) > overwrites current state with something wrong. ## State Locking — What It Actually Prevents **A lock is acquired before any operation that might write state (`plan` acquires a read lock momentarily, `apply` holds a write lock for the operation's duration) and released after — its entire job is making the second scenario below impossible.** ```mermaid sequenceDiagram participant A as Engineer A participant Backend as S3 backend + lock participant B as Engineer B A->>Backend: terraform apply (acquire lock) Backend-->>A: Lock acquired B->>Backend: terraform apply (attempt lock) Backend-->>B: Error: state locked by Engineer A Note over B: B must wait, or investigate
if the lock looks stale A->>Backend: Apply completes, write new state, release lock B->>Backend: Retry — lock now available Backend-->>B: Lock acquired, proceeds safely ``` **This chapter's caption**: the second `apply` doesn't corrupt anything or silently queue — it fails loudly and immediately, which is the correct behavior; a lock that failed *silently* would be far more dangerous than one that blocks. A lock that appears stuck (the holding process crashed mid-apply, or a CI job was killed without cleanup) can be force-released with `terraform force-unlock ` — a command that exists specifically for this recovery case and should never be reached for casually. Force-unlocking while a genuine apply is actually still in progress reopens exactly the corruption window locking exists to prevent. > [!CAUTION] > Before running `force-unlock`, confirm the process that holds the lock is actually dead — check your CI > system for a still-running job, and check with teammates before assuming a lock is stale. A `force-unlock` > against a lock that's still legitimately held is one of the few Terraform operations that can genuinely > corrupt state beyond easy recovery, because it removes the one guarantee protecting a write already in > flight. ## Partial Backend Configuration for CI/CD **A `backend` block's arguments don't all have to be hardcoded in `.tf` files — Terraform supports "partial configuration," where some or all backend settings are supplied at `init` time instead, which is exactly what lets the same configuration target a different bucket/key per environment without editing code.** ```hcl # backend.tf — deliberately incomplete terraform { backend "s3" {} } ``` ```bash # CI supplies the actual values, per environment, at init time terraform init \ -backend-config="bucket=meridian-platform-tfstate" \ -backend-config="key=checkout-service/${ENVIRONMENT}/terraform.tfstate" \ -backend-config="region=us-east-1" \ -backend-config="use_lockfile=true" ``` The same pattern works from a file instead of repeated flags — `-backend-config=prod.backend.hcl` pointing at a small `.hcl` file holding just the backend arguments for that environment. This is the mechanism that makes Part 4's per-environment directory structure and Part 8's CI pipeline actually work together: one shared configuration, a different backend target selected per pipeline run, with no environment-specific values baked into version-controlled `.tf` files at all. > [!TIP] > **Best practice**: never hardcode an environment name into a `bucket` or `key` value inside a committed > `.tf` file if more than one environment will ever run this same configuration — partial configuration keeps > that value where it belongs, supplied by the pipeline (or a per-environment `.hcl` file) at `init` time. ## HCP Terraform as a Managed Backend **HCP Terraform (the current name for what was Terraform Cloud) provides state storage, locking, and versioning as a managed service, plus a run pipeline (Part 8 goes deep on this) — using it as *just* a state backend, with your own CI driving `plan`/`apply`, is a legitimate and common middle-ground choice.** ```hcl terraform { cloud { organization = "meridian-platform" workspaces { name = "checkout-service-network" } } } ``` Every state write goes through HCP Terraform's API instead of directly to a storage bucket you manage — locking, versioning, and a browsable state history UI come for free, with no bucket, IAM policy, or lockfile mechanism to maintain yourself. The tradeoff, covered fully with real numbers in Part 9, is the resource-based pricing that replaced the legacy free tier: for a team running the numbers, self-managed S3 state plus a self-hosted run pipeline can be materially cheaper at scale, at the cost of maintaining that infrastructure yourselves. > [!NOTE] > `cloud` blocks and `backend "remote"` blocks are two different (if related) configuration mechanisms for > pointing at HCP Terraform — `cloud` is the current, actively-developed syntax; `backend "remote"` is the > older form some existing configurations still carry. Prefer `cloud` for anything new. ## Encrypting State and Restricting Who Can Read It **Encryption at rest and access control are the two controls that actually address the secrets-in-state problem Part 1's "Marking Sensitive Values" section flagged but didn't solve.** For an S3 backend, encryption at rest is a bucket-level setting (`encrypt = true` in the backend block enables SSE, and pairing it with a customer-managed KMS key is the stronger option for anything holding genuinely sensitive attributes), and access control is ordinary IAM: a bucket policy scoping read/write to the specific roles that legitimately run Terraform against this state, with no broader read access than that. ```hcl # Bucket policy fragment: only the platform team's CI role and the # platform team's own IAM role can read this specific state prefix. { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:role/platform-ci" }, "Action": ["s3:GetObject", "s3:PutObject"], "Resource": "arn:aws:s3:::meridian-platform-tfstate/checkout-service/*" } ``` Terraform itself also ships a native state-encryption feature (stable in OpenTofu since 1.7, available in Terraform CLI in newer releases) that encrypts the state file's contents before it's ever written to the backend, independent of what encryption the backend storage itself provides — a defense-in-depth layer worth adopting for state holding anything a leaked-but-still-encrypted-at-rest bucket wouldn't fully protect against (an insider with valid IAM read access, for instance). > [!IMPORTANT] > Backend-level encryption (SSE on the bucket) protects against someone getting raw access to the storage > medium. It does **not** protect against someone with legitimate IAM read access to the bucket — that > person reads fully decrypted JSON. Scoping IAM access tightly is not optional hardening on top of > encryption; for state, it's the primary control. ## State Versioning and Recovering From a Bad Apply **Object versioning on the state bucket (S3 bucket versioning, GCS object versioning, Azure blob versioning) turns every state write into a recoverable point in time — the single most useful safety net for a bad apply that state-file corruption or an unwanted destroy can otherwise turn into a genuine incident.** ```bash # List all versions of the state object aws s3api list-object-versions \ --bucket meridian-platform-tfstate \ --prefix checkout-service/network/terraform.tfstate # Restore a specific prior version as the current object aws s3api copy-object \ --bucket meridian-platform-tfstate \ --copy-source "meridian-platform-tfstate/checkout-service/network/terraform.tfstate?versionId=" \ --key checkout-service/network/terraform.tfstate ``` HCP Terraform's equivalent is built directly into its UI — a browsable state version history with a one-click rollback, no manual S3 API calls needed, which is one of the concrete conveniences that pricing tradeoff in the previous section is actually buying. > [!TIP] > **Best practice**: after restoring a prior state version, immediately run `terraform plan` before touching > `apply` — the restored state reflects reality as of that point in time, and anything that changed on the > real infrastructure since then (including whatever the "bad" apply itself did) will show up as drift to > reconcile deliberately, not as a clean slate. ## Splitting State — Why One Giant State File Is an Anti-Pattern **A single state file covering every environment, every service, and every piece of shared infrastructure is the most common structural mistake a growing Terraform codebase makes — and the fix (splitting by service or lifecycle boundary) is a deliberate design decision, not something to back into accidentally.** ```mermaid flowchart LR subgraph Monolith["One state file"] VPC1["VPC"] DB1["checkout DB"] EKS1["EKS cluster"] S31["S3 buckets"] IAM1["IAM roles"] end Monolith --> Slow["Every plan evaluates
ALL of it — slow, and a
typo in checkout's config
can show a diff touching
the shared EKS cluster"] subgraph Split["Split by service/layer"] Network["network state
(VPC, subnets)"] Checkout["checkout-service state"] Catalog["catalog-service state"] Shared["shared-platform state
(EKS, IAM)"] end Split --> Fast["Each plan is small,
fast, and its blast
radius is exactly
that one boundary"] classDef bad fill:#fbe8e6,stroke:#b3261e,color:#10161c classDef good fill:#e5f5ea,stroke:#1f8a4c,color:#10161c class Monolith,Slow bad class Split,Fast good ``` The right boundary to split along is almost always a real ownership or lifecycle boundary, not an arbitrary file-count target: infrastructure with a genuinely different rate of change (a VPC that changes rarely vs. an autoscaling group's launch template that changes weekly), infrastructure owned by a different team, and infrastructure with a genuinely different blast-radius tolerance (a shared EKS cluster vs. one service's own Lambda functions) are all legitimate split points. `checkout-service`, `catalog-service`, and `inventory-service` each getting their own state, with a shared `network` and `platform` state underneath them, is exactly this pattern applied to the throughline system. > [!WARNING] > Splitting too finely has its own cost — dozens of tiny state files with tangled cross-references via > `terraform_remote_state` become their own maintenance burden, and a change that genuinely needs to touch > three of them at once now needs three separate applies, carefully sequenced. Split along real ownership > and blast-radius boundaries, not for the sake of having many small files. ## terraform_remote_state — Reading Outputs Across State Files **Once state is split, the `terraform_remote_state` data source is how one configuration reads another's outputs — the mechanism that lets `checkout-service`'s state reference the shared `network` state's VPC ID without owning or duplicating that resource itself.** ```hcl data "terraform_remote_state" "network" { backend = "s3" config = { bucket = "meridian-platform-tfstate" key = "shared/network/terraform.tfstate" region = "us-east-1" } } resource "aws_db_instance" "checkout" { # ... db_subnet_group_name = data.terraform_remote_state.network.outputs.private_subnet_group } ``` This is a **read-only, one-way** dependency — `checkout-service`'s configuration can consume `network`'s outputs, but it never writes to `network`'s state, and `network`'s own apply has no awareness `checkout-service` exists. This loose coupling is the point: the network team can add resources or refactor internals freely, as long as the exposed outputs' shape stays stable, exactly the same contract a well- designed module's outputs provide (Part 3). > [!TIP] > **Best practice**: treat a foundation state's `outputs.tf` as a real, versioned interface — export whole, > well-named values (`private_subnet_group`, not a raw subnet ID a consumer has to guess the purpose of), and > think twice before removing or renaming an output any other state might already be reading via > `terraform_remote_state`, since Terraform has no way to warn you about that cross-state dependency at plan > time the way it would for an in-module reference. ## State Manipulation Commands, and Their Blast Radius **A small family of `terraform state` subcommands let you directly inspect or edit what's tracked — genuinely necessary tools, and also some of the easiest ways to hurt yourself in the entire CLI, because they bypass the normal plan/review safety net entirely.** | Command | Does | Risk level | |---|---|---| | `terraform state list` | Lists every resource address currently tracked | Read-only, safe | | `terraform state show ` | Prints one resource's full current attributes | Read-only, safe | | `terraform state mv ` | Renames a resource's address in state, without touching real infrastructure | Moderate — gets it wrong and either address can silently vanish from tracking | | `terraform state rm ` | Stops tracking a resource — the real infrastructure is untouched, but Terraform "forgets" it entirely | High — the resource still exists but is now completely unmanaged | | `terraform state pull` / `push` | Downloads/uploads the raw state JSON | Very high — a `push` overwrites the entire remote state file wholesale | `terraform state mv` is the imperative counterpart to the `moved` block covered in Part 6 — both solve "this resource's address needs to change without destroying and recreating it," but `moved` is declarative, reviewable in a PR, and repeatable in CI, while `state mv` is a one-time, local, unreviewed edit run directly against the backend. Prefer `moved` blocks for anything that will be applied through your normal pipeline; reserve direct `state mv` for genuinely one-off, interactive recovery situations. > [!CAUTION] > `terraform state push` overwrites the **entire** remote state object with whatever local file you point it > at — not a merge, not a patch. Running it against the wrong file, or against a file that's out of date > relative to what's actually deployed, can silently make Terraform forget about (or misrepresent) every > resource that existed only in the version it just overwrote. Always `state pull` immediately before, save > that output as a backup, and confirm the file you're about to push is genuinely what you intend before > running it. ## Performance at Scale: -refresh=false and -target **Two flags exist specifically for state files large enough that a full plan cycle becomes slow or risky to run in full — both are deliberate escape hatches, not everyday defaults, and both trade completeness for speed in ways worth understanding before reaching for them under pressure.** ```bash # Skip the refresh step — trust current state as-is, don't re-query real infrastructure terraform plan -refresh=false # Limit planning/apply to one resource (and its dependencies) instead of the whole state terraform apply -target=aws_db_instance.checkout ``` `-refresh=false` skips exactly the step Part 1 identified as how Terraform detects drift — using it means the plan is computed purely from what state *already believes* is true, which is faster (no round-trip to every provider API) but blind to any change made outside Terraform since the last real refresh. It's a reasonable choice for a routine, low-risk change against a large state file where you've refreshed recently and trust nothing has drifted; it is not a substitute for an occasional full, refreshed plan. `-target` narrows a plan/apply to one resource address and whatever it depends on, skipping evaluation of everything else. HashiCorp's own guidance treats this as a break-glass tool for a genuine emergency (a single resource needs an urgent fix and a full-state plan would take too long or touch too much), not a routine workflow — a `-target`ed apply can leave the configuration and the full state subtly out of sync with each other in ways a subsequent *untargeted* plan is needed to reconcile. | Flag | Speeds up | What it sacrifices | When it's appropriate | |---|---|---|---| | `-refresh=false` | Skips provider round-trips during plan | Drift detection for this run | Routine changes against a large, recently-refreshed state | | `-target=` | Skips evaluating unrelated resources | Full-state consistency guarantee | A genuine one-resource emergency fix, followed by an untargeted plan to confirm nothing else needs reconciling | > [!WARNING] > Repeated, habitual use of `-target` instead of fixing why full plans are slow (usually: state that should > have been split per this chapter's earlier section) is a common anti-pattern — it treats the symptom > (slow plans) instead of the actual cause (state too large for its own good), and every `-target`ed apply > leaves a small trust gap until the next full, untargeted plan confirms the whole picture is still > consistent. ## Migrating From Local to Remote State Without Downtime **Adding a `backend` block to an existing local-state configuration and running `terraform init` triggers Terraform's own guided migration — it detects the backend change and offers to copy existing state into the new location, with no manual JSON surgery required for the common case.** ```bash # 1. Add the backend block to your configuration. # 2. Re-initialize — Terraform detects the backend change automatically. terraform init # Terraform prompts: # Initializing the backend... # Do you want to copy existing state to the new backend? # Pre-existing state was found while migrating the previous "local" backend to the # newly configured "s3" backend. ... Enter "yes" to copy... "no" to start with an empty state. ``` Confirming the copy migrates the local `terraform.tfstate` into the new backend intact — every resource stays tracked, no destroy/recreate, and no downtime for the real infrastructure, because nothing about the actual resources changes, only where their tracking record lives. > [!TIP] > **Best practice**: before migrating, run `terraform plan` against the current local state and confirm it > shows zero changes — a clean plan means the migration starts from a known-good baseline, so if anything > looks different immediately after migrating, you know the migration itself (not pre-existing drift) is what > to investigate. ## Inspecting State Programmatically with terraform show -json **`terraform show -json` dumps the current state (or a saved plan file) as structured JSON — the mechanism every serious CI cost-estimation, policy, or drift-reporting tool (Infracost and OPA/Conftest in Part 7 and Part 9 both consume this) actually parses, rather than scraping human-readable CLI output.** ```bash terraform show -json terraform.tfstate | jq '.values.root_module.resources[] | {address, type}' ``` ```json {"address": "aws_db_instance.checkout", "type": "aws_db_instance"} {"address": "aws_vpc.main", "type": "aws_vpc"} ``` The same flag against a saved plan file (`terraform show -json tfplan`) produces a richer structure — every planned change's `actions` array (`["create"]`, `["update"]`, `["delete", "create"]` for a replace), which is exactly what a CI policy check parses to answer "does this plan destroy anything in production" programmatically, without a human having to read prose plan output for every single run. ```mermaid erDiagram STATE ||--o{ RESOURCE : tracks RESOURCE ||--o{ INSTANCE : "has (count/for_each)" RESOURCE }o--|| PROVIDER : "managed by" STATE ||--o{ OUTPUT : exposes OUTPUT }o--o{ TERRAFORM_REMOTE_STATE : "read by other configs" STATE { int version int serial string lineage } RESOURCE { string address string type string mode } OUTPUT { string name bool sensitive } classDef core fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef boundary fill:#f0e9fb,stroke:#6d43c0,color:#10161c class STATE,RESOURCE,INSTANCE,PROVIDER core class OUTPUT,TERRAFORM_REMOTE_STATE boundary ``` **This chapter's caption**: `terraform_remote_state` (bottom right) is the only edge that crosses a state boundary — everything else in this diagram lives inside one state file, which is exactly the split point the earlier section on splitting state draws the line at. > [!TIP] > **Best practice**: build any custom CI tooling (a Slack notification summarizing what a plan will destroy, > a dashboard of resource counts feeding the capacity conversation from earlier in this chapter) against > `terraform show -json` output, never by regex-parsing the human-readable CLI text — the JSON schema is > the stable, documented contract; the prose output format is not. ## Worked Scenario: the destroy Run Against the Wrong Backend Early in the platform team's history, before per-service state splitting, an engineer ran `terraform destroy` intending to tear down a disposable load-testing environment. The working directory's backend configuration, however, still pointed at the shared production `key` from a copy-pasted `backend` block nobody had updated for the new environment — the `.tf` files looked like a fresh load-test setup, but `terraform init` had silently connected to production's actual state. ``` Plan: 0 to add, 0 to change, 34 to destroy. ``` The engineer, trusting the directory name over the actual plan output, typed `yes`. The immediate cause was the stale `key` value in a copy-pasted backend block; the underlying condition was that nothing in the workflow forced a human to look at *which* state was about to be destroyed before confirming — the directory name and the backend's actual target had silently diverged, and nothing surfaced that gap. The team's fix, adopted immediately afterward and now standard across every environment: every backend `key` is generated from the same `local.name_prefix` the rest of the configuration uses (never hand-typed independently), and every `destroy` — anywhere, for any reason — requires the same CI-gated plan review as an `apply` (Part 8), with no local, unreviewed `destroy` permitted against anything but a genuinely disposable sandbox with its own throwaway backend. > [!CAUTION] > A directory's name or its `.tf` file contents tell you nothing about which state it's actually connected > to — only the `backend` block (and, transitively, whatever `key`/`workspace` it resolves to) determines > that. Always read the actual plan's resource count and resource names before confirming a destructive > operation, never just the folder you believe you're standing in. ## Worked Scenario: Splitting the Platform Team's Monolithic State By the time `checkout-service`, `catalog-service`, and `inventory-service` were all live, the team's single state file tracked 140+ resources, and a routine `plan` for a one-line tag change on `catalog-service` took over 90 seconds purely refreshing unrelated resources. The team split it into four states — `network`, `platform` (the shared EKS cluster and IAM roles), and one state per service — using `terraform state mv` for the one-time migration: ```bash # Pull current (monolithic) state as a backup first. terraform state pull > backup-before-split.tfstate # For each resource that belongs in the new checkout-service state, # move it out of the monolith and into the new backend/key. terraform state mv \ -state-out=checkout-service.tfstate \ aws_db_instance.checkout aws_db_instance.checkout ``` Post-split, `catalog-service`'s plans dropped to under 10 seconds, and — the real point — a mistake in `catalog-service`'s configuration could no longer produce a plan that even *mentioned* `checkout-service`'s resources, since they no longer shared a state file at all. The migration itself was done resource-by-resource during a scheduled low-traffic window, verified with a `plan` showing zero changes against each new state before considering that piece of the split complete. ## Worked Scenario: Recovering From a Corrupted State After a Failed Apply An `apply` against `inventory-service` was killed mid-run (a CI runner's spot instance was reclaimed) after successfully creating two new resources but before writing the updated state back to the backend — the in-flight lock was never cleanly released, and the backend's recorded state didn't reflect the two resources that now genuinely existed in AWS. The recovery, in order: first, `terraform force-unlock` (after confirming, per the earlier warning, that the CI job was actually dead, not just slow), then `terraform plan`, which showed the two orphaned resources as `0 to add` for the ones state already knew about and — critically — a `terraform import` (Part 6) needed for the two resources that existed in AWS but nowhere in state, since Terraform had no record of them at all. The team confirmed each imported resource's plan showed zero further changes before considering the recovery complete, and used the bucket-versioning history covered earlier to confirm the pre-crash state as a reference point throughout. > [!NOTE] > This is exactly the class of situation `import` (covered fully in Part 6) exists for — a resource that > genuinely exists in the real infrastructure but has no state record. The scenario here shows *when* you'd > reach for it; Part 6 shows the mechanics in depth, including the newer configuration-driven `import` block > that keeps the recovery itself reviewable in a PR rather than a one-off local command. ## Choosing a Backend — a Decision Framework **With four legitimate backend options on the table across this chapter, the actual decision usually comes down to three questions, not a feature checklist.** | Question | Points toward | |---|---| | Is the team already fully committed to one cloud, with no near-term multi-cloud plan? | That cloud's native object storage backend (S3/GCS/Azure Blob) — no reason to add a cross-cloud dependency purely for state | | Does the team want a managed run pipeline (Part 8) bundled with state storage, and does the resource-based pricing pencil out at current scale? | HCP Terraform — the state-plus-pipeline bundle is genuinely convenient, if the price is acceptable | | Does regulatory or contractual policy require state to never leave infrastructure the company directly controls? | A self-hosted backend (S3/GCS/Azure Blob you own) over any third-party managed service | | Is the team running OpenTofu specifically for license reasons? | A cloud-native backend — Sentinel and HCP Terraform's deepest integration are Terraform-only; OPA/Conftest (Part 9) works with either | > [!TIP] > **Best practice**: don't treat this as a permanent, unchangeable decision — the migration path shown > earlier in this chapter (add a `backend` block, `terraform init`, confirm the copy) works in both > directions. Starting with the simplest option that satisfies today's constraints, and revisiting the > decision once real scale or new requirements arrive, costs far less than over-engineering a backend > strategy for a team of three. ## Part 2 CLI Cheat Sheet | Command | Purpose | |---|---| | `terraform state list` | List every resource address currently tracked | | `terraform state show ` | Print one resource's full current attributes | | `terraform state mv ` | Rename a resource's address without destroying/recreating it | | `terraform state rm ` | Stop tracking a resource (leaves real infrastructure untouched) | | `terraform state pull > backup.tfstate` | Download raw state JSON — always run before a risky operation | | `terraform force-unlock ` | Release a stuck lock — confirm the holder is actually dead first | | `terraform init -reconfigure` | Re-initialize after a backend configuration change | | `aws s3api list-object-versions` | List recoverable prior versions of an S3-backed state object | ## Common Mistakes and Interview Traps | Mistake | Why it's wrong | Correct approach | |---|---|---| | Using local state for anything beyond a solo sandbox | No locking, no versioning, no CI access — the three properties team-scale Terraform actually needs | Configure a remote backend (S3, GCS, Azure Blob, or HCP Terraform) from day one for anything shared | | Assuming `sensitive = true` protects secrets in state | It only redacts CLI/plan output — the state file itself stores the raw value in plain JSON | Encrypt state at rest, scope IAM access tightly, and keep genuine secrets out of Terraform-managed attributes entirely | | Treating one giant state file as simpler to manage | Slower plans, and every change's blast radius spans everything the file tracks | Split state along real ownership/lifecycle boundaries; connect split states via `terraform_remote_state` | | `terraform state push`-ing without pulling a backup first | It's a full overwrite, not a merge — mistakes are not easily reversible without a prior backup | Always `state pull` immediately before any risky manual state operation | | Force-unlocking a lock without confirming the holder is dead | Can corrupt a state write that's still genuinely in progress | Confirm via CI system status and with teammates before force-unlocking | | Confusing directory/folder name with which backend is actually configured | A copy-pasted or stale `backend` block can point anywhere, regardless of what the directory suggests | Always read the actual plan's resource list, never infer target from folder name alone | ## Worked Practice Problems **Problem 1**: Two engineers both run `terraform apply` against the same S3-backed state within seconds of each other, with `use_lockfile = true` configured. What happens, and why is this safer than local state in the same situation? *Answer*: The second `apply` fails immediately with a lock-held error rather than proceeding — S3's conditional-write-based lock (an `If-None-Match` PUT that only succeeds if no lock object already exists) means only one client can hold the lock at a time, and the backend rejects the second attempt outright. With local state, no such coordination exists at all — both applies could proceed independently against separate copies of the state file, with the second write silently clobbering the first with no error and no warning. **Problem 2**: A team's state file lineage doesn't match what a newly cloned CI runner expects, and `terraform plan` fails with a lineage-mismatch error. What does this most likely indicate, and what's the safe way to investigate before doing anything else? *Answer*: It most likely indicates the backend configuration points at a *different* state file than the one this configuration's history was built against — a stale or wrong `key`, a copy-pasted backend block pointed at the wrong bucket/prefix, or a genuinely unrelated state was somehow written to this location. The safe first step is `terraform state pull` on both the expected and the actual target to compare their `resources` lists directly, rather than forcing past the mismatch (there's no safe "override lineage" operation) — this is exactly the class of problem the destroy-scenario in this chapter shows going wrong when skipped. **Problem 3**: A platform team splits one monolithic state into four, using `terraform_remote_state` to connect them. Six months later, the `network` team wants to rename an output from `subnet_group` to `private_subnet_group` for clarity. What's the risk in doing this without checking further, and what should happen first? *Answer*: Any other state's configuration reading `data.terraform_remote_state.network.outputs.subnet_group` has no compile-time or plan-time warning that this output is about to disappear — the rename would only surface as a runtime error the next time a dependent configuration's `plan` runs and finds the referenced output gone. Before renaming, the network team should search (or ask) which other states actually consume this output, coordinate the rename with those teams' own next apply, and — if backward compatibility during a transition period matters — consider exposing both the old and new output names temporarily rather than a single atomic rename. **Problem 4**: A CI pipeline runs `terraform plan -refresh=false` on every pull request for speed, and a full, refreshed `plan` only on a nightly schedule. A production incident occurs where a manually-deleted security group rule (changed directly in the AWS console during an emergency) wasn't caught by any PR's plan for six hours. Was the pipeline's design flawed, and what's the actual tradeoff being made? *Answer*: The design wasn't flawed so much as making a real, known tradeoff explicit — `-refresh=false` on PR-time plans trades drift visibility for speed, which is a defensible choice for routine review, but it does mean any out-of-band change genuinely isn't visible until the next refreshed plan runs. The nightly full-refresh schedule was the safety net for exactly this gap, and a six-hour visibility window matches what a nightly cadence should be expected to produce. The real fix isn't abandoning `-refresh=false` on PRs (that would slow down every single review for a rare event) — it's shortening the refreshed-plan interval (hourly instead of nightly) or, better, triggering an immediate refreshed plan on any out-of-band change detected by a separate drift-monitoring signal (CloudTrail-based alerting on manual console changes to Terraform-managed resources), which several teams layer on top of exactly this kind of scheduled-refresh setup. ## Summary and What's Next State is Terraform's single source of truth for what it manages, and every practice in this chapter follows from taking that seriously: a remote backend with native locking prevents concurrent-write corruption, versioning and encryption make a bad write recoverable and a leaked bucket less catastrophic, splitting state along real ownership boundaries keeps blast radius proportional to the actual change, and the direct state- manipulation commands are powerful enough to deserve real caution, not casual use. `terraform_remote_state` is what makes splitting state a genuine architectural choice rather than a loss of the ability to share values across boundaries. Part 3 moves from where state lives to how configuration itself is structured for reuse: writing modules that the `checkout-service`, `catalog-service`, and `inventory-service` state files from this chapter's split can all consume without duplicating the same VPC, database, or IAM logic three times over — including exactly how a module's own state and this chapter's splitting strategy interact.