# Terraform & Infrastructure as Code — Part 6: Drift Detection, Import & Refactoring Existing Infrastructure > **Series:** Terraform & Infrastructure as Code (6 of 9) > **Part 1:** `01-fundamentals-and-workflow.md` — Fundamentals, HCL & the Plan/Apply Workflow > **Part 2:** `02-state-management-and-remote-backends.md` — State Management & Remote Backends > **Part 3:** `03-modules-and-reusable-design.md` — Modules & Reusable Infrastructure Design > **Part 4:** `04-workspaces-and-environments.md` — Workspaces, Environments & Real-World Repository Structure > **Part 5:** `05-providers-data-sources-and-provisioners.md` — Providers, Data Sources & Provisioners Deep Dive > **Part 6:** This file — Drift Detection, Import & Refactoring Existing Infrastructure > **Part 7:** `07-testing-terraform.md` — Testing Terraform: Static Analysis, the Native Test Framework & Terratest > **Part 8:** `08-cicd-for-terraform.md` — CI/CD for Terraform: Pipelines, Gates & GitOps for Infrastructure > **Part 9:** `09-governance-cost-and-multi-cloud-at-scale.md` — Terraform at Team Scale: Governance, Cost & Multi-Cloud Patterns > **Questions:** `questions.md` Assumes you're comfortable with Part 1's refresh step and Part 2's state manipulation commands — this chapter is the deep, dedicated treatment of everything this series has referenced but not yet fully shown: `moved`, `import`, and the newer `removed` block. ## Table of Contents 1. [What Drift Actually Is, and Why It's Inevitable](#what-drift-actually-is-and-why-its-inevitable) 2. [Detecting Drift: -refresh-only](#detecting-drift--refresh-only) 3. [Reconciling Drift — Three Legitimate Paths](#reconciling-drift--three-legitimate-paths) 4. [Scheduled Drift Detection in CI](#scheduled-drift-detection-in-ci) 5. [Third-Party Drift Tooling](#third-party-drift-tooling) 6. [Preventing Drift at the Source](#preventing-drift-at-the-source) 7. [The moved Block — Refactoring Without Destroying](#the-moved-block--refactoring-without-destroying) 8. [moved Blocks for count-to-for_each Migrations](#moved-blocks-for-count-to-for_each-migrations) 9. [moved Blocks Across Module Boundaries](#moved-blocks-across-module-boundaries) 10. [The removed Block — Letting Go Without Destroying](#the-removed-block--letting-go-without-destroying) 11. [terraform import — the Imperative Original](#terraform-import--the-imperative-original) 12. [The import Block — Declarative, Reviewable Import](#the-import-block--declarative-reviewable-import) 13. [Auto-Generating Configuration From an Import](#auto-generating-configuration-from-an-import) 14. [The Import Workflow, End to End](#the-import-workflow-end-to-end) 15. [Importing Into a Module](#importing-into-a-module) 16. [Bulk Import for a Large, Previously Unmanaged Estate](#bulk-import-for-a-large-previously-unmanaged-estate) 17. [taint and -replace — Forcing a Clean Recreate](#taint-and--replace--forcing-a-clean-recreate) 18. [Drift Severity Across Environments](#drift-severity-across-environments) 19. [A Drift Response Runbook](#a-drift-response-runbook) 20. [Drift With No Human Involved: Provider-Side Default Changes](#drift-with-no-human-involved-provider-side-default-changes) 21. [Worked Scenario: a Manual Console Change Nobody Told Terraform About](#worked-scenario-a-manual-console-change-nobody-told-terraform-about) 22. [Worked Scenario: Importing Three Years of Hand-Built Infrastructure](#worked-scenario-importing-three-years-of-hand-built-infrastructure) 23. [Worked Scenario: a Module Refactor That Nearly Destroyed Production](#worked-scenario-a-module-refactor-that-nearly-destroyed-production) 24. [Part 6 CLI Cheat Sheet](#part-6-cli-cheat-sheet) 25. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps) 26. [Worked Practice Problems](#worked-practice-problems) 27. [Summary and What's Next](#summary-and-whats-next) --- ## What Drift Actually Is, and Why It's Inevitable **Drift is any gap between what Terraform's state believes is true and what's actually true in the real infrastructure — caused by a manual console change, an emergency `kubectl`/CLI fix during an incident, another tool managing the same resource, or the cloud provider itself changing a default.** No team, however disciplined, eliminates drift entirely — incident response routinely requires a fast manual fix under pressure, well before there's time to write and review a proper Terraform change. The goal isn't zero drift; it's *detecting* drift quickly and *reconciling* it deliberately, rather than discovering it as a confusing surprise diff weeks later. ```mermaid flowchart TD Applied["Terraform apply:
state matches reality"] --> Change{"Something changes
the real resource"} Change -->|"Another terraform apply"| StillMatches["State still matches reality"] Change -->|"Manual console/CLI change
(often during an incident)"| Drift["DRIFT: state and reality
now disagree"] Change -->|"A different tool
(autoscaler, another IaC tool)"| Drift Drift --> NextPlan["Next terraform plan's
refresh step surfaces it —
UNLESS caught earlier"] classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c classDef risk fill:#fbeee0,stroke:#b8650f,color:#10161c class Applied,StillMatches ok class Drift,NextPlan risk ``` > [!NOTE] > Part 1 already established that every ordinary `terraform plan` includes a refresh step that would > eventually surface drift — this chapter is about *not waiting* for that eventual, unplanned discovery, and > about the deliberate mechanisms for bringing state and reality back into agreement once drift is found. The > mechanisms in this chapter — `moved`, `removed`, and `import` — are also exactly what Part 3's module > extraction and Part 2's state-splitting scenarios depended on, now shown in full rather than referenced. ## Detecting Drift: -refresh-only **`terraform plan -refresh-only` (Terraform 0.15.4+, the modern replacement for the old standalone `terraform refresh` command) updates Terraform's *understanding* of current reality and shows exactly what changed, without touching real infrastructure and without proposing any remediation — pure detection, fully separated from any remediation decision.** ```bash terraform plan -refresh-only ``` ``` # aws_security_group_rule.checkout_ingress has changed outside of Terraform ~ resource "aws_security_group_rule" "checkout_ingress" { ~ cidr_blocks = [ - "10.0.0.0/16", + "0.0.0.0/0", ] } This is a refresh-only plan, so Terraform will not take any actions to undo these changes. ``` This output is exactly what a "someone widened an ingress rule to the entire internet during an incident and never reverted it" drift incident looks like — a real, security-relevant example of why detection needs to happen faster than "whenever the next unrelated `plan` happens to run." `-detailed-exitcode` makes this usable in automation: exit code `0` means no drift, `2` means drift detected (a real, actionable signal), and `1` means an actual error — a CI job or alert can branch on this directly, which the next section builds on. > [!TIP] > **Best practice**: run drift detection (`-refresh-only`) and ordinary change-planning as two genuinely > separate steps, not conflated into one workflow — a scheduled, unattended `-refresh-only` job answers "has > anything drifted" on its own cadence, independent of whenever the next real feature change happens to be > planned. ## Reconciling Drift — Three Legitimate Paths **Once drift is detected, exactly three responses are legitimate — which one is correct depends entirely on whether the drifted value should actually become the new desired state, or whether Terraform's original configuration was right and the manual change needs to be undone.** ```mermaid flowchart TD Detected["Drift detected via
-refresh-only"] --> Decide{"Was the manual
change actually correct?"} Decide -->|"Yes — it should be
the new desired state"| UpdateConfig["Update .tf files to match
reality, then a normal apply
makes it official going forward"] Decide -->|"No — the manual change
was a mistake or a stopgap"| ApplyRefresh["terraform apply -refresh-only
to accept it into state,
THEN a normal apply reverts
it back to the .tf-declared value"] Decide -->|"The resource shouldn't be
Terraform-managed at all"| IgnoreOrRemove["lifecycle.ignore_changes
for one attribute, or a
removed block (this chapter)
for the whole resource"] classDef path fill:#e5f0fa,stroke:#1d6fb8,color:#10161c class UpdateConfig,ApplyRefresh,IgnoreOrRemove path ``` The middle path deserves care: `terraform apply -refresh-only` **updates state to match reality** but doesn't touch `.tf` files — if the manual change was wrong and the `.tf` files still declare the original, correct value, the very next *ordinary* apply will revert it, which is the intended two-step dance (accept the drift into state first, so the *next* plan's diff is clean and obviously "revert to declared config" rather than a confusing three-way disagreement between old state, new reality, and declared config). | Response | When | Effect | |---|---|---| | Update `.tf` files to match reality | The drift was a legitimate, correct change | Config becomes the new source of truth going forward | | `apply -refresh-only`, then a normal `apply` | The drift was wrong and should be reverted | State first accepts reality, then the normal apply reverts it cleanly | | `ignore_changes` or `removed` | This attribute/resource shouldn't be Terraform's concern | Terraform stops trying to reconcile it at all | ## Scheduled Drift Detection in CI Building directly on `-detailed-exitcode`, most mature teams run a scheduled (nightly, or more frequent for security-sensitive resources) drift-check job entirely separate from the PR-driven pipeline Part 8 covers in depth: ```yaml # .github/workflows/drift-check.yml (illustrative — Part 8 covers full CI/CD) on: schedule: - cron: "0 * * * *" # hourly jobs: drift-check: steps: - run: | terraform plan -refresh-only -detailed-exitcode if [ $? -eq 2 ]; then # Post to Slack, page on-call, whatever this org's alerting policy is echo "Drift detected" fi ``` > [!IMPORTANT] > Part 2's earlier worked practice problem (a six-hour drift-detection gap caused by `-refresh=false` on > every PR plan) is exactly the scenario this scheduled job exists to close — PR-time plans optimize for > review speed, and a separate, regularly-scheduled `-refresh-only` job is what keeps drift visibility from > depending entirely on how often unrelated PRs happen to be opened. ## Third-Party Drift Tooling Beyond the native `-refresh-only` mechanism, a small ecosystem of dedicated drift tools exists for teams wanting continuous, cross-account drift visibility beyond what a single configuration's own scheduled plan covers — `driftctl` (created by Snyk, now in maintenance mode but still functional) was the best-known open-source option; newer entrants (Firefly, and drift-detection features built into commercial run-pipeline platforms like Spacelift and env0) have picked up active development since. | Tool | Scope | Status (2026) | |---|---|---| | `terraform plan -refresh-only` | Per-configuration, native | Actively maintained, the baseline every team has for free | | `driftctl` | Cross-account coverage-scanning, IaC-coverage reporting | Maintenance mode, still usable | | Commercial platforms (Spacelift, env0, HCP Terraform) | Continuous, scheduled, with alerting built in | Actively developed, part of a paid platform | > [!NOTE] > For most teams, native `-refresh-only` on a real schedule (the previous section) closes the overwhelming > majority of the actual gap — reach for a dedicated third-party tool once the need is genuinely broader than > one configuration's own drift (cross-account "what exists that Terraform doesn't even know about at all" > coverage scanning), not as a default first step. ## Preventing Drift at the Source **Detecting and reconciling drift well is necessary, but the cheapest drift to handle is the drift that never happens — restricting who and what can make manual changes to Terraform-managed infrastructure at all is a genuine prevention layer, not just a detection-and-cleanup story.** ```hcl # An SCP (Service Control Policy) at the AWS Organization level (Part 4) — # denies manual console/CLI changes to resources tagged as Terraform-managed, # EXCEPT via the specific role Terraform's CI pipeline itself assumes. { "Effect": "Deny", "Action": ["ec2:*", "rds:*", "s3:*"], "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/ManagedBy": "terraform" }, "StringNotEquals": { "aws:PrincipalArn": "arn:aws:iam::444455556666:role/terraform-prod-deployer" } } } ``` This is a real tradeoff, not a free win — it also blocks the legitimate emergency console fix this chapter's first worked scenario depended on, which is exactly why most teams implement it with a documented break-glass exception process (a separate, audited, time-boxed role that *can* bypass the restriction, used deliberately and logged) rather than an absolute, no-exceptions block. The goal is raising the bar from "any engineer with console access can silently drift a resource" to "drifting a resource requires a deliberate, audited, logged decision to use the break-glass path" — prevention as friction, not prevention as an impossible wall. | Layer | What it prevents | What it can't prevent | |---|---|---| | SCP/IAM deny on tagged resources | Casual, accidental console changes | A deliberate, audited break-glass override (by design) | | Scheduled `-refresh-only` (this chapter) | Drift going undetected for a long window | The drift itself from happening | | A drift response runbook (this chapter) | Inconsistent, ad hoc reconciliation | Nothing — this is purely a *response* layer | > [!TIP] > **Best practice**: treat prevention (restricting console access) and detection (scheduled drift checks) as > complementary, not either/or — prevention reduces how often drift happens; detection catches what > prevention couldn't (a legitimate break-glass exception, a different tool's automated change, a provider- > side default shift) quickly enough to matter. ## The moved Block — Refactoring Without Destroying **A `moved` block declaratively tells Terraform "this resource's address changed, but it's the same real object" — the fix for exactly the "renamed local name → destroy and recreate" trap Part 1 and Part 3 both flagged without yet showing the mechanics.** ```hcl moved { from = aws_instance.web to = aws_instance.app } ``` ``` Terraform will perform the following actions: # aws_instance.web has moved to aws_instance.app resource "aws_instance" "app" { # (no other changes) } Plan: 0 to add, 0 to change, 0 to destroy. ``` Unlike `terraform state mv` (Part 2), a `moved` block lives in version-controlled configuration — it's reviewable in a PR, applies consistently across every environment/workspace that runs this configuration, and runs automatically as part of a normal `plan`/`apply`, with no separate manual step anyone could forget to run against one environment. `moved` blocks are safe to leave in configuration indefinitely (Terraform simply no-ops if the "from" address no longer exists in state), though most teams remove them after confirming every environment has applied the move at least once, to keep the configuration from accumulating stale historical markers forever. > [!TIP] > **Best practice**: any time a resource's address changes for *any* reason — a rename, a move into a > module (Part 3's extraction scenario), a `count`-to-`for_each` conversion (next section) — add a `moved` > block in the same commit as the address change, every time, as a reflex. The cost of an unnecessary > `moved` block is nearly zero; the cost of skipping one is an unplanned destroy-and-recreate landing in > someone's `plan` output. ## moved Blocks for count-to-for_each Migrations Part 1's `count`-vs-`for_each` scenario showed the *problem* a `count`-indexed resource creates when an item is removed from the middle of the list; `moved` blocks are the actual *migration* mechanism for converting an existing `count`-based resource to `for_each` without destroying anything already applied. ```hcl # Before: count = 3, addressed by index # resource "aws_db_instance" "replica" { # count = 3 # ... # } # After: for_each, addressed by a stable key resource "aws_db_instance" "replica" { for_each = toset(["analytics", "reporting", "backup"]) # ... } moved { from = aws_db_instance.replica[0] to = aws_db_instance.replica["analytics"] } moved { from = aws_db_instance.replica[1] to = aws_db_instance.replica["reporting"] } moved { from = aws_db_instance.replica[2] to = aws_db_instance.replica["backup"] } ``` One `moved` block per existing index, mapping it explicitly to its new key — this is precisely the mechanism Part 1's own worked scenario ("count vs. for_each, and getting bitten by it") referenced as the eventual fix the team applied, shown here in full. ## moved Blocks Across Module Boundaries The same mechanism handles Part 3's module-extraction scenario — moving a resource that used to live directly in the root module into a newly-created child module, without destroying it: ```hcl moved { from = aws_vpc.main to = module.network.aws_vpc.main } ``` And, less commonly but just as validly, moving a resource *between* two different child modules, or up out of a module back into the root — the `from`/`to` pair works across any address change, module boundaries included, as long as the underlying resource type is identical on both sides. > [!WARNING] > `moved` cannot change a resource's *type* — only its address. Moving `aws_instance.web` to > `aws_instance.app` works; there's no `moved`-based way to migrate `aws_instance.web` into a conceptually > similar but differently-typed resource (say, if a provider introduced a new resource type replacing an > older one). That kind of migration genuinely requires `import`/`removed` (later in this chapter) or a real > destroy-and-recreate. ## The removed Block — Letting Go Without Destroying **`removed` (Terraform 1.7+) declaratively stops Terraform from managing a resource — with an explicit, reviewable choice about whether the real infrastructure should be destroyed or left alone.** ```hcl removed { from = aws_s3_bucket.legacy_logs lifecycle { destroy = false } } ``` With `destroy = false`, Terraform drops the resource from state entirely, leaving the real bucket untouched — the declarative, version-controlled, PR-reviewable equivalent of `terraform state rm` (Part 2), with the same "leaves real infrastructure orphaned but unmanaged" effect, just expressed as code instead of a one-off local command. Omitting `lifecycle { destroy = false }` (or setting `destroy = true`) instead makes `removed` behave like deleting the resource block outright — a genuine destroy, planned and reviewable exactly like any other planned deletion. | Mechanism | State | Real resource | Reviewable in a PR? | |---|---|---|---| | Delete the `resource` block entirely | Removed | Destroyed | Yes — shows as a planned delete | | `removed { lifecycle { destroy = true } }` | Removed | Destroyed | Yes, with an explicit marker of intent | | `removed { lifecycle { destroy = false } }` | Removed | Untouched, now unmanaged | Yes — the declarative form of `state rm` | | `terraform state rm` (Part 2) | Removed | Untouched, now unmanaged | No — a one-off local command, not in version control | > [!TIP] > **Best practice**: prefer `removed { destroy = false }` over an ad hoc `terraform state rm` for anything > beyond a genuine one-off emergency recovery — it's the same effect, applied consistently across every > environment that runs this configuration, with a clear, permanent record in version control of exactly > when and why a resource was handed off from Terraform's management. ## terraform import — the Imperative Original **The original `terraform import` CLI command brings a real, existing resource under Terraform's state management — but only the state side; you still had to hand-write the matching `.tf` configuration block yourself, and getting that configuration to exactly match the imported state (with zero drift on the very next plan) was, and still is with this form, a real, tedious exercise in trial and error.** ```bash terraform import aws_s3_bucket.legacy_logs meridian-legacy-logs-bucket ``` ```hcl # You must ALSO write this yourself, matching the real resource's # actual configuration closely enough that the next plan shows no changes: resource "aws_s3_bucket" "legacy_logs" { bucket = "meridian-legacy-logs-bucket" # ... every other real attribute, guessed/confirmed by hand } ``` This imperative form is still fully supported and still the fastest option for a genuine one-off, single- resource import — but it's a local, unreviewed command (the same category of concern Part 2 raised about direct `state` manipulation), and the config-writing burden it leaves entirely on you is exactly what the next section's declarative form improves on. ## The import Block — Declarative, Reviewable Import **The `import` block (Terraform 1.5+) moves the same operation into version-controlled configuration, plannable and reviewable exactly like any other change — the modern default over the bare CLI command for anything beyond a quick, disposable one-off.** ```hcl import { to = aws_s3_bucket.legacy_logs id = "meridian-legacy-logs-bucket" } resource "aws_s3_bucket" "legacy_logs" { bucket = "meridian-legacy-logs-bucket" # ... } ``` `terraform plan` with an `import` block present shows the import as an explicit planned action — reviewable in a PR before it ever touches state, exactly the same review discipline Part 1 established for every other kind of change: ``` Terraform will perform the following actions: # aws_s3_bucket.legacy_logs will be imported resource "aws_s3_bucket" "legacy_logs" { id = "meridian-legacy-logs-bucket" # ... } Plan: 1 to import, 0 to add, 0 to change, 0 to destroy. ``` Multiple `import` blocks can exist in one configuration, and — as of newer Terraform releases — `import` blocks now work inside modules, not just the root configuration, closing an early gap this feature launched with. ## Auto-Generating Configuration From an Import **Pairing an `import` block with `-generate-config-out` solves the original CLI command's biggest pain point directly: Terraform inspects the real resource and writes matching HCL for you, rather than leaving you to hand-guess every attribute.** ```bash terraform plan -generate-config-out=generated_resources.tf ``` ```hcl # generated_resources.tf — written BY Terraform, from the import block(s) # in your configuration and the real resource's current attributes resource "aws_s3_bucket" "legacy_logs" { bucket = "meridian-legacy-logs-bucket" # ... every real attribute Terraform could discover } ``` This is a genuinely large improvement over hand-writing import configuration, and still explicitly marked experimental — its output format may change across releases, and it has real gaps: it doesn't work for resources targeted with `count`/`for_each` that don't already exist in configuration, and it doesn't reach into module-internal resources. Treat the generated file as a strong first draft, not a final answer — review it line by line, confirm the very next `plan` shows zero changes, and only then commit it. > [!WARNING] > Never commit generated configuration without reading it first. `-generate-config-out` reflects the > resource's *current* real attributes exactly, including anything accidental or undesirable about how it > was originally, manually created (an overly permissive setting, a missing tag) — importing faithfully is > not the same as importing *correctly*; review before committing, and clean up anything the generated > config faithfully captured but shouldn't have. ## The Import Workflow, End to End **Pulling the last three sections into one sequence — the actual, repeatable process a platform team runs for every real import, not just the individual mechanics in isolation.** ```mermaid sequenceDiagram participant Eng as Engineer participant TF as Terraform participant Cloud as Real infrastructure Eng->>TF: Write an import block (to + id) Eng->>TF: terraform plan -generate-config-out=generated.tf TF->>Cloud: Read the real resource's current attributes Cloud-->>TF: Full attribute set TF-->>Eng: generated.tf written — a first draft, not final Eng->>Eng: Review and clean up generated.tf by hand Eng->>TF: terraform plan (with reviewed config) TF-->>Eng: Confirm "0 to add, 0 to change, 0 to destroy" Eng->>TF: terraform apply (records the import in state) Eng->>Eng: Remove the import block (no longer needed) ``` **This chapter's caption**: the "0 to add, 0 to change, 0 to destroy" confirmation step is the actual gate — skipping straight from generated config to `apply` without confirming a clean plan first is how a subtly wrong generated attribute silently becomes a real, unintended change instead of a faithful import. > [!NOTE] > The `import` block itself is only needed for the one `apply` that performs the import — once state > reflects the resource, the block has no further purpose and is typically removed in a follow-up commit, > the same way a completed database migration script isn't re-run on every future deploy. ## Importing Into a Module Bringing a pre-existing resource into a module (rather than the root configuration) combines this chapter's `import` mechanics with Part 3's module addressing: ```hcl import { to = module.network.aws_vpc.main id = "vpc-0a1b2c3d" } ``` This is exactly the mechanic Part 2's "recovering from a corrupted state" scenario referenced — a resource that genuinely exists in real infrastructure but has no state record needs `import` to bring it back under management, and if that resource conceptually belongs inside an already-extracted module (Part 3), the import target is the module-prefixed address, not a bare resource address. ## Bulk Import for a Large, Previously Unmanaged Estate **A single resource's `import` block is straightforward; bringing an entire pre-existing account under management (this chapter's later "three years of hand-built infrastructure" scenario, at real scale) needs a repeatable, scriptable approach rather than hand-writing dozens or hundreds of import blocks one at a time.** ```bash # Enumerate real resources of a given type via the cloud provider's own CLI, # generate one import block per resource programmatically aws s3api list-buckets --query 'Buckets[].Name' --output text | tr '\t' '\n' | while read -r bucket; do cat >> generated_imports.tf < [!WARNING] > Bulk-generating `import` blocks is a scripting convenience for the *boilerplate*, never a shortcut around > this chapter's per-resource review discipline — every generated configuration still needs a confirmed clean > plan before it's trusted, exactly as the single-resource case demands. A bulk import that skips review > "because there are too many resources to check individually" is exactly how a faithfully-imported but > genuinely misconfigured resource (an overly permissive bucket policy from three years of ad hoc console > changes) quietly becomes permanent, committed configuration instead of getting caught and fixed. > [!TIP] > **Best practice**: pace a bulk import the same way the "importing three years of hand-built infrastructure" > scenario later in this chapter does — lowest-risk resource types first, a confirmed clean plan after every > single resource, and two-person review on anything touching data storage or IAM, even when the volume makes > a faster, less careful pace tempting. ## taint and -replace — Forcing a Clean Recreate **Sometimes the correct reconciliation for drift isn't accepting or reverting a value — it's forcing a clean destroy-and-recreate of a resource that's degraded or misconfigured in a way no attribute-level fix addresses.** ```bash # Modern, apply-scoped form — the current recommended approach terraform apply -replace="aws_instance.worker" # Older, persistent form — marks the resource tainted in STATE until # the next apply, which can surprise a later, unrelated apply terraform taint aws_instance.worker ``` `-replace` is scoped to one specific `apply` invocation — it doesn't persist any marker in state the way the older `taint` command does, which is exactly why it's the currently recommended approach: a forgotten `taint` sitting in state can cause a resource to be unexpectedly replaced by a completely unrelated later apply that nobody remembered the taint was even there for. > [!NOTE] > This is also exactly the mechanism referenced by Part 5's provisioner discussion — a creation-time > provisioner that failed and left a resource `tainted` (the default `on_failure = fail` behavior) will be > replaced on the next apply; `-replace` is the same underlying mechanic, invoked deliberately instead of as > an automatic consequence of provisioner failure. ## Drift Severity Across Environments **Not every environment's drift deserves the same response speed — Part 4's directory-per-environment structure makes it natural to tune drift-check frequency and alerting severity per environment, matching each one's real stakes rather than treating dev and prod identically.** | Environment | Drift-check cadence | Alert routing | Rationale | |---|---|---|---| | `dev` | Daily, or on-demand only | A dashboard, no page | Low stakes; engineers actively experiment here, some drift is expected noise | | `staging` | Hourly | A team Slack channel | Should mirror prod closely enough that drift here is worth investigating promptly | | `prod` | Every 15-30 minutes, for security-sensitive resource types especially | Pages on-call, per severity | Highest stakes — the six-hour gap from Part 2's practice problem is exactly what tighter cadence here closes | This tuning is exactly why Part 4's account-per-environment and directory-per-environment structure pays off again here — each environment's drift-check job runs against its own state/backend independently, so `dev`'s higher noise tolerance never has to be balanced against `prod`'s low tolerance in one shared configuration or one shared alert channel. > [!TIP] > **Best practice**: don't page on-call for `dev` drift, and don't silently log `prod` drift to a dashboard > nobody watches in real time — matching alert severity to actual environment stakes keeps the signal > meaningful; an on-call rotation that gets paged for routine `dev` noise will very predictably start > ignoring drift alerts altogether, the same alert-fatigue failure mode from incident management generally. ## A Drift Response Runbook Pulling this chapter's mechanisms into one operational checklist for a platform team's on-call rotation: - [ ] Confirm the drift with `terraform plan -refresh-only` — read the full diff before deciding anything - [ ] Identify *why* the drift happened — an incident fix, a different tool, a provider default change (feeds directly into whether this specific drift should recur, or was truly one-off) - [ ] Decide which of the three reconciliation paths applies (config update, revert, or ignore/remove) - [ ] If reverting: `apply -refresh-only` first, then a normal `apply`, confirming the plan matches expectations at each step - [ ] If the manual change should become permanent: update `.tf` files, open a normal reviewed PR — never skip review just because the change already exists live - [ ] If the resource shouldn't be Terraform-managed going forward: use `removed { destroy = false }`, not an ad hoc `state rm` - [ ] Document the drift and its resolution somewhere durable — a recurring drift source (the same security group, repeatedly) is itself a signal worth escalating, not just repeatedly reconciling - [ ] If the drift traced back to a genuine emergency console change, confirm the break-glass process (this chapter's prevention section) was actually used, rather than an unrestricted credential - [ ] Confirm the environment's drift-check cadence still matches its actual severity tier — a repeated incident in one environment may justify tightening that environment's own detection frequency - [ ] Close the loop with whoever made the original change, even when it was correct — a quick heads-up keeps the next drift from being an equal surprise to the runbook's own on-call rotation ## Drift With No Human Involved: Provider-Side Default Changes **Not all drift traces back to a person — a provider version bump (Part 1's module-upgrade scenario, applied here to provider versions directly) can change a resource's default value between API versions, producing drift with zero manual changes anywhere.** A concrete, real pattern: a cloud provider's API introduces a new default for a previously-optional field (similar in spirit to the `map_public_ip_on_launch` default change Part 3's module scenario hit), and the *next* provider version upgrade that picks up the new API behavior shows every existing resource of that type as drifted — not because anything about the real resource changed, but because the provider's own interpretation of "what does this field's absence mean" changed underneath it. ``` ~ resource "aws_db_instance" "checkout" { ~ storage_type = "gp2" -> "gp3" # Provider version bump changed the # inferred default for an unset argument } ``` | Drift source | Human involved? | Detected by | |---|---|---| | A manual console/CLI change | Yes | `-refresh-only` | | A different tool managing the same resource | Indirectly (someone configured that tool) | `-refresh-only` | | A provider version bump changing an inferred default | No | `-refresh-only`, surfacing immediately after the provider upgrade | > [!NOTE] > This is exactly why Part 1's "review the changelog, not just the version diff" guidance for provider > upgrades matters here too — a provider changelog entry noting a changed default for an unset argument is > the advance warning that would let a team set the argument explicitly *before* upgrading, avoiding a > surprise drift diff appearing immediately after the bump with no human action in between. ## Worked Scenario: a Manual Console Change Nobody Told Terraform About During a genuine production incident, an on-call engineer widened `checkout-service`'s security group to allow traffic from a debugging tool's IP range, directly in the AWS console, to unblock an active outage — the right call under the circumstances, but never followed up afterward. The platform team's scheduled `-refresh-only` drift check (this chapter's dedicated CI job) caught the change the following night, well before it would have otherwise surfaced in the next unrelated feature PR's plan. The team's response followed the drift runbook directly: confirmed the change was a genuine incident stopgap (not an intentional permanent widening), ran `apply -refresh-only` to accept the current state, then a normal `apply` to revert it back to the originally-declared, narrower rule — closing the gap within a day of the incident rather than leaving an overly-permissive rule live indefinitely until someone happened to notice. ## Worked Scenario: Importing Three Years of Hand-Built Infrastructure Before the platform team existed, `inventory-service`'s original infrastructure was built entirely by hand in the AWS console over roughly three years — no Terraform, no consistent tagging, real configuration drift even against itself across environments. Bringing it under management used `import` blocks paired with `-generate-config-out`, resource by resource, starting with the lowest-risk, easiest-to-verify pieces (S3 buckets, IAM roles) before working up to the database and its security groups. Each generated configuration file was reviewed by two engineers before being committed — not a rubber stamp, because the generated HCL faithfully captured real inconsistencies (three years of ad hoc console changes had left slightly different tagging on nearly every resource) that needed cleaning up, not blind preservation. The full import took several weeks, deliberately paced rather than rushed, with a `terraform plan` showing zero changes confirmed after every single resource's import before moving to the next — exactly the discipline this chapter's import-block section recommends, applied at real scale. ## Worked Scenario: a Module Refactor That Nearly Destroyed Production A well-intentioned refactor renamed several internal resource names inside the `database` module (Part 3) for clarity — `aws_db_instance.this` to `aws_db_instance.primary`, in preparation for adding a future read replica — without adding matching `moved` blocks, on the mistaken assumption that "it's just an internal module implementation detail, callers won't notice." The very next `plan` against every environment calling this module showed `1 to destroy, 1 to add` for every single database the module managed — the internal rename was, from Terraform's perspective, exactly as consequential as any other address change, module- internal or not. The plan-review discipline (Part 1) caught it before any `apply` reached production, and the fix was exactly this chapter's `moved` block mechanism, applied retroactively: ```hcl moved { from = aws_db_instance.this to = aws_db_instance.primary } ``` > [!CAUTION] > "It's just an internal rename, nobody outside the module will notice" is a dangerous assumption — Terraform > tracks resource addresses, not human intent, and an address change is consequential regardless of whether a > human reader considers it "internal." Any resource rename, anywhere, in any module, needs a `moved` block > as a reflex, not a judgment call about whether it "counts." ## Part 6 CLI Cheat Sheet | Command | Purpose | |---|---| | `terraform plan -refresh-only -detailed-exitcode` | Detect drift, scriptable (exit 2 = drift found) | | `terraform apply -refresh-only` | Accept real infrastructure's current state into Terraform state | | `terraform import ` | Imperative, one-off import (CLI form) | | `terraform plan -generate-config-out=` | Generate HCL for pending `import` blocks | | `terraform apply -replace=` | Force a clean destroy-and-recreate for one apply | | `terraform state rm ` | Ad hoc, unreviewed equivalent of `removed { destroy = false }` — prefer the block form | ## Common Mistakes and Interview Traps | Mistake | Why it's wrong | Correct approach | |---|---|---| | Waiting for the next unrelated `plan` to discover drift | Can leave a security-relevant or costly drift live for days/weeks | Run scheduled, dedicated `-refresh-only` drift checks, separate from feature-change plans | | Renaming a resource without a `moved` block, "because it's just internal" | Terraform tracks addresses, not intent — any rename is a destroy/recreate without one | Add a `moved` block for every address change, module-internal or not | | Using `terraform state rm` as a routine practice | Unreviewed, local-only, not applied consistently across environments | Prefer `removed { lifecycle { destroy = false } }` for anything beyond a genuine emergency | | Committing `-generate-config-out` output without reviewing it | Faithfully captures any pre-existing misconfiguration or inconsistency | Review generated config line by line before committing, exactly like any other change | | Leaving a `taint` marker instead of using `-replace` | Persists in state and can surprise a later, unrelated apply | Prefer `-replace`, scoped to one specific apply invocation | | Treating `apply -refresh-only` as "fixing" the drift | It only updates state to match reality — it doesn't revert anything | Follow with a normal `apply` if the drift should actually be reverted | ## Worked Practice Problems **Problem 1**: A `terraform plan -refresh-only` shows a security group rule changed from a narrow CIDR block to `0.0.0.0/0`. What are the two possible correct responses, and what determines which one is right? *Answer*: Either update the `.tf` configuration to declare `0.0.0.0/0` (if the change was legitimate and should persist) or run `apply -refresh-only` followed by a normal `apply` to revert it back to the originally declared narrow CIDR (if the change was a mistake or an incident stopgap that was never meant to be permanent). Which is correct depends entirely on investigating *why* the change happened — the drift itself carries no information about intent, only about the fact that a disagreement exists. **Problem 2**: An engineer renames `aws_instance.web` to `aws_instance.frontend` in one PR, with no `moved` block, reasoning "the plan will just show it as a rename, not a destroy." What actually happens, and why is that reasoning wrong? *Answer*: The plan shows `1 to add (aws_instance.frontend), 1 to destroy (aws_instance.web)` — Terraform has no concept of "rename" as a first-class plan action; it only ever sees two distinct addresses, one gone and one new, and its default inference is destroy-then-create. The reasoning is wrong because it assumes Terraform reasons about *semantic* similarity between two resources, when it actually reasons purely about *address* identity — a `moved` block is what supplies the "these are actually the same object" information Terraform has no other way to infer. **Problem 3**: A team runs `terraform import aws_s3_bucket.legacy mybucket` successfully, then finds that their very next `terraform plan` still shows several changes rather than a clean "no changes." What does this indicate, and what's the correct next step — re-running the import? *Answer*: This indicates the hand-written (or `-generate-config-out`-generated but not fully verified) `.tf` configuration for `aws_s3_bucket.legacy` doesn't yet exactly match the real bucket's current attributes — the import itself succeeded (state now tracks the resource), but the *configuration* is still out of sync with it. Re-running the import doesn't fix this — the import already succeeded once and doesn't need repeating. The correct next step is reading the `plan` diff carefully and adjusting the `.tf` configuration's arguments to match what the diff shows, repeating until `plan` shows zero changes, exactly the same iterative process the "importing three years of hand-built infrastructure" scenario describes at real scale. **Problem 4**: An organization implements an SCP denying manual changes to Terraform-tagged resources, with no break-glass exception process. Six months later, a production incident requires an emergency security group change, and the on-call engineer's console attempt is denied outright, with no documented alternative. What went wrong in the policy's design, and what should have been included from the start? *Answer*: The SCP achieved pure prevention with no accounted-for exception path — exactly the failure mode this chapter's prevention section warns against, treating prevention as "an impossible wall" instead of "a deliberate, audited friction." A production incident occasionally does need a fast manual fix before there's time to write and review a proper Terraform change (this chapter's very first worked scenario depends on this being possible); the fix is a documented, separate, time-boxed break-glass role that can bypass the SCP, used deliberately and logged, paired with the expectation that any break-glass change gets caught and reconciled by the next scheduled `-refresh-only` drift check — prevention and detection working together, not prevention alone with no escape valve for genuine emergencies. ## Summary and What's Next Drift is inevitable, not a sign of process failure — the discipline that matters is detecting it on a deliberate schedule (`-refresh-only`, scripted with `-detailed-exitcode`) rather than waiting for it to surface as a confusing surprise, and reconciling it through one of exactly three legitimate paths rather than guessing. `moved` and `removed` blocks turn what used to require unreviewed, local `state` commands (Part 2) into version-controlled, PR-reviewable, consistently-applied configuration — and `import` blocks, especially paired with `-generate-config-out`, do the same for bringing previously-unmanaged infrastructure under Terraform's control. Every mechanism in this chapter shares one theme: making an operation that used to be imperative, local, and easy to forget into something declarative, reviewable, and consistent across every environment that runs the configuration. Part 7 shifts from reactive (detecting and fixing problems after they exist) to proactive: testing Terraform configuration and modules *before* a bad change ever reaches a real environment at all — static analysis with `tflint` and `checkov`, the native `terraform test` framework, and Terratest for genuine integration testing against real, ephemeral infrastructure. Several of that chapter's test cases exist specifically to catch the exact regressions this chapter's worked scenarios showed slipping past a human reviewer, turning a lesson learned the hard way into an automated check nobody has to remember to run by hand.