# Terraform & Infrastructure as Code — Part 8: CI/CD for Terraform: Pipelines, Gates & GitOps for Infrastructure > **Series:** Terraform & Infrastructure as Code (8 of 9) > **Part 1:** `01-fundamentals-and-workflow.md` — Fundamentals, HCL & the Plan/Apply Workflow > **Part 2:** `02-state-management-and-remote-backends.md` — State Management & Remote Backends > **Part 3:** `03-modules-and-reusable-design.md` — Modules & Reusable Infrastructure Design > **Part 4:** `04-workspaces-and-environments.md` — Workspaces, Environments & Real-World Repository Structure > **Part 5:** `05-providers-data-sources-and-provisioners.md` — Providers, Data Sources & Provisioners Deep Dive > **Part 6:** `06-drift-detection-import-and-refactoring.md` — Drift Detection, Import & Refactoring Existing Infrastructure > **Part 7:** `07-testing-terraform.md` — Testing Terraform: Static Analysis, the Native Test Framework & Terratest > **Part 8:** This file — CI/CD for Terraform: Pipelines, Gates & GitOps for Infrastructure > **Part 9:** `09-governance-cost-and-multi-cloud-at-scale.md` — Terraform at Team Scale: Governance, Cost & Multi-Cloud Patterns > **Questions:** `questions.md` Assumes you're comfortable with Part 1's saved-plan-file mechanics, Part 4's environment/account structure, and Part 7's full test suite — this chapter is where all three finally get wired together into one running pipeline. ## Table of Contents 1. [Why Terraform CI/CD Isn't Application CI/CD](#why-terraform-cicd-isnt-application-cicd) 2. [The Plan/Apply Split, as a Pipeline Contract](#the-planapply-split-as-a-pipeline-contract) 3. [OIDC — Eliminating Long-Lived Cloud Credentials in CI](#oidc--eliminating-long-lived-cloud-credentials-in-ci) 4. [A GitHub Actions Pipeline, Stage by Stage](#a-github-actions-pipeline-stage-by-stage) 5. [Posting Plan Output as a PR Comment](#posting-plan-output-as-a-pr-comment) 6. [Secrets in the Pipeline — Injection, Not Hardcoding](#secrets-in-the-pipeline--injection-not-hardcoding) 7. [Redacting Sensitive Values From Posted Plan Output](#redacting-sensitive-values-from-posted-plan-output) 8. [Approval Gates — Environments and Required Reviewers](#approval-gates--environments-and-required-reviewers) 9. [Wiring In Part 7's Full Test Suite](#wiring-in-part-7s-full-test-suite) 10. [Atlantis — PR-Comment-Driven Terraform Automation](#atlantis--pr-comment-driven-terraform-automation) 11. [Atlantis vs. a Hand-Rolled Pipeline](#atlantis-vs-a-hand-rolled-pipeline) 12. [State Locking in CI — Preventing Concurrent Pipeline Runs](#state-locking-in-ci--preventing-concurrent-pipeline-runs) 13. [Promotion Through Environments, in the Pipeline](#promotion-through-environments-in-the-pipeline) 14. [GitOps for Infrastructure — Pull vs. Push](#gitops-for-infrastructure--pull-vs-push) 15. [What "Rollback" Actually Means for Terraform](#what-rollback-actually-means-for-terraform) 16. [Skipping a No-Op Apply](#skipping-a-no-op-apply) 17. [Worked Scenario: Two PRs, One Stale Plan](#worked-scenario-two-prs-one-stale-plan) 18. [Worked Scenario: Building checkout-service's Pipeline End to End](#worked-scenario-building-checkout-services-pipeline-end-to-end) 19. [Worked Scenario: an OIDC Trust Policy Too Loosely Scoped](#worked-scenario-an-oidc-trust-policy-too-loosely-scoped) 20. [Part 8 Pipeline Cheat Sheet](#part-8-pipeline-cheat-sheet) 21. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps) 22. [Worked Practice Problems](#worked-practice-problems) 23. [Summary and What's Next](#summary-and-whats-next) --- ## Why Terraform CI/CD Isn't Application CI/CD **An application pipeline builds an artifact and deploys it — mostly reversible, mostly idempotent in the "redeploy the old version" sense. A Terraform pipeline's "artifact" is a *plan against live infrastructure*, and applying it can be destructive, can cost real money, and — per Part 1's very first worked scenario — can delete a production database if nobody reads it carefully.** Every practice in this chapter follows from that one difference: a Terraform pipeline needs the plan to be visible, reviewed by a human, and applied *exactly* as reviewed — not re-computed at apply time, not approved based on a description of the change rather than the change itself. ```mermaid flowchart LR AppCICD["Application CI/CD:
build artifact, deploy,
rollback = redeploy
previous artifact"] --> AppRisk["Mostly reversible —
the old artifact still exists"] TFCICD["Terraform CI/CD:
plan against LIVE state,
apply = real infra change"] --> TFRisk["Can be destructive,
costly, and hard to
reverse — 'rollback' means
something different entirely"] classDef app fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef tf fill:#fbeee0,stroke:#b8650f,color:#10161c class AppCICD,AppRisk app class TFCICD,TFRisk tf ``` ## The Plan/Apply Split, as a Pipeline Contract **Part 1 introduced `-out=tfplan` for exact-plan application; a real pipeline turns that into a hard contract between two separate CI jobs — a `plan` job producing a reviewable artifact, and a completely separate `apply` job consuming *exactly* that artifact, with no re-computation in between.** ```mermaid sequenceDiagram participant PR as Pull request participant PlanJob as CI: plan job participant Human as Reviewer participant ApplyJob as CI: apply job (on merge) PR->>PlanJob: Triggered on push PlanJob->>PlanJob: terraform plan -out=tfplan PlanJob->>PR: Post plan output as a comment PlanJob->>PlanJob: Upload tfplan as a CI artifact Human->>PR: Reads the plan, approves PR->>ApplyJob: Merge triggers apply job ApplyJob->>ApplyJob: Download the SAME tfplan artifact ApplyJob->>ApplyJob: terraform apply tfplan (no re-plan) ``` **This chapter's caption**: the artifact uploaded by the plan job and consumed by the apply job must be the *exact same file* — if the apply job instead runs a fresh `terraform plan` internally, everything Part 1 warned about (a plan drifting between review time and apply time) is back in play, silently, even with a human-reviewed comment sitting right there in the PR. > [!IMPORTANT] > This is the single most important structural rule in this entire chapter: **never let the apply stage > re-run `plan`.** A pipeline that looks correct (it has a plan stage, a review step, an apply stage) but > internally re-plans at apply time has quietly defeated the entire purpose of showing a human the plan at > all — the reviewer approved one thing; the pipeline may apply a different one. ## OIDC — Eliminating Long-Lived Cloud Credentials in CI **Rather than storing a long-lived AWS access key as a CI secret (a real, ongoing liability — leaked secrets, manual rotation, broad standing permissions), OIDC lets CI request short-lived, narrowly-scoped credentials fresh on every single run, with nothing long-lived stored anywhere.** ```mermaid sequenceDiagram participant GHA as GitHub Actions job participant GH as GitHub's OIDC provider participant AWS as AWS STS GHA->>GH: Request an OIDC token (id-token: write permission) GH-->>GHA: Signed JWT — repo, branch, workflow claims embedded GHA->>AWS: AssumeRoleWithWebIdentity(JWT) AWS->>AWS: Validate JWT signature + trust policy's sub/aud claims AWS-->>GHA: Short-lived STS credentials, scoped to this one job's role ``` ```yaml permissions: id-token: write contents: read jobs: plan: steps: - uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: arn:aws:iam::333344445555:role/github-actions-checkout-plan aws-region: us-east-1 ``` ```json // The AWS IAM role's trust policy — scoped to a specific repo AND branch, // not "any workflow from anywhere in this org" { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::333344445555:oidc-provider/token.actions.githubusercontent.com" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "token.actions.githubusercontent.com:aud": "sts.amazonaws.com" }, "StringLike": { "token.actions.githubusercontent.com:sub": "repo:meridian-platform/infrastructure-live:ref:refs/heads/main" } } } ``` The `sub` claim condition is the entire security model — scoping it to `ref:refs/heads/main` specifically (rather than a broad `repo:meridian-platform/infrastructure-live:*`) means only workflow runs against the `main` branch can assume this particular role, closing off a fork or a feature-branch PR from ever obtaining production-capable credentials, even from within the same repository. > [!TIP] > **Best practice**: use OIDC over static access keys for every CI-to-cloud connection, full stop — it > eliminates an entire class of secret-leak risk (there's no long-lived key to leak), removes manual rotation > entirely, and its short-lived, narrowly-scoped nature means a compromised CI run has a far smaller and > shorter-lived blast radius than a leaked static key would. ## A GitHub Actions Pipeline, Stage by Stage Pulling Part 1's saved-plan contract and OIDC together into a real, minimal pipeline: ```yaml name: terraform on: pull_request: paths: ["environments/prod/checkout-service/**"] push: branches: [main] paths: ["environments/prod/checkout-service/**"] permissions: id-token: write contents: read pull-requests: write jobs: plan: if: github.event_name == 'pull_request' runs-on: ubuntu-latest defaults: run: working-directory: environments/prod/checkout-service steps: - uses: actions/checkout@v5 - uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: arn:aws:iam::333344445555:role/github-actions-checkout-plan aws-region: us-east-1 - run: terraform init - run: terraform plan -out=tfplan - uses: actions/upload-artifact@v4 with: { name: tfplan, path: environments/prod/checkout-service/tfplan } apply: if: github.event_name == 'push' runs-on: ubuntu-latest environment: production # Gate — covered in the next section defaults: run: working-directory: environments/prod/checkout-service steps: - uses: actions/checkout@v5 - uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: arn:aws:iam::333344445555:role/github-actions-checkout-apply aws-region: us-east-1 - uses: actions/download-artifact@v4 with: { name: tfplan, path: environments/prod/checkout-service } - run: terraform init - run: terraform apply tfplan # The SAME artifact — no re-plan ``` Notice two **distinct** IAM roles — `github-actions-checkout-plan` and `github-actions-checkout-apply` — rather than one shared role for both stages. This follows directly from Part 4's least-privilege guidance: the plan role only needs read permissions (enough to compute a diff), while the apply role needs the actual write permissions to create/modify/destroy — a compromised or buggy plan-stage run should never be capable of mutating real infrastructure at all. ## Posting Plan Output as a PR Comment **A plan sitting only in CI logs gets read far less reliably than one posted directly into the PR — every mature pipeline surfaces the plan where the review is actually happening.** ```yaml - name: Post plan to PR uses: actions/github-script@v7 with: script: | const output = `#### Terraform Plan 📖 \`\`\` ${{ steps.plan.outputs.stdout }} \`\`\` `; github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body: output }); ``` Pairing this with Part 7's Infracost diff and a summary of any Checkov/tflint findings in the same comment (or a clearly-linked companion comment) means a reviewer sees everything relevant — the actual change, its cost impact, and any static-analysis findings — in one place, without needing to dig through separate CI job logs for each. > [!TIP] > **Best practice**: update the same PR comment on each new push (rather than posting a fresh comment every > time) — most GitHub Actions marketplace actions for this support a "find and update" mode, keyed on a > hidden marker in the comment body. A PR with a dozen stale, superseded plan comments is genuinely harder to > review than one comment that always reflects the current state. ## Secrets in the Pipeline — Injection, Not Hardcoding **Beyond the cloud credentials OIDC handles, a real pipeline often needs other secrets — a database password Terraform sets on creation, a third-party API token a provider needs — and these deserve the same "never hardcoded, never long-lived-in-CI-config" discipline as cloud credentials themselves.** ```yaml - name: Fetch DB password from Secrets Manager run: | echo "TF_VAR_db_password=$(aws secretsmanager get-secret-value \ --secret-id checkout/prod/db-password \ --query SecretString --output text)" >> "$GITHUB_ENV" # Fetched fresh every run, via the same OIDC-derived credentials — # never stored as a GitHub Actions secret directly. ``` | Anti-pattern | Why it's worse | Better approach | |---|---|---| | Hardcoding a secret directly in workflow YAML | Visible in plain text to anyone who can read the repo | Never do this, full stop | | Storing it as a GitHub Actions repo secret | Better than hardcoding, but static and manually rotated, and visible to any workflow with access | Fetch fresh from a secrets manager at run time, using the same short-lived OIDC credentials | | Fetching fresh from Secrets Manager/Vault at run time | Nothing long-lived stored in CI at all; rotation happens independently of the pipeline | The current best practice | > [!TIP] > **Best practice**: apply the same "nothing long-lived, fetched fresh every run" philosophy this chapter's > OIDC section established for cloud credentials to every other secret a pipeline needs — a GitHub Actions > repo secret is a real improvement over hardcoding, but a secrets-manager fetch using already-short-lived > OIDC credentials is strictly better, and often no more work to set up. ## Redacting Sensitive Values From Posted Plan Output **Part 1 established that `sensitive = true` redacts a value from CLI plan output — this protection carries through directly to a PR-posted plan comment, but only if the pipeline doesn't accidentally undo it, which a naive JSON-based plan-summary tool can.** ```bash # Safe — respects sensitive = true, redacts automatically terraform plan -out=tfplan terraform show tfplan # Human-readable, sensitive values shown as (sensitive value) # RISKY if posted directly without further redaction — terraform show -json's # machine-readable output includes sensitive value markers, but a naive script # extracting values from it can defeat the redaction if not careful terraform show -json tfplan | jq '.resource_changes[].change.after' ``` > [!WARNING] > Any custom tooling built on top of `terraform show -json` (Part 2's CI-tooling guidance) for generating a > PR comment must explicitly check each field's own `sensitive` marker in the JSON structure and redact > accordingly — piping raw JSON output through a generic formatter can bypass the same redaction the plain > `terraform show`/CLI plan output respects automatically. Test any custom plan-comment tooling specifically > against a resource with a `sensitive = true` argument before trusting it in production. ## Approval Gates — Environments and Required Reviewers **GitHub Environments (referenced in the apply job above via `environment: production`) provide the actual gating mechanism — a required-reviewers rule that blocks the apply job from running at all until an authorized person approves it, independent of the PR's own review/approval.** ```mermaid flowchart TD Merge["PR merged to main"] --> ApplyTriggered["apply job triggered,
targets 'production'
GitHub Environment"] ApplyTriggered --> Gate{"Environment protection rule:
required reviewers"} Gate -->|"Not yet approved"| Waiting["Job WAITS —
does not run"] Gate -->|"Approved by an
authorized reviewer"| Runs["apply job proceeds"] classDef wait fill:#fbeee0,stroke:#b8650f,color:#10161c classDef go fill:#e5f5ea,stroke:#1f8a4c,color:#10161c class Waiting wait class Runs go ``` This is a genuinely separate approval from the PR merge itself — a team can allow any engineer to merge a PR once its own review passes, while still requiring a *second*, specifically-authorized approval (often a smaller platform-team subset) before the `apply` job targeting `production` is allowed to actually run, matching the promotion discipline Part 4 established for moving a change into a higher-stakes environment. | Gate | Answers | |---|---| | PR review/approval | "Is this change correct?" | | Environment required-reviewer approval | "Is NOW, by THIS specific person, the right moment to actually apply it to this specific environment?" | ## Wiring In Part 7's Full Test Suite Extending the plan job with every static and unit-test layer from Part 7, in the fast-to-slow order that chapter established: ```yaml validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - run: terraform fmt -check -recursive - run: terraform validate - run: tflint --recursive - run: checkov -d . --framework terraform --compact - run: terraform test - run: infracost diff --path . --compare-to infracost-base.json ``` `validate` runs as its own job, in parallel with (or as a required precondition to) the `plan` job — a failure here should block the PR from even reaching a human reviewer, exactly Part 7's "fast checks block" guidance, now expressed as actual pipeline wiring rather than a table describing intent. > [!NOTE] > Terratest (Part 7's real-infrastructure integration layer) deliberately doesn't appear in this every-PR > job — per Part 7's own CI-placement guidance, it runs on a schedule and/or a label-triggered run scoped to > the specific module changing, not on every PR across the whole monorepo. ## Atlantis — PR-Comment-Driven Terraform Automation **Atlantis takes a fundamentally different shape from a hand-rolled Actions pipeline — instead of YAML workflow stages, it's a standalone service watching for PR events, running `plan` automatically and posting results, then waiting for an explicit `atlantis apply` comment before applying.** ``` # A PR is opened touching environments/prod/checkout-service/ Atlantis (automatically): Ran `terraform plan` in `environments/prod/checkout-service` Plan: 0 to add, 1 to change, 0 to destroy [view full plan] # A reviewer, after reading the plan: > atlantis apply Atlantis: Ran `terraform apply` in `environments/prod/checkout-service` Apply complete! Resources: 0 added, 1 changed, 0 destroyed. ``` ```yaml # atlantis.yaml — repo-level configuration version: 3 projects: - name: checkout-service-prod dir: environments/prod/checkout-service workflow: default apply_requirements: [approved, mergeable] ``` `apply_requirements: [approved, mergeable]` is Atlantis's own built-in equivalent of this chapter's GitHub Environment gate — `atlantis apply` itself is refused unless the PR already has the required approvals and is in a mergeable state, folding the review-gate logic into the tool rather than a separate CI job configuration. ## Atlantis vs. a Hand-Rolled Pipeline | | Hand-rolled GitHub Actions | Atlantis | |---|---|---| | Setup | Workflow YAML per repo (or a shared reusable workflow) | A standalone service to host, plus repo-level `atlantis.yaml` | | Plan trigger | Any push to a PR | Automatic on PR open/push | | Apply trigger | Merge to main (this chapter's pattern) or a manual workflow dispatch | An explicit `atlantis apply` PR comment | | Locking across concurrent PRs | Terraform's own backend locking (Part 2) only | Built-in PR-level locking, preventing two PRs from planning/applying the same project concurrently | | Multi-project awareness | Custom path-filtering logic you write yourself | Native — `atlantis.yaml` declares every project, and Atlantis figures out which changed | | Flexibility | Full control — any CI logic you can write | Constrained to Atlantis's own workflow model, extensible via custom workflows | > [!NOTE] > Neither is universally "better" — a hand-rolled pipeline gives full control at the cost of building and > maintaining the review/comment/locking logic this chapter's earlier sections showed by hand; Atlantis gives > that logic for free at the cost of adopting a service with its own operational surface and a somewhat more > constrained workflow model. Many teams use Atlantis specifically to avoid re-implementing exactly the > PR-comment and cross-PR-locking mechanics shown earlier in this chapter as bespoke YAML. ## State Locking in CI — Preventing Concurrent Pipeline Runs **Part 2's state locking prevents two concurrent `apply` operations from corrupting state — but a CI-specific risk sits one layer above that: two PRs touching the *same* project, both queued to apply around the same time, can each compute a valid plan against a state that changes out from under the second one before its own apply runs.** ```mermaid sequenceDiagram participant PR1 as PR #1 (merged first) participant PR2 as PR #2 (merged shortly after) participant State as Backend state PR1->>State: apply job runs, state updated (serial N+1) Note over PR2: PR #2's plan was computed against
the OLDER state (serial N) — BEFORE
PR #1's apply landed PR2->>State: apply job attempts to apply its
now-stale saved plan State-->>PR2: Terraform detects the plan's
expected prior state doesn't match
current state — apply REFUSED ``` **This chapter's caption**: Terraform itself is the actual safety net here — an `apply` against a saved plan file whose expected starting state no longer matches reality fails outright rather than silently applying a now-stale plan, which is exactly the state-locking-adjacent protection Part 2's `serial` field (this diagram's `N+1` vs `N`) was built to provide. The pipeline-level mitigation, beyond trusting Terraform's own refusal: **serialize the apply stage per project** (a concurrency group in GitHub Actions, or Atlantis's native per-project locking) so a second PR's apply job simply waits for the first to complete rather than racing it and hitting the stale-plan refusal as its first indication anything was wrong. ```yaml apply: concurrency: group: apply-checkout-service-prod cancel-in-progress: false ``` ## Promotion Through Environments, in the Pipeline Turning Part 4's promotion diagram into actual pipeline structure — each environment directory (Part 4) gets its own path-filtered workflow, and a promotion is a real, separate PR per environment, exactly as that chapter described conceptually: ```yaml on: pull_request: paths: - "environments/staging/checkout-service/**" push: branches: [main] paths: - "environments/staging/checkout-service/**" ``` A near-identical workflow file (differing only in its `paths` filter and target IAM role/GitHub Environment) exists per environment — `dev`, `staging`, `prod` — which is exactly the kind of structural repetition Part 4's Terragrunt/Stacks section addressed for the underlying Terraform configuration; the same DRY tooling typically generates the matching CI workflow repetition too, rather than three hand-maintained, slowly-diverging YAML files. ## GitOps for Infrastructure — Pull vs. Push **Every pipeline pattern shown so far is "push-based" — CI actively runs `apply` and pushes the change out. A "pull-based" GitOps model instead has an agent running *inside* the target environment, continuously reconciling against the desired state declared in Git — a meaningfully different architecture with real tradeoffs.** | | Push-based (this chapter's default) | Pull-based (GitOps agent) | |---|---|---| | Who initiates apply | CI, on merge | An in-cluster/in-account agent, on its own reconciliation loop | | Credentials needed in CI | Yes — CI needs write access to the target environment | No — CI only needs to update the Git repo; the agent already has local access | | Drift correction | Manual (Part 6's runbook) | Often automatic — the agent's next reconciliation loop reverts drift on its own | | Maturity for Terraform specifically | Very mature, the industry default | Less mature than for Kubernetes-native GitOps (Argo CD, Flux) — HCP Terraform's own run-triggers and some third-party platforms (Atlantis in a polling mode, Spacelift) approximate it | > [!NOTE] > GitOps is a much more established pattern for Kubernetes manifests specifically (see this site's > Kubernetes Deep Dive series) than for Terraform itself — most Terraform pipelines, including every one > shown in this chapter, remain push-based, with CI holding (scoped, short-lived, OIDC-issued) write > credentials rather than an in-account polling agent. Know the distinction and the terminology, but don't > expect to find as mature a pull-based Terraform ecosystem as exists for Kubernetes. ## What "Rollback" Actually Means for Terraform **Unlike an application deploy, there's no "redeploy the previous artifact" for Terraform — the closest equivalent is reverting the `.tf` configuration to its prior state via Git and running that reverted configuration through the exact same plan/apply pipeline, which itself computes a new, real plan (very possibly another destroy-and-recreate) rather than instantly restoring anything.** ```mermaid flowchart TD Bad["Apply #2 (bad change)
lands, causes a problem"] --> Revert["git revert the commit
that introduced the bad change"] Revert --> NewPlan["A NEW terraform plan runs
against the reverted config —
NOT an instant restore"] NewPlan --> Outcome{"What does THIS plan
actually show?"} Outcome -->|"A clean reversal"| Good["Applies cleanly,
infrastructure restored"] Outcome -->|"Depends on now-changed
real-world state"| Complicated["May show unexpected
changes — e.g. reverting an
instance type change after data
already migrated to the new size"] classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c classDef risk fill:#fbeee0,stroke:#b8650f,color:#10161c class Good ok class Complicated risk ``` **This chapter's caption**: a `git revert` produces a *new* plan to review, not an automatic restoration — treat a Terraform "rollback" with exactly the same plan-review discipline as any other change, since it's mechanically just another apply, not a special, trusted-by-default operation. > [!WARNING] > The "complicated" branch above is real and common: reverting an instance-type or storage-size *increase* > after real data has already grown into the larger size can itself become a destructive operation (shrinking > storage often isn't even possible in-place for many resource types). "Roll back the Terraform change" is > not automatically safe just because it's labeled a rollback — read the resulting plan with the same care as > any forward change. ## Skipping a No-Op Apply **`-detailed-exitcode` (Part 6's drift-detection mechanism, reused here) also makes a pipeline skip its apply stage entirely when a plan shows genuinely no changes — avoiding a pointless apply job run, an empty PR comment, and a confusing "applied" notification for a change that didn't actually change anything.** ```yaml - id: plan run: | terraform plan -out=tfplan -detailed-exitcode continue-on-error: true - name: Skip apply if no changes if: steps.plan.outputs.exitcode == '0' run: echo "No changes — skipping apply stage entirely." ``` Exit code `0` (no changes) and `2` (changes present) both represent a *successful* plan — only `1` is a genuine error — so `continue-on-error: true` combined with checking the specific exit code (rather than just pass/fail) is what lets the pipeline distinguish "nothing to do" from "plan failed" and "changes are pending," routing each to the right next step. | Exit code | Meaning | Pipeline action | |---|---|---| | `0` | No changes | Skip apply, no PR comment needed (or a lightweight "no changes" comment) | | `1` | Plan failed | Fail the job loudly, block the PR | | `2` | Changes detected | Post the plan comment, proceed to the normal review/apply flow | > [!TIP] > **Best practice**: wire this into any pipeline expected to run frequently against configuration that > doesn't always have real changes pending (a scheduled drift-check-adjacent plan, or a PR that only touches > documentation but happens to be in a path-filtered directory) — it keeps the pipeline's signal-to-noise > ratio high, so a posted plan comment reliably means "there's something here worth reviewing." ## Worked Scenario: Two PRs, One Stale Plan Two engineers, working independently, both open PRs modifying different resources within `checkout-service`'s same `staging` state — one resizing a database instance, one adding a new S3 bucket. Both PRs' `plan` jobs run and post clean, non-conflicting-looking output. The database PR merges first and applies successfully. The S3-bucket PR, merged twelve minutes later, hits exactly the stale-plan refusal this chapter's state-locking-in-CI section diagrammed — its saved plan's expected prior state (`serial N`) no longer matched the backend's actual current state (`serial N+1`, from the database PR's apply). The team's response was exactly what Terraform's own refusal is designed to prompt: the S3-bucket PR's apply job failed cleanly with a clear error rather than silently corrupting anything, and re-running its `plan` job (picking up the now-current state) produced a fresh, still-clean plan that applied successfully on retry. The follow-up process change was adding the `concurrency` group from this chapter's mitigation section, so future same-project PRs queue automatically rather than each engineer discovering the conflict manually via a failed apply. ## Worked Scenario: Building checkout-service's Pipeline End to End Assembling every piece from this chapter for `checkout-service`'s `prod` environment: a `validate` job (Part 7's full static/unit-test suite) required on every PR touching that path; a `plan` job posting output as a PR comment alongside an Infracost diff; a GitHub Environment named `production` requiring two specific platform-team members' approval; an `apply` job, gated on that environment, consuming the exact saved plan artifact; and a `concurrency` group serializing applies against that one project. The first real production incident this pipeline caught, within its first month live: a PR's plan showed an unexpected `1 to destroy` on the production RDS instance, buried in an otherwise-unremarkable-looking diff — caught by a reviewer specifically because the plan was posted directly and legibly in the PR (this chapter's comment-posting section), not left in a CI log a reviewer would have had to actively seek out. The change was rejected and reworked using a `moved` block (Part 6) instead of the accidental resource rename that had triggered the destroy. ## Worked Scenario: an OIDC Trust Policy Too Loosely Scoped An early version of the platform team's OIDC trust policy scoped `sub` to `repo:meridian-platform/infrastructure-live:*` — matching *any* ref in the repository, including feature branches and PR-triggered workflow runs, not just `main`. A routine security review caught that a maliciously-crafted PR from a compromised or careless contributor's feature branch could, in principle, trigger a workflow run capable of assuming the production-apply role, entirely bypassing the "only merges to `main` trigger apply" logic that lived only in the *workflow YAML*, not in the IAM trust boundary itself. The fix tightened the trust policy's `sub` condition to `ref:refs/heads/main` specifically (this chapter's own OIDC example), closing the gap between "what the workflow YAML intends to allow" and "what the IAM trust policy actually permits" — a reminder that a CI pipeline's logical structure (which job runs when) and its actual security boundary (which credentials a given run can obtain) are two different things, and only the second one is a genuine security control. > [!CAUTION] > Never assume workflow YAML logic ("this job only runs `if: github.event_name == 'push'`") is itself a > security boundary — the actual boundary is whatever the OIDC trust policy's conditions permit, independent > of how carefully the workflow file is written. Scope `sub` conditions as tightly as the real security > requirement, not just tightly enough to match the happy path. ## Part 8 Pipeline Cheat Sheet | Concept | Purpose | |---|---| | `-out=tfplan` + upload/download artifact | Guarantees apply uses the exact reviewed plan, never a re-computed one | | OIDC (`id-token: write` + `configure-aws-credentials`) | Short-lived, narrowly-scoped credentials, no long-lived secrets | | GitHub Environment + required reviewers | A second, deliberate approval gate distinct from PR review | | `concurrency` group on the apply job | Serializes applies against one project, avoiding stale-plan races | | Atlantis `apply_requirements: [approved, mergeable]` | Atlantis's built-in equivalent of an environment gate | | `git revert` + a fresh plan | The actual "rollback" mechanism — always re-reviewed, never instant | ## Common Mistakes and Interview Traps | Mistake | Why it's wrong | Correct approach | |---|---|---| | Letting the apply job re-run `terraform plan` internally | Defeats the entire point of a human-reviewed plan — apply may not match what was approved | Apply must consume the exact saved plan artifact from the plan job, never re-plan | | Storing a long-lived AWS access key as a CI secret | Standing liability — leak risk, manual rotation, broad permissions | Use OIDC for short-lived, narrowly-scoped, per-run credentials | | Scoping an OIDC trust policy's `sub` claim too broadly (e.g., any branch) | Lets a PR from an untrusted branch potentially assume a production-capable role | Scope `sub` to the specific branch/ref that should be trusted, e.g. `ref:refs/heads/main` | | Treating workflow YAML conditionals as a security boundary | The IAM trust policy, not the workflow logic, is what actually governs which credentials a run can obtain | Verify the trust policy's own conditions are as tight as the intended access, independent of workflow logic | | Assuming a `git revert` instantly restores prior infrastructure | It only reverts configuration — the resulting plan is a new, real change to review | Read the revert's resulting plan with the same care as any forward change | | Using one shared IAM role for both plan and apply stages | Gives the read-only plan stage unnecessary write/destroy permissions | Use distinct, least-privilege roles per pipeline stage | ## Worked Practice Problems **Problem 1**: A team's apply job is defined as `terraform apply -auto-approve` (with no saved plan file involved) rather than `terraform apply tfplan`. A PR was reviewed and approved based on a plan showing `0 to destroy`; the actual apply, run minutes later, destroys a resource nobody expected. What's the root structural cause? *Answer*: The apply job re-computed its own fresh plan internally (implicit in a bare `-auto-approve` with no saved plan file) rather than applying the exact plan a human reviewed — if anything about real infrastructure changed between the review and the apply (another apply landing first, manual drift), the freshly re-computed plan can differ from what was reviewed, and nothing in this pipeline shape would ever surface that gap to a human. The fix is exactly this chapter's core contract: `plan -out=tfplan`, upload as an artifact, and `apply tfplan` — never a bare `-auto-approve` with no saved plan. **Problem 2**: Two PRs targeting the same project's `staging` environment are both approved and merged within a minute of each other. What prevents the second apply from silently corrupting state, and what's the pipeline-level improvement that avoids the resulting failed run entirely? *Answer*: Terraform's own state-serial check (Part 2) prevents corruption — the second PR's saved plan was computed against an older state serial, and Terraform refuses to apply a plan whose expected starting state no longer matches current reality, failing loudly rather than corrupting anything. The pipeline-level improvement is a `concurrency` group (or Atlantis's native per-project locking) on the apply stage, so the second PR's apply job queues and waits for the first to finish, then computes a fresh, valid plan against the now-current state — turning a failed run and a manual retry into a smooth, automatic sequential apply. **Problem 3**: An OIDC trust policy's `sub` condition is set to `repo:meridian-platform/infrastructure-live:*` rather than scoped to a specific branch. What's the concrete risk, and why doesn't the workflow YAML's own `if: github.event_name == 'push'` condition mitigate it? *Answer*: The risk is that any ref in the repository — including a feature branch, or a PR-triggered workflow run from an untrusted or compromised source — can obtain credentials for the role this trust policy grants, regardless of what the workflow YAML's own conditional logic intends to restrict. The workflow YAML's `if` condition only controls whether *this specific job, as currently written* runs under normal circumstances — it provides no actual security guarantee, because it's just configuration a sufficiently different workflow run (or a modified workflow file on an attacker-controlled branch) could bypass entirely. The IAM trust policy's own `sub` condition is the only thing that actually, unconditionally restricts which ref can obtain the credentials in the first place. **Problem 4**: A team builds a custom Slack notification that parses `terraform show -json`'s plan output directly and posts a summary of every changed attribute, including a database's `password` argument, which is marked `sensitive = true` in the configuration. The Slack message shows the real password in plain text. What went wrong, and what's the fix? *Answer*: The custom tooling read raw attribute values out of the JSON structure without checking each field's own `sensitive` marker (also present in `terraform show -json`'s output, alongside the value) — `sensitive = true` only automatically redacts output through Terraform's own built-in formatters (`terraform show` without `-json`, standard `plan`/`apply` CLI output); a hand-rolled tool consuming the raw JSON is responsible for checking and respecting that marker itself. The fix is updating the custom notification code to check each `resource_changes[].change.after_sensitive` (or the equivalent field for the Terraform version in use) and redact any attribute the plan JSON itself marks as sensitive, rather than assuming JSON output is automatically as safe as the CLI's default formatting. ## Summary and What's Next A production-grade Terraform pipeline is built on one non-negotiable contract — the plan a human reviews is exactly the plan that gets applied, never re-computed — surrounded by OIDC for credential hygiene, an explicit approval gate distinct from PR review, Part 7's full test suite wired in at the speed-appropriate stage, and concurrency control that turns Terraform's own state-serial safety check from a failed-run surprise into a smooth, automatic queue. Atlantis offers the same guarantees through a different, more opinionated shape, trading some flexibility for less bespoke YAML to maintain. "Rollback," for Terraform, is never instant — it's always a new, real plan, reviewed with the same care as any forward change. Part 9, the series' final chapter, zooms out from any single pipeline to the organizational and financial layer above it: policy as code enforced at the pipeline level (Sentinel and OPA/Conftest, introduced in Part 1's ecosystem discussion, shown in full here), cost governance building on Part 7's Infracost integration, the real 2026 pricing landscape from Part 1's opening chapter applied concretely, and genuine multi-cloud provisioning patterns extending Part 4's directory-level cloud separation into actual cross-cloud configuration.