The series closer. Assumes the full stack from Parts 1-8 — state, modules, structure, providers, drift handling, testing, and CI/CD — and adds the organizational and financial layer that sits above all of it once Terraform is running dozens of services across multiple accounts, teams, and (in this chapter) more than one cloud.
Table of Contents#
- Why Governance Becomes Unavoidable at Scale
- Sentinel — Policy as Code, HCP-Native
- OPA and Conftest — Portable Policy as Code
- Sentinel vs. OPA/Conftest — the Real Decision
- Soft-Mandatory vs. Hard-Mandatory Policies
- The 2026 Terraform Pricing Landscape, Applied
- The Middle Ground: Third-Party TACOS Platforms
- Cost Governance Beyond a Single PR's Infracost Diff
- Tagging Strategy as the Foundation of Cost Attribution
- A Module Registry as a Governance Surface
- Provider Aliasing for Genuine Multi-Cloud Provisioning
- When to Alias vs. When to Fully Separate
- A Real Multi-Cloud Pattern: Active-Passive Disaster Recovery
- Avoiding a False Abstraction Across Clouds
- Choosing Terraform vs. OpenTofu, Revisited With Full Context
- Audit Trails — Answering "Who Changed What, and Why"
- Onboarding a New Team Onto the Platform
- Building a Platform Team's Governance Charter
- Measuring Whether Governance Is Actually Working
- Cross-Team Communication — the Human Layer Above All of This
- Worked Scenario: a Sentinel Policy That Blocked a Legitimate Emergency Change
- Worked Scenario: the Free-Tier Migration, Revisited With a Real Budget
- Worked Scenario: Standing Up checkout-service's Full DR Posture
- Worked Scenario: the Governance Charter's First Annual Review
- Part 9 Governance Cheat Sheet
- Common Mistakes and Interview Traps
- Worked Practice Problems
- Summary and What's Next
Why Governance Becomes Unavoidable at Scale#
Everything through Part 8 makes one team's Terraform usage safe and fast. At the scale of dozens of teams, hundreds of services, and real budget accountability, a new class of problem appears that no single team's pipeline discipline solves on its own: how does an organization enforce a rule ("every S3 bucket must be encrypted," "no resource may cost more than $500/month without a named approver") consistently across every team's independent pipeline, without a human manually reviewing every single plan for policy compliance forever?
This chapter's caption: nothing about the individual-team practices from Parts 1-8 stops working at scale — the problem is purely one of consistency across many independent pipelines, which is exactly what policy as code exists to solve.
Sentinel — Policy as Code, HCP-Native#
HashiCorp Sentinel evaluates policy against a Terraform plan/state/config, deeply integrated with HCP Terraform/Enterprise's run pipeline — the native choice for an organization already committed to HCP Terraform.
import "tfplan/v2" as tfplan
mandatory_tags = ["Environment", "Service", "ManagedBy"]
ec2_instances = filter tfplan.resource_changes as _, rc {
rc.type is "aws_instance" and rc.change.actions contains "create"
}
mandatory_instance_tags = rule {
all ec2_instances as _, instance {
all mandatory_tags as tag {
instance.change.after.tags contains tag
}
}
}
main = rule {
mandatory_instance_tags
}This policy directly enforces Part 4's naming/tagging convention — every new aws_instance in a plan must
carry Environment, Service, and ManagedBy tags, or the run is blocked before it ever reaches apply.
Sentinel's tfplan/v2 import gives policy code structured access to exactly the same plan data a human
reviewer would read, but evaluated automatically and consistently on every single run.
Note
Sentinel policies run as part of HCP Terraform/Enterprise's own managed run pipeline — they aren't a
standalone tool you invoke separately the way tflint/Checkov are in Part 7's CI steps. This is exactly
the coupling Part 1's ecosystem table flagged: Sentinel only runs inside HCP Terraform, and doesn't work
with OpenTofu at all.
OPA and Conftest — Portable Policy as Code#
Open Policy Agent (OPA), typically invoked via Conftest as a CLI wrapper, evaluates policy written in Rego against a Terraform plan's JSON output — vendor-neutral, works identically with Terraform or OpenTofu, and runs as an ordinary CI step (Part 8's pipeline) rather than requiring HCP Terraform at all.
package terraform.policies.no_open_ingress
deny[msg] {
rc := input.resource_changes[_]
rc.type == "aws_security_group"
rc.change.after.ingress[_].cidr_blocks[_] == "0.0.0.0/0"
msg := sprintf("%v allows ingress from 0.0.0.0/0 — see Part 6's SG-widening incident", [rc.address])
}terraform show -json tfplan > plan.json
conftest test plan.json --policy policies/FAIL - plan.json - terraform.policies.no_open_ingress - aws_security_group.checkout allows ingress from 0.0.0.0/0
This is precisely Part 2's terraform show -json pattern, reused one more time — every programmatic
tool this series has introduced (Infracost, custom CI notifications, and now OPA) consumes the same stable,
documented JSON contract rather than parsing CLI text output.
Tip
Best practice: wire Conftest into the same validate CI job Part 8 established for tflint/Checkov —
policy-as-code belongs at the same fast, every-PR, blocking layer as the rest of Part 7's static analysis,
not as a separate, later gate.
Sentinel vs. OPA/Conftest — the Real Decision#
| Factor | Points toward Sentinel | Points toward OPA/Conftest |
|---|---|---|
| Already on HCP Terraform, want the tightest native integration | Yes | — |
| Need portability across Terraform, OpenTofu, and non-Terraform tools (Kubernetes, APIs) | — | Yes — OPA is genuinely general-purpose |
| Team already knows/uses Rego for Kubernetes policy (Gatekeeper) | — | Yes — one policy language across the whole platform |
| Want policy evaluation as a normal CI step, no managed-pipeline dependency | — | Yes |
Value HashiCorp's official support and pre-built policy library (terraform-sentinel-policies) | Yes | — |
Note
This is the same OpenTofu-licensing-adjacent decision Part 1 raised in the ecosystem discussion, now made concrete: a team on OpenTofu specifically has no Sentinel option at all, which typically settles the question outright. A team fully committed to HCP Terraform has a genuine, real tradeoff to weigh rather than a forced choice.
Soft-Mandatory vs. Hard-Mandatory Policies#
Not every policy should block a run outright — Sentinel and most OPA/Conftest CI wiring both support a graduated enforcement level, and picking the right level per policy is itself a governance decision.
| Level | Behavior | Use for |
|---|---|---|
| Advisory | Logged, visible, never blocks | A new policy being rolled out gradually, or genuinely informational guidance |
| Soft-mandatory | Blocks by default, but an authorized override can proceed anyway | Most real governance rules — strict by default, with an accountable escape hatch |
| Hard-mandatory | Blocks unconditionally, no override possible | Only for the smallest set of truly non-negotiable rules (a prevent_destroy-adjacent guarantee, a compliance-mandated control) |
Warning
Defaulting every new policy to hard-mandatory is exactly the mistake this chapter's first worked scenario below explores — a policy engine with zero override path for a genuine, time-sensitive emergency recreates Part 6's "SCP with no break-glass exception" failure mode one layer up, at the policy-engine level instead of the IAM level.
The 2026 Terraform Pricing Landscape, Applied#
Part 1 introduced the licensing/pricing shift in the abstract; at governance scale, this is a real, recurring line-item decision a platform team owns. HCP Terraform's legacy Free plan reached end of life March 31, 2026; the replacement free tier caps at 500 managed resources, and paid tiers bill per resource under management — Essentials $0.10, Standard $0.47, Premium $0.99 per resource/month, published February 2026.
For the platform team's own footprint — checkout-service, catalog-service, inventory-service, each
across dev/staging/prod, plus shared network/platform infrastructure — a rough count in the low thousands of
resources puts a Standard-tier HCP Terraform bill in real four-figure monthly territory, which is exactly the
kind of number that turns "self-hosted S3 backend + Atlantis" (Parts 2 and 8) from an academic alternative
into a genuine budget conversation with a CFO.
| Question | Answer that shapes the decision |
|---|---|
| Current/projected managed-resource count | Directly determines HCP Terraform's RUM-based bill |
| Value of HCP Terraform's managed run pipeline, Sentinel, and state UI | Weighed against the self-hosted alternative's engineering-time cost |
| Team's tolerance for owning backend/pipeline infrastructure | A small platform team may not want another service to operate |
Tip
Best practice: track managed-resource count as a real, reviewed metric (Part 1's "resource count is a capacity signal" framing) well before a pricing tier boundary forces the conversation — the free-tier migration scenario later in this chapter shows exactly what happens when nobody was tracking it.
The Middle Ground: Third-Party TACOS Platforms#
Between "fully self-hosted S3 backend plus Atlantis" (Parts 2 and 8) and "HCP Terraform" sits a third real option worth naming explicitly: commercial "Terraform Automation and Collaboration Software" (TACOS) platforms — Spacelift, env0, and Scalr among them — offering a managed run pipeline, policy engine, and cost visibility, similar in spirit to HCP Terraform but independently priced and, notably, first-class OpenTofu support.
| Option | Backend | Run pipeline | Policy engine | OpenTofu support |
|---|---|---|---|---|
| Self-hosted (Parts 2, 8) | S3/GCS/Azure Blob, self-managed | Atlantis or hand-rolled CI | OPA/Conftest | Full |
| HCP Terraform | Managed | Managed | Sentinel | None |
| A TACOS platform (Spacelift, env0, Scalr) | Managed | Managed | Usually OPA-based | Full, typically a first-class feature |
For an organization that's already decided on OpenTofu (per Part 1's licensing discussion) but still wants a managed run pipeline rather than operating Atlantis themselves, a TACOS platform is often the actual answer — it isn't a compromise between the two options this chapter has emphasized so far, it's a genuine third point on the same tradeoff space, worth pricing out alongside HCP Terraform and the fully self-hosted option before committing.
Note
This series doesn't endorse a specific TACOS vendor — the point is structural: "managed run pipeline" and "HCP Terraform specifically" are not synonyms, and an organization committed to OpenTofu still has a managed-pipeline option available, not just a binary choice between HCP Terraform and fully self-hosted tooling.
Cost Governance Beyond a Single PR's Infracost Diff#
Part 7's Infracost integration shows the cost impact of one PR; org-wide cost governance needs the aggregate view across every service, every environment, continuously — a different scope than any single plan.
infracost breakdown --path . --format json --out-file infracost-report.json
# Aggregated across every environment/service in a scheduled job,
# feeding a dashboard or a monthly cost-review meeting| Layer | Scope | Cadence |
|---|---|---|
Part 7's infracost diff | One PR's change | Every PR |
Org-wide infracost breakdown aggregation | Every service, every environment | Scheduled (weekly/monthly) |
| A cloud provider's own cost-and-usage reports | Actual billed spend, including non-Terraform-managed resources | Continuous, provider-native |
Note
Infracost estimates cost from planned configuration — it cannot see actual usage-driven costs (data transfer, request volume, storage growth over time). Pairing Infracost's pre-merge estimates with the cloud provider's own actual billing data (AWS Cost Explorer, GCP Billing) closes that gap — this chapter's earlier "what testing can't tell you" honesty (Part 7) applies to cost estimation too.
Tagging Strategy as the Foundation of Cost Attribution#
Part 4's common_tags local, applied consistently across every resource, is what makes any of this cost
governance actually queryable — without consistent Environment/Service/Team tags, a cloud bill is an
undifferentiated total with no way to attribute spend back to the team or service responsible for it.
# The SAME common_tags local from Part 4, now load-bearing for cost governance too
locals {
common_tags = {
Environment = var.environment
Service = var.service
Team = var.owning_team
ManagedBy = "terraform"
CostCenter = var.cost_center
}
}A Sentinel or OPA policy enforcing mandatory_instance_tags (this chapter's own example) is, in practice,
often written specifically to guarantee this cost-attribution data exists on every resource — the security-
sounding "policy as code" framing and the accounting-sounding "cost attribution" framing are frequently the
exact same enforced tag set, serving two audiences from one mechanism.
A Module Registry as a Governance Surface#
Part 3 established modules as a reuse mechanism; at governance scale, a private module registry becomes the primary enforcement surface too — a well-designed "golden path" module bakes tagging, encryption, and naming compliance in by default, making the compliant choice the easy choice.
module "checkout_database" {
source = "app.terraform.io/meridian-platform/rds/aws"
version = "~> 3.0"
service = "checkout"
environment = "prod"
# storage_encrypted, mandatory tags, and backup retention are all
# baked into the module's own defaults — a caller has to actively
# override them to be non-compliant, not actively configure them
# correctly from scratch every time.
}| Enforcement layer | Catches | When |
|---|---|---|
| A "golden path" module's own defaults | Most non-compliance, structurally, by default | Before the caller even writes non-compliant config |
| Sentinel/OPA policy in the pipeline | Whatever slips past the module (a raw resource, an overridden default) | At plan time |
| A scheduled Checkov/drift scan | Anything that slipped past both | Post-apply, ongoing |
Tip
Best practice: treat a golden-path module as the primary governance mechanism and policy-as-code as the backstop — a module that makes the compliant path the default, easiest path prevents far more non-compliance than any number of policies catching it after the fact, which only ever fires once someone has already written non-compliant configuration.
Provider Aliasing for Genuine Multi-Cloud Provisioning#
Part 4 established cloud-level directory separation as the default; this section covers the narrower case where a single configuration genuinely needs to provision into more than one cloud at once — DNS failover records, a cross-cloud VPN tunnel, or (this chapter's own closing scenario) disaster-recovery replication.
provider "aws" {
region = "us-east-1"
}
provider "google" {
project = "meridian-dr-project"
region = "us-central1"
}
resource "aws_db_instance" "checkout_primary" {
# ...
}
resource "google_sql_database_instance" "checkout_replica" {
# References the AWS resource's endpoint directly —
# this ONE configuration genuinely spans two clouds
settings {
ip_configuration {
authorized_networks {
value = aws_db_instance.checkout_primary.address
}
}
}
}This is the deliberate exception to Part 4's "split at the cloud level first" default — used specifically when the relationship between two clouds' resources (not just their independent existence) is what the configuration needs to express, and Terraform's own dependency graph (Part 1) is the actual mechanism making that cross-cloud reference safe and ordered.
When to Alias vs. When to Fully Separate#
| Situation | Points toward |
|---|---|
| Two clouds' resources need to reference each other's real-time attributes (an IP, an endpoint) in one apply | Provider aliasing, one configuration |
| Two clouds' resources are independently provisioned with no direct relationship | Full directory/state separation (Part 4's default) |
| The relationship is narrow and stable (one DNS record, one VPN tunnel) | A small, dedicated "bridge" configuration aliasing both providers, separate from each cloud's main state |
| The relationship is broad and evolving (an entire service's cross-cloud DR posture) | Still separate per-cloud state, connected via terraform_remote_state (Part 2) rather than one shared apply |
Important
Even when aliasing both providers in one configuration, prefer keeping that configuration's state narrow and dedicated to just the cross-cloud relationship itself — not merged into either cloud's own broader state. A small "DR bridge" state that only owns the cross-cloud DNS/VPN resources keeps Part 2's blast-radius discipline intact even while the configuration itself spans two providers.
A Real Multi-Cloud Pattern: Active-Passive Disaster Recovery#
Building on Part 4's GCP disaster-recovery scenario, the full pattern: AWS remains the active, primary environment for all three services; GCP holds a continuously-replicated standby, promoted only during an actual declared disaster.
This chapter's caption: the DNS failover record — not a shared Terraform state, not a shared provider block — is the actual cutover mechanism; both clouds' infrastructure stays independently managed (Part 4's default), with only the narrow "bridge" pieces (replication configuration, the failover record itself) genuinely spanning both providers.
Avoiding a False Abstraction Across Clouds#
Part 4's DR scenario already established this — worth restating here as the chapter's own explicit governance guidance: never build one shared module pretending AWS RDS and GCP Cloud SQL (or any two providers' conceptually-similar-but-structurally-different services) are interchangeable behind one interface.
# DON'T: a false abstraction hiding real provider differences
module "managed_database" {
source = "./modules/cloud-agnostic-database"
cloud = "aws" # or "gcp" — the module internally branches on this,
# accumulating conditionals for every real difference
}# DO: genuinely separate, provider-native modules, each simple on its own
module "checkout_primary_db" {
source = "./modules/aws-rds"
# ...
}
module "checkout_dr_db" {
source = "./modules/gcp-cloudsql"
# ...
}This is Part 3's "god module" anti-pattern, recurring one level up — a cross-cloud abstraction accumulates the same kind of ever-growing conditional complexity a single environment-spanning module does, for the same underlying reason: it's trying to hide genuine structural differences behind one interface instead of letting each concern (each cloud's own module) stay simple and provider-native.
Choosing Terraform vs. OpenTofu, Revisited With Full Context#
Part 1 opened with this decision in the abstract; by this point in the series, every factor that actually matters has a name and a chapter behind it:
| Factor | Now grounded in |
|---|---|
| Pricing/budget impact | This chapter's HCP Terraform RUM pricing section |
| Policy engine choice | This chapter's Sentinel vs. OPA/Conftest section |
| Backend/pipeline flexibility | Part 2 (self-hosted S3 backend) and Part 8 (Atlantis vs. HCP Terraform) |
Feature currency (state encryption, provider for_each) | Part 1's ecosystem table |
Tip
Best practice: revisit this decision explicitly, with real numbers (resource count, current HCP Terraform spend or projected spend, actual policy-engine usage), rather than treating Part 1's introduction as a one-time decision made in the abstract and never revisited — an organization's real constraints (and OpenTofu's own feature set) both continue to evolve.
Audit Trails — Answering "Who Changed What, and Why"#
Every practice across this series contributes to one governance question auditors and incident responders ask constantly: who changed this resource, when, why, and was it reviewed — answering it well is a natural consequence of the discipline this series built, not a separate system to bolt on.
| Question | Answered by |
|---|---|
| What changed, exactly? | The saved plan artifact (Part 8) and the PR diff itself |
| Who approved it? | GitHub Environment required-reviewer records (Part 8) |
| Why was it made? | The PR description and any linked ticket |
| Who actually ran the apply, and when? | CI job logs, tied to the OIDC-derived, short-lived identity (Part 8) — never a shared, anonymous credential |
| Was a policy overridden? | The soft-mandatory override's audit log (this chapter) |
| Did the real infrastructure match what was applied? | The state file's own version history (Part 2) |
Because every credential in this series' pipeline is short-lived and OIDC-derived (Part 8), and every apply consumes a specific, artifact-pinned plan (Part 8), a complete audit trail already exists as a natural byproduct — nobody had to build a separate change-tracking system, because Git history, CI logs, and state versioning together already answer every one of these questions.
Tip
Best practice: when a compliance framework (SOC 2, ISO 27001, or an internal audit) asks for evidence of "change management for infrastructure," the honest answer is pointing directly at this series' own mechanisms — the PR, its required approval, the CI logs, and the state version history — rather than standing up a parallel, manually-maintained change log that duplicates what the pipeline already records automatically and more reliably.
Onboarding a New Team Onto the Platform#
Closing the practical loop on this chapter's governance charter — the actual checklist a new team (say, a
future payments-service) works through to start using everything this series established, rather than
reinventing any of it independently:
- Clone the
infrastructure-liverepository structure (Part 4) for the new service, under the correct domain-parent directory - Consume the golden-path modules from the shared registry (this chapter) rather than writing raw resources for anything a golden-path module already covers
- Confirm the new service's OIDC trust policy is scoped correctly (Part 8) — a distinct role per environment, never a shared, broadly-scoped one
- Confirm state is split along real ownership/lifecycle boundaries (Part 2) from day one, rather than starting monolithic and needing a disruptive split later once the service has grown
- Wire the standard CI pipeline (Part 8) —
validate,plan, environment-gatedapply— using the shared reusable workflow rather than a bespoke one - Confirm the mandatory tag set (this chapter) is applied via
common_tags, not reinvented per service - Add the new service to the scheduled drift-check rotation (Part 6) at the appropriate severity tier (Part 6's per-environment cadence table), and confirm alert routing matches an on-call rotation that actually exists for this team
- Review the governance charter's override/escalation path with the new team before their first incident, not during it
- Confirm the new team knows where the companion
questions.mdself-check and this series' worked scenarios live, as a reference for the incident classes this checklist exists to prevent
Note
A team that can complete this checklist in under a day, using entirely pre-built shared infrastructure (modules, CI workflows, policies), is the concrete evidence that this series' patterns actually compound — each new service benefits from every prior chapter's discipline without re-deriving any of it, which is the real payoff of treating infrastructure as a shared, governed platform rather than each team's private practice.
Building a Platform Team's Governance Charter#
Pulling this entire chapter into one operational artifact — the kind of short, living document a platform team actually maintains and points every consuming team at:
- Which policies are hard-mandatory, and why (a short list, deliberately kept short per this chapter's soft/hard-mandatory guidance)
- Which policies are soft-mandatory, and who is authorized to override, with what audit trail
- The mandatory tag set every resource must carry, and which module(s) bake it in by default
- The current HCP Terraform tier (or self-hosted equivalent) and the resource-count threshold that triggers a re-evaluation
- Which module registry is authoritative for "golden path" infrastructure, and the review process for adding a new golden-path module
- The escalation path for a policy blocking a genuine, time-sensitive change (this chapter's break-glass equivalent, mirroring Part 6's console-access break-glass process)
Note
This charter is deliberately a living document, not a one-time artifact — every worked scenario across all
nine parts of this series (a stale credential, a loosely-scoped OIDC policy, a module upgrade nobody read
the changelog for) represents exactly the kind of incident that should feed back into updating a document
like this one, the same way .claude/rules-style living documentation works for any other engineering
discipline.
Measuring Whether Governance Is Actually Working#
A governance charter that nobody measures against tends to drift out of sync with reality the same way unreviewed infrastructure drifts (Part 6) — a small set of concrete, trackable metrics is what keeps it a living practice rather than a document nobody revisits.
A rising override rate for one specific policy (as in the illustrative chart above) is a genuine signal worth investigating, not just tolerating — it can mean the policy itself is miscalibrated (too strict for a legitimate, recurring case) or that a real process gap is being routinely worked around rather than fixed. Either conclusion is more useful than letting the override count climb unexamined.
| Metric | What a healthy trend looks like | What a concerning trend looks like |
|---|---|---|
| Soft-mandatory policy override rate | Low, roughly flat | Rising, or concentrated on one specific policy |
| Time from PR open to production apply | Stable, matching the team's own SLA expectations | Growing, possibly signaling gate friction worth addressing |
| Checkov/tflint findings per PR | Low and flat, or trending down | Trending up — a signal training or a golden-path module gap |
| Managed-resource count vs. pricing-tier threshold | Tracked proactively, reviewed quarterly | Discovered only when a bill or an EOL notice forces the conversation, exactly Part 1's opening incident |
| Drift incidents per environment per quarter | Low in prod, matching Part 6's severity-tuned cadence | Rising in any environment, especially prod |
| New-team onboarding time (this chapter's checklist) | Consistently under a day, using shared infrastructure | Growing, or requiring frequent one-off exceptions to the standard path |
Tip
Best practice: review these metrics at the same cadence as the governance charter itself (this chapter's earlier "living document" guidance) — a quarterly fifteen-minute look at override rates and finding counts is enough to catch a policy quietly becoming friction-without-value long before it erodes trust in the governance process altogether.
None of these metrics need a dedicated dashboard product to start — a scheduled query against CI logs and the policy engine's own override log, summarized into a short quarterly document, is enough to make this review a real, recurring habit rather than an aspiration nobody gets back to. Start small and keep it going — a metric reviewed inconsistently is barely more useful than one nobody tracks at all.
Cross-Team Communication — the Human Layer Above All of This#
Every mechanism this series has built — modules, policies, pipelines, drift checks — still depends on actual communication between the platform team and the teams consuming what it builds; no amount of tooling substitutes for a genuine feedback channel.
- A visible, low-friction way for a consuming team to propose a change to a golden-path module (a PR against the module registry itself, reviewed by the platform team, not a request routed through a ticket queue)
- A regular (not just incident-triggered) forum where consuming teams can flag friction — a policy that blocks more than it should, a module missing a genuinely common configuration option
- A clear, documented distinction between "the platform team's opinion" and "an enforced policy" — not every recommendation in this series needs to become a hard-mandatory Sentinel/OPA rule; some are genuinely best left as strong defaults a team can deliberately override with justification
Note
The series' own throughline is a useful closing illustration of this: checkout-service,
catalog-service, and inventory-service are three different teams' services sharing one platform team's
infrastructure practice. Every worked scenario across all nine chapters — the security group rule Part 1's
reviewer caught, the module version bump Part 3 flagged, the OIDC trust policy Part 8 tightened — was
caught by a human paying attention, informed by tooling, not by tooling alone. The tooling in this series
raises the floor and removes the tedious, repetitive checking; it was never meant to remove the humans
from the loop entirely, and a governance practice that forgets this tends to optimize for passing checks
instead of for the actual outcomes those checks were written to protect.
Worked Scenario: a Sentinel Policy That Blocked a Legitimate Emergency Change#
A hard-mandatory Sentinel policy required every aws_instance to be launched only within a specific,
pre-approved set of instance types, with zero override path — reasonable as written, until a genuine capacity
emergency during a traffic spike required an urgent, temporary instance-type change outside that approved
list to keep checkout-service responsive. The policy blocked the change outright, with no mechanism for an
authorized human to proceed anyway, even under a declared incident.
The team's response mirrored Part 6's break-glass fix directly: the policy was reclassified from hard-mandatory to soft-mandatory, with an explicit, audited override path (a specific on-call role, logged, requiring a linked incident ticket) — preserving the policy's normal-operations enforcement while adding exactly the escape valve a genuine emergency needs, the same lesson this series has now taught at the IAM layer (Part 6), the workflow layer (Part 8's OIDC scoping), and now the policy-engine layer.
Worked Scenario: the Free-Tier Migration, Revisited With a Real Budget#
Extending Part 1's opening worked scenario (the team that hadn't been tracking managed-resource count and
landed just over the new 500-resource free-tier cap) with the full governance picture: post-migration, the
platform team added managed-resource count as a standing line item in their governance charter's quarterly
review, alongside the actual Standard-tier bill this chapter's pricing section modeled. The concrete
follow-up decision — informed by real numbers, not a hypothetical — was migrating the lowest-stakes dev
environments to a self-hosted S3 backend (Part 2) while keeping staging and prod on HCP Terraform for its
managed run pipeline and Sentinel integration, a deliberate, split decision rather than an all-or-nothing
migration.
Worked Scenario: Standing Up checkout-service's Full DR Posture#
Closing the series' throughline: checkout-service's complete disaster-recovery posture, assembled entirely
from patterns this series already built — a network module (Part 3) and its own state (Part 2) in both AWS
and GCP, provisioned independently (Part 4's directory-per-cloud default); a narrow, dedicated "DR bridge"
configuration (this chapter) owning only the cross-cloud replication settings and the DNS failover record;
Sentinel/OPA policy (this chapter) enforcing that both the primary and standby databases carry identical
mandatory tags; and the entire DR bridge configuration running through the exact same Part 8 CI/CD pipeline
— plan posted to a PR, reviewed, gated behind an environment approval — as every other piece of
infrastructure in this series, with no special-cased, less-reviewed path for "the DR stuff."
Note
Nothing about disaster-recovery infrastructure specifically exempted it from any practice established earlier in this series — the same state discipline, module boundaries, testing, and pipeline gates apply identically, which is itself the point: a consistent set of practices, applied without exception, is what "production-grade Terraform" actually means at the end of this series, not a special set of extra-careful rules reserved only for the infrastructure that happens to be labeled critical.
Worked Scenario: the Governance Charter's First Annual Review#
One year after adopting the governance charter this chapter describes, the platform team ran their first
full annual review — pulling together every metric from this chapter's measurement section across all three
throughline services. The findings were mixed in exactly the way a healthy, honestly-measured governance
practice should be: the mandatory-tagging policy's override rate had dropped to near zero (the golden-path
modules had made compliance the default, exactly as this chapter's "primary enforcement layer" guidance
predicted), while one specific Checkov check — a strict encryption-in-transit rule originally written for
prod — was showing a rising override rate specifically in dev, where the team had never actually needed
that strength of guarantee for genuinely disposable, non-production data.
Rather than treating the rising override count as a discipline problem to fix with more enforcement, the
team read it as exactly the signal this chapter's measurement section describes — a policy miscalibrated for
one environment, not a team ignoring governance. The fix was environment-scoping the check itself
(hard-mandatory in staging/prod, advisory-only in dev), which immediately eliminated the override
pattern because the policy now matched what each environment actually needed, rather than applying prod's
strictness uniformly and relying on individual engineers to keep overriding it correctly.
This chapter's caption, and the series' own closing point: a metric showing a policy being worked around is a prompt to investigate why, not an automatic justification for more enforcement — the same "investigate before reacting" discipline Part 6 applied to drift, applied here one final time to governance itself.
Part 9 Governance Cheat Sheet#
| Tool/Practice | Purpose |
|---|---|
Sentinel (tfplan/v2 import) | HCP-native policy as code |
Conftest + Rego, against terraform show -json | Portable, vendor-neutral policy as code |
| Soft-mandatory + audited override | The break-glass pattern for policy enforcement |
| A golden-path module registry | Structural compliance by default, the primary enforcement layer |
infracost breakdown (aggregated, scheduled) | Org-wide cost visibility beyond a single PR |
| A governance charter (living document) | The single source of truth for what's enforced, how, and who can override |
Common Mistakes and Interview Traps#
| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Making every new policy hard-mandatory by default | No override path for a genuine emergency — recreates Part 6's SCP-without-break-glass failure at the policy layer | Default new policies to soft-mandatory with an audited override; reserve hard-mandatory for truly non-negotiable rules |
| Relying on policy-as-code as the only enforcement layer | Only fires after someone has already written non-compliant configuration | Treat a golden-path module as the primary layer; policy as code is the backstop |
| Choosing Sentinel while planning to adopt OpenTofu | Sentinel doesn't work with OpenTofu at all | Choose OPA/Conftest for any OpenTofu-committed or portability-conscious organization |
| Building one shared module abstracting two clouds' structurally different services | Recreates Part 3's god-module anti-pattern, one level up | Keep provider-native modules genuinely separate; connect via narrow bridge configurations only where needed |
| Merging a cross-cloud "bridge" configuration's state into either cloud's main state | Loses Part 2's blast-radius discipline for the bridge resources | Keep bridge configurations in their own narrow, dedicated state |
| Not tracking managed-resource count until a pricing-tier boundary forces the conversation | Exactly Part 1's free-tier migration surprise, recurring without deliberate tracking | Track resource count as a standing governance-charter metric, reviewed regularly |
| Responding to a rising policy-override rate with stricter enforcement, by default | Often a calibration problem, not a discipline problem — tightening further just adds friction without fixing the mismatch | Investigate why overrides are rising before reacting; recalibrate the policy itself where warranted |
| Treating a compliance auditor's request as needing a new, bespoke change-management system | Duplicates evidence the pipeline (Part 8) already generates automatically and more reliably | Point directly at PR history, environment-approval records, and state version history as primary evidence |
Worked Practice Problems#
Problem 1: An organization commits to OpenTofu specifically for its permissive MPL 2.0 license. A consultant recommends Sentinel for policy enforcement. What's wrong with this recommendation, and what should be used instead?
Answer: Sentinel only runs inside HCP Terraform/Enterprise's managed run pipeline and has no OpenTofu integration at all — recommending it to an OpenTofu-committed organization is a straightforward mismatch with the earlier licensing decision. OPA/Conftest is the correct recommendation: vendor-neutral, works identically against Terraform or OpenTofu plan JSON, and doesn't require any particular managed run pipeline to function.
Problem 2: A platform team builds a single cloud_database module accepting a cloud = "aws" or
cloud = "gcp" variable, internally branching to create either aws_db_instance or
google_sql_database_instance resources. Six months in, the module has 40 variables, half of which only
apply to one cloud or the other. What's the underlying design mistake, and what's the fix?
Answer: This is a false abstraction — AWS RDS and GCP Cloud SQL are conceptually similar but structurally
different enough that forcing them behind one shared interface accumulates exactly the kind of
per-cloud-conditional sprawl this chapter's "avoiding a false abstraction" section warns against, the same
underlying problem as Part 3's god-module anti-pattern. The fix is splitting into two genuinely separate,
provider-native modules (aws-rds and gcp-cloudsql), each simple and idiomatic for its own cloud, composed
together at the root-module level (Part 3) rather than hidden behind one shared, ever-growing interface.
Problem 3: A team's governance charter lists a mandatory tag policy as hard-mandatory with no override,
justified as "tags are never actually urgent, so there's no legitimate reason to bypass this." A production
incident requires an emergency resource creation where the on-call engineer doesn't have time to determine
the correct CostCenter tag value before the fix needs to be live. Was the "tags are never urgent"
reasoning sound?
Answer: No — the reasoning conflates "this specific policy's subject matter is rarely urgent" with "this policy should never need an emergency override," but the actual trigger for needing an override is the situation (a live incident with limited time), not whether the policy's subject matter sounds inherently urgent. Any hard-mandatory policy blocking infrastructure creation during a genuine incident creates exactly this chapter's Sentinel worked-scenario failure mode, regardless of whether the policy itself concerns tags, instance types, or anything else — the fix is the same soft-mandatory-with-audited-override pattern, applied here too, likely paired with a follow-up requirement to backfill the correct tag value once the incident is resolved.
Problem 4: An auditor asks a platform team to demonstrate that every change to production infrastructure over the last quarter was reviewed and approved by an authorized person before being applied. The team has no dedicated change-management system. What should they point to, and why is this sufficient?
Answer: The team should point directly to the existing pipeline artifacts this series already produces as a byproduct of normal operation — the PR history (each showing the actual diff and its plan), the GitHub Environment's required-reviewer approval records (Part 8) for every production apply, and the CI job logs tied to short-lived, OIDC-derived identities rather than a shared credential. This is sufficient because it answers exactly the audit question (who approved what, when, and what actually ran) with primary evidence generated automatically at the time each change happened, rather than a separately-maintained log that could drift out of sync with what actually occurred — a stronger, not weaker, form of evidence than a manually kept change log would provide.
Problem 5: A platform team notices a specific Checkov check's override rate climbing steadily in dev
over two consecutive quarters, while the same check's override rate in staging and prod stays near zero.
What does this pattern suggest, and what's the recommended response per this chapter's own closing worked
scenario?
Answer: The pattern strongly suggests the check is well-calibrated for staging/prod but genuinely
too strict for dev's actual risk profile — exactly the environment-specific miscalibration this chapter's
governance-review scenario walks through in full. The recommended response is investigating why the pattern
is environment-specific (not organization-wide) before assuming a discipline problem, then, if the
investigation confirms a genuine mismatch, scoping the check's enforcement level per environment (hard-
mandatory where it was already working, advisory-only or removed where it isn't) rather than either ignoring
the trend or tightening enforcement further in the one environment where it's clearly not fitting the actual
need.
Summary and What's Next#
Across nine chapters, this series went from a single terraform apply (Part 1) to a full, production-grade
practice: state that's shared, locked, and recoverable (Part 2); modules with real interface discipline
(Part 3); a repository and account structure that makes a dev-to-prod mistake structurally hard, not just
conventionally discouraged (Part 4); providers, data sources, and provisioners used precisely and only when
genuinely warranted (Part 5); drift treated as inevitable but manageable through deliberate, version-controlled
mechanisms (Part 6); a layered test suite catching an entire class of mistake before a human ever has to
(Part 7); a CI/CD pipeline where the plan a human reviews is exactly the plan that gets applied (Part 8); and,
closing the series, the governance, cost, and multi-cloud discipline that keeps all of the above consistent
across an entire organization rather than one team's private best practice.
The throughline — checkout-service, catalog-service, inventory-service, and the platform team behind
them — started as a teaching device in Part 1 and, by this final chapter, looks like a genuinely complete,
realistic platform engineering practice: not a toy example, but the shape a real team's Terraform usage
converges toward once every chapter's lesson has actually been applied, incident by incident, exactly the way
this series' worked scenarios showed it happening in practice.
Where to go from here depends on where your own team currently stands. If Terraform is brand new to your organization, revisit Part 1 through Part 4 first and get the foundation genuinely solid — a remote backend with locking, a real module boundary, and an account structure that makes a mistake structurally hard — before layering on Part 7's test suite or Part 9's policy engines; governance without a solid foundation underneath it mostly just formalizes existing problems rather than fixing them. If your team already runs Terraform at real scale but has been accumulating exactly the kind of ad hoc, undocumented practice this series' worked scenarios describe, start with an honest audit against Part 6's drift-detection and Part 8's plan-review disciplines specifically — those two chapters catch the highest-frequency, highest-severity real incidents this series covers, and are worth prioritizing over the more advanced governance material in this final chapter if something has to come first.
For hands-on practice putting these patterns together in a real, runnable repository, check the companion
devops-msrashed-com-handson repo's growing Terraform examples as they're added — and this site's Kubernetes
Deep Dive and AWS Cloud Architecture series for the infrastructure this series' own Terraform modules were
built to provision in the first place, since the throughline system in all three series is the same one.