# Terraform & Infrastructure as Code — Part 4: Workspaces, Environments & Real-World Repository Structure
> **Series:** Terraform & Infrastructure as Code (4 of 9)
> **Part 1:** `01-fundamentals-and-workflow.md` — Fundamentals, HCL & the Plan/Apply Workflow
> **Part 2:** `02-state-management-and-remote-backends.md` — State Management & Remote Backends
> **Part 3:** `03-modules-and-reusable-design.md` — Modules & Reusable Infrastructure Design
> **Part 4:** This file — Workspaces, Environments & Real-World Repository Structure
> **Part 5:** `05-providers-data-sources-and-provisioners.md` — Providers, Data Sources & Provisioners Deep Dive
> **Part 6:** `06-drift-detection-import-and-refactoring.md` — Drift Detection, Import & Refactoring Existing Infrastructure
> **Part 7:** `07-testing-terraform.md` — Testing Terraform: Static Analysis, the Native Test Framework & Terratest
> **Part 8:** `08-cicd-for-terraform.md` — CI/CD for Terraform: Pipelines, Gates & GitOps for Infrastructure
> **Part 9:** `09-governance-cost-and-multi-cloud-at-scale.md` — Terraform at Team Scale: Governance, Cost & Multi-Cloud Patterns
> **Questions:** `questions.md`
Assumes you're comfortable with Part 2's state splitting and Part 3's module design — this chapter is where
those two ideas combine into the actual directory layout and repository shape a real platform team runs day
to day, at the scale of dozens of services, several environments, and (by the end of the chapter) more than
one cloud.
## Table of Contents
1. [Why Repository Structure Deserves Its Own Chapter](#why-repository-structure-deserves-its-own-chapter)
2. [Terraform Workspaces — What They Actually Are](#terraform-workspaces--what-they-actually-are)
3. [Workspaces vs. Directory-Per-Environment — the Real Tradeoff](#workspaces-vs-directory-per-environment--the-real-tradeoff)
4. [The Directory-Per-Environment Pattern in Practice](#the-directory-per-environment-pattern-in-practice)
5. [The DRY Problem Directory-Per-Environment Creates](#the-dry-problem-directory-per-environment-creates)
6. [Terragrunt and the Infrastructure-Live Pattern](#terragrunt-and-the-infrastructure-live-pattern)
7. [A Variable Hierarchy: Global, Account, Region, Environment](#a-variable-hierarchy-global-account-region-environment)
8. [Monorepo vs. Polyrepo for Terraform at Scale](#monorepo-vs-polyrepo-for-terraform-at-scale)
9. [A Concrete Repository Layout for a Mid-Size Platform Team](#a-concrete-repository-layout-for-a-mid-size-platform-team)
10. [Bootstrapping the State Backend Itself](#bootstrapping-the-state-backend-itself)
11. [Promoting a Change From Dev to Prod](#promoting-a-change-from-dev-to-prod)
12. [Multi-Account AWS — Landing Zones and Organizations](#multi-account-aws--landing-zones-and-organizations)
13. [Cross-Account Access — the Hub-and-Spoke assume_role Pattern](#cross-account-access--the-hub-and-spoke-assume_role-pattern)
14. [Multi-Cloud Directory Structure — Keeping Clouds Cleanly Separated](#multi-cloud-directory-structure--keeping-clouds-cleanly-separated)
15. [HCP Terraform Stacks — a Native Alternative to Terragrunt](#hcp-terraform-stacks--a-native-alternative-to-terragrunt)
16. [Naming Conventions Across the Tree](#naming-conventions-across-the-tree)
17. [Choosing Your Structure — a Decision Framework](#choosing-your-structure--a-decision-framework)
18. [Worked Scenario: Migrating Off Workspaces Into Directory-Per-Environment](#worked-scenario-migrating-off-workspaces-into-directory-per-environment)
19. [Worked Scenario: Standing Up a Second Cloud for Disaster Recovery](#worked-scenario-standing-up-a-second-cloud-for-disaster-recovery)
20. [Worked Scenario: the tfvars Override That Silently Hit the Wrong Account](#worked-scenario-the-tfvars-override-that-silently-hit-the-wrong-account)
21. [Part 4 CLI Cheat Sheet](#part-4-cli-cheat-sheet)
22. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
23. [Worked Practice Problems](#worked-practice-problems)
24. [Summary and What's Next](#summary-and-whats-next)
---
## Why Repository Structure Deserves Its Own Chapter
**Almost every serious Terraform incident that isn't a state-management problem (Part 2) is a repository-
structure problem — a change meant for staging that silently applied to production, a `.tfvars` file copied
from the wrong environment, a shared module change that rippled into every account at once because nothing
separated them.** None of this is about HCL syntax. It's about how a codebase is physically laid out: which
directory maps to which real environment, which account, and — once a company has more than one — which
cloud, and how much of that mapping is enforced by structure itself versus trusted to a human reading a
folder name correctly under pressure.
This is also the chapter where the throughline system stops being a convenient teaching device and starts
looking like what a real platform team actually runs: `checkout-service`, `catalog-service`, and
`inventory-service`, each needing dev/staging/prod, all inside one AWS Organization, with a second cloud
(GCP) added for disaster recovery by the end of this chapter — and the actual directory tree that holds all
of it.
> [!NOTE]
> Everything in this chapter is provider-agnostic in principle but AWS-flavored in its concrete examples
> (account structure, `assume_role`), consistent with the rest of this series — the same directory-structure
> and DRY-vs-explicit tradeoffs apply identically under Azure subscriptions or GCP projects, which the
> multi-cloud section later in this chapter addresses directly.
## Terraform Workspaces — What They Actually Are
**A CLI workspace is a named, isolated state file within the same backend configuration — `terraform
workspace new staging` creates a second, independent state under the same `bucket`/`key`, addressable via
`terraform.workspace` inside your configuration.** This is a genuinely different feature from an "HCP
Terraform workspace" (a whole managed run environment in the SaaS product) despite the identical name — a
recurring source of confusion worth clearing up immediately.
```bash
terraform workspace new staging
terraform workspace new prod
terraform workspace select staging
terraform workspace list
# default
# * staging
# prod
```
```hcl
locals {
instance_type = terraform.workspace == "prod" ? "m6i.large" : "t3.micro"
}
```
| Term | What it actually is |
|---|---|
| CLI workspace (`terraform workspace`) | A named state file, same backend, same configuration — a free-tier, built-in feature |
| HCP Terraform workspace | An entire managed run environment (its own variables, run history, VCS connection) in HashiCorp's SaaS product |
> [!WARNING]
> These two concepts sharing one name is a genuine, ongoing source of confusion in job interviews and real
> conversations alike — always clarify which one is meant before answering a question that uses the bare
> word "workspace." This series uses "CLI workspace" explicitly whenever the distinction matters.
## Workspaces vs. Directory-Per-Environment — the Real Tradeoff
**CLI workspaces switch which state file you're pointed at without changing which `.tf` files are loaded —
which is exactly their strength for genuinely ephemeral, structurally-identical environments, and exactly
their weakness for anything long-lived and meaningfully different.**
```mermaid
flowchart TD
Workspaces["CLI workspaces:
same .tf files, different
state per workspace"] --> WGood["Good fit: ephemeral,
structurally-identical
environments — a PR preview
environment, a load-test sandbox"]
Workspaces --> WBad["Risky fit: long-lived environments
that genuinely differ (prod has a
Multi-AZ RDS, dev doesn't) — the
difference lives in conditional
logic, easy to get wrong"]
Directories["Directory-per-environment:
separate .tf/.tfvars per
environment, explicit backend
config each"] --> DGood["Good fit: long-lived, structurally-
different environments — what
prod IS is explicit and readable
on its own, no conditional to trace"]
Directories --> DBad["Cost: some duplication across
environment directories, unless
paired with modules (Part 3) or
Terragrunt (this chapter)"]
classDef good fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
classDef risk fill:#fbeee0,stroke:#b8650f,color:#10161c
class WGood,DGood good
class WBad,DBad risk
```
The single sharpest real-world danger with CLI workspaces: **switching workspaces is a silent, local CLI
state change** — `terraform workspace select prod` followed by an `apply` intended for staging, because the
engineer forgot which workspace they'd last selected, is a genuinely common way to apply a staging-sized
change against production. Directory-per-environment structurally prevents this exact mistake: the working
directory itself (`environments/prod/`) is what you'd have to be in, and that's visible in every terminal
prompt, every `cd`, every CI job's `working-directory` setting — there's no separate, invisible "current
workspace" state to forget about.
| Question | Points toward |
|---|---|
| Are the environments genuinely, structurally identical (a load-test copy of prod, a per-PR preview)? | CLI workspaces |
| Do environments have real structural differences (prod is Multi-AZ, dev is single-AZ; prod has WAF, dev doesn't)? | Directory-per-environment |
| Is accidentally applying to the wrong environment catastrophic? | Directory-per-environment — the explicit directory is a real safety layer |
| Are environments created and destroyed constantly (dozens of short-lived PR previews)? | CLI workspaces — creating a new directory per PR doesn't scale |
> [!TIP]
> **Best practice**, confirmed by wide production consensus: use directories for permanent environments
> (dev/staging/prod) and reserve CLI workspaces for genuinely ephemeral ones (a feature-branch preview, a
> load-test sandbox spun up and destroyed within hours). Many mature platform teams use **both** at once —
> directories for the permanent shape, workspaces layered inside a directory for that directory's own
> short-lived variants.
## The Directory-Per-Environment Pattern in Practice
**The straightforward version of this pattern is one directory per environment, each a complete root module
in its own right, each with its own backend configuration and its own `.tfvars` file.**
```
environments/
├── dev/
│ ├── main.tf
│ ├── backend.tf
│ └── terraform.tfvars
├── staging/
│ ├── main.tf
│ ├── backend.tf
│ └── terraform.tfvars
└── prod/
├── main.tf
├── backend.tf
└── terraform.tfvars
```
```hcl
# environments/prod/main.tf
module "checkout_database" {
source = "../../modules/database"
instance_class = "db.r6g.large"
multi_az = true
backup_retention = 30
}
```
```hcl
# environments/dev/main.tf
module "checkout_database" {
source = "../../modules/database"
instance_class = "db.t4g.small"
multi_az = false
backup_retention = 1
}
```
Each environment's `main.tf` is a real, honest description of what that environment actually is — a reader
opening `environments/prod/main.tf` sees `multi_az = true` right there, with no conditional expression to
trace through to figure out what prod actually gets. This directness is the entire value proposition of the
pattern, and it's also exactly what creates the next section's problem.
## The DRY Problem Directory-Per-Environment Creates
**The moment three environment directories share 90% identical structure and differ only in a handful of
values, plain directory-per-environment starts to hurt — a change to `checkout_database`'s module call
(a new argument, a renamed variable) now has to be made in three places, by hand, and it's easy for one
environment to quietly drift out of sync with the other two.**
```mermaid
quadrantChart
title Structure vs. duplication cost as environment count grows
x-axis Few, similar environments --> Many, structurally distinct environments
y-axis Low duplication tolerance --> High duplication tolerance
quadrant-1 Plain directories still fine
quadrant-2 Terragrunt or Stacks earn their cost
quadrant-3 CLI workspaces may suffice
quadrant-4 Plain directories become painful
"3 environments, near-identical shape": [0.2, 0.75]
"3 accounts x 3 regions x 2 environments": [0.85, 0.25]
"20 identical PR-preview environments": [0.15, 0.15]
"6 environments, genuinely different shapes": [0.6, 0.6]
```
This is precisely the gap the two mechanisms in the next two sections exist to close — a variable hierarchy
(pulling shared defaults up and out) and Terragrunt (a wrapper generating the repetitive parts) both attack
the same underlying problem from different angles: keep each environment's *actual differences* explicit and
readable, while eliminating the *accidental* duplication of everything that's genuinely the same.
## Terragrunt and the Infrastructure-Live Pattern
**Terragrunt is a thin wrapper around the `terraform` CLI, popularized by Gruntwork, that generates
boilerplate (backend config, provider config, common variables) from a DRY hierarchy of small `.hcl` files —
the most widely adopted answer to directory-per-environment's duplication problem.**
```
infrastructure-live/
├── terragrunt.hcl # Root: shared backend config generation
├── prod/
│ ├── account.hcl # Account-level values (account ID, environment name)
│ ├── us-east-1/
│ │ ├── region.hcl # Region-level values
│ │ └── checkout-database/
│ │ └── terragrunt.hcl # Points at the module + this deployment's own overrides
```
```hcl
# prod/us-east-1/checkout-database/terragrunt.hcl
include "root" {
path = find_in_parent_folders()
}
terraform {
source = "git::https://github.com/meridian-platform/terraform-modules.git//database?ref=v2.3.1"
}
inputs = {
instance_class = "db.r6g.large"
multi_az = true
backup_retention = 30
}
```
The `include "root"` block pulls in shared configuration (backend generation, provider generation) defined
once at the repository root, so every leaf `terragrunt.hcl` file only ever states what's genuinely specific
to that one deployment — the module source/version and the environment-specific input values, nothing else.
Running `terragrunt apply` in this leaf directory generates the full Terraform configuration (backend block,
provider block, module call) on the fly from the hierarchy, applies it, and leaves the generated files as
disposable artifacts, not something committed to version control.
| Concern | Plain Terraform directories | Terragrunt |
|---|---|---|
| Backend config repetition | Copy-pasted per environment | Generated once, inherited everywhere |
| Provider config repetition | Copy-pasted per environment | Generated once, inherited everywhere |
| Module version pinning | Per environment's `main.tf` | Per leaf `terragrunt.hcl`, same DRY inheritance |
| Learning curve | None beyond Terraform itself | A second tool, its own HCL-like syntax and mental model |
| `terraform plan -all`-equivalent across many deployments | Not built in | `terragrunt run-all plan` — built-in orchestration across the whole tree |
> [!TIP]
> **Best practice**: adopt Terragrunt once genuine, growing duplication (per the quadrant chart above) is a
> real, felt pain — not preemptively for a three-environment setup that plain directories already handle
> comfortably. It's a real second tool with its own learning curve and failure modes; the DRY payoff needs
> to be worth that cost, which it usually is somewhere between "a handful of environments" and "dozens of
> accounts times regions times environments," not before.
## A Variable Hierarchy: Global, Account, Region, Environment
**Whether implemented via Terragrunt's `find_in_parent_folders()` mechanism or a simpler hand-rolled
`.tfvars` merge, the underlying idea is the same: values that genuinely vary by scope should be *defined*
at that scope, once, and inherited downward — not copy-pasted into every leaf that needs them.**
```mermaid
flowchart TD
Global["global.hcl:
org_name = 'meridian'
(true for everything, everywhere)"] --> Account["account.hcl:
account_id, environment name
(true for this account only)"]
Account --> Region["region.hcl:
aws_region, availability_zones
(true for this region only)"]
Region --> Leaf["Leaf terragrunt.hcl:
instance_class, multi_az
(true for THIS deployment only)"]
classDef broad fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
classDef narrow fill:#f0e9fb,stroke:#6d43c0,color:#10161c
class Global,Account broad
class Region,Leaf narrow
```
**This chapter's caption**: each layer only ever states what's genuinely new at that scope — `region.hcl`
never repeats `org_name`, and the leaf `terragrunt.hcl` never repeats the region's AZ list; every layer below
inherits everything above it automatically.
A hand-rolled equivalent without Terragrunt is achievable too, using Terraform's own `-var-file` layering
(later files override earlier ones):
```bash
terraform apply \
-var-file=../../global.tfvars \
-var-file=../account.tfvars \
-var-file=./region.tfvars \
-var-file=./prod.tfvars
```
> [!NOTE]
> This layered-`.tfvars` approach gets real DRY benefit without adopting a second tool, at the cost of the
> caller (a human or a CI job) needing to remember the correct file order every time — a real, ongoing
> discipline burden Terragrunt's automatic parent-folder inheritance removes entirely. Weigh this against the
> earlier "adopt Terragrunt once the pain is real" guidance rather than treating either as an automatic
> default.
## Monorepo vs. Polyrepo for Terraform at Scale
**Whether all of this infrastructure code lives in one repository or is split across many is a genuinely
separate decision from directory structure — and, like CLI-workspaces-vs-directories, the right answer
tracks team size and ownership boundaries more than any technical property of Terraform itself.**
| Factor | Favors monorepo | Favors polyrepo |
|---|---|---|
| Team size | 2-15 engineers, tightly collaborating | 50+ engineers, autonomous teams |
| Module change frequency | Modules and their consumers change together often | Modules have independent release cycles from consumers |
| Access control granularity | Everyone reasonably needs broad read access | Different teams need genuinely separate repo-level permissions |
| Shared tooling | One CI pipeline, one linting config, applies everywhere | Each team wants its own pipeline cadence and gating rules |
| Cross-cutting refactors | One PR touches every affected environment at once | A refactor requires coordinating PRs across several repos |
For the platform team running `checkout-service`, `catalog-service`, and `inventory-service`, a monorepo
(`infrastructure-live/`, one repository, directory-per-service-per-environment inside it) fits comfortably —
one small team, tightly coupled ownership, and the cross-cutting refactor benefit (Part 3's module extraction
scenario touched all three services in one coordinated change) genuinely matters. A larger organization with
dozens of autonomous product teams, each owning its own infrastructure independently, would reasonably split
by team or by service instead — Google and Meta's internal monorepo-at-massive-scale approach works because
of enormous, purpose-built tooling investment most organizations don't have and shouldn't try to replicate.
> [!TIP]
> **Best practice**: default to a monorepo for infrastructure code up to the point where a genuine
> cross-team ownership or access-control boundary appears — splitting later, once that boundary is real, is
> far less painful than prematurely fragmenting a small team's infrastructure across repositories they all
> need broad visibility into anyway.
## A Concrete Repository Layout for a Mid-Size Platform Team
Pulling every pattern in this chapter together, here's the platform team's actual `infrastructure-live`
repository, at the point in the series where `checkout-service`, `catalog-service`, and `inventory-service`
all run across dev/staging/prod in one AWS Organization:
```
infrastructure-live/
├── terragrunt.hcl # Root: backend + provider generation
├── global.hcl # org_name, common tags
├── dev/
│ ├── account.hcl # Dev account ID
│ └── us-east-1/
│ ├── region.hcl
│ ├── network/terragrunt.hcl
│ ├── checkout-service/
│ │ ├── database/terragrunt.hcl
│ │ └── eks-node-group/terragrunt.hcl
│ ├── catalog-service/...
│ └── inventory-service/...
├── staging/ # Same shape as dev/
├── prod/
│ └── us-east-1/
│ ├── network/terragrunt.hcl
│ ├── checkout-service/...
│ ├── catalog-service/...
│ └── inventory-service/...
└── modules/ # (or a separate terraform-modules repo, per Part 3)
├── network/
├── database/
└── eks-node-group/
```
Each service's directory under each environment mirrors Part 2's state-splitting boundaries exactly — one
Terragrunt deployment (and therefore one state file) per service per environment, all consuming the shared
`network` deployment's outputs via `dependency` blocks (Terragrunt's own equivalent of Part 2's
`terraform_remote_state`, resolved automatically from the tree structure rather than a hand-typed backend
config).
> [!IMPORTANT]
> Notice the structure directly encodes the org chart this chapter opened with: environment (dev/staging/
> prod) is the outermost split, matching the blast-radius priority from Part 2 — a mistake in `dev` should
> never even be *structurally capable* of touching `prod`, and this layout enforces that by construction,
> not by convention alone.
## Bootstrapping the State Backend Itself
**A genuine chicken-and-egg problem every new environment/account hits: Part 2's remote backend (an S3
bucket with locking enabled) is itself infrastructure — so what manages *it*, if Terraform needs a working
backend before it can safely run at all?**
The standard resolution is a small, deliberately separate "bootstrap" configuration, using **local** state
(the one legitimate long-term use of local state this series endorses, beyond a throwaway sandbox) purely
for the handful of resources — the state bucket, its versioning and locking configuration, and the IAM
policy scoping access to it — that every other configuration in that account will depend on:
```
infrastructure-live/
├── bootstrap/
│ └── prod/
│ ├── main.tf # Creates the prod account's own state bucket
│ └── terraform.tfstate # LOCAL state — deliberately, for this one directory only
├── prod/
│ └── us-east-1/
│ └── network/ # Uses the bucket bootstrap/ just created, as ITS backend
```
```hcl
# bootstrap/prod/main.tf
resource "aws_s3_bucket" "tfstate" {
bucket = "meridian-platform-tfstate-prod"
}
resource "aws_s3_bucket_versioning" "tfstate" {
bucket = aws_s3_bucket.tfstate.id
versioning_configuration { status = "Enabled" }
}
```
Once applied, every other configuration in that account references this bucket in its own `backend "s3"`
block — but the bootstrap configuration itself has nowhere "further down" to delegate its own state to,
which is exactly why it's the one deliberate, permanent exception to "always use a remote backend." Its
local state file is small, rarely changes after initial creation, and — critically — should still be backed
up (committed to a tightly-access-controlled internal repository, or copied somewhere durable) since it's the
literal foundation every other state in that account depends on existing correctly.
> [!IMPORTANT]
> This bootstrap configuration should be the **smallest possible** surface — the state bucket and its
> locking/versioning/IAM configuration, nothing else. Resist the temptation to add "just one more thing"
> (a shared IAM role, a KMS key) to it once it exists; each addition makes the one configuration in the
> whole system without a proper remote backend larger and riskier to lose track of.
## Promoting a Change From Dev to Prod
**With environments split into their own directories (or Terragrunt deployments), "promotion" is the actual
mechanism by which a change reaches production — and the discipline here is what turns directory isolation
from a nice structural property into an enforced deployment process.**
```mermaid
flowchart LR
PR["PR opened,
changes module or
dev/ deployment"] --> DevApply["Auto-plan + apply
to dev on merge
(low stakes, fast feedback)"]
DevApply --> Soak["Soak period —
automated + manual
verification in dev"]
Soak --> StagingPR["Second PR: bump the
SAME module version/config
into staging/"]
StagingPR --> StagingApply["Plan + apply to staging,
gated by review"]
StagingApply --> ProdPR["Third PR: bump into
prod/, identical change"]
ProdPR --> ProdApply["Plan + apply to prod,
gated by review + approval"]
classDef fast fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
classDef gated fill:#fbeee0,stroke:#b8650f,color:#10161c
class DevApply,Soak fast
class StagingApply,ProdApply gated
```
**This chapter's caption**: the same underlying change (a module version bump, or a `.tfvars` value) moves
through three separate, reviewed PRs rather than one PR touching all three environments simultaneously —
each promotion step is its own decision point, not an automatic cascade.
The concrete mechanism varies by how modules are sourced: if `staging/` and `prod/` pin a module by git tag
or registry version (Part 3), promotion is literally bumping that pinned version in one directory's
`terragrunt.hcl` (or `main.tf`) at a time, in order, each its own reviewed PR — never one PR editing all
three environments' pins simultaneously, which would defeat the entire point of graduated, observed
promotion.
> [!TIP]
> **Best practice**: never let staging or prod point at an unpinned branch ref ("always deploy latest") the
> way a dev environment reasonably might for fast iteration — pinned, deliberately-bumped versions are what
> make "promote this exact, already-verified-in-dev change" a meaningful, auditable action rather than an
> ambiguous "whatever the branch currently contains right now."
## Multi-Account AWS — Landing Zones and Organizations
**At real scale, "environment" and "AWS account" become the same boundary — dev, staging, and prod each get
their own AWS account under one AWS Organization, not just separate directories or `.tfvars` files inside a
single shared account.** This is the single strongest blast-radius control available: an IAM policy mistake,
a leaked credential, or a runaway resource in the `dev` account has no path to reach `prod` at all, because
they're different accounts with different credentials entirely — not a permissions boundary inside one
account that a misconfiguration could erode.
```mermaid
flowchart TD
Org["AWS Organization
(management account)"] --> SecurityOU["Security OU"]
Org --> InfraOU["Infrastructure OU"]
Org --> WorkloadsOU["Workloads OU"]
SecurityOU --> LogArchive["log-archive account"]
SecurityOU --> Audit["audit account"]
InfraOU --> SharedServices["shared-services account
(CI/CD, DNS, Terraform state bucket)"]
WorkloadsOU --> Dev["dev account"]
WorkloadsOU --> Staging["staging account"]
WorkloadsOU --> Prod["prod account"]
classDef guard fill:#fbeee0,stroke:#b8650f,color:#10161c
classDef shared fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
classDef workload fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
class SecurityOU,LogArchive,Audit guard
class InfraOU,SharedServices shared
class WorkloadsOU,Dev,Staging,Prod workload
```
**This chapter's caption**: `dev`, `staging`, and `prod` are peer accounts under one OU, not nested inside
each other — none of them can reach another by default, and every cross-account interaction (state bucket
access, CI role assumption) has to be explicitly granted, which is exactly the property a single-account,
directory-only isolation strategy can't offer.
For a mid-size organization this typically lands at 10-50 accounts (fewer, coarser OUs); a large enterprise
runs 100-500+. AWS Control Tower (or the newer Terraform-native "Account Factory for Terraform," AFT) is the
common automation layer for account *creation* itself — genuinely out of scope for a Terraform-content-only
series, but worth knowing exists as the tool that provisions the accounts this chapter's directory structure
then targets.
> [!TIP]
> **Best practice**: even a small team should seriously consider account-per-environment over
> account-per-team-with-directory-isolation the moment the cost of a `dev`-to-`prod` blast-radius mistake is
> genuinely unacceptable — the account boundary is a structural guarantee that requires deliberate,
> explicit, and auditable configuration to cross, where directory isolation alone only requires a human
> reading the right folder correctly.
## Cross-Account Access — the Hub-and-Spoke assume_role Pattern
**With separate AWS accounts, a Terraform run needs a way to act *in* the target account without holding
that account's own long-lived credentials directly — the standard pattern is a "hub" identity (a CI role, or
a human's SSO identity) that assumes a "spoke" role scoped to exactly one target account.**
```hcl
provider "aws" {
alias = "prod"
region = "us-east-1"
assume_role {
role_arn = "arn:aws:iam::444455556666:role/terraform-prod-deployer"
session_name = "terraform-ci-${terraform.workspace}"
}
}
```
```mermaid
sequenceDiagram
participant CI as CI runner (hub identity)
participant STS as AWS STS
participant Prod as prod account role
CI->>STS: AssumeRole(role_arn=prod-deployer, session_name=...)
STS->>Prod: Check trust policy — does prod-deployer
trust the CI hub identity?
Prod-->>STS: Trust confirmed
STS-->>CI: Temporary credentials, scoped to prod-deployer's
permissions, expiring in 1 hour
CI->>Prod: terraform apply, using temporary credentials
```
The target account's `terraform-prod-deployer` role's trust policy is the actual security control — it names
exactly which hub identity is permitted to assume it, and the role's own IAM policy scopes exactly what that
session can do once assumed. No long-lived credential for the `prod` account ever needs to exist outside that
account itself; the CI hub only ever holds credentials for *assuming into* other accounts, each session
expiring automatically.
> [!TIP]
> **Best practice**: scope each spoke role as tightly as the workload genuinely needs (least privilege, not
> a blanket `AdministratorAccess` "because it's easier") and give each *environment* its own distinct spoke
> role rather than one shared "terraform-deployer" role reused with different session names — a distinct
> role per environment means the trust policy and permission boundary are independently auditable and
> independently revocable per environment, not one shared blast radius across all of them.
## Multi-Cloud Directory Structure — Keeping Clouds Cleanly Separated
**When a second cloud provider enters the picture — GCP for disaster recovery, per this chapter's closing
scenario — the same discipline that separates environments applies again, one level up: never mix two
clouds' resources inside the same state file, and keep cloud-specific configuration in its own directory
branch rather than interleaved with the primary cloud's.**
```
infrastructure-live/
├── aws/
│ ├── dev/us-east-1/...
│ ├── staging/us-east-1/...
│ └── prod/us-east-1/...
└── gcp/
└── dr/us-central1/
├── network/terragrunt.hcl
└── checkout-service-replica/terragrunt.hcl
```
Splitting at the cloud level, above environment, keeps each cloud's account/project structure, IAM model,
and provider configuration fully independent — an AWS Organizational Unit and a GCP Folder are conceptually
similar but never actually related, and forcing them into a shared directory branch (interleaving
`aws-prod/` next to `gcp-prod/` as siblings under one `prod/` folder, for instance) tends to produce
configuration that quietly assumes cross-cloud symmetry that doesn't actually exist.
| Practice | Why it matters across clouds |
|---|---|
| Never share one state file across two providers | State corruption on one cloud's API hiccup shouldn't be able to affect the other cloud's tracked resources at all |
| Cloud-specific modules stay cloud-specific | An `aws-vpc` module and a `gcp-vpc` module solve conceptually similar problems with entirely different arguments — don't force one shared interface |
| A consistent tagging/labeling convention across clouds | AWS tags and GCP labels are different mechanisms, but naming them consistently (`team`, `environment`, `managed-by`) is what makes cross-cloud cost/ownership reporting (Part 9) possible at all |
| One root-level split by cloud, not interleaved per-environment | Keeps each cloud's real account/project/IAM structure independently legible, rather than forcing an artificial parallel structure |
> [!NOTE]
> Part 9 goes deep on the actual multi-cloud *provisioning* patterns (provider aliasing across clouds in one
> configuration, when that's appropriate vs. when full directory separation is better) — this chapter's
> concern is specifically the repository/directory shape, which is the decision that has to be made before
> any of Part 9's provisioning patterns are written.
## HCP Terraform Stacks — a Native Alternative to Terragrunt
**Stacks, HashiCorp's own answer to the DRY-multi-deployment problem, reached broader general availability
through 2026 (including monorepo support) — a native, first-party alternative to Terragrunt worth knowing
about even for a team not currently on HCP Terraform, since it represents where the ecosystem is heading.**
A Stack defines reusable "components" (similar in spirit to modules) and "deployments" (similar in spirit to
Terragrunt's per-environment leaf configuration), with HCL-native orchestration across them instead of a
separate wrapper tool — the goal being Terragrunt's DRY, multi-environment ergonomics without introducing a
second tool and syntax dialect on top of Terraform itself.
| | Terragrunt | HCP Terraform Stacks |
|---|---|---|
| Maturity (2026) | Long-established, widely adopted | Newly GA'd, actively expanding feature set |
| Tooling | Separate CLI, separate `.hcl` dialect | Native to Terraform/HCP Terraform, no second tool |
| Works with self-hosted backends | Yes, backend-agnostic | Tied to HCP Terraform |
| Works with OpenTofu | Yes | No — HCP Terraform/Terraform-specific |
> [!NOTE]
> Given Stacks' current HCP Terraform coupling, a team already committed to a self-hosted backend (Part 2's
> S3-native-locking pattern) or to OpenTofu specifically (Part 1's licensing discussion) doesn't have a
> Stacks option today — Terragrunt remains the practical DRY answer for that combination. Teams already
> planning to consolidate onto HCP Terraform for its managed run pipeline (Part 8) are the ones for whom
> evaluating Stacks against Terragrunt is a live, current decision.
## Naming Conventions Across the Tree
**A consistent naming and tagging scheme, applied identically across every environment/account/cloud
directory, is what makes the structure this chapter builds actually navigable and query-able at scale — a
different naming habit per environment quietly undermines every other practice in this chapter.**
```hcl
locals {
name_prefix = "${var.org}-${var.environment}-${var.service}"
# e.g. "meridian-prod-checkout"
common_tags = {
Organization = var.org
Environment = var.environment
Service = var.service
ManagedBy = "terraform"
Repository = "infrastructure-live"
}
}
resource "aws_db_instance" "this" {
identifier = "${local.name_prefix}-db"
tags = local.common_tags
}
```
| Convention | Example | Why it matters |
|---|---|---|
| A single, shared `name_prefix` local, computed identically everywhere | `meridian-prod-checkout-db` | Makes every resource's environment and service ownership readable from its name alone, in the AWS console, in a cost report, anywhere |
| `common_tags` merged into every taggable resource | `Environment`, `Service`, `ManagedBy`, `Repository` | Feeds Part 9's cost-attribution and governance queries directly — untagged or inconsistently-tagged resources are invisible to them |
| A `Repository` tag pointing back to the exact repo (and, ideally, exact directory) that manages a resource | `infrastructure-live` | Answers "where's the code for this" instantly during an incident, without guessing across a monorepo/polyrepo split |
> [!TIP]
> **Best practice**: define `name_prefix` and `common_tags` once, in the shared root-level configuration
> every environment inherits from (Terragrunt's `global.hcl`, or a shared `locals.tf` sourced by every
> directory) — never let each environment or service redefine its own tagging shape independently. A single
> naming/tagging bug fixed once, at the shared source, is far better than the same bug silently diverging
> three different ways across dev/staging/prod.
## Choosing Your Structure — a Decision Framework
**Pulling this entire chapter into one decision sequence:**
```mermaid
flowchart TD
Start{"How many environments,
and how similar?"} -->|"Few, near-identical,
ephemeral"| WS["CLI workspaces"]
Start -->|"Few, structurally
distinct, long-lived"| Dirs{"Growing duplication
pain across directories?"}
Dirs -->|"Not yet"| PlainDirs["Plain directory-per-environment"]
Dirs -->|"Yes, real and growing"| DRY{"On HCP Terraform
already, or committed to it?"}
DRY -->|"Yes"| Stacks["HCP Terraform Stacks"]
DRY -->|"No / on OpenTofu / self-hosted"| TG["Terragrunt"]
Start -->|"Multiple accounts,
possibly multiple clouds"| Accounts["Account-per-environment +
cloud-level directory split,
THEN apply the above within it"]
classDef simple fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
classDef dry fill:#f0e9fb,stroke:#6d43c0,color:#10161c
classDef scale fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
class WS,PlainDirs simple
class Stacks,TG dry
class Accounts scale
```
## Worked Scenario: Migrating Off Workspaces Into Directory-Per-Environment
The platform team's earliest Terraform setup used CLI workspaces for dev/staging/prod, driven by
`terraform.workspace` conditionals throughout. As `checkout-service` grew genuinely divergent prod-only
requirements (a read replica, WAF rules, a different backup schedule), the conditional logic sprawled
across a dozen `terraform.workspace == "prod" ? ... : ...` expressions scattered through the configuration,
increasingly hard for a new team member to reconstruct "what does prod actually look like" from.
The migration itself used the state-splitting mechanics from Part 2 — each CLI workspace's state was
`terraform state pull`ed independently, and each became the seed state for a new, dedicated
`environments//` directory with its own explicit backend `key`. No resources were destroyed or
recreated; only the organizing structure around already-applied infrastructure changed. Six months post-
migration, `environments/prod/main.tf` reads as a complete, honest description of production on its own,
with zero conditional expressions — exactly the payoff this chapter's second section promised.
## Worked Scenario: Standing Up a Second Cloud for Disaster Recovery
Following an executive mandate for cross-cloud disaster recovery (not just cross-region), the team stood up
a GCP replica of `checkout-service`'s critical path — Cloud SQL as the replicated read target, GKE as the
standby compute layer — under a brand-new `gcp/` directory branch, exactly per this chapter's multi-cloud
structure guidance. The team deliberately did **not** try to reuse the AWS-side `database` or
`eks-node-group` modules with cloud-specific conditionals bolted in — a lesson learned directly from Part 3's
"god module" anti-pattern — and instead wrote genuinely separate `gcp-cloudsql` and `gke-node-pool` modules,
accepting real duplication of *structure* (both provision "a managed database" and "a Kubernetes node pool"
conceptually) in exchange for each module staying simple, provider-native, and easy to reason about on its
own.
> [!NOTE]
> This is the practical, worked version of the "multi-cloud directory structure" table's guidance from
> earlier in the chapter — the team chose duplicated, cloud-native modules over one forced shared interface,
> and considers that the correct call in hindsight, specifically because AWS RDS and GCP Cloud SQL differ
> enough in their real configuration surface that a shared abstraction would have either leaked provider
> details anyway or forced awkward, disabled-feature compromises on both sides.
## Worked Scenario: the tfvars Override That Silently Hit the Wrong Account
An engineer running a routine `catalog-service` change in `staging` used a shell alias that, on this
particular machine, still pointed `AWS_PROFILE` at the `prod` account from an unrelated task earlier that
day. The `staging/` directory's own explicit `backend.tf` correctly targeted the staging state bucket — Part
2's directory-explicit-backend discipline worked exactly as designed — but the *provider* credentials used
to actually execute the plan came from the shell's ambient AWS profile, which the directory structure has no
way to enforce on its own.
The plan showed changes against real `prod` resources (visible instance IDs, ARNs the engineer immediately
recognized as wrong), and the review discipline from Part 1 caught it before `apply` — but the near-miss
prompted a structural fix: every environment directory's own generated backend/provider configuration
(via Terragrunt, adopted shortly after this incident) now also asserts the AWS account ID it expects to be
operating against, failing `plan` outright with a clear error if the assumed role's actual account ID
doesn't match what that directory declares.
```hcl
data "aws_caller_identity" "current" {}
resource "terraform_data" "account_guard" {
lifecycle {
precondition {
condition = data.aws_caller_identity.current.account_id == "222233334444"
error_message = "This directory targets the staging account (222233334444) — got ${data.aws_caller_identity.current.account_id}."
}
}
}
```
> [!CAUTION]
> Directory structure alone communicates *intent* — it doesn't enforce *which credentials actually get
> used*. An ambient environment variable, a stale AWS profile, or a misconfigured CI secret can all silently
> override what the directory name implies. An explicit account-ID assertion (shown above) or an equivalent
> provider-level guard is the actual enforcement mechanism; treat directory naming as documentation for
> humans, and a runtime check as the real safety net.
## Part 4 CLI Cheat Sheet
| Command | Purpose |
|---|---|
| `terraform workspace list` / `new` / `select` | Manage CLI workspaces |
| `terraform apply -var-file=prod.tfvars` | Apply with an explicit, layered variable file |
| `terragrunt plan` / `apply` (in a leaf directory) | Generate config from the DRY hierarchy and run Terraform |
| `terragrunt run-all plan` | Plan every deployment under the current tree at once |
| `aws sts assume-role --role-arn ... --role-session-name ...` | Manually test a cross-account role assumption outside Terraform |
| `aws sts get-caller-identity` | Confirm which account/identity is actually active before running anything destructive |
## Common Mistakes and Interview Traps
| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Confusing CLI workspaces with HCP Terraform workspaces | They're unrelated concepts sharing one name | Always clarify "CLI workspace" vs. "HCP Terraform workspace" explicitly |
| Using CLI workspaces for long-lived, structurally different environments | Differences live in scattered conditionals, easy to get subtly wrong, and switching workspaces is a silent local state change | Use directory-per-environment for permanent, structurally distinct environments |
| Adopting Terragrunt (or Stacks) before duplication is a real, felt problem | Adds a second tool's learning curve and failure modes for a benefit that doesn't yet exist | Start with plain directories; adopt DRY tooling once duplication pain is genuinely growing |
| Isolating environments by directory alone, with a shared AWS account | A directory boundary doesn't stop a misconfigured credential from crossing it | Use account-per-environment for anything where a cross-environment mistake is unacceptable |
| Assuming directory structure enforces which credentials get used | Ambient env vars/profiles can silently override the directory's implied target | Add an explicit account/project-ID assertion as a runtime guard, not just directory naming |
| Interleaving two clouds' resources under one shared per-environment folder | Encourages a false assumption of cross-cloud symmetry that rarely actually holds | Split at the cloud level first, environment second, within each cloud's own branch |
## Worked Practice Problems
**Problem 1**: A team runs three near-identical, disposable load-test environments, created and destroyed
multiple times per week, alongside a permanent, structurally distinct production environment. What structure
fits each, and why shouldn't they use the same mechanism for both?
*Answer*: The load-test environments fit CLI workspaces well — they're genuinely ephemeral and structurally
identical to each other, and creating/destroying a workspace is fast with no new directory scaffolding
needed each time. Production should be its own explicit directory (and, following this chapter's stronger
guidance, its own AWS account) — it's long-lived, structurally distinct from anything else, and the stakes
of an accidental wrong-target apply are high enough to want the structural, not just conventional,
protection a dedicated directory (and account) provides. Using CLI workspaces for both would put production
one `terraform workspace select` mistake away from an accidental apply; using a full directory-per-instance
structure for the load-test environments would mean constant directory scaffolding for something meant to
be quick and disposable.
**Problem 2**: A platform team has 4 engineers, one shared set of infrastructure modules, and 3 services
that change together frequently during releases. Should they use a monorepo or split into per-service
repositories, and what would change their answer?
*Answer*: A monorepo fits this team well — small, tightly-collaborating team, modules and consumers that
change together often, and no genuine need yet for per-repository access control. The concrete condition
that would change this answer: the team growing into several genuinely autonomous groups each owning one
service independently, needing separate release cadences and separate repository-level permissions — at
that point, the polyrepo factors from this chapter's comparison table start outweighing the monorepo's
current cross-cutting-refactor convenience.
**Problem 3**: An engineer's `terraform plan` for the `staging` directory shows changes against resources
with ARNs the engineer doesn't recognize from staging at all. The directory's `backend.tf` correctly points
at the staging state bucket. What's the most likely explanation, and what structural fix prevents this class
of mistake going forward?
*Answer*: The backend (state location) is correct, but the *provider* credentials actually being used
during this run are pointed at a different account than the directory implies — an ambient
`AWS_PROFILE`/session left over from unrelated work, exactly per this chapter's tfvars-override scenario.
The state backend being correct doesn't guarantee the provider's actual target account matches; the
structural fix is an explicit account-ID (or project-ID, on GCP) assertion via a `precondition` (or
equivalent guard) that fails the plan outright the moment the assumed identity's real account doesn't match
what that directory declares, rather than relying on directory naming and human attention alone.
## Summary and What's Next
Repository and directory structure is where Part 2's state-splitting and Part 3's module design actually
meet the real world: CLI workspaces for genuinely ephemeral, identical environments; explicit directories
for anything permanent and structurally distinct; a DRY tool (Terragrunt today, Stacks increasingly as it
matures) once duplication pain across those directories is real; and, at genuine scale, account-per-
environment and cloud-level directory separation as the actual enforcement mechanisms behind what a folder
name only implies. None of this is Terraform syntax — it's the organizational discipline that keeps a
mistake in `dev` from ever having a structural path to reach `prod`, which is worth more than almost any
single language feature this series covers.
Part 5 returns to configuration-language depth: providers beyond the single-region default this chapter's
examples assumed, data sources for reading infrastructure Terraform doesn't manage, and provisioners' narrow,
correctly-scoped role — all of it informed by the directory and account structure this chapter just
established as the ground everything else in the series stands on.