# Automation, CI/CD & GitOps — Part 2: Infrastructure as Code > **Series:** Automation, CI/CD & GitOps (2 of 14) > **Part 1:** `01-cicd-fundamentals.md` — CI/CD Fundamentals > **Part 2:** This file — Infrastructure as Code > **Part 3:** `03-gitops.md` — GitOps > **Part 4:** `04-github-actions.md` — GitHub & GitHub Actions > **Part 5:** `05-github-security-governance.md` — GitHub Security & Governance > **Part 6:** `06-gitlab-cicd.md` — GitLab & GitLab CI/CD > **Part 7:** `07-bitbucket-pipelines.md` — Bitbucket & Bitbucket Pipelines > **Part 8:** `08-azure-devops.md` — Azure DevOps > **Part 9:** `09-jenkins.md` — Jenkins > **Part 10:** `10-circleci.md` — CircleCI > **Part 11:** `11-tekton.md` — Tekton > **Part 12:** `12-monorepo-cicd.md` — Monorepo CI/CD Strategies > **Part 13:** `13-progressive-delivery.md` — Progressive Delivery with Argo Rollouts & Flagger > **Part 14:** `14-self-hosted-runner-scaling.md` — Self-Hosted Runner Scaling & Cost Optimization > **Questions:** `questions.md` ## Table of Contents 1. [The Problem IaC Actually Solves](#the-problem-iac-actually-solves) 2. [Configuration Management vs Provisioning — Two Different Jobs](#configuration-management-vs-provisioning--two-different-jobs) 3. [Idempotency — The Single Most Important IaC Property](#idempotency--the-single-most-important-iac-property) 4. [Declarative vs Imperative IaC](#declarative-vs-imperative-iac) 5. [Terraform — Core Concepts](#terraform--core-concepts) 6. [The Terraform Workflow](#the-terraform-workflow) 7. [Terraform State — The Most Important, Most Misunderstood Concept](#terraform-state--the-most-important-most-misunderstood-concept) 8. [State Locking — Preventing a Real, Common Disaster](#state-locking--preventing-a-real-common-disaster) 9. [Drift — When Reality Disagrees With the Code](#drift--when-reality-disagrees-with-the-code) 10. [Terraform Modules — Reusability](#terraform-modules--reusability) 11. [Ansible — A Different Tool for a Different Job](#ansible--a-different-tool-for-a-different-job) 12. [Terraform vs Ansible — Choosing the Right Tool](#terraform-vs-ansible--choosing-the-right-tool) 13. [IaC Scanning, Revisited](#iac-scanning-revisited) 14. [Common Mistakes](#common-mistakes) 15. [Worked Practice Problems](#worked-practice-problems) 16. [Summary and What's Next](#summary-and-whats-next) --- ## The Problem IaC Actually Solves Before IaC, provisioning infrastructure meant a human manually clicking through a cloud console, or manually SSHing into servers and running commands — every single time, remembered correctly (or not) from memory or a wiki page. ```mermaid graph TD Manual["Manual infrastructure
changes: click through a
console, run commands by
hand"] --> ManualProb["❌ Not repeatable, not
versioned, not reviewable —
exactly the TOIL problem
from the SRE Fundamentals
series, applied to
infrastructure itself"] IaC["Infrastructure as Code:
infrastructure defined in
VERSION-CONTROLLED files,
applied by a TOOL"] --> IaCGood["✅ Repeatable, reviewable
(via pull requests, exactly
like application code),
and AUDITABLE — every
change has a git history"] ``` **A clean, memorable interview line:** "Infrastructure as Code takes the exact same discipline we apply to application code — version control, code review, automated testing — and applies it to servers, networks, and databases. It's the direct antidote to the manual, undocumented, tribal-knowledge way infrastructure used to be managed." --- ## Configuration Management vs Provisioning — Two Different Jobs A genuinely important distinction, and a very common early interview question — the IaC ecosystem actually splits into two related-but-different jobs. ```mermaid graph TD Provisioning["PROVISIONING:
creating the actual
INFRASTRUCTURE itself —
a VM, a network, a
database instance,
an S3 bucket"] --> ProvTools["Tools: Terraform,
CloudFormation, Pulumi"] ConfigMgmt["CONFIGURATION
MANAGEMENT: configuring
WHAT RUNS ON that
infrastructure once it
exists — installing
packages, managing config
files, running services"] --> ConfigTools["Tools: Ansible, Chef,
Puppet"] ``` **Simple analogy:** provisioning is building the house (walls, plumbing, electrical) — configuration management is furnishing and setting up everything *inside* it once the house exists (which room has which furniture, which lights are on). Both are necessary, but they're genuinely different jobs, often (though not always) handled by different tools. --- ## Idempotency — The Single Most Important IaC Property This is arguably the single most important concept in this entire tutorial — the property that makes IaC actually safe and trustworthy to use repeatedly. ```mermaid graph TD Idempotent["IDEMPOTENT: running the
SAME operation MULTIPLE
TIMES produces the SAME
end result as running it
ONCE — no matter how many
times you run it"] --> IdempotentGood["✅ SAFE to re-run —
whether it's the first
run, or the hundredth,
after a failure, after a
partial success —
ALWAYS converges to the
SAME correct state"] NonIdempotent["NON-IDEMPOTENT: running
the SAME operation
multiple times keeps
ADDING to or CHANGING
the result each time"] --> NonIdempotentBad["❌ DANGEROUS to re-run —
e.g. a script that BLINDLY
runs 'CREATE a new server'
every time, creating
DUPLICATE servers on
every accidental re-run"] ``` **A concrete, worked example worth having ready:** a script that says `run: create-server --name web-1` is NOT idempotent — running it twice creates two servers (or errors confusingly on the second attempt). A properly idempotent tool instead expresses the *desired state* — "a server named web-1 should exist, with these properties" — and checks first: if it already exists and matches, do nothing; if it doesn't exist, create it; if it exists but doesn't match, update it to match. **This is EXACTLY the same reconciliation loop pattern from the Kubernetes Deep Dive series (Part 1) — Terraform, Ansible, and Kubernetes controllers all share this identical underlying philosophy, just applied at different layers.** --- ## Declarative vs Imperative IaC Already introduced conceptually in the Kubernetes Deep Dive series (Part 1) — here's the direct, concrete IaC-specific version of that same distinction. ```mermaid graph TD Imperative["IMPERATIVE IaC: 'run
THESE commands, in THIS
order' — e.g. a shell
script: create VPC, THEN
create subnet, THEN
create server..."] --> ImperativeNote["You must track the
CURRENT state yourself,
and figure out what
STEPS get you to the
desired end state"] Declarative["DECLARATIVE IaC: 'here's
the state I want the
END RESULT to look like' —
the TOOL figures out
what needs to change to
get there"] --> DeclarativeNote["The tool compares current
vs desired state and
computes the DIFF itself —
you never manually track
'what step comes next'"] ``` **Terraform, CloudFormation, and Kubernetes manifests are all declarative** — this is the dominant, modern approach for exactly the same reasons declarative Kubernetes configuration is preferred (Kubernetes Deep Dive series, Part 1): it's idempotent by design, and safely re-runnable regardless of the starting state. --- ## Terraform — Core Concepts **Terraform** (by HashiCorp) is the most widely used, cloud-agnostic infrastructure provisioning tool — genuinely worth deep, hands-on familiarity. ```hcl # main.tf — a simple, real Terraform configuration resource "aws_instance" "web" { ami = "ami-0abcdef1234567890" instance_type = "t3.medium" tags = { Name = "web-server" } } resource "aws_s3_bucket" "data" { bucket = "my-app-data-bucket" } ``` ```mermaid graph TD HCL["HCL (HashiCorp
Configuration Language):
the DECLARATIVE syntax
Terraform config files
are written in"] --> Provider["Provider: a PLUGIN
(AWS, GCP, Azure,
Kubernetes, hundreds more)
that knows how to talk to
a SPECIFIC platform's API"] Provider --> Resource["Resource: a SPECIFIC
piece of real
infrastructure (a VM, a
bucket, a database) to
create/manage"] ``` --- ## The Terraform Workflow ```mermaid sequenceDiagram participant Dev as Developer participant TF as Terraform participant State as State File participant Cloud as Real Cloud API Dev->>TF: terraform plan TF->>State: Read CURRENT known state TF->>Cloud: (Often) verify actual
real-world state too TF->>Dev: Show a DIFF: what will
be created/changed/destroyed Dev->>Dev: Review the plan
(exactly like a code review) Dev->>TF: terraform apply TF->>Cloud: Make the ACTUAL API
calls to create/update/
destroy resources TF->>State: Update the state file
to reflect the new reality ``` ```bash # The core Terraform workflow, in order terraform init # download providers/modules terraform plan # show what WOULD change, without changing anything terraform apply # actually make the changes terraform destroy # tear down everything Terraform manages ``` **Why `terraform plan` is such a genuinely important safety step, worth stating explicitly:** it's a **dry run** — showing exactly what would be created, modified, or (critically) **destroyed**, before anything actually happens, giving a human (or an automated policy gate, directly reusing the OPA/Gatekeeper concept from the DevSecOps series) a chance to catch a mistake before it becomes real, irreversible infrastructure damage. --- ## Terraform State — The Most Important, Most Misunderstood Concept This deserves the single largest section in this tutorial, because getting it wrong is one of the most common, most damaging real-world Terraform mistakes. ```mermaid graph TD State["The STATE FILE
(terraform.tfstate):
Terraform's own RECORD of
what it BELIEVES exists in
the real world, and how
each resource maps to
your configuration"] --> Why["WHY it's needed: Terraform
can't just ask the cloud
'show me everything you
manage for me' — the state
file is the ONLY thing
connecting your HCL code
to the SPECIFIC real
resources it created"] ``` **Why losing the state file is a genuinely severe problem, worth stating explicitly:** without it, Terraform has no memory of what it previously created — running `terraform apply` again could attempt to create brand-new, duplicate resources instead of recognizing existing ones, or Terraform might believe resources need to be destroyed and recreated that are actually fine as-is. **This is exactly why the state file itself needs to be treated with the same seriousness as the etcd backup discussion from the Kubernetes Deep Dive series** — it's a small file with an outsized, critical importance. ```mermaid graph TD Local["LOCAL state file
(on one person's laptop)"] --> LocalProb["❌ Not shared — a SECOND
person running Terraform
has NO idea what the
first person already
created
❌ Easily lost (laptop dies,
file accidentally deleted)"] Remote["REMOTE state
(e.g. an S3 bucket, or
Terraform Cloud)"] --> RemoteGood["✅ SHARED — the whole
team sees the SAME,
current state
✅ Durable, backed up,
versioned"] ``` ```hcl # Configuring remote state storage — a near-universal # best practice for any real team using Terraform terraform { backend "s3" { bucket = "my-terraform-state" key = "prod/network/terraform.tfstate" region = "us-east-1" } } ``` --- ## State Locking — Preventing a Real, Common Disaster A directly important extension of the remote-state discussion, worth its own callout since it prevents a genuinely common, real-world corruption scenario. ```mermaid sequenceDiagram participant DevA as Developer A participant DevB as Developer B participant State as Shared Remote State DevA->>State: terraform apply (reads
current state, starts
making changes) DevB->>State: ALSO runs terraform apply
AT THE SAME TIME (reads
the SAME starting state) Note over DevA,DevB: 🚨 WITHOUT locking: both
write CONFLICTING updates
to the state file —
CORRUPTION, and the state
no longer matches reality ``` **State locking fixes this exactly the way a database transaction lock prevents two concurrent writers from corrupting the same row:** while Developer A's `apply` is in progress, the state is **locked** — Developer B's simultaneous `apply` attempt is blocked (or fails immediately with a clear "state is locked" error) until A's operation completes and releases the lock. Most remote backends (like S3 combined with a DynamoDB table for locking) support this automatically, and it's a genuinely essential, non-optional safety feature for any team with more than one person running Terraform. --- ## Drift — When Reality Disagrees With the Code **Drift** happens when the real, actual infrastructure no longer matches what the Terraform state (and configuration) believe it should be — usually because someone made a manual change directly in the cloud console, bypassing Terraform entirely. ```mermaid graph TD A["Terraform config says:
instance_type = 't3.medium'"] --> B["Someone manually resizes
the SAME instance to
't3.large' directly in
the AWS console
(bypassing Terraform)"] B --> C["🚨 DRIFT: the REAL world
(t3.large) no longer
matches what Terraform's
state/config BELIEVES
(t3.medium)"] C --> D["Next 'terraform plan'
shows an UNEXPECTED
change - 'downgrade back
to t3.medium' - which may
surprise/confuse whoever
runs it, or WORSE, get
blindly applied,
accidentally undoing the
manual change"] ``` ```bash # Detect drift explicitly, without applying anything terraform plan -detailed-exitcode # Exit code 0 = no changes (no drift) # Exit code 2 = changes detected (possible drift) ``` **The direct, practical fix worth stating explicitly, and a genuinely important discipline: treat the Terraform configuration as the single source of truth, and NEVER make manual changes directly in the cloud console for anything managed by Terraform** — any change, however small, should go through the same `plan` -> review -> `apply` workflow. This exact discipline is precisely what GitOps (Part 3 of this series) formalizes and enforces even more strictly. --- ## Terraform Modules — Reusability Exactly the same motivation as functions in a programming language, or Helm charts in Kubernetes (Kubernetes Deep Dive series, Part 4) — avoid repeating the same configuration over and over. ```hcl # Using a reusable module for a standard "web application" pattern module "checkout_service" { source = "./modules/web-app" app_name = "checkout" instance_type = "t3.medium" min_instances = 3 max_instances = 10 } module "payments_service" { source = "./modules/web-app" app_name = "payments" instance_type = "t3.large" min_instances = 5 max_instances = 20 } ``` **Why this matters practically, directly reusing the "consistency across teams" principle from the DevSecOps series (Part 5, on CI/CD pipeline security):** a well-designed module can bake in an organization's security and reliability best practices (correct network isolation, correct IAM least-privilege scoping, correct tagging for cost tracking) once, centrally — every team consuming the module automatically inherits those best practices, rather than each team reinventing (and potentially getting wrong) the same infrastructure pattern independently. --- ## Ansible — A Different Tool for a Different Job **Ansible** is the most widely used configuration management tool — it doesn't provision infrastructure (that's Terraform's job); it configures what's already running on existing servers. ```yaml # A simple Ansible playbook — configuring an already-existing server - name: Configure web servers hosts: webservers become: true tasks: - name: Install nginx apt: name: nginx state: present # idempotent - does nothing if already installed - name: Ensure nginx is running service: name: nginx state: started enabled: true - name: Deploy nginx config template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf notify: restart nginx handlers: - name: restart nginx service: name: nginx state: restarted ``` ```bash # Run a playbook against a defined set of hosts ansible-playbook -i inventory.ini configure-web.yml ``` **Notice `state: present` and `state: started` — this is Ansible's own idempotency in action, worth pointing out explicitly:** running this playbook 100 times in a row produces the exact same end result as running it once — nginx installed and running — rather than 100 redundant install attempts. **A genuinely important architectural detail worth knowing: Ansible is agentless** — it connects over standard SSH and runs its logic remotely, with no permanent agent software needing to be pre-installed on the target servers, which is a real, distinctive difference from some other configuration management tools (like Chef and Puppet, which traditionally do require an installed agent). --- ## Terraform vs Ansible — Choosing the Right Tool ```mermaid flowchart TD Q{"What are you actually
trying to do?"} --> Q1{"Create/destroy/resize
the INFRASTRUCTURE
itself (a VM, a network,
a database)?"} Q1 -->|Yes| TF["Terraform
(provisioning)"] Q1 -->|No| Q2{"Configure/install
software ON already-
existing infrastructure?"} Q2 -->|Yes| Ansible["Ansible
(configuration management)"] ``` **A strong, senior-level interview answer, worth stating explicitly: these tools are frequently used TOGETHER, not as competitors — a very common real-world pattern is Terraform provisioning the VMs/network/infrastructure, and then Ansible (sometimes triggered directly by Terraform's own `provisioner` blocks, though a separate, decoupled pipeline stage is often cleaner) configuring what actually runs on those newly-created servers.** In a fully containerized/Kubernetes-native environment, Ansible's role shrinks considerably (since container images already bake in "what software is installed," and Kubernetes itself handles a lot of what Ansible would otherwise do) — but it remains extremely common and valuable for traditional VM-based infrastructure. --- ## IaC Scanning, Revisited Already covered in depth in the DevSecOps series (Part 5) — worth a brief, explicit callback here, since it's directly relevant to this tutorial's subject matter, not just a security afterthought. ```bash # From the DevSecOps series — scanning Terraform BEFORE it's ever applied tfsec . checkov -d . ``` **Why this belongs in both series, worth stating explicitly: IaC scanning is fundamentally a shift-left security practice (DevSecOps series) applied specifically to the infrastructure-provisioning workflow this tutorial covers** — catching a misconfigured, publicly-exposed resource in the `terraform plan` output, before `apply` ever runs, is exactly the same "catch it as early and cheaply as possible" principle woven throughout this entire course. --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Storing Terraform state locally, on one person's machine | Not shared with the team, easily lost, and a recipe for confusing, conflicting applies | Use remote state (S3, Terraform Cloud, etc.) as a near-universal default | | No state locking configured for a shared remote backend | Two simultaneous `apply` operations can corrupt the state file | Enable locking (e.g., S3 + DynamoDB) alongside remote state | | Making manual changes directly in the cloud console for Terraform-managed resources | Causes drift — reality no longer matches the code, and the next `apply` can produce confusing or dangerous unexpected changes | Treat the Terraform configuration as the sole source of truth; route ALL changes through `plan`/`apply` | | Writing imperative, ordered shell scripts instead of declarative IaC | Not idempotent, not safely re-runnable, requires manually tracking current state | Use declarative tools (Terraform, Ansible with idempotent modules) that compute the diff themselves | | Treating `terraform plan` as optional, going straight to `apply` | Skips the one safety check that shows exactly what will be created/changed/DESTROYED before it happens | Always review the plan output, especially for anything showing a destroy/replace action | | Using Ansible to try to provision cloud infrastructure it wasn't designed for, or Terraform to configure software inside a running server | Uses the wrong tool for the job, fighting against each tool's actual design | Use Terraform for provisioning, Ansible (or container images) for configuration — often together, not as substitutes | --- ## Worked Practice Problems **Problem 1:** A team's Terraform state file was stored locally on a departed engineer's laptop, which has since been wiped. The infrastructure it managed is still running fine in production. What's the actual risk, and what would you do? *Answer:* The real infrastructure is fine right now, but Terraform itself has effectively lost all memory of what it created and how — running `terraform apply` again with no matching state could attempt to create duplicate resources, or Terraform might not recognize existing resources at all, risking accidental deletion/recreation on the next run. The fix: use `terraform import` to rebuild a fresh state file by explicitly telling Terraform "this existing real resource corresponds to this specific block in my configuration," resource by resource, reconnecting the code to reality — and immediately move to remote, shared state storage afterward to prevent this exact scenario from ever happening again. **Problem 2:** During a routine `terraform plan`, an engineer notices Terraform wants to "destroy and recreate" a production database resource that nobody intentionally changed. What's the most likely explanation, and what should they do before running `apply`? *Answer:* This is a strong signal of drift — someone likely made a manual change directly in the cloud console (or a different automation process touched the resource) that doesn't match what the Terraform configuration expects, and for certain resource attributes, Terraform's only way to reconcile a mismatch is a destroy-and-recreate rather than an in-place update. Before running `apply`, they should absolutely NOT blindly accept this — for a database specifically, a destroy-and-recreate could mean real, catastrophic data loss. They should investigate what actually changed (checking cloud provider audit logs, comparing the resource's real current configuration against the Terraform code) and either update the Terraform configuration to match the intentional manual change, or revert the manual change to match the code — resolving the drift deliberately, never by blindly running `apply` on a plan showing an unexpected destroy. **Problem 3:** A team wants to provision 10 new microservices' infrastructure, each needing an identical pattern (a VPC, an autoscaling group, a load balancer, correctly-scoped IAM roles). What Terraform feature would you recommend to avoid copy-pasting the same configuration 10 times, and what's the concrete benefit beyond just less typing? *Answer:* A reusable Terraform module encapsulating the standard "microservice infrastructure" pattern, parameterized by the few things that genuinely differ per service (name, instance size, scaling limits). Beyond just reducing repetition, the real benefit is baking correct security and reliability practices (proper network isolation, least-privilege IAM scoping) into the module ONCE — every team consuming it automatically inherits those correct defaults, rather than each of the 10 teams independently writing (and potentially getting subtly wrong) their own version of the same pattern, directly mirroring the consistency benefit of a shared library in application code. --- ## Summary and What's Next - **Infrastructure as Code** applies the same version control, review, and audit discipline used for application code to infrastructure itself — directly attacking the manual, undocumented "toil" problem from the SRE Fundamentals series. - **Provisioning** (Terraform — creating the infrastructure itself) and **configuration management** (Ansible — configuring what runs on it) are two related but genuinely different jobs, often used together. - **Idempotency** — running the same operation any number of times produces the same end result — is the single most important property making IaC safe to re-run, and it's the exact same underlying philosophy as the Kubernetes reconciliation loop. - **Declarative** IaC (describe the desired end state; let the tool compute the diff) is the dominant modern approach, for exactly the same reasons declarative Kubernetes configuration is preferred. - The **Terraform state file** is the critical, easy-to-underestimate link between your code and real infrastructure — it must be stored remotely, shared, and locked to prevent corruption from concurrent applies. - **Drift** (reality diverging from code, usually via manual console changes) causes confusing, potentially dangerous unexpected plans — the fix is treating the code as the sole source of truth, a discipline GitOps (Part 3) formalizes even further. - **Terraform modules** provide the same reusability and baked-in best-practice benefits as a shared library or a Helm chart. - **Ansible** is agentless, idempotent configuration management for already-existing servers — a genuinely different job from Terraform's provisioning role, though the two are very commonly used together. **Continue to Part 3** (`03-gitops.md`) to see how these IaC principles get taken one step further — using Git itself as the enforced, single source of truth for what's actually running, continuously reconciled by automated tooling.