Infrastructure as Code

Table of Contents#

1. The Problem IaC Actually Solves
2. Configuration Management vs Provisioning — Two Different Jobs
3. Idempotency — The Single Most Important IaC Property
4. Declarative vs Imperative IaC
5. Terraform — Core Concepts
6. The Terraform Workflow
7. Terraform State — The Most Important, Most Misunderstood Concept
8. State Locking — Preventing a Real, Common Disaster
9. Drift — When Reality Disagrees With the Code
10. Terraform Modules — Reusability
11. Ansible — A Different Tool for a Different Job
12. Terraform vs Ansible — Choosing the Right Tool
13. IaC Scanning, Revisited
14. Common Mistakes
15. Worked Practice Problems
16. Summary and What's Next

---

The Problem IaC Actually Solves#

Before IaC, provisioning infrastructure meant a human manually clicking through a cloud console, or manually SSHing into servers and running commands — every single time, remembered correctly (or not) from memory or a wiki page.

Diagram

A clean, memorable interview line: "Infrastructure as Code takes the exact same discipline we apply to application code — version control, code review, automated testing — and applies it to servers, networks, and databases. It's the direct antidote to the manual, undocumented, tribal-knowledge way infrastructure used to be managed."

---

Configuration Management vs Provisioning — Two Different Jobs#

A genuinely important distinction, and a very common early interview question — the IaC ecosystem actually splits into two related-but-different jobs.

Diagram

Simple analogy: provisioning is building the house (walls, plumbing, electrical) — configuration management is furnishing and setting up everything inside it once the house exists (which room has which furniture, which lights are on). Both are necessary, but they're genuinely different jobs, often (though not always) handled by different tools.

---

Idempotency — The Single Most Important IaC Property#

This is arguably the single most important concept in this entire tutorial — the property that makes IaC actually safe and trustworthy to use repeatedly.

Diagram

A concrete, worked example worth having ready: a script that says run: create-server --name web-1 is NOT idempotent — running it twice creates two servers (or errors confusingly on the second attempt). A properly idempotent tool instead expresses the desired state — "a server named web-1 should exist, with these properties" — and checks first: if it already exists and matches, do nothing; if it doesn't exist, create it; if it exists but doesn't match, update it to match. This is EXACTLY the same reconciliation loop pattern from the Kubernetes Deep Dive series (Part 1) — Terraform, Ansible, and Kubernetes controllers all share this identical underlying philosophy, just applied at different layers.

---

Declarative vs Imperative IaC#

Already introduced conceptually in the Kubernetes Deep Dive series (Part 1) — here's the direct, concrete IaC-specific version of that same distinction.

Diagram

Terraform, CloudFormation, and Kubernetes manifests are all declarative — this is the dominant, modern approach for exactly the same reasons declarative Kubernetes configuration is preferred (Kubernetes Deep Dive series, Part 1): it's idempotent by design, and safely re-runnable regardless of the starting state.

---

Terraform — Core Concepts#

Terraform (by HashiCorp) is the most widely used, cloud-agnostic infrastructure provisioning tool — genuinely worth deep, hands-on familiarity.

Copy
# main.tf — a simple, real Terraform configuration
resource "aws_instance" "web" {
  ami           = "ami-0abcdef1234567890"
  instance_type = "t3.medium"

  tags = {
    Name = "web-server"
  }
}

resource "aws_s3_bucket" "data" {
  bucket = "my-app-data-bucket"
}

Diagram

---

The Terraform Workflow#

Diagram

Copy
# The core Terraform workflow, in order
terraform init      # download providers/modules
terraform plan      # show what WOULD change, without changing anything
terraform apply     # actually make the changes
terraform destroy   # tear down everything Terraform manages

Why terraform plan is such a genuinely important safety step, worth stating explicitly: it's a dry run — showing exactly what would be created, modified, or (critically) destroyed, before anything actually happens, giving a human (or an automated policy gate, directly reusing the OPA/Gatekeeper concept from the DevSecOps series) a chance to catch a mistake before it becomes real, irreversible infrastructure damage.

---

Terraform State — The Most Important, Most Misunderstood Concept#

This deserves the single largest section in this tutorial, because getting it wrong is one of the most common, most damaging real-world Terraform mistakes.

Diagram

Why losing the state file is a genuinely severe problem, worth stating explicitly: without it, Terraform has no memory of what it previously created — running terraform apply again could attempt to create brand-new, duplicate resources instead of recognizing existing ones, or Terraform might believe resources need to be destroyed and recreated that are actually fine as-is. This is exactly why the state file itself needs to be treated with the same seriousness as the etcd backup discussion from the Kubernetes Deep Dive series — it's a small file with an outsized, critical importance.

Diagram

Copy
# Configuring remote state storage — a near-universal
# best practice for any real team using Terraform
terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "prod/network/terraform.tfstate"
    region = "us-east-1"
  }
}

---

State Locking — Preventing a Real, Common Disaster#

A directly important extension of the remote-state discussion, worth its own callout since it prevents a genuinely common, real-world corruption scenario.

Diagram

State locking fixes this exactly the way a database transaction lock prevents two concurrent writers from corrupting the same row: while Developer A's apply is in progress, the state is locked — Developer B's simultaneous apply attempt is blocked (or fails immediately with a clear "state is locked" error) until A's operation completes and releases the lock. Most remote backends (like S3 combined with a DynamoDB table for locking) support this automatically, and it's a genuinely essential, non-optional safety feature for any team with more than one person running Terraform.

---

Drift — When Reality Disagrees With the Code#

Drift happens when the real, actual infrastructure no longer matches what the Terraform state (and configuration) believe it should be — usually because someone made a manual change directly in the cloud console, bypassing Terraform entirely.

Diagram

Copy
# Detect drift explicitly, without applying anything
terraform plan -detailed-exitcode
# Exit code 0 = no changes (no drift)
# Exit code 2 = changes detected (possible drift)

The direct, practical fix worth stating explicitly, and a genuinely important discipline: treat the Terraform configuration as the single source of truth, and NEVER make manual changes directly in the cloud console for anything managed by Terraform — any change, however small, should go through the same plan -> review -> apply workflow. This exact discipline is precisely what GitOps (Part 3 of this series) formalizes and enforces even more strictly.

---

Terraform Modules — Reusability#

Exactly the same motivation as functions in a programming language, or Helm charts in Kubernetes (Kubernetes Deep Dive series, Part 4) — avoid repeating the same configuration over and over.

Copy
# Using a reusable module for a standard "web application" pattern
module "checkout_service" {
  source = "./modules/web-app"

  app_name      = "checkout"
  instance_type = "t3.medium"
  min_instances = 3
  max_instances = 10
}

module "payments_service" {
  source = "./modules/web-app"

  app_name      = "payments"
  instance_type = "t3.large"
  min_instances = 5
  max_instances = 20
}

Why this matters practically, directly reusing the "consistency across teams" principle from the DevSecOps series (Part 5, on CI/CD pipeline security): a well-designed module can bake in an organization's security and reliability best practices (correct network isolation, correct IAM least-privilege scoping, correct tagging for cost tracking) once, centrally — every team consuming the module automatically inherits those best practices, rather than each team reinventing (and potentially getting wrong) the same infrastructure pattern independently.

---

Ansible — A Different Tool for a Different Job#

Ansible is the most widely used configuration management tool — it doesn't provision infrastructure (that's Terraform's job); it configures what's already running on existing servers.

Copy
# A simple Ansible playbook — configuring an already-existing server
- name: Configure web servers
  hosts: webservers
  become: true
  tasks:
    - name: Install nginx
      apt:
        name: nginx
        state: present     # idempotent - does nothing if already installed

    - name: Ensure nginx is running
      service:
        name: nginx
        state: started
        enabled: true

    - name: Deploy nginx config
      template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
      notify: restart nginx

  handlers:
    - name: restart nginx
      service:
        name: nginx
        state: restarted

Copy
# Run a playbook against a defined set of hosts
ansible-playbook -i inventory.ini configure-web.yml

Notice state: present and state: started — this is Ansible's own idempotency in action, worth pointing out explicitly: running this playbook 100 times in a row produces the exact same end result as running it once — nginx installed and running — rather than 100 redundant install attempts. A genuinely important architectural detail worth knowing: Ansible is agentless — it connects over standard SSH and runs its logic remotely, with no permanent agent software needing to be pre-installed on the target servers, which is a real, distinctive difference from some other configuration management tools (like Chef and Puppet, which traditionally do require an installed agent).

---

Terraform vs Ansible — Choosing the Right Tool#

Diagram

A strong, senior-level interview answer, worth stating explicitly: these tools are frequently used TOGETHER, not as competitors — a very common real-world pattern is Terraform provisioning the VMs/network/infrastructure, and then Ansible (sometimes triggered directly by Terraform's own provisioner blocks, though a separate, decoupled pipeline stage is often cleaner) configuring what actually runs on those newly-created servers. In a fully containerized/Kubernetes-native environment, Ansible's role shrinks considerably (since container images already bake in "what software is installed," and Kubernetes itself handles a lot of what Ansible would otherwise do) — but it remains extremely common and valuable for traditional VM-based infrastructure.

---

IaC Scanning, Revisited#

Already covered in depth in the DevSecOps series (Part 5) — worth a brief, explicit callback here, since it's directly relevant to this tutorial's subject matter, not just a security afterthought.

Copy
# From the DevSecOps series — scanning Terraform BEFORE it's ever applied
tfsec .
checkov -d .

Why this belongs in both series, worth stating explicitly: IaC scanning is fundamentally a shift-left security practice (DevSecOps series) applied specifically to the infrastructure-provisioning workflow this tutorial covers — catching a misconfigured, publicly-exposed resource in the terraform plan output, before apply ever runs, is exactly the same "catch it as early and cheaply as possible" principle woven throughout this entire course.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A team's Terraform state file was stored locally on a departed engineer's laptop, which has since been wiped. The infrastructure it managed is still running fine in production. What's the actual risk, and what would you do?

Answer: The real infrastructure is fine right now, but Terraform itself has effectively lost all memory of what it created and how — running terraform apply again with no matching state could attempt to create duplicate resources, or Terraform might not recognize existing resources at all, risking accidental deletion/recreation on the next run. The fix: use terraform import to rebuild a fresh state file by explicitly telling Terraform "this existing real resource corresponds to this specific block in my configuration," resource by resource, reconnecting the code to reality — and immediately move to remote, shared state storage afterward to prevent this exact scenario from ever happening again.

Problem 2: During a routine terraform plan, an engineer notices Terraform wants to "destroy and recreate" a production database resource that nobody intentionally changed. What's the most likely explanation, and what should they do before running apply?

Answer: This is a strong signal of drift — someone likely made a manual change directly in the cloud console (or a different automation process touched the resource) that doesn't match what the Terraform configuration expects, and for certain resource attributes, Terraform's only way to reconcile a mismatch is a destroy-and-recreate rather than an in-place update. Before running apply, they should absolutely NOT blindly accept this — for a database specifically, a destroy-and-recreate could mean real, catastrophic data loss. They should investigate what actually changed (checking cloud provider audit logs, comparing the resource's real current configuration against the Terraform code) and either update the Terraform configuration to match the intentional manual change, or revert the manual change to match the code — resolving the drift deliberately, never by blindly running apply on a plan showing an unexpected destroy.

Problem 3: A team wants to provision 10 new microservices' infrastructure, each needing an identical pattern (a VPC, an autoscaling group, a load balancer, correctly-scoped IAM roles). What Terraform feature would you recommend to avoid copy-pasting the same configuration 10 times, and what's the concrete benefit beyond just less typing?

Answer: A reusable Terraform module encapsulating the standard "microservice infrastructure" pattern, parameterized by the few things that genuinely differ per service (name, instance size, scaling limits). Beyond just reducing repetition, the real benefit is baking correct security and reliability practices (proper network isolation, least-privilege IAM scoping) into the module ONCE — every team consuming it automatically inherits those correct defaults, rather than each of the 10 teams independently writing (and potentially getting subtly wrong) their own version of the same pattern, directly mirroring the consistency benefit of a shared library in application code.

---

Summary and What's Next#

• Infrastructure as Code applies the same version control, review, and audit discipline used for application code to infrastructure itself — directly attacking the manual, undocumented "toil" problem from the SRE Fundamentals series.
• Provisioning (Terraform — creating the infrastructure itself) and configuration management (Ansible — configuring what runs on it) are two related but genuinely different jobs, often used together.
• Idempotency — running the same operation any number of times produces the same end result — is the single most important property making IaC safe to re-run, and it's the exact same underlying philosophy as the Kubernetes reconciliation loop.
• Declarative IaC (describe the desired end state; let the tool compute the diff) is the dominant modern approach, for exactly the same reasons declarative Kubernetes configuration is preferred.
• The Terraform state file is the critical, easy-to-underestimate link between your code and real infrastructure — it must be stored remotely, shared, and locked to prevent corruption from concurrent applies.
• Drift (reality diverging from code, usually via manual console changes) causes confusing, potentially dangerous unexpected plans — the fix is treating the code as the sole source of truth, a discipline GitOps (Part 3) formalizes even further.
• Terraform modules provide the same reusability and baked-in best-practice benefits as a shared library or a Helm chart.
• Ansible is agentless, idempotent configuration management for already-existing servers — a genuinely different job from Terraform's provisioning role, though the two are very commonly used together.

Continue to Part 3 (03-gitops.md) to see how these IaC principles get taken one step further — using Git itself as the enforced, single source of truth for what's actually running, continuously reconciled by automated tooling.