# Terraform & Infrastructure as Code — Part 7: Testing Terraform: Static Analysis, the Native Test Framework & Terratest
> **Series:** Terraform & Infrastructure as Code (7 of 9)
> **Part 1:** `01-fundamentals-and-workflow.md` — Fundamentals, HCL & the Plan/Apply Workflow
> **Part 2:** `02-state-management-and-remote-backends.md` — State Management & Remote Backends
> **Part 3:** `03-modules-and-reusable-design.md` — Modules & Reusable Infrastructure Design
> **Part 4:** `04-workspaces-and-environments.md` — Workspaces, Environments & Real-World Repository Structure
> **Part 5:** `05-providers-data-sources-and-provisioners.md` — Providers, Data Sources & Provisioners Deep Dive
> **Part 6:** `06-drift-detection-import-and-refactoring.md` — Drift Detection, Import & Refactoring Existing Infrastructure
> **Part 7:** This file — Testing Terraform: Static Analysis, the Native Test Framework & Terratest
> **Part 8:** `08-cicd-for-terraform.md` — CI/CD for Terraform: Pipelines, Gates & GitOps for Infrastructure
> **Part 9:** `09-governance-cost-and-multi-cloud-at-scale.md` — Terraform at Team Scale: Governance, Cost & Multi-Cloud Patterns
> **Questions:** `questions.md`
Assumes you're comfortable with Part 3's module structure and Part 6's `moved`/`removed`/`import` mechanics
— several of this chapter's testing patterns exist specifically to catch the class of mistake those chapters
showed going wrong in production.
## Table of Contents
1. [A Testing Pyramid for Infrastructure Code](#a-testing-pyramid-for-infrastructure-code)
2. [terraform validate and fmt -check as the Floor](#terraform-validate-and-fmt--check-as-the-floor)
3. [tflint — Provider-Aware Linting](#tflint--provider-aware-linting)
4. [Checkov — Security and Compliance Static Analysis](#checkov--security-and-compliance-static-analysis)
5. [Suppressing a Checkov Finding Correctly](#suppressing-a-checkov-finding-correctly)
6. [Writing a Custom Checkov Policy](#writing-a-custom-checkov-policy)
7. [tflint vs. Checkov — Why Teams Run Both](#tflint-vs-checkov--why-teams-run-both)
8. [The Native terraform test Framework](#the-native-terraform-test-framework)
9. [Plan-Only Tests vs. Apply Tests](#plan-only-tests-vs-apply-tests)
10. [Mocking Providers for Fast, Offline Tests](#mocking-providers-for-fast-offline-tests)
11. [Overriding Specific Resources or Data Sources](#overriding-specific-resources-or-data-sources)
12. [Terratest — Real Integration Testing Against Real Infrastructure](#terratest--real-integration-testing-against-real-infrastructure)
13. [Terratest Patterns: Retry, Cleanup, and Parallelism](#terratest-patterns-retry-cleanup-and-parallelism)
14. [Choosing Between terraform test and Terratest](#choosing-between-terraform-test-and-terratest)
15. [Infracost — Cost as a Testable Property](#infracost--cost-as-a-testable-property)
16. [What This Testing Strategy Can't Tell You](#what-this-testing-strategy-cant-tell-you)
17. [Assembling the Full Test Suite for One Module](#assembling-the-full-test-suite-for-one-module)
18. [Part 7 CI Placement — Fast Checks Block, Slow Checks Gate](#part-7-ci-placement--fast-checks-block-slow-checks-gate)
19. [Worked Scenario: a tflint Rule That Would Have Caught Part 5's Timing Bug](#worked-scenario-a-tflint-rule-that-would-have-caught-part-5s-timing-bug)
20. [Worked Scenario: a Checkov Finding the Team Disagreed With](#worked-scenario-a-checkov-finding-the-team-disagreed-with)
21. [Worked Scenario: a terraform test Catching a Cross-Variable Regression](#worked-scenario-a-terraform-test-catching-a-cross-variable-regression)
22. [Part 7 CLI Cheat Sheet](#part-7-cli-cheat-sheet)
23. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
24. [Worked Practice Problems](#worked-practice-problems)
25. [Summary and What's Next](#summary-and-whats-next)
---
## A Testing Pyramid for Infrastructure Code
**The classic application-testing pyramid (many fast unit tests, fewer integration tests, a handful of true
end-to-end tests) maps onto Terraform surprisingly well — the layers just look different, trading "unit test
a function" for "statically analyze configuration" and "integration test a service" for "actually provision
real, ephemeral infrastructure and assert against it."**
```mermaid
flowchart TD
E2E["Terratest: real infra,
ephemeral, slow, expensive —
the fewest of these"] --> Apply["terraform test (apply mode):
real infra, module-scoped,
fewer than plan-only tests"]
Apply --> PlanOnly["terraform test (plan mode):
no real infra, fast,
more of these"]
PlanOnly --> Static["tflint + Checkov:
no infra at all, seconds,
the MOST of these — run on every save"]
classDef slow fill:#fbeee0,stroke:#b8650f,color:#10161c
classDef fast fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
class E2E,Apply slow
class PlanOnly,Static fast
```
**This chapter's caption**: cost and speed increase moving up the pyramid, exactly like the application-
testing version — the discipline is running many cheap, fast checks constantly and reserving the
expensive, slow, real-infrastructure tests for what genuinely needs them, not inverting that ratio.
| Layer | Tool | Real infra? | Speed | Catches |
|---|---|---|---|---|
| Static analysis | `tflint`, Checkov | No | Seconds | Syntax mistakes, deprecated arguments, security misconfigurations |
| Plan-only unit tests | `terraform test` (`command = plan`) | No | Seconds | Logic errors — wrong conditional, wrong computed value |
| Apply tests | `terraform test` (`command = apply`) | Yes, ephemeral | Minutes | Whether real provider behavior matches expectations |
| Integration tests | Terratest | Yes, ephemeral | Minutes to tens of minutes | End-to-end behavior across multiple resources/modules |
## terraform validate and fmt -check as the Floor
Part 1 already introduced both — worth restating here as the absolute floor every other layer in this
chapter builds on top of, and the fastest possible feedback loop (no provider credentials needed, runs in
well under a second on a typical module):
```bash
terraform fmt -check -recursive # Exits non-zero if anything isn't canonically formatted
terraform validate # Syntax and internal reference validity
```
> [!TIP]
> **Best practice**: wire both into a pre-commit hook (Part 1 referenced `pre-commit-terraform`), not just
> CI — catching a formatting or syntax issue before a commit even exists is strictly cheaper than catching it
> in a PR's CI run.
## tflint — Provider-Aware Linting
**`tflint` catches provider-specific mistakes `validate` can't see — a deprecated argument, an invalid
instance type string, an unused declared variable — informed by the actual provider's schema, not just
generic HCL syntax.**
```hcl
# .tflint.hcl
plugin "aws" {
enabled = true
version = "0.35.0"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}
rule "aws_instance_invalid_type" {
enabled = true
}
rule "terraform_unused_declarations" {
enabled = true
}
```
```bash
tflint --init # Install configured plugins
tflint --recursive # Lint every module in the tree
```
```
checkout-app/main.tf
1:1 warning "t3.micro-typo" is an invalid instance type. (aws_instance_invalid_type)
```
`tflint`'s AWS ruleset specifically knows the real, current set of valid instance types, region names, and
deprecated arguments — catching a typo'd instance type at lint time, in seconds, well before it would
otherwise surface as a provider-level API rejection during `apply`.
> [!NOTE]
> `tflint` does **not** flag security misconfigurations (an open security group, an unencrypted bucket) —
> that's Checkov's job, covered next. Running `tflint` alone and considering static analysis "done" leaves a
> real, common category of finding uncaught.
## Checkov — Security and Compliance Static Analysis
**Checkov scans Terraform configuration against a large library of built-in policies covering security
misconfigurations and compliance gaps — the class of finding `tflint` explicitly doesn't cover.**
```bash
checkov -d . --framework terraform
```
```
Check: CKV_AWS_16: "Ensure that encryption is enabled for RDS instances"
FAILED for resource: aws_db_instance.checkout
File: /modules/database/main.tf:12-24
12 | resource "aws_db_instance" "checkout" {
...
20 | # storage_encrypted not set — defaults to false
```
This is exactly the kind of finding worth catching in seconds during CI, before it reaches Part 4's
production account at all — an unencrypted RDS instance is a real, checkable, well-known misconfiguration
class, and Checkov's built-in policy library (thousands of checks across AWS/Azure/GCP/Kubernetes) covers the
overwhelming majority of "well-known bad practice" without an organization needing to author its own rule for
every common case.
## Suppressing a Checkov Finding Correctly
**Not every flagged finding is actually wrong for the specific context — Checkov supports deliberate,
documented suppression, which is meaningfully different from just ignoring or disabling the check globally.**
```hcl
resource "aws_s3_bucket" "public_assets" {
#checkov:skip=CKV_AWS_20:This bucket intentionally serves public static assets via CloudFront OAC
bucket = "checkout-public-assets"
}
```
The inline `#checkov:skip=:` comment is scoped to exactly this one resource, and — this is
the part that matters — the reason is a permanent, version-controlled record of *why* this specific instance
is an intentional exception, reviewable the same as any other code, rather than a silent global disable that
a future engineer has no way to discover the reasoning behind.
| Suppression mechanism | Scope | When to use |
|---|---|---|
| Inline `#checkov:skip=` comment | One specific resource | A genuine, deliberate, resource-specific exception |
| `.checkov.yml` global skip list | Every resource, org-wide | A check that's fundamentally inapplicable to this organization's context |
| `--soft-fail` CLI flag | The whole CI run | Rolling Checkov out gradually, without yet blocking merges — a temporary adoption stage, not a permanent setting |
> [!WARNING]
> A global skip in `.checkov.yml` with no per-instance reasoning is a much weaker record than an inline
> skip — six months later, nobody can tell from the config alone whether that check was disabled because it's
> genuinely inapplicable everywhere, or because someone got tired of fixing findings under a deadline. Prefer
> the inline, resource-scoped form whenever the exception is genuinely resource-specific, which is the common
> case.
## Writing a Custom Checkov Policy
**Beyond the built-in library, Checkov supports organization-specific custom policies — the mechanism for
enforcing a rule that's real and important to *this* organization but has no general-purpose equivalent in
the built-in set** (Part 4's naming-convention discipline is a strong candidate):
```python
# custom_policies/EnsureNamePrefixConvention.py
from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck
from checkov.common.models.enums import CheckCategories, CheckResult
class EnsureNamePrefixConvention(BaseResourceCheck):
def __init__(self):
super().__init__(
name="Ensure resource names follow the org/env/service prefix convention",
id="CKV_MERIDIAN_1",
categories=[CheckCategories.CONVENTION],
supported_resources=["aws_db_instance", "aws_s3_bucket"],
)
def scan_resource_conf(self, conf):
identifier = conf.get("identifier", conf.get("bucket", [""]))[0]
if identifier and not identifier.startswith("meridian-"):
return CheckResult.FAILED
return CheckResult.PASSED
```
```bash
checkov -d . --external-checks-dir custom_policies/
```
This is a concrete, automatable enforcement of exactly the naming discipline Part 4 established as
"important but only ever a convention a human has to remember" — a custom Checkov policy turns it into
something CI actively verifies on every PR, closing the gap between "documented convention" and "actually
enforced."
## tflint vs. Checkov — Why Teams Run Both
| | `tflint` | Checkov |
|---|---|---|
| Catches | Provider-specific syntax/logic mistakes | Security/compliance misconfigurations |
| Knows about | Real, current provider schemas (valid instance types, deprecated args) | A curated policy library, plus custom org rules |
| Misses | Security posture entirely | Provider-schema-level mistakes (an invalid instance type isn't a security issue, so it's out of scope) |
| Typical CI placement | Every PR, fast, blocking | Every PR, fast, blocking (with `--soft-fail` during initial adoption) |
> [!TIP]
> **Best practice**: run both, every PR, as separate CI steps with separate pass/fail reporting — treating
> either one alone as "the linter" leaves a real, distinct category of problem uncaught. Neither tool is a
> superset of the other.
## The Native terraform test Framework
**Terraform 1.6+ ships `terraform test` natively — HCL-native test files (`.tftest.hcl`), no separate
language or tool required, evaluating `assert` conditions against a real (or, per the next sections, mocked)
plan or apply.**
```
modules/network/
├── main.tf
├── variables.tf
├── outputs.tf
└── tests/
└── network.tftest.hcl
```
```hcl
# tests/network.tftest.hcl
variables {
vpc_cidr = "10.0.0.0/16"
environment = "test"
}
run "vpc_cidr_matches_input" {
command = plan
assert {
condition = aws_vpc.main.cidr_block == var.vpc_cidr
error_message = "VPC CIDR block did not match the input variable."
}
}
run "subnet_count_matches_az_count" {
command = plan
assert {
condition = length(aws_subnet.private) == length(var.availability_zones)
error_message = "Expected one private subnet per availability zone."
}
}
```
```bash
terraform test
```
```
tests/network.tftest.hcl... in progress
run "vpc_cidr_matches_input"... pass
run "subnet_count_matches_az_count"... pass
tests/network.tftest.hcl... tearing down
tests/network.tftest.hcl... pass
```
Each `run` block executes independently, in file order, and can reference outputs and resource attributes
from *earlier* `run` blocks in the same file — enabling multi-step scenarios (create, then verify a
follow-up plan shows no changes) within one test file.
## Plan-Only Tests vs. Apply Tests
**`command = plan` and `command = apply` answer genuinely different questions, and defaulting to `plan` for
anything that doesn't specifically need `apply` is both faster and avoids provisioning real (even if
ephemeral) infrastructure for a logic check that doesn't need it.**
| | `command = plan` | `command = apply` |
|---|---|---|
| Question answered | "Would Terraform compute the right values?" | "Does the value match reality, after real creation?" |
| Speed | Seconds | Minutes (real provider API calls) |
| Needs real credentials? | Only enough to plan (often none, with mocking) | Yes, real create/destroy permissions |
| Catches | Logic errors, wrong conditionals, wrong references | Provider-level surprises `plan` can't predict (a `(known after apply)` attribute's real value) |
```hcl
run "instance_type_correct_for_prod" {
command = plan
variables { environment = "prod" }
assert {
condition = aws_instance.worker.instance_type == "m6i.large"
error_message = "Prod should use m6i.large, per the sizing table in Part 1."
}
}
run "real_ami_resolves_and_boots_correctly" {
command = apply # Genuinely needs a real AMI lookup + real instance creation
assert {
condition = aws_instance.worker.public_ip != null
error_message = "Instance should have a public IP after apply."
}
}
```
> [!TIP]
> **Best practice**: default every `run` block to `command = plan` unless the specific assertion genuinely
> requires an attribute only known after real creation — most logic bugs (a wrong conditional, a
> misconfigured `for_each`, an incorrect computed local) are fully catchable at plan time, at a fraction of
> the cost of a real apply.
## Mocking Providers for Fast, Offline Tests
**`mock_provider` (Terraform 1.7+) replaces a real provider with a fake one generating plausible-but-fake
values — turning even `command = apply` tests fast and fully offline, with no real credentials or
infrastructure involved at all.**
```hcl
mock_provider "aws" {}
run "network_module_creates_expected_subnet_count" {
command = apply # Fast and offline — mock_provider means no real API calls happen
variables {
availability_zones = ["us-east-1a", "us-east-1b", "us-east-1c"]
}
assert {
condition = length(aws_subnet.private) == 3
error_message = "Expected 3 subnets, one per AZ."
}
}
```
With `mock_provider "aws" {}` in effect, every `aws_*` resource in this test run is faked — no real AWS API
call happens, no real credentials are needed, and the test runs in the same seconds-level timeframe as a
plan-only test, while still exercising the `apply`-mode code path (useful for asserting on attributes that
only exist post-apply, using a plausible fake value instead of a real one).
> [!NOTE]
> Mocked computed attributes are randomly generated on each run unless pinned — an assertion depending on a
> *specific* value (not just "is this non-null" or "does this count match") needs the override mechanism
> from the next section to get a deterministic, assertable value.
## Overriding Specific Resources or Data Sources
**`override_resource` and `override_data` pin specific values within a mocked (or even a real) provider
context — the fix for the previous section's "randomly generated, not deterministic" gap, and also useful
for simulating a specific data-source result without depending on real infrastructure existing.**
```hcl
mock_provider "aws" {
override_resource {
target = aws_db_instance.checkout
values = {
endpoint = "checkout-db.mock.us-east-1.rds.amazonaws.com:5432"
}
}
override_data {
target = data.aws_ami.app
values = {
id = "ami-mock12345678"
}
}
}
run "db_connection_string_uses_expected_endpoint" {
command = apply
assert {
condition = output.db_connection_string == "postgres://app:***@checkout-db.mock.us-east-1.rds.amazonaws.com:5432/checkout"
error_message = "Connection string did not use the expected mocked endpoint."
}
}
```
Beyond determinism, overrides are genuinely useful for **speeding up** a test that would otherwise wait on a
real, slow-to-provision resource (an RDS instance can take 10+ minutes to actually create) — overriding it
to a fake-but-plausible endpoint lets a test exercise everything *downstream* of that resource (an output
string built from its endpoint, in this example) without paying the real provisioning time cost.
## Terratest — Real Integration Testing Against Real Infrastructure
**Terratest (a Gruntwork-maintained Go library) genuinely applies a Terraform configuration against real
infrastructure, runs assertions against the real, live result, then destroys everything — the layer above
`terraform test` for anything that needs to verify actual, observed behavior, not just planned/mocked
values.**
```go
package test
import (
"testing"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/stretchr/testify/assert"
)
func TestNetworkModuleCreatesReachableSubnets(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../examples/basic",
Vars: map[string]interface{}{
"vpc_cidr": "10.0.0.0/16",
"availability_zones": []string{"us-east-1a", "us-east-1b"},
},
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
subnetIds := terraform.OutputList(t, terraformOptions, "private_subnet_ids")
assert.Equal(t, 2, len(subnetIds))
}
```
`defer terraform.Destroy(...)` immediately after setting up options — before `InitAndApply` even runs — is
the standard Terratest idiom: Go's `defer` guarantees the destroy call runs when the test function returns,
*including* when it fails partway through, which is what keeps a failed test from leaving real, orphaned
infrastructure behind.
> [!WARNING]
> Terratest genuinely provisions real infrastructure and incurs real cost, even for a short-lived test run —
> run it in a dedicated, isolated test account (per Part 4's account-per-environment discipline), never
> against a shared dev/staging account, and always via CI on a schedule/PR trigger rather than routinely by
> hand, to keep both cost and the risk of an orphaned resource (from a crashed test run that never reached its
> `defer`) contained.
## Terratest Patterns: Retry, Cleanup, and Parallelism
**Beyond the basic apply-assert-destroy shape, three patterns show up in nearly every real Terratest suite.**
```go
import "github.com/gruntwork-io/terratest/modules/retry"
// Retry an assertion that depends on eventual consistency
// (exactly Part 5's remote-exec timing problem, but on the TEST side)
retry.DoWithRetry(t, "Waiting for instance to become reachable", 10, 15*time.Second, func() (string, error) {
statusCode := getHTTPStatus(publicIP)
if statusCode != 200 {
return "", fmt.Errorf("expected 200, got %d", statusCode)
}
return "Reachable", nil
})
```
```go
func TestChecoutAndCatalogInParallel(t *testing.T) {
t.Parallel() // This test can run concurrently with other t.Parallel() tests
// ...
}
```
| Pattern | Solves |
|---|---|
| `defer terraform.Destroy(...)` first | Guarantees cleanup even on test failure |
| `retry.DoWithRetry` | Handles genuine eventual-consistency delays (DNS propagation, an instance's boot time) without a flaky, fixed `sleep` |
| `t.Parallel()` | Runs independent test suites concurrently, reducing total CI wall-clock time for a large suite |
> [!TIP]
> **Best practice**: reach for `retry.DoWithRetry` (a bounded, deliberate retry with a clear timeout) rather
> than a fixed `time.Sleep()` for anything genuinely eventually-consistent — the same underlying lesson as
> Part 5's `remote-exec` timing scenario, now applied to test code that has to tolerate the same real-world
> propagation delays it's testing against.
## Choosing Between terraform test and Terratest
| Question | Points toward |
|---|---|
| Am I testing internal module logic (conditionals, computed locals, `for_each` behavior)? | `terraform test` (plan mode) |
| Do I need a deterministic value from a resource that would otherwise be slow/expensive to create for real? | `terraform test` with `mock_provider`/overrides |
| Am I verifying genuinely observed, real-world behavior (an instance is actually reachable over HTTP, a DNS record actually resolves)? | Terratest |
| Am I testing across multiple modules/a full example composition, closer to how a real caller would use it? | Terratest, or `terraform test` in `apply` mode against a full example |
| Do I want the test suite to live entirely in HCL, with no second language? | `terraform test` |
| Do I need Go's broader ecosystem (HTTP clients, retry libraries, custom assertions) for verification logic? | Terratest |
> [!NOTE]
> These aren't mutually exclusive — most mature module test suites use `terraform test` for the fast,
> frequent, logic-level checks and reserve Terratest for a smaller number of genuine end-to-end scenarios,
> exactly mirroring this chapter's opening pyramid.
## Infracost — Cost as a Testable Property
**Infracost parses a Terraform plan and estimates its monthly cost impact against real cloud pricing — worth
including in this chapter because a cost regression is exactly as testable, and exactly as worth catching in
CI, as a security misconfiguration.**
```bash
infracost breakdown --path .
```
```
Name Monthly Qty Unit Monthly Cost
aws_db_instance.checkout
├─ Database instance 730 hours $146.00
├─ Storage (gp3) 100 GB $11.50
Total $157.50
```
```bash
infracost diff --path . --compare-to infracost-base.json
```
`infracost diff` against a saved baseline is the CI-relevant form — posting a PR comment showing exactly how
much a proposed change would increase (or decrease) monthly spend, using the same `terraform plan` JSON
mechanics Part 2 introduced for programmatic tooling generally.
> [!TIP]
> **Best practice**: post Infracost's diff as a PR comment (Part 8 covers the CI wiring) rather than a
> separate dashboard nobody checks proactively — cost visibility at the exact point a change is being
> reviewed, alongside the security/lint findings from earlier in this chapter, is far more likely to actually
> change a decision than a report discovered after the fact.
## What This Testing Strategy Can't Tell You
**Being honest about the limits of everything in this chapter matters as much as the tools themselves — none
of this replaces the actual plan review Part 1 established, and several real failure classes fall outside
what any of these layers catch.**
| This chapter's tools catch | They do NOT catch |
|---|---|
| Syntax mistakes, deprecated arguments (`tflint`) | Whether the *business logic* behind a change is actually correct |
| Known security misconfiguration patterns (Checkov) | A novel misconfiguration outside the policy library's coverage |
| Module-internal logic regressions (`terraform test`) | Whether a change interacts badly with *another* module/service it doesn't reference directly |
| Real, observed infrastructure behavior for one module (Terratest) | Emergent, system-wide behavior only visible with everything running together in a real environment |
| A cost regression against a static baseline (Infracost) | Usage-driven cost changes (the same infrastructure costing more because traffic grew) |
The provider-depends-on-a-resource trap from Part 5, and the "minor" module version bump that silently
changed a default from Part 3, are both excellent illustrations of this gap — neither is a syntax error,
neither trips a common security-misconfiguration pattern, and neither is guaranteed to be caught by a
module's own isolated `terraform test` suite unless someone specifically thought to write a test for that
exact scenario. This is exactly why Part 1's human plan-review discipline remains load-bearing even with a
complete Part 7 test suite in place — testing narrows what a reviewer needs to check manually; it doesn't
eliminate the need for a human to read the plan.
> [!IMPORTANT]
> Treat a green CI run across every tool in this chapter as "the known, checkable classes of mistake are
> ruled out," never as "this change is definitely correct and safe to merge unread." A thorough automated
> test suite raises the floor; it doesn't replace the plan-review ceiling.
## Assembling the Full Test Suite for One Module
Pulling this whole chapter together, here's what a genuinely complete test suite for the `database` module
(Part 3) looks like in CI, in the order it actually runs — fast, cheap checks first, per the pyramid:
```mermaid
flowchart LR
Fmt["terraform fmt -check"] --> Validate["terraform validate"]
Validate --> Lint["tflint"]
Lint --> Sec["checkov"]
Sec --> UnitTest["terraform test
(plan mode, mocked)"]
UnitTest --> Cost["infracost diff"]
Cost --> AppTest["terraform test
(apply mode, real)"]
AppTest --> Integration["Terratest
(scheduled, not every PR)"]
classDef fast fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
classDef slow fill:#fbeee0,stroke:#b8650f,color:#10161c
class Fmt,Validate,Lint,Sec,UnitTest,Cost fast
class AppTest,Integration slow
```
**This chapter's caption**: everything left of `AppTest` runs on every single PR in seconds; the two
rightmost stages — genuinely provisioning real infrastructure — are reserved for pre-merge gating on the
module's own PRs and a separate scheduled run, not run on every unrelated PR across the whole monorepo.
## Part 7 CI Placement — Fast Checks Block, Slow Checks Gate
**Not every check in this chapter's pyramid should run at the same point in a PR's lifecycle — matching each
check's speed and cost to where it runs is what keeps a well-tested pipeline from also becoming a slow one.**
| Check | Runs on | Blocks the PR? |
|---|---|---|
| `fmt -check`, `validate` | Every push | Yes — instant, no reason not to |
| `tflint`, Checkov | Every push | Yes — seconds, catches real mistakes cheaply |
| `terraform test` (plan/mocked) | Every push | Yes — still fast, still cheap |
| Infracost diff | Every push | No — informational PR comment, human judgment call on cost tradeoffs |
| `terraform test` (real apply mode) | Every push to the module's own PR, or a label-triggered run | Yes, but scoped narrowly to avoid slowing unrelated PRs |
| Terratest | Scheduled (nightly) + on-demand for the module's own PRs | No, for unrelated PRs — informational until a scheduled/deliberate run confirms |
> [!TIP]
> **Best practice**: the fast, cheap layers (top of the pyramid) should run on literally every push, with no
> exceptions — the cost of running them is low enough that any friction to skip them isn't worth the
> inconsistency. The slow, real-infrastructure layers should be scoped to the specific module actually
> changing, not the whole monorepo, and should lean on scheduled runs rather than blocking every unrelated
> PR's merge — Part 8 covers the actual CI wiring that implements this split.
## Worked Scenario: a tflint Rule That Would Have Caught Part 5's Timing Bug
Retrospectively analyzing Part 5's flaky `remote-exec` incident, the platform team added a custom `tflint`
rule flagging any `provisioner "remote-exec"` block anywhere in the codebase as a warning (not a hard
failure, since a small number of genuinely justified uses still exist per Part 5's decision tree) — turning
"is anyone using a provisioner, and does the PR reviewer happen to notice" into an automated, visible signal
on every relevant PR.
```hcl
# .tflint.hcl (custom rule, illustrative — tflint's plugin SDK is the real mechanism)
rule "no_remote_exec_provisioner" {
enabled = true
severity = "warning"
}
```
This doesn't prevent a genuinely justified `remote-exec` from being merged — it makes the decision visible
and deliberate, forcing a PR author to either remove it in favor of one of Part 5's alternatives or leave an
explicit justification, exactly the same "convert an implicit convention into an enforced, visible check"
pattern as this chapter's custom Checkov naming-convention policy.
## Worked Scenario: a Checkov Finding the Team Disagreed With
Checkov flagged `CKV_AWS_79` ("Ensure Instance Metadata Service Version 1 is disabled") against a legacy
`aws_instance` resource still running an older internal tool that hadn't yet been updated to support IMDSv2.
The team's response followed this chapter's suppression guidance exactly: rather than a blanket
`--soft-fail` or a silent global skip, they added an inline `#checkov:skip=CKV_AWS_79:legacy-tool-imdsv1-only,
tracked in JIRA-4821, IMDSv2 migration scheduled Q3` — a permanent, reviewable, time-bounded record of
exactly why this one instance is an exception, distinct from every other instance in the fleet (which do
enforce IMDSv2, and would still fail the check if the exception were accidentally broadened).
> [!NOTE]
> The specific ticket reference inside the skip comment is a small but real practice worth adopting broadly
> — it turns "this is an intentional exception" into "this is an intentional, *tracked, time-bounded*
> exception," which is meaningfully more accountable than a suppression comment with no forcing function to
> ever revisit it.
## Worked Scenario: a terraform test Catching a Cross-Variable Regression
Part 3's cross-variable validation example (`max_size >= min_size`) was, in the platform team's real module,
accompanied by a `terraform test` case asserting exactly that invariant — and it caught a real regression
when a later, unrelated refactor accidentally removed the `validation` block while cleaning up the
variable's `description` text:
```hcl
run "max_size_must_be_at_least_min_size" {
command = plan
variables {
min_size = 5
max_size = 2 # Deliberately invalid — this run should fail to plan
}
expect_failures = [
var.max_size,
]
}
```
`expect_failures` inverts the usual assertion logic — this test *passes* specifically because the plan is
expected to fail validation, and it would have failed loudly (in CI, before merge) the moment the
`validation` block was accidentally dropped, since the plan would have unexpectedly succeeded instead of
failing as this test expects.
## Part 7 CLI Cheat Sheet
| Command | Purpose |
|---|---|
| `terraform fmt -check -recursive` | Fail if anything isn't canonically formatted |
| `tflint --recursive` | Provider-aware linting across the whole tree |
| `checkov -d . --framework terraform` | Security/compliance static analysis |
| `terraform test` | Run every `.tftest.hcl` file under `tests/` |
| `terraform test -filter=tests/network.tftest.hcl` | Run one specific test file |
| `infracost diff --path . --compare-to ` | Cost impact of the current plan vs. a baseline |
| `go test ./test/... -v` | Run a Terratest suite |
## Common Mistakes and Interview Traps
| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Running only `tflint` and considering static analysis "done" | Misses the entire security/compliance category `tflint` doesn't cover | Run both `tflint` and Checkov — neither is a superset of the other |
| Suppressing a Checkov finding via a global `.checkov.yml` skip with no per-instance reasoning | Leaves no record of *why*, and silently exempts every future instance too | Prefer an inline, resource-scoped `#checkov:skip=` comment with a documented reason |
| Defaulting every `terraform test` run block to `command = apply` | Slower, needs real credentials, provisions real (if ephemeral) infrastructure for checks that don't need it | Default to `command = plan`; reserve `apply` for assertions genuinely needing post-apply values |
| Using a fixed `time.Sleep()` in a Terratest suite for eventual consistency | Flaky — too short fails intermittently, too long wastes CI time | Use `retry.DoWithRetry` with a bounded, deliberate retry loop |
| Running Terratest against a shared dev/staging account | Real cost and real risk of an orphaned resource from a crashed test run | Use a dedicated, isolated test account, per Part 4's account-per-environment discipline |
| Treating mocked computed attributes as deterministic without pinning them | Mocked values are randomly generated per run unless overridden | Use `override_resource`/`override_data` for any assertion depending on a specific value |
## Worked Practice Problems
**Problem 1**: A module's `terraform test` suite has ten `run` blocks, all using `command = apply` against
real AWS resources, taking 12 minutes total in CI. What's the likely design problem, and what would fixing it
look like?
*Answer*: Most of those ten checks are very likely verifying logic (conditionals, computed values, `for_each`
behavior) that doesn't actually require a real apply — the design problem is defaulting to the most
expensive test mode rather than the cheapest one that answers the question. The fix is auditing each `run`
block: convert any that only need plan-time values to `command = plan` (near-instant), and reserve
`command = apply` for the small number that genuinely need a real, post-apply attribute — likely reducing 12
minutes to well under a minute for the bulk of the suite, per this chapter's plan-vs-apply guidance.
**Problem 2**: A Terratest suite fails partway through `terraform.InitAndApply`, and the CI job's logs show
several orphaned AWS resources still exist afterward, costing money. What's the most likely code-level cause,
and what's the fix?
*Answer*: The most likely cause is `defer terraform.Destroy(t, terraformOptions)` being placed *after*
`InitAndApply` instead of immediately after `terraformOptions` is constructed — if `InitAndApply` itself
panics or fails before the `defer` statement is ever reached, the deferred destroy call never gets registered
and never runs. The fix is the standard Terratest idiom: register the `defer terraform.Destroy(...)` call
first, immediately after building `terraformOptions`, so Go's `defer` mechanism guarantees it runs on the
way out of the function regardless of where or how the test fails afterward.
**Problem 3**: A team wants to enforce that every S3 bucket in the organization follows the
`meridian--` naming convention from Part 4, but no built-in Checkov check covers this. What's
the correct mechanism, and why is a `tflint` custom rule not the better fit here?
*Answer*: A custom Checkov policy (per this chapter's example) is the correct mechanism — this is
organization-specific convention enforcement, exactly Checkov's custom-policy use case, not a provider-schema
correctness issue (`tflint`'s domain) or a security/compliance issue from Checkov's *built-in* library.
`tflint` is the wrong fit specifically because its ruleset is oriented around provider-schema correctness
(valid instance types, deprecated arguments) — while `tflint` does support custom rules via its plugin SDK
too, Checkov's custom-policy mechanism is the more natural, more commonly-adopted home for an organization-
specific convention check like a naming pattern, and keeps all "our own rules" in one place rather than split
across two tools' custom-extension mechanisms.
**Problem 4**: Every check in a team's CI pipeline passes — `fmt`, `validate`, `tflint`, Checkov, and a full
`terraform test` suite — and the change is merged and applied. Two weeks later, an unrelated service that
consumes this module's output via `terraform_remote_state` breaks. Did the testing strategy fail, and what
should the team take away from this?
*Answer*: The testing strategy didn't fail at what it was designed to check — every layer in this chapter
verifies a module's *own* internal correctness, not its effect on *other* configurations consuming its
outputs elsewhere, which is exactly the gap this chapter's "what this strategy can't tell you" section names
directly. The takeaway isn't "add more tests until this can't happen" (an unbounded, likely-impossible goal)
— it's recognizing that cross-configuration contracts (Part 2's `terraform_remote_state` outputs, Part 3's
module interface) need their own deliberate discipline (treating outputs as append-only, a changelog, a
version bump) precisely because no amount of one module's own test suite can substitute for that.
## Summary and What's Next
A complete Terraform testing strategy layers cheap-and-frequent above expensive-and-rare, mirroring the
classic testing pyramid: `fmt`/`validate` as an instant floor, `tflint` and Checkov catching schema and
security issues in seconds on every PR, `terraform test` (mocked where possible) verifying module logic
without real infrastructure, and Terratest reserved for genuine end-to-end verification against real,
ephemeral resources. Infracost extends the same "catch it in CI, not after merge" discipline to cost. None of
these tools replace the plan-review discipline Part 1 established — they're what makes that review faster and
more trustworthy, catching an entire class of mistake before a human reviewer ever needs to notice it
manually.
Part 8 is where every tool in this chapter actually gets wired together into a real, running pipeline — the
plan/apply gates, the PR-comment integrations, the approval workflows, and the GitOps patterns that turn
"we have all these testing tools" into "every single change is automatically, consistently checked before it
ever reaches production."