CI/CD Pipeline & Supply Chain Security

Table of Contents#

1. Why the Pipeline Itself Is a Target
2. The SolarWinds Wake-Up Call
3. CI/CD Pipeline Threat Model
4. Hardening the Pipeline Itself
5. Third-Party Actions and Dependency Pinning
6. Infrastructure as Code Scanning
7. IaC Scanning in Practice: tfsec and Checkov
8. What Is a Software Supply Chain, Really
9. SBOM — Software Bill of Materials
10. Generating and Using an SBOM
11. Artifact Signing with Sigstore/Cosign
12. SLSA — Supply Chain Levels for Software Artifacts
13. Provenance — Proving Where an Artifact Came From
14. in-toto — the Attestation Framework Underneath SLSA
15. Real-World Supply Chain Attacks Beyond SolarWinds
16. Cross-Platform Attestation — GitHub, GitLab, Bitbucket, and Azure DevOps
17. Continuous SBOM Monitoring with Dependency-Track
18. Putting It All Together: A Secure Pipeline Blueprint
19. Common Mistakes
20. Worked Practice Problems
21. Summary and What's Next

Why the Pipeline Itself Is a Target#

Every tutorial in this series so far has focused on securing the application — its code, its dependencies, its containers, its secrets. This part asks a different question: what if the attack isn't against your application at all, but against the machinery that builds and deploys it?

Why this is such a high-leverage attack for a real adversary: if you compromise one popular library's build process, or one company's CI/CD pipeline, you don't need to individually breach each of their thousands of downstream customers — the customers will voluntarily, trustingly pull in the poisoned artifact themselves, believing it's legitimate. This is exactly the reasoning behind supply chain attacks, and it's why this has become one of the most actively discussed areas in all of security over the last several years.

The SolarWinds Wake-Up Call#

Worth knowing by name as the canonical real-world example that made supply chain security a mainstream, board-level concern (disclosed in December 2020).

The core lesson worth stating explicitly in an interview: "SolarWinds demonstrated that even a company with mature application security can be catastrophically compromised through their build/CI pipeline specifically — the attackers never needed to find a bug in SolarWinds' actual product code, because they compromised the process that builds and signs the product instead. This is exactly why supply chain security (SBOM, provenance, artifact signing) has become a first-class discipline, not an afterthought."

CI/CD Pipeline Threat Model#

Applying the STRIDE framework from Part 1 specifically to a CI/CD pipeline:

The "Pwn Request" Pattern — A Real, Specific Attack#

A genuinely important, specific pattern worth knowing by name: many CI systems (GitHub Actions being the most commonly cited) have a workflow trigger (pull_request_target) that runs with access to the target repository's secrets, even for a pull request submitted from a completely untrusted fork.

Why this matters and how to prevent it: pull_request_target should only ever be used when the workflow does NOT check out and execute the untrusted fork's own code — if it does need to run the PR's code (e.g., to run tests), use the regular pull_request trigger instead, which deliberately runs with no access to repository secrets for exactly this reason. This is a concrete, specific, real-world gotcha worth citing to demonstrate genuine hands-on pipeline security knowledge.

Hardening the Pipeline Itself#

A practical checklist of CI/CD-specific hardening measures:

# Require signed, GPG-verified commits on a protected branch
# (a GitHub branch protection setting, shown here via the API)
gh api repos/OWNER/REPO/branches/main/protection \
  --method PUT \
  -f required_signatures=true

Third-Party Actions and Dependency Pinning#

Every CI/CD pipeline pulls in third-party building blocks — GitHub Actions from the Marketplace, Jenkins plugins, published container base images. Each one is effectively a dependency, with exactly the same trust and versioning concerns as the SCA discussion from Part 2 — but often overlooked because it "feels like infrastructure, not code."

# BAD: mutable tag — trusts whatever "v4" points to, forever
- uses: actions/checkout@v4

# BETTER: pinned to an exact, immutable commit SHA
- uses: actions/checkout@8f4b7f84864484a7bde019a09fc0e2d5b7c8f5f2

A concrete, real-world incident worth citing: in 2024, a popular GitHub Action (tj-actions/changed-files) was compromised, with a malicious update pushed under an existing version tag, causing CI pipelines across many organizations that used the mutable @vX reference (rather than a pinned commit SHA) to unknowingly execute malicious code and leak secrets. This is the exact "pipeline itself is a supply-chain target" lesson made concrete and current.

Infrastructure as Code Scanning#

Just as SAST scans application code (Part 2), IaC scanning tools scan Terraform, CloudFormation, and Kubernetes manifests for known misconfiguration patterns — before that infrastructure is ever actually provisioned.

Why this is such high-leverage shift-left security: a misconfigured cloud resource (a publicly readable S3 bucket, an overly permissive security group, an unencrypted database) is one of the single most common real-world sources of actual breaches — and IaC scanning catches it before the resource is ever created, rather than discovering it after the fact via a cloud security posture scan of already-live infrastructure.

IaC Scanning in Practice: tfsec and Checkov#

# tfsec — Terraform-specific static analysis
tfsec .

# Checkov — broader coverage (Terraform, CloudFormation,
# Kubernetes, Dockerfile, and more, all in one tool)
checkov -d .

# Run Checkov specifically against Kubernetes manifests
checkov -d ./k8s-manifests --framework kubernetes

# CI gate example
checkov -d . --compact --quiet --check CKV_AWS_20,CKV_AWS_21

A realistic finding from Checkov against a Terraform S3 bucket resource:

# The fix — remove public ACL, explicitly block public access
resource "aws_s3_bucket" "data" {
  bucket = "my-app-data"
}

resource "aws_s3_bucket_public_access_block" "data" {
  bucket                  = aws_s3_bucket.data.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

What Is a Software Supply Chain, Really#

A helpful, complete mental model of everything involved in getting code from a developer's keyboard into a running production system — worth being able to draw from memory.

Every single arrow in this diagram is a potential attack point — this is exactly why supply chain security requires layered controls at every stage, not just one: SCA for dependencies (Part 2), image scanning for artifacts (Part 3), pipeline hardening for the build system (above), and the two remaining pieces covered next — knowing exactly what's inside an artifact (SBOM) and proving where it actually came from (signing and provenance).

SBOM — Software Bill of Materials#

An SBOM is a complete, formal, machine-readable inventory of every single component — every library, every transitive dependency, every version — that went into building a piece of software. Think of it as an ingredients label for software.

Why this matters practically, tied directly back to Part 2's transitive dependency discussion: imagine a brand-new, severe CVE is announced for log4j (the actual real-world 2021 "Log4Shell" scenario). Without an SBOM, answering "which of our hundreds of applications actually use this library, even transitively, and in which specific versions?" could take days or weeks of manual, panicked investigation across every team. With an SBOM already generated and stored for every artifact, that question becomes a simple, instant search.

Generating and Using an SBOM#

# Generate an SBOM for a container image, using Trivy
# (in CycloneDX format, one of the two dominant SBOM standards)
trivy image --format cyclonedx --output sbom.json myapp:1.2.3

# Or in SPDX format (the other dominant standard)
trivy image --format spdx-json --output sbom.spdx.json myapp:1.2.3

# Generate an SBOM directly from source, using Syft
# (another very widely used SBOM generation tool)
syft myapp:1.2.3 -o cyclonedx-json > sbom.json

# Later — search a stored SBOM for a specific vulnerable component
grep -i "log4j" sbom.json

The two dominant SBOM formats worth knowing by name: CycloneDX (originated in the OWASP ecosystem, strong security-tooling focus) and SPDX (originated in the Linux Foundation, strong license-compliance focus, now an ISO standard). Both are widely supported by modern tooling; knowing both names (even without deep format-level detail) is enough for most interview purposes.

Artifact Signing with Sigstore/Cosign#

An SBOM tells you what's inside an artifact. Signing tells you where it actually came from, and that nobody tampered with it after it was built.

# Sign a container image (using keyless signing via Sigstore's
# Fulcio + Rekor — no long-lived private key to manage or leak!)
cosign sign myregistry.io/myapp:1.2.3

# Verify a signature before deploying
cosign verify \
  --certificate-identity="https://github.com/myorg/myapp/.github/workflows/build.yml@refs/heads/main" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  myregistry.io/myapp:1.2.3

Why "keyless signing" is worth understanding specifically: traditional signing requires managing a long-lived private key — which is itself a secret that can leak (exactly the Part 4 problem, applied to signing). Sigstore's keyless model instead issues a very short-lived certificate tied to a verified identity (e.g., "this specific GitHub Actions workflow, triggered from this specific repo"), using an OIDC token as proof of identity, and records the signing event in a public, tamper-evident transparency log (Rekor) — eliminating the long-lived-key leak risk entirely, while still providing strong, verifiable provenance.

SLSA — Supply Chain Levels for Software Artifacts#

SLSA (pronounced "salsa," originated at Google, now a broader open industry framework) defines a tiered maturity model specifically for supply chain integrity — genuinely useful as a structured way to describe "how mature is our build security" in an interview, similar in spirit to the DevSecOps maturity model from Part 1.

A great, concrete interview line: "SLSA gives a shared vocabulary for supply chain maturity, similar to how CVSS gives a shared vocabulary for vulnerability severity. Level 1 is basically 'we have a real build script, not a person manually typing commands.' Level 3-4 means the build platform itself is hardened against tampering, with signed, verifiable provenance for every artifact — that's the SolarWinds-style attack this framework is specifically designed to prevent."

An important update worth knowing precisely, since it's a common source of stale information: the diagram above describes SLSA's original pre-1.0 model, still widely referenced in older articles and even some tooling documentation. SLSA v1.0 (the first stable release) restructured the framework into separate tracks, each with its own independent level scale, rather than one single monolithic 1-4 ladder:

Why this reorganization matters practically: the pre-1.0 model conflated source-repository controls (two-person review, branch protection) with build-platform controls (isolation, provenance) into one single ladder, which made it awkward to describe a project that had excellent build-platform hardening but, say, a source repo with looser review requirements. Splitting into independent tracks lets an organization describe its actual posture precisely — "Build Track Level 3, Source Track in progress" — rather than being forced into one blended number. For interview purposes, knowing that SLSA v1.0 uses tracks (Build being the primary one) rather than one flat 1-4 scale is the current, correct answer — citing the older flat model without that caveat is a signal of stale, un-researched knowledge specifically on this topic.

Provenance — Proving Where an Artifact Came From#

Provenance is a signed, verifiable statement answering: "exactly which source code, built by exactly which pipeline, at exactly what time, produced this specific artifact?"

Why this matters, tying directly back to the SolarWinds lesson: provenance lets a downstream consumer (or an automated policy gate) verify "this artifact genuinely came from the source code and build process I trust" — rather than blindly trusting an artifact simply because it's sitting in the right registry with the right name and tag, which is exactly the trust assumption a build-system compromise like SolarWinds exploited.

in-toto — the Attestation Framework Underneath SLSA#

Everything covered so far (provenance statements, SBOMs, signatures) needs a common, machine-readable format to actually be interoperable — a signature verifier, an admission controller, and a compliance dashboard all need to parse the same structure. in-toto is that format: an open specification for a signed attestation — a structured statement (predicate) about a subject (an artifact, identified by its cryptographic digest), wrapped in a standard envelope and signed.

The key insight worth stating plainly: SLSA provenance is just ONE specific kind of in-toto attestation — predicateType set to a SLSA-defined provenance schema. An SBOM can also be published as an in-toto attestation (predicateType pointing to a CycloneDX or SPDX schema instead), as can a vulnerability scan result, a test report, or a code-review record. This is why GitHub's actions/attest-build-provenance (covered in the Automation, CI/CD & GitOps series' GitHub chapters) and GitLab's/Azure's own attestation mechanisms can all interoperate at a tooling level — they're all producing and consuming the same underlying in-toto envelope shape, just with different predicate content.

# Generate a generic in-toto attestation for an SBOM, not just provenance
cosign attest --predicate sbom.cyclonedx.json --type cyclonedx myregistry.io/myapp:1.2.3

# Verify it later, checking the predicate type specifically
cosign verify-attestation --type cyclonedx myregistry.io/myapp:1.2.3

Why this generality matters for a real organization's tooling strategy: rather than inventing a bespoke format for "how do we attach an SBOM to an artifact" and a separate bespoke format for "how do we attach provenance," standardizing on in-toto means one verification pipeline (cosign verify-attestation, or an admission controller's policy engine) can check any kind of attestation a project chooses to publish — provenance, SBOM, scan results, sign-off records — without needing format-specific parsing logic for each one.

Real-World Supply Chain Attacks Beyond SolarWinds#

SolarWinds (Part 5's opening example) is the canonical build-system compromise. It's worth knowing a few more recent, differently-shaped incidents, since interviewers increasingly expect familiarity with more than one case study — each of the following exploited a genuinely different point in the supply chain diagram from earlier in this chapter.

The xz-utils case deserves particular attention as the most sophisticated of these, and it directly reframes "supply chain security" as something broader than a pure tooling problem. The attacker spent roughly two years building a legitimate-looking open-source contribution history and community trust before being granted co-maintainer access — at which point they modified the project's build scripts themselves (not the readable source code most reviewers would scrutinize) to inject a backdoor only during the actual release-tarball build process, specifically evading detection by source-code review while compromising the built artifact. The single most transferable lesson: every control this chapter covers (SBOM, signing, provenance, SLSA maturity) assumes the build process itself is trustworthy — xz-utils demonstrates that a sufficiently patient, socially-engineered attacker can compromise that assumption at its root, which is exactly why SLSA's higher build-track levels specifically emphasize a hardened, isolated build platform that can't be arbitrarily influenced even by someone with legitimate-looking commit access to the build definition.

The Codecov incident is worth flagging for a much more mundane, far more common root cause: a curl https://some-vendor.com/script.sh | bash pattern in a CI pipeline — fetching and immediately executing a remote script with no pinning, no hash verification, no review — is functionally identical to the unpinned-Action risk covered earlier in this chapter, just applied to a shell script instead of a GitHub Action. Any pipeline step that fetches and executes remote code without pinning to an immutable reference is exposed to this exact attack pattern, regardless of which specific vendor or tool is involved.

Cross-Platform Attestation — GitHub, GitLab, Bitbucket, and Azure DevOps#

The concepts in this chapter — provenance, signing, SBOM attachment — are platform-agnostic, but each major CI/CD platform now ships its own concrete implementation, covered in full mechanical detail in this course's Automation, CI/CD & GitOps series. Worth a summary table here specifically to connect that series' platform-specific mechanics back to this chapter's conceptual framework:

The pattern worth internalizing, tying directly back to the earlier OIDC discussion in Part 4 of this series: every platform's attestation/signing mechanism ultimately reuses the same OIDC identity token that already eliminates long-lived cloud credentials — a signing operation is, structurally, just another thing a short-lived, workflow-scoped identity can be trusted to do, alongside authenticating to a cloud provider. An organization that's already adopted OIDC for deployment authentication (this series' Part 4) is most of the way to also adopting keyless signing — the identity infrastructure is the same; only the downstream verifier (a cloud IAM trust policy vs. Sigstore's Fulcio CA) differs.

Continuous SBOM Monitoring with Dependency-Track#

Generating an SBOM per build (covered earlier) answers "what's in this specific artifact." Dependency-Track (an OWASP project) answers the follow-up question this chapter's log4j/SBOM example implicitly assumed but didn't fully address: how do you continuously monitor hundreds of already-generated SBOMs against newly-published vulnerabilities, without re-scanning anything?

# A CI pipeline step uploading its generated SBOM after every build
curl -X POST "https://dtrack.internal/api/v1/bom" \
  -H "X-Api-Key: $DTRACK_API_KEY" \
  -F "project=checkout-service" \
  -F "projectVersion=1.2.3" \
  -F "bom=@sbom.json"

The distinction worth being precise about, since it's easy to conflate the two: an SBOM generation tool (Syft/Trivy, covered earlier) answers "what's in this artifact, right now, at build time." Dependency-Track answers the continuously-updating question "of everything we've EVER built and stored an SBOM for, what's affected by vulnerability data published just today" — turning the earlier "instant search across all stored SBOMs" scenario from a manual grep exercise into an automated, continuously-running, alerting system that requires zero new scans to answer a brand-new CVE the instant it's published, since the matching happens against already-collected component inventories, not against re-scanned artifacts.

Putting It All Together: A Secure Pipeline Blueprint#

A realistic, complete CI/CD pipeline incorporating every concept from Parts 2, 3, 4, and 5 of this series into one coherent flow:

This single diagram is genuinely one of the strongest artifacts you can have ready for a DevSecOps interview — being able to walk through it stage by stage, naming the specific tool category and threat each stage addresses, demonstrates the full breadth of this entire series in one coherent, memorable picture.

Common Mistakes#

Worked Practice Problems#

Problem 1: Your CI pipeline references a third-party GitHub Action using some-org/some-action@v2. The action's maintainer account is later compromised, and the attacker pushes malicious code under the existing v2 tag. What happens to your pipeline, and how would pinning have prevented it?

Answer: Because @v2 is a mutable reference, every subsequent CI run automatically pulls and executes whatever code the tag currently points to — meaning your pipeline would silently begin executing the attacker's malicious code the very next time it ran, with no change on your end at all, and with access to whatever secrets/permissions that step of the pipeline had. Pinning to an exact commit SHA (@a1b2c3d...) instead guarantees your pipeline always runs the exact code you originally reviewed and approved — a tag can be moved by the maintainer (or an attacker who compromises them), but a specific commit hash cannot be silently changed underneath you.

Problem 2: A new critical CVE is announced in a widely-used logging library. Your organization has 300 microservices. How would having SBOMs for every deployed artifact change your response time and process compared to not having them?

Answer: Without SBOMs, determining which of the 300 services actually use the vulnerable library — including transitively, buried several dependency levels deep — would require each team manually auditing their own dependency tree, likely taking days and prone to human error/omission. With SBOMs already generated and centrally stored for every artifact, this becomes a single, instant search across all 300 SBOMs for the specific vulnerable library and version range, immediately producing a precise, complete list of affected services — turning a days-long, error-prone manual fire drill into a minutes-long, reliable query.

Problem 3: A deployment pipeline currently pulls container images directly from a public registry with no signature verification, trusting the image purely because it has the expected name and tag. What specific attack does this leave the organization open to, and how would artifact signing close the gap?

Answer: This leaves the organization open to a tampered or maliciously substituted image being deployed — an attacker who compromises the registry, or performs a supply-chain-style substitution earlier in the pipeline, could publish a malicious image under the exact expected name/tag, and the deployment pipeline would have no way to distinguish it from the legitimate one. Enforcing signature verification (e.g., via cosign, requiring the image be signed by the organization's actual trusted CI pipeline identity) closes this gap — a deploy-time policy gate would reject any image that isn't signed by the expected, verified source, regardless of whether its name/tag looks correct, directly mirroring the exact trust failure that enabled the SolarWinds attack.

Summary and What's Next#

• Attackers increasingly target the build pipeline itself, not just the application — compromising one build system can poison software trusted by thousands of downstream victims, as demonstrated concretely by the SolarWinds attack.
• CI/CD pipelines need their own threat modeling — watch specifically for the "pwn request" pattern (pull_request_target exposing secrets to untrusted fork code) and always use least-privilege, ephemeral, isolated build credentials/environments.
• Third-party CI actions/plugins are dependencies too — pin them to exact commit SHAs, never mutable version tags, exactly as demonstrated by the real 2024 tj-actions/changed-files compromise.
• IaC scanning (tfsec, Checkov) catches misconfigured infrastructure (public buckets, open security groups) before it's ever provisioned — one of the highest-leverage shift-left security practices, since misconfiguration is a leading real-world breach cause.
• An SBOM (Software Bill of Materials — CycloneDX or SPDX format) is a complete, machine-readable inventory of every component in an artifact, turning "are we affected by this new CVE" from a days-long manual audit into an instant search.
• Artifact signing (cosign/Sigstore, often using modern "keyless" signing tied to a verified CI identity) proves an artifact genuinely came from your trusted build process and wasn't tampered with — directly closing the exact trust gap SolarWinds exploited.
• SLSA provides a shared, tiered maturity model (Level 1 through 4) for describing and improving overall supply chain integrity, similar in spirit to CVSS for vulnerability severity or the DevSecOps maturity model from Part 1.
• A fully secure pipeline layers every concept from this entire series — secret scanning, SAST/SCA, IaC scanning, image scanning, SBOM generation, signing, DAST, and policy gating — at the appropriate stage, cheapest checks first.

Continue to Part 6 (06-compliance-and-cheatsheet.md) for compliance frameworks (SOC 2, ISO 27001, PCI-DSS, GDPR), and a consolidated master cheat sheet tying every tool and concept from this entire DevSecOps series together in one quick-reference document.