# DevSecOps — Part 5: CI/CD Pipeline & Supply Chain Security > **Series:** DevSecOps (5 of 6) > **Part 1:** `01-fundamentals-and-shift-left.md` — Fundamentals & Shift-Left Security > **Part 2:** `02-sast-dast-sca.md` — SAST, DAST, SCA & Dependency Scanning > **Part 3:** `03-container-and-kubernetes-security.md` — Container & Kubernetes Security > **Part 4:** `04-secrets-management-and-iam.md` — Secrets Management & IAM > **Part 5:** This file — CI/CD & Supply Chain Security > **Part 6:** `06-compliance-and-cheatsheet.md` — Compliance Frameworks & Master Cheat Sheet > **Questions:** `questions.md` ## Table of Contents 1. [Why the Pipeline Itself Is a Target](#why-the-pipeline-itself-is-a-target) 2. [The SolarWinds Wake-Up Call](#the-solarwinds-wake-up-call) 3. [CI/CD Pipeline Threat Model](#cicd-pipeline-threat-model) 4. [Hardening the Pipeline Itself](#hardening-the-pipeline-itself) 5. [Third-Party Actions and Dependency Pinning](#third-party-actions-and-dependency-pinning) 6. [Infrastructure as Code Scanning](#infrastructure-as-code-scanning) 7. [IaC Scanning in Practice: tfsec and Checkov](#iac-scanning-in-practice-tfsec-and-checkov) 8. [What Is a Software Supply Chain, Really](#what-is-a-software-supply-chain-really) 9. [SBOM — Software Bill of Materials](#sbom--software-bill-of-materials) 10. [Generating and Using an SBOM](#generating-and-using-an-sbom) 11. [Artifact Signing with Sigstore/Cosign](#artifact-signing-with-sigstorecosign) 12. [SLSA — Supply Chain Levels for Software Artifacts](#slsa--supply-chain-levels-for-software-artifacts) 13. [Provenance — Proving Where an Artifact Came From](#provenance--proving-where-an-artifact-came-from) 14. [in-toto — the Attestation Framework Underneath SLSA](#in-toto--the-attestation-framework-underneath-slsa) 15. [Real-World Supply Chain Attacks Beyond SolarWinds](#real-world-supply-chain-attacks-beyond-solarwinds) 16. [Cross-Platform Attestation — GitHub, GitLab, Bitbucket, and Azure DevOps](#cross-platform-attestation--github-gitlab-bitbucket-and-azure-devops) 17. [Continuous SBOM Monitoring with Dependency-Track](#continuous-sbom-monitoring-with-dependency-track) 18. [Putting It All Together: A Secure Pipeline Blueprint](#putting-it-all-together-a-secure-pipeline-blueprint) 19. [Common Mistakes](#common-mistakes) 20. [Worked Practice Problems](#worked-practice-problems) 21. [Summary and What's Next](#summary-and-whats-next) --- ## Why the Pipeline Itself Is a Target Every tutorial in this series so far has focused on securing the **application** — its code, its dependencies, its containers, its secrets. This part asks a different question: **what if the attack isn't against your application at all, but against the machinery that builds and deploys it?** ```mermaid graph TD A["Attacker's goal: compromise
thousands of downstream
customers of a company"] --> Q{"Attack EACH customer
individually? OR..."} Q -->|"Hard — thousands
of separate targets"| Individual["Attack each one directly"] Q -->|"MUCH more efficient —
ONE target, MASSIVE reach"| Pipeline["Compromise the SOFTWARE
VENDOR's build pipeline —
poison the software EVERY
customer will legitimately
download and trust"] ``` **Why this is such a high-leverage attack for a real adversary:** if you compromise one popular library's build process, or one company's CI/CD pipeline, you don't need to individually breach each of their thousands of downstream customers — the customers will voluntarily, trustingly pull in the poisoned artifact themselves, believing it's legitimate. This is exactly the reasoning behind **supply chain attacks**, and it's why this has become one of the most actively discussed areas in all of security over the last several years. --- ## The SolarWinds Wake-Up Call Worth knowing by name as the canonical real-world example that made supply chain security a mainstream, board-level concern (disclosed in December 2020). ```mermaid flowchart TD A["Attackers compromised
SolarWinds' BUILD SYSTEM
(not any customer directly)"] --> B["Malicious code injected
into a LEGITIMATE software
update, signed with
SolarWinds' own valid
certificate"] B --> C["~18,000 organizations
installed the update,
trusting it completely —
it looked 100% legitimate"] C --> D["Included multiple US
government agencies and
Fortune 500 companies —
one build-system compromise,
massive downstream reach"] ``` **The core lesson worth stating explicitly in an interview:** "SolarWinds demonstrated that even a company with mature application security can be catastrophically compromised through their build/CI pipeline specifically — the attackers never needed to find a bug in SolarWinds' actual product code, because they compromised the *process that builds and signs* the product instead. This is exactly why supply chain security (SBOM, provenance, artifact signing) has become a first-class discipline, not an afterthought." --- ## CI/CD Pipeline Threat Model Applying the STRIDE framework from Part 1 specifically to a CI/CD pipeline: ```mermaid graph TD Pipeline["CI/CD Pipeline"] --> T1["Spoofing: an attacker
impersonates a legitimate
CI job or webhook trigger"] Pipeline --> T2["Tampering: build artifacts
modified BETWEEN build
and deploy"] Pipeline --> T3["Info Disclosure: pipeline
secrets/credentials leaked
via a malicious PR
('pwn request')"] Pipeline --> T4["Elevation of Privilege: a
compromised third-party
CI plugin/action runs with
full pipeline permissions"] ``` ### The "Pwn Request" Pattern — A Real, Specific Attack A genuinely important, specific pattern worth knowing by name: many CI systems (GitHub Actions being the most commonly cited) have a workflow trigger (`pull_request_target`) that runs with access to the **target repository's secrets**, even for a pull request submitted from a completely untrusted fork. ```mermaid sequenceDiagram participant Attacker participant Fork as Attacker's Fork participant PR as Pull Request participant CI as CI Pipeline (misconfigured) participant Secrets as Repo Secrets Attacker->>Fork: Modifies workflow file
to exfiltrate secrets Fork->>PR: Opens a pull request
against the real repo PR->>CI: Triggers CI using
'pull_request_target'
(WRONG trigger for this!) CI->>Secrets: Runs attacker's code
WITH ACCESS to real secrets Secrets-->>Attacker: Secrets exfiltrated,
e.g. printed to a log or
sent to an external server ``` **Why this matters and how to prevent it:** `pull_request_target` should only ever be used when the workflow does NOT check out and execute the untrusted fork's own code — if it does need to run the PR's code (e.g., to run tests), use the regular `pull_request` trigger instead, which deliberately runs with **no access to repository secrets** for exactly this reason. This is a concrete, specific, real-world gotcha worth citing to demonstrate genuine hands-on pipeline security knowledge. --- ## Hardening the Pipeline Itself A practical checklist of CI/CD-specific hardening measures: ```mermaid graph TD Harden[Pipeline Hardening] --> H1["Least-privilege pipeline
credentials (Part 4) —
never a broad, shared
admin token"] Harden --> H2["Isolated, ephemeral build
runners — a fresh, disposable
environment per build, not
a long-lived shared machine"] Harden --> H3["No secrets exposed to
untrusted PR code
(the pwn-request fix above)"] Harden --> H4["Branch protection — require
reviews before merge,
disallow force-pushes to
main/protected branches"] Harden --> H5["Signed commits — verify
WHO actually authored
a change"] ``` ```bash # Require signed, GPG-verified commits on a protected branch # (a GitHub branch protection setting, shown here via the API) gh api repos/OWNER/REPO/branches/main/protection \ --method PUT \ -f required_signatures=true ``` --- ## Third-Party Actions and Dependency Pinning Every CI/CD pipeline pulls in third-party building blocks — GitHub Actions from the Marketplace, Jenkins plugins, published container base images. **Each one is effectively a dependency, with exactly the same trust and versioning concerns as the SCA discussion from Part 2 — but often overlooked because it "feels like infrastructure, not code."** ```mermaid graph TD Bad["action: some-org/some-action@v3
(a MUTABLE tag —
the maintainer could push
a NEW, malicious v3
tomorrow, and your pipeline
would silently use it)"] --> BadRisk["❌ Your CI pipeline now
trusts whatever code
THAT tag points to,
WHENEVER it changes"] Good["action: some-org/some-action@a1b2c3d4
(pinned to an EXACT,
immutable commit SHA)"] --> GoodSafe["✅ Guaranteed to run the
EXACT code you reviewed —
a tag can be moved, a
commit SHA cannot"] ``` ```yaml # BAD: mutable tag — trusts whatever "v4" points to, forever - uses: actions/checkout@v4 # BETTER: pinned to an exact, immutable commit SHA - uses: actions/checkout@8f4b7f84864484a7bde019a09fc0e2d5b7c8f5f2 ``` **A concrete, real-world incident worth citing:** in 2024, a popular GitHub Action (`tj-actions/changed-files`) was compromised, with a malicious update pushed under an existing version tag, causing CI pipelines across many organizations that used the mutable `@vX` reference (rather than a pinned commit SHA) to unknowingly execute malicious code and leak secrets. **This is the exact "pipeline itself is a supply-chain target" lesson made concrete and current.** --- ## Infrastructure as Code Scanning Just as SAST scans application code (Part 2), **IaC scanning tools scan Terraform, CloudFormation, and Kubernetes manifests for known misconfiguration patterns** — before that infrastructure is ever actually provisioned. ```mermaid flowchart LR TF["Terraform code
(never yet applied)"] --> Scanner["IaC Scanner
(tfsec, Checkov)"] Scanner --> Finding["Finding: 'aws_s3_bucket.data
has acl = public-read —
this exposes data publicly'"] ``` **Why this is such high-leverage shift-left security:** a misconfigured cloud resource (a publicly readable S3 bucket, an overly permissive security group, an unencrypted database) is one of the single most common real-world sources of actual breaches — and IaC scanning catches it **before the resource is ever created**, rather than discovering it after the fact via a cloud security posture scan of already-live infrastructure. --- ## IaC Scanning in Practice: tfsec and Checkov ```bash # tfsec — Terraform-specific static analysis tfsec . # Checkov — broader coverage (Terraform, CloudFormation, # Kubernetes, Dockerfile, and more, all in one tool) checkov -d . # Run Checkov specifically against Kubernetes manifests checkov -d ./k8s-manifests --framework kubernetes # CI gate example checkov -d . --compact --quiet --check CKV_AWS_20,CKV_AWS_21 ``` A realistic finding from Checkov against a Terraform S3 bucket resource: ``` Check: CKV_AWS_20: "S3 Bucket has an ACL defined which allows public READ access" FAILED for resource: aws_s3_bucket.data File: /main.tf:12-16 12 | resource "aws_s3_bucket" "data" { 13 | bucket = "my-app-data" 14 | acl = "public-read" 15 | } ``` ```hcl # The fix — remove public ACL, explicitly block public access resource "aws_s3_bucket" "data" { bucket = "my-app-data" } resource "aws_s3_bucket_public_access_block" "data" { bucket = aws_s3_bucket.data.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true } ``` --- ## What Is a Software Supply Chain, Really A helpful, complete mental model of everything involved in getting code from a developer's keyboard into a running production system — worth being able to draw from memory. ```mermaid graph LR Dev["Developer's
code"] --> Deps["Third-party
dependencies
(Part 2)"] Deps --> Build["Build system
(CI/CD)"] Build --> Artifact["Built artifact
(container image,
binary, package)"] Artifact --> Registry["Artifact registry
(Docker Hub, npm,
internal registry)"] Registry --> Deploy["Deployment
pipeline"] Deploy --> Prod["Production
runtime"] ``` **Every single arrow in this diagram is a potential attack point** — this is exactly why supply chain security requires layered controls at *every* stage, not just one: SCA for dependencies (Part 2), image scanning for artifacts (Part 3), pipeline hardening for the build system (above), and the two remaining pieces covered next — **knowing exactly what's inside an artifact (SBOM)** and **proving where it actually came from (signing and provenance)**. --- ## SBOM — Software Bill of Materials An **SBOM** is a complete, formal, machine-readable inventory of every single component — every library, every transitive dependency, every version — that went into building a piece of software. Think of it as an ingredients label for software. ```mermaid graph TD SBOM["SBOM for myapp:1.2.3"] --> C1["express@4.18.0"] SBOM --> C2["lodash@4.17.21"] SBOM --> C3["... (200+ more entries,
including EVERY transitive
dependency)"] ``` **Why this matters practically, tied directly back to Part 2's transitive dependency discussion:** imagine a brand-new, severe CVE is announced for `log4j` (the actual real-world 2021 "Log4Shell" scenario). Without an SBOM, answering **"which of our hundreds of applications actually use this library, even transitively, and in which specific versions?"** could take days or weeks of manual, panicked investigation across every team. **With an SBOM already generated and stored for every artifact, that question becomes a simple, instant search.** ```mermaid flowchart LR NewCVE["A severe new CVE
is announced"] --> Query["Search ALL stored SBOMs:
'which artifacts contain
this exact library/version?'"] Query --> Answer["Instant, precise answer —
not a days-long manual
audit across every team"] ``` --- ## Generating and Using an SBOM ```bash # Generate an SBOM for a container image, using Trivy # (in CycloneDX format, one of the two dominant SBOM standards) trivy image --format cyclonedx --output sbom.json myapp:1.2.3 # Or in SPDX format (the other dominant standard) trivy image --format spdx-json --output sbom.spdx.json myapp:1.2.3 # Generate an SBOM directly from source, using Syft # (another very widely used SBOM generation tool) syft myapp:1.2.3 -o cyclonedx-json > sbom.json # Later — search a stored SBOM for a specific vulnerable component grep -i "log4j" sbom.json ``` **The two dominant SBOM formats worth knowing by name:** **CycloneDX** (originated in the OWASP ecosystem, strong security-tooling focus) and **SPDX** (originated in the Linux Foundation, strong license-compliance focus, now an ISO standard). Both are widely supported by modern tooling; knowing both names (even without deep format-level detail) is enough for most interview purposes. --- ## Artifact Signing with Sigstore/Cosign An SBOM tells you *what's inside* an artifact. **Signing** tells you *where it actually came from*, and that *nobody tampered with it* after it was built. ```mermaid sequenceDiagram participant CI as CI Pipeline participant Cosign as cosign participant Registry as Container Registry participant Deploy as Deployment (verifier) CI->>Cosign: Sign the built image
using cosign Cosign->>Registry: Push the SIGNATURE
alongside the image Note over Deploy: Later, at deploy time... Deploy->>Registry: Pull the image + its signature Deploy->>Cosign: Verify the signature alt Signature valid Cosign-->>Deploy: ✅ Verified — this EXACT
image was built by our
trusted CI pipeline,
untampered Deploy->>Deploy: Proceed with deployment else Signature invalid/missing Cosign-->>Deploy: ❌ REJECTED — refuse
to deploy an unsigned
or tampered image end ``` ```bash # Sign a container image (using keyless signing via Sigstore's # Fulcio + Rekor — no long-lived private key to manage or leak!) cosign sign myregistry.io/myapp:1.2.3 # Verify a signature before deploying cosign verify \ --certificate-identity="https://github.com/myorg/myapp/.github/workflows/build.yml@refs/heads/main" \ --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \ myregistry.io/myapp:1.2.3 ``` **Why "keyless signing" is worth understanding specifically:** traditional signing requires managing a long-lived private key — which is itself a secret that can leak (exactly the Part 4 problem, applied to signing). Sigstore's keyless model instead issues a very **short-lived** certificate tied to a verified identity (e.g., "this specific GitHub Actions workflow, triggered from this specific repo"), using an OIDC token as proof of identity, and records the signing event in a public, tamper-evident transparency log (Rekor) — eliminating the long-lived-key leak risk entirely, while still providing strong, verifiable provenance. --- ## SLSA — Supply Chain Levels for Software Artifacts **SLSA** (pronounced "salsa," originated at Google, now a broader open industry framework) defines a tiered maturity model specifically for supply chain integrity — genuinely useful as a structured way to describe "how mature is our build security" in an interview, similar in spirit to the DevSecOps maturity model from Part 1. ```mermaid graph TD L1["SLSA Level 1:
Build process is
SCRIPTED/automated
(not just a person
manually running commands)"] --> L2 L2["SLSA Level 2:
Build runs on a HOSTED,
shared build service;
provenance is generated
and SIGNED"] --> L3 L3["SLSA Level 3:
Build platform itself is
HARDENED against tampering
(isolated, ephemeral
build environments)"] --> L4 L4["SLSA Level 4:
Two-person review REQUIRED
for all changes; HERMETIC,
fully reproducible builds"] ``` **A great, concrete interview line:** "SLSA gives a shared vocabulary for supply chain maturity, similar to how CVSS gives a shared vocabulary for vulnerability severity. Level 1 is basically 'we have a real build script, not a person manually typing commands.' Level 3-4 means the build platform itself is hardened against tampering, with signed, verifiable provenance for every artifact — that's the SolarWinds-style attack this framework is specifically designed to prevent." **An important update worth knowing precisely, since it's a common source of stale information:** the diagram above describes SLSA's original pre-1.0 model, still widely referenced in older articles and even some tooling documentation. **SLSA v1.0 (the first stable release) restructured the framework into separate tracks**, each with its own independent level scale, rather than one single monolithic 1-4 ladder: ```mermaid graph TD V1["SLSA v1.0"] --> BuildTrack["Build Track
(the primary, most
mature track)"] V1 --> OtherTracks["Additional tracks
(e.g. Source Track)
— newer, less mature,
evolving independently"] BuildTrack --> BL1["Build L1: provenance exists,
but is NOT tamper-resistant"] BuildTrack --> BL2["Build L2: provenance is
signed and tamper-resistant,
generated by a HOSTED
build platform"] BuildTrack --> BL3["Build L3: the build platform
itself is hardened —
isolated between builds,
can't be influenced by the
build definition it's running"] ``` **Why this reorganization matters practically:** the pre-1.0 model conflated source-repository controls (two-person review, branch protection) with build-platform controls (isolation, provenance) into one single ladder, which made it awkward to describe a project that had excellent build-platform hardening but, say, a source repo with looser review requirements. Splitting into independent tracks lets an organization describe its actual posture precisely — "Build Track Level 3, Source Track in progress" — rather than being forced into one blended number. **For interview purposes, knowing that SLSA v1.0 uses tracks (Build being the primary one) rather than one flat 1-4 scale is the current, correct answer** — citing the older flat model without that caveat is a signal of stale, un-researched knowledge specifically on this topic. --- ## Provenance — Proving Where an Artifact Came From **Provenance** is a signed, verifiable statement answering: "exactly which source code, built by exactly which pipeline, at exactly what time, produced this specific artifact?" ```mermaid graph TD Artifact["myapp:1.2.3 container
image"] --> Provenance["Provenance statement
(signed):
- Built from commit abc123
- Built by github.com/myorg/
myapp workflow build.yml
- Built at 2026-06-01T14:00Z
- Build inputs: (full list)"] ``` **Why this matters, tying directly back to the SolarWinds lesson:** provenance lets a downstream consumer (or an automated policy gate) verify "this artifact genuinely came from the source code and build process I trust" — rather than blindly trusting an artifact simply because it's sitting in the right registry with the right name and tag, which is exactly the trust assumption a build-system compromise like SolarWinds exploited. --- ## in-toto — the Attestation Framework Underneath SLSA Everything covered so far (provenance statements, SBOMs, signatures) needs a common, machine-readable *format* to actually be interoperable — a signature verifier, an admission controller, and a compliance dashboard all need to parse the same structure. **in-toto** is that format: an open specification for a signed **attestation** — a structured statement (`predicate`) about a `subject` (an artifact, identified by its cryptographic digest), wrapped in a standard envelope and signed. ```mermaid graph TD Envelope["in-toto ATTESTATION
(a signed envelope)"] --> Subject["subject: WHAT this is
about — an artifact,
identified by its digest
(sha256:abc123...)"] Envelope --> PredType["predicateType: WHAT KIND
of statement this is —
a URI identifying the
schema (e.g. SLSA
provenance, an SBOM,
a vulnerability scan result)"] Envelope --> Predicate["predicate: the actual
STATEMENT content,
shaped according to
predicateType's schema"] Envelope --> Signature["signature: cryptographically
signs the whole envelope"] ``` **The key insight worth stating plainly: SLSA provenance is just ONE specific *kind* of in-toto attestation** — `predicateType` set to a SLSA-defined provenance schema. An SBOM can *also* be published as an in-toto attestation (`predicateType` pointing to a CycloneDX or SPDX schema instead), as can a vulnerability scan result, a test report, or a code-review record. This is why GitHub's `actions/attest-build-provenance` (covered in the Automation, CI/CD & GitOps series' GitHub chapters) and GitLab's/Azure's own attestation mechanisms can all interoperate at a tooling level — they're all producing and consuming the same underlying in-toto envelope shape, just with different predicate content. ```bash # Generate a generic in-toto attestation for an SBOM, not just provenance cosign attest --predicate sbom.cyclonedx.json --type cyclonedx myregistry.io/myapp:1.2.3 # Verify it later, checking the predicate type specifically cosign verify-attestation --type cyclonedx myregistry.io/myapp:1.2.3 ``` **Why this generality matters for a real organization's tooling strategy:** rather than inventing a bespoke format for "how do we attach an SBOM to an artifact" and a separate bespoke format for "how do we attach provenance," standardizing on in-toto means one verification pipeline (`cosign verify-attestation`, or an admission controller's policy engine) can check *any* kind of attestation a project chooses to publish — provenance, SBOM, scan results, sign-off records — without needing format-specific parsing logic for each one. --- ## Real-World Supply Chain Attacks Beyond SolarWinds SolarWinds (Part 5's opening example) is the canonical *build-system compromise*. It's worth knowing a few more recent, differently-shaped incidents, since interviewers increasingly expect familiarity with more than one case study — each of the following exploited a genuinely different point in the supply chain diagram from earlier in this chapter. | Incident | Year | What was actually compromised | Which arrow in the supply-chain diagram | |---|---|---|---| | **xz-utils backdoor** (CVE-2024-3094) | 2024 | A trusted maintainer identity, patiently built up over ~2 years of legitimate contributions, then used to slip a backdoor into a core Linux compression library's build scripts | Developer's code → dependency (a maintainer *becoming* the insider threat) | | **event-stream** | 2018 | A popular npm package's maintainer handed control to an unknown volunteer, who added a targeted, obfuscated payload | Third-party dependency | | **Codecov Bash Uploader** | 2021 | Attackers modified a widely-used CI script (fetched and executed directly via `curl \| bash` in thousands of pipelines) to exfiltrate CI environment secrets | Build system (a fetched, unpinned script, not even a formally versioned dependency) | | **ua-parser-js** | 2021 | A compromised npm maintainer account used to publish malicious versions directly to the registry | Artifact registry | **The xz-utils case deserves particular attention as the most sophisticated of these, and it directly reframes "supply chain security" as something broader than a pure tooling problem.** The attacker spent roughly two years building a legitimate-looking open-source contribution history and community trust before being granted co-maintainer access — at which point they modified the project's *build scripts themselves* (not the readable source code most reviewers would scrutinize) to inject a backdoor only during the actual release-tarball build process, specifically evading detection by source-code review while compromising the built artifact. **The single most transferable lesson:** every control this chapter covers (SBOM, signing, provenance, SLSA maturity) assumes the build *process* itself is trustworthy — xz-utils demonstrates that a sufficiently patient, socially-engineered attacker can compromise that assumption at its root, which is exactly why SLSA's higher build-track levels specifically emphasize a *hardened, isolated build platform* that can't be arbitrarily influenced even by someone with legitimate-looking commit access to the build definition. **The Codecov incident is worth flagging for a much more mundane, far more common root cause:** a `curl https://some-vendor.com/script.sh | bash` pattern in a CI pipeline — fetching and immediately executing a remote script with no pinning, no hash verification, no review — is functionally identical to the unpinned-Action risk covered earlier in this chapter, just applied to a shell script instead of a GitHub Action. **Any pipeline step that fetches and executes remote code without pinning to an immutable reference is exposed to this exact attack pattern, regardless of which specific vendor or tool is involved.** --- ## Cross-Platform Attestation — GitHub, GitLab, Bitbucket, and Azure DevOps The concepts in this chapter — provenance, signing, SBOM attachment — are platform-agnostic, but each major CI/CD platform now ships its own concrete implementation, covered in full mechanical detail in this course's **Automation, CI/CD & GitOps** series. Worth a summary table here specifically to connect that series' platform-specific mechanics back to this chapter's conceptual framework: | Platform | Attestation mechanism | Underlying identity for signing | |---|---|---| | **GitHub Actions** | `actions/attest-build-provenance`, verified via `gh attestation verify` | OIDC — the same token used for cloud auth | | **GitLab CI/CD** | Container image signing via `cosign` + GitLab's own ID tokens for keyless signing | GitLab ID tokens (OIDC) | | **Bitbucket Pipelines** | Assembled from third-party Pipes (no first-party attestation product) + `oidc: true` for the identity token | Bitbucket's OIDC step token | | **Azure DevOps** | No dedicated first-party attestation task equivalent to GitHub's; commonly assembled via `cosign` tasks + Workload Identity Federation | Azure AD federated identity | **The pattern worth internalizing, tying directly back to the earlier OIDC discussion in Part 4 of this series:** every platform's attestation/signing mechanism ultimately reuses the *same* OIDC identity token that already eliminates long-lived cloud credentials — a signing operation is, structurally, just another thing a short-lived, workflow-scoped identity can be trusted to do, alongside authenticating to a cloud provider. An organization that's already adopted OIDC for deployment authentication (this series' Part 4) is most of the way to also adopting keyless signing — the identity infrastructure is the same; only the downstream verifier (a cloud IAM trust policy vs. Sigstore's Fulcio CA) differs. --- ## Continuous SBOM Monitoring with Dependency-Track Generating an SBOM per build (covered earlier) answers "what's in this specific artifact." **Dependency-Track** (an OWASP project) answers the follow-up question this chapter's log4j/SBOM example implicitly assumed but didn't fully address: **how do you continuously monitor *hundreds* of already-generated SBOMs against newly-published vulnerabilities, without re-scanning anything?** ```mermaid flowchart TD CI["Every CI pipeline
uploads its SBOM
after each build"] --> DT["Dependency-Track
(central SBOM store)"] NVD["NVD / OSV / GitHub
Advisory feeds
(continuously updated)"] --> DT DT --> Match["Continuously RE-MATCHES
every stored SBOM against
the latest vulnerability data
— not just at upload time"] Match --> Alert["New CVE published today
→ instantly flags EVERY
already-stored SBOM that's
affected, org-wide"] ``` ```bash # A CI pipeline step uploading its generated SBOM after every build curl -X POST "https://dtrack.internal/api/v1/bom" \ -H "X-Api-Key: $DTRACK_API_KEY" \ -F "project=checkout-service" \ -F "projectVersion=1.2.3" \ -F "bom=@sbom.json" ``` **The distinction worth being precise about, since it's easy to conflate the two:** an SBOM generation tool (Syft/Trivy, covered earlier) answers "what's in this artifact, right now, at build time." Dependency-Track answers the continuously-updating question "of everything we've EVER built and stored an SBOM for, what's affected by vulnerability data published just today" — turning the earlier "instant search across all stored SBOMs" scenario from a manual `grep` exercise into an automated, continuously-running, alerting system that requires zero new scans to answer a brand-new CVE the instant it's published, since the matching happens against already-collected component inventories, not against re-scanned artifacts. --- ## Putting It All Together: A Secure Pipeline Blueprint A realistic, complete CI/CD pipeline incorporating every concept from Parts 2, 3, 4, and 5 of this series into one coherent flow: ```mermaid flowchart TD Commit["1. Commit
(pre-commit secret
scan — Part 4)"] --> PR["2. Pull Request
(no secrets exposed to
untrusted fork code)"] PR --> SAST["3. SAST scan
(Semgrep — Part 2)"] PR --> SCA["4. SCA scan
(Snyk/Dependabot — Part 2)"] PR --> IaCScan["5. IaC scan
(tfsec/Checkov — this Part)"] SAST --> Build["6. Build (isolated,
ephemeral runner,
pinned dependency SHAs)"] SCA --> Build IaCScan --> Build Build --> ImageScan["7. Container image
scan (Trivy — Part 3)"] ImageScan --> SBOMGen["8. Generate SBOM
(Syft/Trivy)"] SBOMGen --> Sign["9. Sign artifact +
generate provenance
(cosign/Sigstore)"] Sign --> DAST["10. DAST scan against
staging (ZAP — Part 2)"] DAST --> PolicyGate["11. Admission policy gate:
verify signature + SBOM
before allowing deploy
(OPA Gatekeeper — Part 3)"] PolicyGate --> Deploy["12. Deploy to production
(least-privilege deploy
credentials — Part 4)"] Deploy --> Runtime["13. Runtime monitoring
(Falco — Part 3)"] ``` **This single diagram is genuinely one of the strongest artifacts you can have ready for a DevSecOps interview** — being able to walk through it stage by stage, naming the specific tool category and threat each stage addresses, demonstrates the full breadth of this entire series in one coherent, memorable picture. --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Referencing third-party CI actions/plugins by a mutable version tag | The maintainer (or an attacker who compromises their account) could push malicious code under that same tag later, silently | Pin to an exact, immutable commit SHA | | Using `pull_request_target` when checking out untrusted fork code | Exposes repository secrets to arbitrary code from an untrusted contributor ("pwn request") | Use `pull_request` (no secret access) unless you specifically know why `pull_request_target` is needed and have excluded untrusted code execution | | Treating IaC as "just config," exempt from the same scrutiny as application code | Misconfigured infrastructure (public buckets, open security groups) is one of the most common real breach causes | Scan IaC (tfsec, Checkov) exactly like application code, in the same pipeline | | No SBOM generated for artifacts | Answering "are we affected by this new CVE" becomes a days-long manual audit instead of an instant search | Generate and store an SBOM for every built artifact | | Deploying artifacts with no signature verification | A tampered or unauthorized artifact could be deployed with no way to detect it | Sign artifacts (cosign/Sigstore) and enforce signature verification at deploy time via a policy gate | | Treating supply chain security as "someone else's problem" (the vendor's) | Your own build pipeline is just as much a target as any vendor's — see SolarWinds | Apply the same hardening (isolated runners, least privilege, signed provenance) to your own pipeline | --- ## Worked Practice Problems **Problem 1:** Your CI pipeline references a third-party GitHub Action using `some-org/some-action@v2`. The action's maintainer account is later compromised, and the attacker pushes malicious code under the existing `v2` tag. What happens to your pipeline, and how would pinning have prevented it? *Answer:* Because `@v2` is a mutable reference, every subsequent CI run automatically pulls and executes whatever code the tag currently points to — meaning your pipeline would silently begin executing the attacker's malicious code the very next time it ran, with no change on your end at all, and with access to whatever secrets/permissions that step of the pipeline had. Pinning to an exact commit SHA (`@a1b2c3d...`) instead guarantees your pipeline always runs the exact code you originally reviewed and approved — a tag can be moved by the maintainer (or an attacker who compromises them), but a specific commit hash cannot be silently changed underneath you. **Problem 2:** A new critical CVE is announced in a widely-used logging library. Your organization has 300 microservices. How would having SBOMs for every deployed artifact change your response time and process compared to not having them? *Answer:* Without SBOMs, determining which of the 300 services actually use the vulnerable library — including transitively, buried several dependency levels deep — would require each team manually auditing their own dependency tree, likely taking days and prone to human error/omission. With SBOMs already generated and centrally stored for every artifact, this becomes a single, instant search across all 300 SBOMs for the specific vulnerable library and version range, immediately producing a precise, complete list of affected services — turning a days-long, error-prone manual fire drill into a minutes-long, reliable query. **Problem 3:** A deployment pipeline currently pulls container images directly from a public registry with no signature verification, trusting the image purely because it has the expected name and tag. What specific attack does this leave the organization open to, and how would artifact signing close the gap? *Answer:* This leaves the organization open to a tampered or maliciously substituted image being deployed — an attacker who compromises the registry, or performs a supply-chain-style substitution earlier in the pipeline, could publish a malicious image under the exact expected name/tag, and the deployment pipeline would have no way to distinguish it from the legitimate one. Enforcing signature verification (e.g., via cosign, requiring the image be signed by the organization's actual trusted CI pipeline identity) closes this gap — a deploy-time policy gate would reject any image that isn't signed by the expected, verified source, regardless of whether its name/tag looks correct, directly mirroring the exact trust failure that enabled the SolarWinds attack. --- ## Summary and What's Next - Attackers increasingly target the **build pipeline itself**, not just the application — compromising one build system can poison software trusted by thousands of downstream victims, as demonstrated concretely by the **SolarWinds** attack. - CI/CD pipelines need their own threat modeling — watch specifically for the **"pwn request"** pattern (`pull_request_target` exposing secrets to untrusted fork code) and always use **least-privilege, ephemeral, isolated** build credentials/environments. - Third-party CI actions/plugins are dependencies too — **pin them to exact commit SHAs**, never mutable version tags, exactly as demonstrated by the real 2024 `tj-actions/changed-files` compromise. - **IaC scanning** (tfsec, Checkov) catches misconfigured infrastructure (public buckets, open security groups) **before** it's ever provisioned — one of the highest-leverage shift-left security practices, since misconfiguration is a leading real-world breach cause. - An **SBOM** (Software Bill of Materials — CycloneDX or SPDX format) is a complete, machine-readable inventory of every component in an artifact, turning "are we affected by this new CVE" from a days-long manual audit into an instant search. - **Artifact signing** (cosign/Sigstore, often using modern "keyless" signing tied to a verified CI identity) proves an artifact genuinely came from your trusted build process and wasn't tampered with — directly closing the exact trust gap SolarWinds exploited. - **SLSA** provides a shared, tiered maturity model (Level 1 through 4) for describing and improving overall supply chain integrity, similar in spirit to CVSS for vulnerability severity or the DevSecOps maturity model from Part 1. - A fully secure pipeline **layers every concept from this entire series** — secret scanning, SAST/SCA, IaC scanning, image scanning, SBOM generation, signing, DAST, and policy gating — at the appropriate stage, cheapest checks first. **Continue to Part 6** (`06-compliance-and-cheatsheet.md`) for compliance frameworks (SOC 2, ISO 27001, PCI-DSS, GDPR), and a consolidated master cheat sheet tying every tool and concept from this entire DevSecOps series together in one quick-reference document.