Developers merge small code changes frequently, with an automated process building and testing every single merge immediately — shift-left applied to integration bugs.

Delivery: every change that passes CI is automatically made ready to deploy, but a human still decides when to actually release it. Deployment: that human gate is removed too — every passing change goes live automatically. Most real organizations practice Delivery, not full Deployment.

So a broken build fails and reports back within seconds, not after waiting many minutes for a slow test suite to even start — the same "fail fast, cheap checks first" principle from the DevSecOps series' layered scanning pipeline.

Recreate: stop everything old, then start everything new (causes downtime). Rolling: gradually replace instances, no downtime, but both versions briefly coexist. Blue-Green: two full environments, instant traffic switch and near-instant rollback, but double infrastructure cost. Canary: small % of traffic to the new version first, limiting blast radius.

Rolling back a blue-green switch just means flipping the router back to the still-running old environment — near-instant. Rolling back a rolling deployment means running the same gradual replacement process again in reverse, which takes real time proportional to fleet size.

By exposing only a small % of traffic to a risky change, a team deliberately spends a small, controlled sliver of error budget to validate it — rather than exposing 100% of users and burning a much larger chunk of budget to find the same bug.

A canary controls what % of traffic hits NEW CODE. A feature flag controls what % of traffic sees NEW BEHAVIOR, even within the exact same running code. They're complementary — you can canary-deploy code that itself contains a feature flag.

They decouple deploying code from releasing a feature — a bad feature can be disabled instantly by flipping a flag, with no new deploy (and none of a deploy's associated risk) needed at all.

Deployment Frequency, Lead Time for Changes, Change Failure Rate, Time to Restore Service.

Elite performers are simultaneously fast (frequent deploys, short lead time) AND stable (low change failure rate, fast recovery) — speed and stability aren't actually a tradeoff at the organizational level, contradicting the instinct that "moving fast breaks things."

It replaces manual, undocumented, click-through-the-console infrastructure changes with version-controlled, reviewable, auditable code — directly attacking the "toil" problem from the SRE Fundamentals series, applied to infrastructure.

Provisioning creates the actual infrastructure itself (Terraform, CloudFormation). Configuration management configures what runs ON already-existing infrastructure (Ansible, Chef, Puppet).

Running the same operation any number of times produces the same end result as running it once. It's what makes IaC safe to re-run after a failure or partial success — the exact same underlying philosophy as the Kubernetes reconciliation loop.

It's a dry run showing exactly what will be created, changed, or destroyed BEFORE anything actually happens — giving a human (or an automated policy gate) a chance to catch a mistake before it becomes real, potentially irreversible damage.

It's Terraform's only record connecting your code to the specific real resources it created. Without it, a re-run could create duplicate resources or fail to recognize existing ones — a small file with outsized, critical importance.

Two simultaneous apply operations against the same unlocked shared state can write conflicting updates, corrupting it. Locking blocks a second apply until the first one completes and releases the lock.

Reality (the real infrastructure) no longer matching what the code/state believes it should be — usually caused by someone making a manual change directly in the cloud console, bypassing the IaC tool entirely.

Treat the code as the sole source of truth and never make manual console changes to anything Terraform manages — every change goes through plan/review/apply. (GitOps, Part 3, enforces this even more strictly and automatically.)

They solve genuinely different jobs — Terraform provisions the infrastructure (the VM, the network), Ansible configures what runs on it once it exists. A common pattern: Terraform creates the servers, Ansible configures them.

Agentless — it connects over standard SSH and runs remotely, with no permanent agent software needing to be pre-installed on target servers, a real, distinctive difference from some other configuration management tools.

Git becomes the single, ENFORCED source of truth, with a dedicated tool continuously and automatically reconciling actual state to match it — not just "infrastructure is defined in code," but a tool actively, continuously enforcing that the code IS what's running.

Declarative, versioned and immutable (stored in Git), pulled automatically (agents pull, rather than being pushed to), continuously reconciled.

Push: an external CI pipeline actively pushes changes to the cluster, holding standing production credentials to do so. Pull: an agent running INSIDE the cluster continuously watches Git and applies changes itself — nothing external needs any credentials to modify the cluster.

It eliminates the need for any external system (like a CI pipeline) to hold standing production write credentials — directly reducing the exact class of supply-chain attack surface that enabled the SolarWinds incident, where a compromised build system had a direct path to modify what customers trusted.

The two dominant GitOps tools for Kubernetes, both CNCF projects implementing the same core reconciliation principles — continuously comparing Git's desired state against the live cluster and converging any difference.

prune: if something is removed from Git, it gets deleted from the real cluster too. selfHeal: if someone manually changes something directly in the cluster (drift), it's automatically reverted back to match Git on the next reconciliation pass.

Plain Terraform relies on team DISCIPLINE — everyone consistently choosing to go through code, never the console. GitOps self-healing is AUTOMATICALLY, CONTINUOUSLY enforced by the tool itself, not just relied upon from every individual engineer.

It's just a Git commit/PR updating a value (like an image tag) in the target environment's directory — fully auditable, fully reversible, requiring no special tooling beyond Git and the GitOps agent already watching the repo.

Git is fundamentally the wrong place for raw secrets (deleting one doesn't remove it from history). Resolved via Sealed Secrets (the value is encrypted before committing, decryptable only by the target cluster) or External Secrets Operator (Git stores only a reference/pointer; the actual value is fetched live from a real secrets manager like Vault).

git revert on the bad commit — the GitOps agent sees the reverted commit as the new desired state and automatically reconciles the cluster back to the previous, known-good configuration. It's strong because it reuses the exact same, already-trusted mechanism (Git history) as every other change, with no special rollback tooling needed.

Since Git already is the complete, declarative source of truth for everything that should exist, recovering from a total cluster loss is just pointing a brand-new cluster's agent at the same repo — the same reconciliation loop that handles everyday drift correction does the entire rebuild automatically.

The pipeline definition lives in the same repository as the code, at .github/workflows/, triggered natively by events GitHub already tracks (push, PR, release) — no separate system, no webhook plumbing, and pipeline changes are reviewed in the same PR as the code they build.

A composite action reuses a sequence of steps within a job (no runs-on of its own). A reusable workflow reuses one or more entire jobs, including their own runner and matrix, invoked via uses: at the job level with workflow_call.

The default (true) cancels every other matrix combination the instant any one fails, potentially hiding results from combinations that would have shown a different, independently useful failure — false lets every leg finish so the full compatibility picture is visible.

A cache speeds up future runs (best-effort, may be evicted) — typically dependencies. An artifact passes data between jobs or preserves a build output, with a guaranteed retention window — never rely on cache for anything correctness-critical.

A workflow is Continuous Deployment by default (fully automatic). Adding required reviewers to the target Environment turns it into Continuous Delivery — the pipeline still gets a build all the way to deployable, but a human must approve the final release step.

Anyone can open a pull request; if a workflow using a self-hosted runner runs on pull_request, an external contributor's code can execute directly on the organization's own infrastructure.

Least privilege — a compromised dependency pulled in during the job inherits only what the token can do; escalating specific scopes only in the specific job that needs them limits the blast radius of a supply-chain compromise.

A tag is mutable — the maintainer (or an attacker who compromises them) can move it to point at different code with no review. A commit SHA is immutable by Git's own design, guaranteeing the exact same code runs every time until deliberately bumped.

It replaces a long-lived, stored cloud credential with a short-lived signed token minted fresh per run — a leaked OIDC-derived credential expires in about an hour and cannot be regenerated by an attacker, versus a leaked static key valid until manually rotated.

Using pull_request_target (which grants the base repo's write-capable token) while also checking out and executing the PR's own untrusted head branch code — letting an external contributor's code run with elevated, real credentials.

Secret scanning push protection — it blocks a secret at the cheapest possible point (before it's ever accepted into Git history), avoiding the painful history-rewrite cleanup required once a secret is already committed and pushed.

That a specific artifact sitting in a registry right now was actually produced by the real pipeline, from the real source — closing the gap where someone with registry write access could swap an image after the fact, verified independently at deploy time.

GitLab markets itself as one integrated DevOps platform (planning, SCM, CI/CD, security scanning, registry, all one product) rather than a git host with CI/CD as one feature — shown concretely in how close to free built-in security scanning is versus GitHub's more ecosystem-driven, opt-in model.

GitLab's default is strictly stage-sequential — every job in one stage waits for the entire previous stage to finish. needs: lets a specific job start as soon as its actual dependencies finish, regardless of stage, avoiding wasted wall-clock time.

workflow:rules: decides whether the entire pipeline is created at all for a given event. A job's own rules: then decides whether that specific job runs within a pipeline that already exists — a job never runs if the pipeline itself was never created, regardless of its own rules.

At high merge volume, several other MRs can land between an MR's last passing pipeline and its actual merge, silently invalidating that test run. A merge train tests each MR against a simulated state including every MR ahead of it in the queue, guaranteeing the tested state matches the actual merge state.

It's injected centrally at the group/framework level — an individual project maintainer cannot remove or bypass it even with full admin rights over their own .gitlab-ci.yml, unlike a required status check whose defining workflow file a maintainer could simply delete.

A zero-configuration, auto-generated full pipeline (build/test/scan/deploy) for a conventionally-structured app deploying to Kubernetes. It's most valuable as a fast on-ramp for new projects, progressively overridable — not a universal fit for a bespoke or legacy release process.

Native Jira integration — Smart Commits and automatic deployment tracking surface directly on a linked Jira issue with zero separate integration tooling, because both products are built by the same vendor specifically to interlock, unlike a third-party GitHub/GitLab-Jira integration.

The most specific match replaces, not supplements — a push to main matching an explicit branches: { main: ... } entry runs only that pipeline definition, never default: in addition.

No native matrix-build primitive (parallel steps must be hand-written), and a materially thinner built-in security-scanning story (mostly assembled from third-party Pipes rather than near-one-line built-in templates).

A focused, single-purpose packaged integration (deploy to AWS, post to Slack) — narrower in scope than a GitHub Action or GitLab Component, which can define arbitrary multi-step reusable job logic; Bitbucket has no first-party equivalent of a cross-repo reusable custom pipeline.

Whether the variable is visible only to pipelines running on a protected branch/tag, versus visible to a pipeline on any branch — the same least-privilege secrets-scoping principle as GitHub Environment secrets and GitLab Protected variables, as a single checkbox.

It can build and deploy from any Git repository — Azure Repos, GitHub, GitLab, Bitbucket, or a generic Git server — not only its own repo hosting; "using Azure DevOps" and "using Azure Repos" are two independent decisions.

Classic Pipelines are pure UI/database configuration outside version control, invisible to code review — the exact problem CI/CD-as-code exists to solve. YAML Pipelines are checked into the repo and reviewed in the same PR as the code they build.

template: is opt-in composition — the calling pipeline still fully controls its own structure and can skip or reorder around it. extends: inverts control — the template defines the overall pipeline structure, and the caller can only fill in exposed parameter slots, making it structurally non-bypassable.

An organization-level extension that injects steps into every pipeline automatically, with zero reference required from any individual pipeline file — stronger than extends: because extends: still requires the pipeline author to use it in the first place.

A Service Connection configured with Workload Identity Federation handles the trust relationship through a largely automatic, portal-driven setup, and the pipeline YAML needs no explicit permissions:/id_tokens: declaration — the complexity is absorbed into the Service Connection abstraction.

A first-class, structured deployment-strategy primitive (named phases like preDeploy/deploy/routeTraffic, plus on: failure:/on: success: hooks) directly in the pipeline YAML — on GitHub, GitLab, and Bitbucket, an equivalent rollout has to be hand-authored as ordinary script steps or delegated entirely to the deployment target.

Its self-hosted, fully open, plugin-extensible model still fits organizations with hard on-prem/air-gapped requirements or a large legacy investment — the tradeoff this series has repeated throughout: self-hosted flexibility against the operational burden of running the platform yourself.

The controller schedules jobs, serves the UI, and holds configuration/plugin state; agents are the actual machines (static, Docker, or dynamic Kubernetes Pods) that execute a pipeline's steps — the controller never runs build work itself in a properly hardened setup.

Freestyle jobs are UI/database configuration, invisible to code review — the same CI/CD-as-code problem Classic Pipelines have in Azure DevOps. A Jenkinsfile is checked into the repo, reviewed in the same PR as the code it builds.

Reusable pipeline logic (steps, functions, even whole stage definitions) versioned in its own repo and imported with @Library, avoiding copy-pasted Jenkinsfile logic across many repositories — Jenkins's own answer to GitHub reusable workflows or GitLab CI/CD Components.

A static pool is fixed-size, always-on VMs — the same idle-cost problem this series' Part 14 argues against. Kubernetes pod-template agents are provisioned per-build and torn down after, giving Jenkins the same scale-to-zero property as ARC or GitLab's Kubernetes executor.

Jenkins's own controller-level settings (security realm, credentials stores, plugin configuration) historically lived only in UI-driven, undiffable XML state; JCasC expresses that configuration as version-controlled YAML, extending pipeline-as-code discipline to the controller itself, not just individual jobs.

Enabling CSRF protection (crumb issuer), enforcing matrix-based RBAC instead of the legacy "logged-in users can do anything" default, and running the Groovy sandbox/Script Security plugin to prevent arbitrary unapproved script execution.

Its scale and age are also its liability — thousands of community-maintained plugins of wildly varying maintenance quality, each a potential supply-chain and security surface, unlike a SaaS platform's own tightly curated, vendor-maintained integration set.

Compute credits, consumed at different rates depending on the executor's resource class (a larger machine consumes credits faster than a small one) — a more granular cost model than a flat per-minute rate.

A job is one unit of work running in one executor; a workflow orchestrates multiple jobs together via requires: dependencies, fan-out/fan-in, and approval gates — the same job/pipeline separation this series has covered per-platform throughout.

Orbs — shareable, versioned packages of jobs, commands, and executors, published to a registry — functionally parallel to GitHub reusable workflows/composite actions or GitLab CI/CD Components.

Sharing a named set of secrets/environment variables across multiple projects without duplicating them per-project — access is restricted by security group membership, similar in spirit to GitHub/GitLab's org-level secret scoping.

CircleCI issues a short-lived, cryptographically signed OIDC token per job that a cloud IAM trust policy verifies before granting temporary credentials — the same trust-without-long-lived-secrets pattern already covered for GitHub, GitLab, and Azure DevOps earlier in this series.

Parallelism combined with intelligent test splitting — CircleCI's own timing data from prior runs is used to divide a test suite across parallel containers so each finishes in roughly the same wall-clock time, rather than a naive even-count split.

It's built around a long-running agent process polling for jobs, not a Kubernetes-native, ephemeral-Pod-per-job controller — a genuine architectural difference (covered in Part 14's platform comparison), not simply a feature gap CircleCI is expected to close over time.

A way to reuse previously built Docker image layers across CI runs and to build/run Docker containers from within a job even when the executor itself isn't a Docker-native environment — directly speeding up container-heavy pipelines.

It's a set of Kubernetes-native building-block primitives (CRDs), not a finished platform — there's no built-in UI-driven job definition, no vendor SaaS; a team assembles Tekton into a CI/CD system rather than adopting one, the explicit build-vs-buy tradeoff this chapter frames directly.

A Task is a reusable definition of steps; a TaskRun is one actual execution of a Task. A Pipeline composes multiple Tasks into a DAG; a PipelineRun is one actual execution of that Pipeline — the same definition/execution split at both the single-unit and composed levels.

Strong isolation between steps by design — unlike a Jenkins stage's shared shell environment, each Step is a genuinely separate container, sharing state only through explicit Workspaces, not ambient filesystem/environment leakage between steps.

The explicit mechanism for sharing data (source code, build artifacts, caches) between Tasks in a Pipeline, since Tasks otherwise run in fully isolated containers with no implicit shared state.

Automatic supply-chain attestation — Chains observes completed TaskRuns and generates in-toto/SLSA-format provenance attestations without requiring the Pipeline author to add attestation steps manually, directly supporting the DevSecOps series' SLSA coverage.

Turning incoming webhooks (a GitHub push, a GitLab merge event) into new PipelineRuns automatically — Tekton's core primitives have no built-in event-listening mechanism of their own, so Triggers is the component that closes that gap.

Using Kaniko (or an equivalent rootless builder) inside a Task step, which builds an OCI image entirely in user space without requiring Docker-in-Docker or a privileged container — a meaningfully stronger security posture than mounting the host's Docker socket.

A reusable Step definition that can be shared across multiple Tasks, similar in spirit to a small, focused function — finer-grained reuse than sharing a whole Task, useful when only one step's logic (not an entire Task) needs to be shared.

Both are Kubernetes-native, CRD-based pipeline engines with genuine overlap; Tekton is more CI/CD-purpose-built (SCM triggers, image-build-focused primitives via Chains/Hub), while Argo Workflows leans more general-purpose DAG/data-pipeline orchestration — the choice usually comes down to which ecosystem's other tools (Argo CD, Argo Rollouts) a team is already standardized on.

A single push can touch code belonging to many independently deployable services at once, so "build and test everything on every push" becomes prohibitively slow at scale — the core problem this chapter's tooling (affected-only builds, remote caching) exists to solve.

It only sees which files literally changed, not which services actually depend on those files transitively — a shared library change won't correctly trigger every consumer's build unless the filtering tool understands the real dependency graph, which plain path filtering doesn't.

Nx builds an explicit project graph and computes nx affected from real dependency relationships; Turborepo defines task pipelines (build depends on upstream builds) and layers a remote cache on top — both solve affected-only execution, with different graph-construction philosophies (inferred vs. declared).

Fully hermetic, reproducible builds — Bazel sandboxes every build action so it cannot silently depend on unstated inputs, at real setup-cost expense; Nx/Turborepo optimize an existing JS/TS toolchain rather than replacing the build model itself.

It shares build/test results across every engineer and every CI run, not just within one machine's local cache — the second engineer building unchanged code gets an instant cache hit from the first engineer's (or a prior CI run's) already-computed result.

Caching reuses previously computed results; distributed execution actually spreads the work of building uncached targets across multiple machines in parallel — a genuinely different, complementary lever for when cache misses are still too slow serially.

High merge-request volume against one shared trunk creates a much higher chance two merges individually pass CI but conflict when combined — a merge queue tests each candidate against the true, most-current trunk state before actually merging, avoiding a broken main.

Sparse checkout / partial clone (only fetching the subset of the repo a given job actually needs) and shallow clone (limiting history depth) — both reduce the sheer I/O cost of checking out a repository that has grown very large.

A monorepo can build/test many services together while still deploying each one independently and on its own cadence — build scope (what CI verifies together) and deploy scope (what actually ships together) are deliberately decoupled, not the same boundary.

Path-scoped ownership and access control matter more, since a single repository now spans many teams' code — without careful CODEOWNERS-style scoping, one team's compromised credentials or careless merge can reach code far outside their actual area of ownership.

Argo Rollouts introduces its own Rollout CRD that replaces a standard Kubernetes Deployment outright; Flagger's Canary CRD instead wraps an existing Deployment, leaving it in place and managing traffic shifting around it.

It replaces the standard Kubernetes Deployment object — meaning existing manifests must be migrated to Rollout, a real, one-time adoption cost Flagger's wrapping approach avoids by design.

Defines the metric queries (against Prometheus, Datadog, or another provider) and pass/fail thresholds used to automatically judge whether a canary step should proceed, pause, or trigger an automated rollback — turning progressive delivery from a manual, human-watched process into a genuinely automated one.

The same error-budget concept from this course's SRE Fundamentals series becomes the actual gating metric an AnalysisTemplate checks — a canary step only proceeds if it isn't burning error budget faster than the SLO allows, connecting rollout automation directly to reliability targets rather than an arbitrary fixed threshold.

Running a comparison between two versions without ever promoting either one to be the primary — useful for A/B-style measurement or side-by-side evaluation that's explicitly not meant to end in a full rollout.

Canary gradually shifts a growing percentage of traffic to the new version via setWeight steps, analyzed at each step; blue-green keeps both versions fully deployed and switches all traffic at once via a router/Service change — the same tradeoff (blast-radius control vs. instant full-traffic rollback) already established generally in Part 1.

A fixed threshold checks a metric against one static number; Kayenta-style statistical analysis compares the canary's full metric distribution against the baseline's, catching regressions a single static threshold could miss or false-positive on due to normal noise.

A canary and the stable version run simultaneously against the same database — a migration must remain compatible with both the old and new application code during that overlap window, which is exactly the expand/contract migration pattern this chapter (and Part 3's GitOps chapter) requires.

Flagger calls out to configurable webhooks at defined points in a canary's lifecycle (pre-rollout, rollout, post-rollout, confirm-promotion) — letting a team hook in custom checks (load testing, manual approval gates) without Flagger needing to natively understand every possible verification step.

A short-lived, fully deployed environment spun up automatically for a single PR/branch and torn down when it closes — covered as one of Argo Rollouts' progressive-delivery-adjacent capabilities, letting reviewers test a real running version of a change before merge.

Pod-level autoscaling (the CI platform's own controller — ARC, GitLab's Kubernetes executor — scaling runner Pods against job queue depth) and node-level autoscaling (Cluster Autoscaler or Karpenter, scaling the underlying compute those Pods actually schedule onto) — Pod-level autoscaling alone accomplishes nothing without node capacity to match.

It's the concrete YAML expression of true scale-to-zero — zero standing runner Pods (and therefore zero standing compute cost) exist when no jobs are queued, the property that makes ephemeral, Kubernetes-based runners fundamentally cheaper than a static VM pool sized for peak.

The two properties that make a workload risky on spot — long-running and stateful — are exactly what CI jobs are not: they're short-lived and (with correctly externalized caching) effectively stateless, making an interruption cheap to simply retry rather than dangerous.

The usage volume (illustratively, ~50,000 monthly build-minutes) above which self-hosted infrastructure cost — including real, honestly counted operational burden, not just raw compute pricing — becomes cheaper than staying on hosted-runner minutes.

No state persists between jobs, closing the same "a compromised job can't infect the next one" risk covered in the platform-specific security chapters, while simultaneously having zero idle time between jobs — the same design decision resolves both concerns at once rather than trading one against the other.

GitHub introduced a $0.002/minute platform fee on self-hosted runner minutes for private repositories, effective March 2026 — self-hosted runners no longer cost GitHub literally $0, though they remain meaningfully cheaper than hosted-runner rates.

Cluster Autoscaler scales pre-defined, fixed-shape node groups up and down; Karpenter provisions individual nodes matching a Pod's actual resource requirements directly, with no pre-defined node group needed — giving it materially better bin-packing and node-shape flexibility for bursty, heterogeneous CI workloads.

It's built around a long-running agent process polling for jobs, not the Kubernetes-native, ephemeral-Pod-per-job model that GitHub's ARC, GitLab's Kubernetes executor, and Jenkins' Kubernetes plugin all converge on — a genuine architectural difference, not a maturity gap.

It expands an organization's own audit scope to genuinely include the runner infrastructure itself (node patching cadence, CVE response, physical/cloud security posture) — work previously carried entirely by the hosted platform vendor and outside the org's own compliance boundary.

When genuine, staffed Kubernetes operational capacity isn't actually available — adopting this chapter's full stack without real ongoing expertise to maintain it produces a fragile fleet that costs more in firefighting than it saves in infrastructure spend.