CI/CD Fundamentals

Table of Contents#

1. Why Automation Is the Backbone of This Entire Course
2. Continuous Integration — What It Actually Means
3. Continuous Delivery vs Continuous Deployment
4. The Anatomy of a Pipeline
5. A Full Worked Pipeline
6. Deployment Strategies — The Big Four
7. Recreate Deployment
8. Rolling Deployment
9. Blue-Green Deployment
10. Canary Deployment
11. Feature Flags — Decoupling Deploy From Release
12. The DORA / Four Keys Metrics
13. Common Mistakes
14. Worked Practice Problems
15. Summary and What's Next

---

Why Automation Is the Backbone of This Entire Course#

Nearly every tutorial in this course has referenced "toil" (SRE Fundamentals), "shift-left" (DevSecOps), or "self-healing" (Kubernetes) — all of it points back to one core idea: humans doing the same repeatable task by hand is slow, inconsistent, and error-prone; automating it is fast, consistent, and safe to repeat. This series is about the two biggest automation disciplines in modern operations: CI/CD (automating how code gets built, tested, and deployed) and GitOps (automating how infrastructure and deployments stay in sync with a source of truth).

---

Continuous Integration — What It Actually Means#

Continuous Integration (CI) means developers merge their code changes into a shared branch frequently (multiple times a day, ideally), with an automated process building and testing every single merge immediately.

Diagram

Simple analogy: CI is like proofreading a document paragraph by paragraph as it's written, rather than writing the entire 300-page book first and only then discovering, all at once, that chapter 3 contradicts chapter 12. This directly reuses the "cost of a bug grows the later it's found" principle from the DevSecOps series — CI is shift-left applied to integration and correctness bugs specifically, not just security ones.

---

Continuous Delivery vs Continuous Deployment#

A genuinely common, frequently-confused pair of terms — worth being precise about the distinction, since it's a classic interview trap.

Diagram

A clean, memorable interview line: "Continuous Delivery means you could deploy to production at any moment, with one click — the pipeline gets you all the way there, but a human still decides when. Continuous Deployment removes even that click — every change that passes the pipeline goes live automatically." Most real organizations practice Continuous Delivery, not full Continuous Deployment — even mature, fast-moving teams often keep at least a lightweight manual approval gate for production releases, especially for anything with real business/compliance stakes.

---

The Anatomy of a Pipeline#

A CI/CD pipeline is a sequence of automated stages, each one gating progress to the next — directly reusing the "fail fast, cheap checks first" principle already established in the DevSecOps series' layered scanning pipeline.

Diagram

Why the stage ORDER matters, worth stating explicitly, and directly reusing a principle from the DevSecOps series: cheap, fast checks run first (unit tests, taking seconds), and expensive, slow checks run later (full end-to-end tests against a real, running environment, taking minutes) — so a broken build fails and reports back within seconds, not after waiting many minutes for a slow test suite to even start.

---

A Full Worked Pipeline#

A realistic GitHub Actions pipeline, tying multiple stages together concretely:

Copy
name: CI/CD Pipeline
on:
  push:
    branches: [main]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: docker build -t myapp:${{ github.sha }} .
      - name: Unit tests
        run: docker run myapp:${{ github.sha }} npm test
      - name: Security scan (DevSecOps series)
        run: trivy image --exit-code 1 --severity CRITICAL myapp:${{ github.sha }}

  deploy-staging:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to staging
        run: kubectl set image deployment/myapp app=myapp:${{ github.sha }} -n staging
      - name: Smoke test
        run: ./scripts/smoke-test.sh https://staging.example.com

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment:
      name: production   # requires manual approval in GitHub —
                          # this IS continuous DELIVERY, not deployment
    steps:
      - name: Deploy to production
        run: kubectl set image deployment/myapp app=myapp:${{ github.sha }} -n production

Notice the environment: production block, worth calling out specifically: this is exactly the "human clicks go" moment that distinguishes Continuous Delivery from full Continuous Deployment — everything up to that point is fully automatic, but this specific job pauses for manual approval before touching production.

---

Deployment Strategies — The Big Four#

Once code is ready to actually go live, how it gets rolled out to real users is its own important decision — directly extending the rolling update mechanics already covered in the Kubernetes Deep Dive series (Part 2), now placed alongside the other major strategies.

Diagram

---

Recreate Deployment#

The simplest, and most disruptive, strategy: stop every old instance, then start every new instance.

Diagram

When this is actually acceptable, worth naming explicitly: internal tools with tolerable downtime, or situations where running two versions of the application simultaneously would genuinely be unsafe (e.g., a breaking database schema change that both versions can't coexist against safely) — otherwise, this strategy is rarely acceptable for any real, user-facing production service.

---

Rolling Deployment#

Already covered mechanically in the Kubernetes Deep Dive series (Part 2) — gradually replace old instances with new ones, a few at a time, with zero full downtime.

Diagram

The real cost worth restating here, at the general CI/CD level, not just the Kubernetes-specific level: during the transition, both versions are simultaneously live and receiving real traffic — this means the application (and its database schema) must be able to handle both versions running side by side without breaking, which is a genuinely important design constraint (directly connecting to the backward-compatible schema migration discussion later in this Part).

---

Blue-Green Deployment#

Maintain two complete, independent, identically-sized production environments ("blue" and "green") — only one is ever receiving real traffic at a time.

Diagram

Diagram

Why this is such a strong strategy specifically for rollback speed, worth stating explicitly: if something goes wrong immediately after the switch, rolling back means simply switching the router back to Blue — near-instant, since Blue never actually stopped running. This is a genuinely different, stronger rollback guarantee than a rolling deployment, where "rolling back" means running the same gradual replacement process again, in reverse, taking real time.

The real cost worth naming, directly connecting to the Capacity Planning & Performance series: blue-green requires running TWO full, complete production-sized environments simultaneously (at least briefly) — meaningfully more infrastructure cost than a rolling deployment, which never needs more than a small percentage of "extra" capacity at any given moment.

---

Canary Deployment#

Route a small percentage of real traffic to the new version first, monitor closely, and gradually increase that percentage as confidence grows — directly reusing the Istio traffic-splitting example from the Kubernetes Deep Dive series (Part 4).

Diagram

Why this is genuinely the strongest strategy for limiting blast radius, worth stating explicitly: if the new version has a real, undetected bug, only a small fraction of real users are ever exposed to it before the automated (or human) monitoring catches the problem and the rollout is halted or reversed — directly connecting to the error-budget-consumption discussion from the SRE Fundamentals series, a canary deliberately spends only a small, controlled sliver of error budget to validate a risky change, rather than exposing the full user base at once.

Automated canary analysis, worth knowing exists as a real, concrete practice: mature organizations don't just manually eyeball dashboards during a canary — tools can automatically compare the canary's RED metrics (error rate, latency) against the stable version's, and automatically halt/rollback the rollout the instant the canary's metrics look meaningfully worse, closing the loop between deployment and observability entirely.

---

Feature Flags — Decoupling Deploy From Release#

A genuinely important, related-but-distinct concept worth its own callout, since it changes the whole mental model of "deploying" vs. "releasing" a feature.

Diagram

Copy
# A simple feature flag check in application code
if feature_flags.is_enabled("new_checkout_flow", user_id=current_user.id):
    return new_checkout_flow()
else:
    return legacy_checkout_flow()

Why this is worth stating explicitly as its own, distinct concept from deployment strategies: a canary deployment controls what percentage of traffic hits new code; a feature flag controls what percentage of traffic sees a new behavior, even within the exact same running code — the two are complementary (you might canary-deploy code that itself contains a feature flag), and together they give extremely fine-grained, low-risk control over exactly how a change reaches real users.

---

The DORA / Four Keys Metrics#

Already briefly previewed in the Monitoring Methodologies series (Part 3) as a distinct framework — this Part is where it gets its full, proper treatment, since it's specifically about measuring the health of the software delivery process this whole tutorial covers.

Diagram

Why these four specific metrics, and not something else, worth understanding: they were identified (through years of large-scale industry research by the DORA — DevOps Research and Assessment — team) as the metrics that most reliably distinguish high-performing engineering organizations from low-performing ones. The genuinely important, counter-intuitive finding worth citing: high performers are fast (frequent deploys, short lead time) AND stable (low change failure rate, fast recovery) — speed and stability are NOT actually a tradeoff against each other at the organizational level, which contradicts a lot of instinctive assumptions about "moving fast breaks things."

A strong, senior-level interview line: "DORA metrics are specifically about delivery performance, not runtime system health — they're the CI/CD counterpart to the Golden Signals/RED/USE metrics from the Monitoring Methodologies series, which measure whether a running system is healthy. Together, they give a complete picture: DORA tells you if your team can ship changes safely and quickly; RED/USE tells you if what you shipped is actually working well in production."

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A team wants the fastest possible rollback if a new release has a critical bug, and has the budget for extra infrastructure. Which deployment strategy would you recommend, and why not a simpler rolling deployment?

Answer: Blue-green deployment — rolling back means switching the router back to the still-running "blue" environment, which is near-instant, since blue never actually stopped serving. A rolling deployment's rollback instead requires running the same gradual replacement process in reverse, which takes real, meaningful time proportional to the fleet size — genuinely slower in a true emergency. The tradeoff is real: blue-green requires running two full-sized production environments simultaneously (at least briefly), which is exactly the extra infrastructure cost/budget the team has already indicated they're willing to accept for this speed guarantee.

Problem 2: A team's Change Failure Rate has been climbing for several months, even as Deployment Frequency has also increased. A leader argues "this is expected — moving faster means more things break." How would you respond, using DORA's own research findings?

Answer: DORA's actual, well-established research finding directly contradicts this assumption — elite-performing organizations are simultaneously fast (frequent deploys, short lead time) AND stable (low change failure rate, fast recovery); speed and stability aren't a real tradeoff at the organizational level. A climbing change failure rate alongside increasing deployment frequency is a genuine warning sign that something in the delivery pipeline needs attention (insufficient automated testing, inadequate canary/rollback safety nets, or genuinely risky changes being rushed out) — not an expected, acceptable cost of moving faster.

Problem 3: A team deploys a risky new feature using a canary strategy, starting at 5% traffic. Ten minutes in, error rates for the canary version spike sharply while the stable version's error rate stays flat. What should happen, and how does this connect to the error budget concept from the SRE Fundamentals series?

Answer: This is exactly the scenario canary deployments and automated canary analysis are designed to catch — the rollout should be automatically (or immediately, manually) halted and rolled back, since the canary's own metrics are clearly showing a real regression isolated to the new version, confirmed by the stable version's flat error rate ruling out a broader, unrelated issue. This directly connects to error budgets: by limiting exposure to only 5% of traffic, the team deliberately spent a small, controlled sliver of their error budget to safely discover this bug, rather than exposing 100% of users (and burning a much larger, more damaging chunk of budget) to find out the same thing the hard way.

---

Summary and What's Next#

• Continuous Integration means merging small changes frequently, with automated build/test on every merge — shift-left applied to integration bugs, exactly the same "find it cheap and early" principle from the DevSecOps series.
• Continuous Delivery means every change is automatically made ready to deploy, with a human still deciding when; Continuous Deployment removes that human gate entirely — most real organizations practice Delivery, not full Deployment.
• A pipeline's stages should run cheapest and fastest first, so failures are caught in seconds, not minutes.
• The four deployment strategies — Recreate (simplest, causes downtime), Rolling (gradual, no downtime, but both versions coexist temporarily), Blue-Green (near-instant rollback, costs double infrastructure), Canary (limits blast radius by exposing only a small % of traffic first) — each represent a different point on the speed/safety/cost tradeoff.
• Feature flags decouple deploying code from releasing a feature to users — a bad feature can be disabled instantly with a flag flip, with no new deploy needed at all.
• The DORA / Four Keys metrics (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Time to Restore Service) measure delivery-process health specifically, complementing (not replacing) the runtime-health metrics from the Monitoring Methodologies series — and the research behind them shows speed and stability genuinely reinforce each other, rather than trading off.

Continue to Part 2 (02-infrastructure-as-code.md) to cover how the infrastructure itself — not just application code — gets automated, versioned, and deployed through code.