Progressive Delivery with Argo Rollouts & Flagger

Table of Contents#

1. Where Progressive Delivery Fits — From Concept to Automated Mechanism
2. Argo Rollouts vs. Flagger — the Core Architectural Difference
3. Argo Rollouts — the Rollout CRD
4. Canary Strategy in Argo Rollouts — Steps and setWeight
5. A Minimal Argo Rollouts Canary, Built Up Step by Step
6. Analysis — AnalysisTemplate and Metric Providers
7. Automated Rollback — When Analysis Fails
8. Blue-Green Strategy in Argo Rollouts
9. Traffic Management — Service Mesh and Ingress Integration
10. The Experiment CRD — Comparison Without Promotion
11. SLO-Gated Rollouts — Tying Progressive Delivery to Error Budgets
12. Ephemeral Preview Environments per Pull Request
13. Progressive Delivery for Non-HTTP Workloads
14. Flagger — the Canary CRD Wrapping a Deployment
15. Flagger's Webhook System
16. Flagger vs. Argo Rollouts — a Direct Comparison
17. Canary Weight Curves — Choosing How Aggressively to Ramp
18. Multi-Cluster and Multi-Region Progressive Delivery
19. Progressive Delivery + GitOps — Argo Rollouts and Argo CD Together
20. Progressive Delivery + CI/CD Pipelines
21. A/B Testing — the Third Progressive Delivery Pattern
22. Fixed Thresholds vs. Statistical Analysis (Kayenta)
23. A Full Realistic Example: Canary with Automated Analysis and Rollback
24. Database Migrations and Progressive Delivery — a Genuine Complication
25. Common Mistakes
26. Worked Practice Problems
27. Summary and What's Next

---

Where Progressive Delivery Fits — From Concept to Automated Mechanism#

Part 1 of this series introduced canary and blue-green deployments as concepts — a canary exposes a new version to a small percentage of traffic first, limiting blast radius; blue-green switches traffic instantly between two complete environments. What Part 1 didn't cover, because no platform context existed yet, is how a canary rollout's promotion decision actually gets made — who or what decides "the canary looks healthy, increase its traffic" versus "the canary is failing, roll back immediately"?

Diagram

Progressive delivery, as a term, names this automation specifically: a deployment process where promotion between stages (10% → 50% → 100% traffic, or blue → green) is driven by automated analysis of real production metrics (error rate, latency, custom business metrics) rather than a fixed time delay or a human's manual judgment call. Both tools this chapter covers — Argo Rollouts (part of the broader Argo project already introduced via Argo CD in Part 3) and Flagger (a CNCF project, commonly paired with Flux, GitOps tooling structurally similar to Argo CD) — are Kubernetes controllers that implement exactly this: watch a canary's real metrics during rollout, and automatically promote or roll back based on whether those metrics stay within defined, acceptable bounds.

Both tools are, deliberately, Kubernetes-specific — worth stating explicitly as a scope boundary for this chapter, since it's a genuine limitation rather than an oversight. Everything covered here assumes workloads running as Kubernetes Pods, with a Kubernetes-native traffic-management layer available to enforce weighted routing. A team running workloads on traditional VMs, or a serverless platform with no direct Kubernetes substrate, needs a different mechanism entirely for the same underlying progressive-delivery goal — commonly a cloud provider's own native traffic-shifting feature (weighted load balancer target groups, serverless traffic-splitting between function versions), conceptually similar in spirit but mechanically unrelated to anything this chapter covers in detail.

Read this chapter, more than any prior chapter in this series, as depending heavily on this course's Kubernetes deep-dive and Monitoring Methodologies series for its prerequisites — the mechanics covered here assume real, working familiarity with Kubernetes CRDs and controllers, and with querying real metrics from a system like Prometheus, rather than introducing either from scratch.

The upfront investment this chapter's tooling requires — a working service mesh, an already-instrumented metrics pipeline, real SLOs to gate against — is itself worth naming as a precondition, not an afterthought: an organization without these already in place gets comparatively little immediate value from adopting Argo Rollouts or Flagger specifically, and is better served investing in that underlying observability and traffic-management foundation first, per this course's own earlier series, before layering progressive delivery on top of it.

---

Argo Rollouts vs. Flagger — the Core Architectural Difference#

Before any syntax, the single most important structural fact distinguishing the two tools, worth understanding precisely since it shapes everything else in this chapter:

Diagram

This distinction is worth internalizing before anything else, because it determines the actual adoption cost and blast radius of each tool. Argo Rollouts requires converting existing Deployment manifests to the Rollout kind — a real, if usually mechanical, migration across every workload that wants progressive delivery, and any tooling that specifically expects a Deployment object (some older dashboards, some third-party integrations) needs to be checked for Rollout compatibility. Flagger's wrapping approach means the underlying workload remains a completely standard Deployment, visible and manageable by any tool that already understands Kubernetes Deployments — Flagger's Canary resource sits alongside it, managing traffic shifting and promotion without altering the workload resource's own kind at all. Neither approach is unconditionally better — Argo Rollouts' CRD-replacement approach enables deeper, more integrated control over the rollout process itself (visible directly in kubectl get rollouts, with its own rich status), while Flagger's wrapping approach has a lower migration barrier and plays more transparently with existing Deployment-based tooling.

Both tools are genuinely mature, CNCF-adjacent, and production-proven at real scale — neither is an experimental or fringe choice, and this architectural difference is the primary axis worth reasoning about when choosing between them, not a maturity gap.

Both projects also maintain active, real integration work with each other's broader ecosystems — Flagger has documented Argo CD compatibility, and community tooling exists bridging Argo Rollouts with Flux — meaning the GitOps-pairing preference described later in this chapter is a genuine, real consideration worth weighing, not an absolute, hard technical restriction locking a team into only one specific combination.

---

Argo Rollouts — the Rollout CRD#

A Rollout looks almost identical to a standard Kubernetes Deployment, with the strategy field replaced by Argo Rollouts' own richer strategy definitions:

Copy
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-service
spec:
  replicas: 10
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: checkout
          image: myregistry.io/checkout:1.2.3
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

Everything above strategy: is deliberately, almost identically familiar from a standard Kubernetes Deployment spec — replicas, selector, template (the Pod spec) are unchanged. The entire value-add is concentrated in strategy:, where Argo Rollouts' canary/blue-green logic replaces the Deployment's simpler, all-or-nothing (or basic rolling-update) behavior with an explicit, stepped, controllable rollout process.

Migrating an existing Deployment to a Rollout is, in the common case, close to mechanical — copy the spec, change kind: Deployment to kind: Rollout, and add a strategy: block; the Argo Rollouts controller then manages the resulting object going forward, with the Argo Rollouts CLI and Dashboard (a separately-installable web UI, distinct from Argo CD's own UI though commonly run alongside it) providing visibility into rollout status that plain kubectl get deployments never had a concept of in the first place — richer, purpose-built observability into exactly which step a rollout is on, its current traffic weight, and its analysis history, is a genuine, tangible benefit of the migration beyond the canary mechanics themselves.

Other Kubernetes objects that reference a workload by label selector (a HorizontalPodAutoscaler, a PodDisruptionBudget, a NetworkPolicy) continue working entirely unchanged against a Rollout, since Rollout-managed Pods carry the exact same labels a Deployment-managed Pod would — the migration genuinely is scoped to the workload definition itself, not a cascading set of changes across every other resource that happens to reference it.

---

Canary Strategy in Argo Rollouts — Steps and setWeight#

The steps: list is a Rollout's canary strategy expressed as an explicit, ordered sequence — directly implementing the "successive waves" canary pattern already covered conceptually in Part 1, and structurally similar to Azure Pipelines' native canary: deployment strategy from Part 8, but as a standing Kubernetes controller rather than a one-time pipeline execution.

Diagram

Copy
# Trigger a new rollout (e.g. by updating the image, exactly like a Deployment)
kubectl argo rollouts set image checkout-service checkout=myregistry.io/checkout:1.2.4

# Watch the rollout progress live, including the current step and traffic weight
kubectl argo rollouts get rollout checkout-service --watch

# Manually promote past the current step immediately (skip the remaining pause)
kubectl argo rollouts promote checkout-service

# Manually abort and roll back
kubectl argo rollouts abort checkout-service

A pause: step with no duration: at all pauses indefinitely, requiring an explicit kubectl argo rollouts promote to continue — this is Argo Rollouts' manual-approval-gate mechanism, the direct equivalent of every CI/CD platform's manual approval gate already covered across Parts 4-11, except here implemented as a property of the deployment controller itself rather than a CI/CD pipeline step. A team can freely mix indefinite pauses (human-gated) and timed pauses (automatic, metric-driven, covered next) within one rollout's steps: list, matching exactly how cautious a given release needs to be.

A setCanaryScale step is worth a brief mention alongside setWeight, since the two are easy to conflate: setWeight controls what percentage of traffic reaches the canary; setCanaryScale controls how many actual canary Pods are running, independent of the traffic percentage. This distinction matters concretely when a team wants to validate a canary's resource behavior (memory usage, startup time) under a realistic Pod count before committing to full traffic-weight promotion, decoupling "how many canary instances exist" from "how much traffic each one receives" — two genuinely separate scaling questions a rollout can control independently rather than assuming they must always move in lockstep.

Argo Rollouts also supports a dynamicStableScale option, letting the stable version's own replica count shrink as the canary's traffic share grows, rather than always running both versions at full production capacity simultaneously throughout the rollout — a genuine cost optimization for a rollout with a long total duration, at the cost of slightly less rollback headroom if the stable version needs to rapidly absorb traffic back in a hurry.

---

A Minimal Argo Rollouts Canary, Built Up Step by Step#

Step 1 — the simplest possible canary, no analysis, just staged traffic:

Copy
strategy:
  canary:
    steps:
      - setWeight: 20
      - pause: { duration: 10m }
      - setWeight: 100

Step 2 — more granular steps, reducing the size of each traffic jump:

Copy
strategy:
  canary:
    steps:
      - setWeight: 5
      - pause: { duration: 5m }
      - setWeight: 20
      - pause: { duration: 5m }
      - setWeight: 50
      - pause: { duration: 5m }
      - setWeight: 100

Step 3 — adding automated analysis at each step (covered fully next section), replacing blind timed pauses with metric-driven ones:

Copy
strategy:
  canary:
    analysis:
      templates:
        - templateName: success-rate
      startingStep: 1        # begin analysis from the second step onward
    steps:
      - setWeight: 5
      - pause: { duration: 2m }
      - setWeight: 20
      - pause: { duration: 2m }
      - setWeight: 50
      - pause: { duration: 2m }
      - setWeight: 100

Step 4 — adding a scaledown delay, so the OLD version's pods aren't immediately terminated on full promotion (a genuinely important safety property, covered further below):

Copy
strategy:
  canary:
    analysis:
      templates: [{ templateName: success-rate }]
    steps:
      - setWeight: 5
      - pause: { duration: 2m }
      - setWeight: 50
      - pause: { duration: 2m }
      - setWeight: 100
    scaleDownDelaySeconds: 300   # keep the OLD ReplicaSet running 5 more minutes after full promotion

The scaledown delay directly addresses a real, specific risk worth naming precisely: without it, the instant a canary reaches setWeight: 100, Argo Rollouts scales the previous version's Pods down immediately — meaning an issue that only manifests moments after full promotion (a slow memory leak, a delayed downstream effect) has no fast, already-warm rollback target available; a fresh rollback would need to spin up new Pods of the old version from scratch. Keeping the old ReplicaSet alive a few extra minutes after full promotion gives a genuinely instant rollback path for exactly that class of "looked fine during the canary, broke moments after" failure.

---

Analysis — AnalysisTemplate and Metric Providers#

An AnalysisTemplate defines the actual metric query and the pass/fail threshold Argo Rollouts evaluates automatically during a canary — this is the concrete mechanism that turns "canary looks healthy" from a human's subjective judgment into an automated, repeatable decision.

Copy
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="checkout",status!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{app="checkout"}[2m]))

Diagram

failureLimit: 3 is worth understanding precisely, since it's the setting controlling how tolerant the analysis is of transient noise versus a genuine, sustained problem — a single failed check (a momentary metric blip, not necessarily a real issue) doesn't trigger an abort; only after 3 consecutive failed checks does Argo Rollouts conclude the canary is genuinely unhealthy and act. This directly reuses the "avoid alerting on noise, alert on sustained signal" principle already established in this course's Monitoring Methodologies series, applied here to an automated deployment decision rather than a human-facing alert.

A rollout can reference multiple metrics within a single AnalysisTemplate, each independently evaluated — a real production analysis commonly checks success rate and p99 latency and a custom business metric (e.g. checkout completion rate) simultaneously, with the rollout failing if any one of them breaches its own threshold. This composability matters because a single metric alone is rarely a complete picture of "is this canary actually healthy" — a canary maintaining a perfect success rate while its p99 latency triples is very much still a real regression, just one a success-rate-only check would completely miss.

AnalysisTemplate also supports parameterized arguments, letting one shared template be reused across many different Rollouts by substituting in the specific service name or threshold each one needs at reference time — the same "define once, parameterize per-consumer" reusability principle already established for CI/CD platform templates throughout this series (GitLab CI/CD Components in Part 6, Azure Pipelines templates in Part 8), applied here to canary analysis definitions specifically.

The provider list — Prometheus, Datadog, New Relic, CloudWatch, and several others — is worth knowing exists specifically because it means analysis isn't locked to any one observability stack, directly reusing whatever metrics platform an organization has already invested in (this course's Observability series covered the broader case for this investment) rather than requiring a separate, dedicated system just for canary analysis.

A web provider is also worth knowing about as a genuine escape hatch beyond the named observability-vendor integrations — it queries an arbitrary HTTP endpoint and evaluates the JSON response, letting an organization plug in literally any internal metrics system with an HTTP API, not just the specific named providers Argo Rollouts ships first-party integrations for.

A job provider takes this further still, running an arbitrary Kubernetes Job (any container, any script) as the actual analysis check — the Argo Rollouts equivalent of Flagger's webhook system, for validation logic too custom to express as a simple metric-threshold query at all.

Between the web, job, and named-vendor providers, an AnalysisTemplate genuinely covers the same range of validation flexibility Flagger's webhook system provides, even though the two tools arrive at it through different-looking mechanisms — worth remembering when the earlier comparison table describes Flagger's webhooks as "more central to the design," since Argo Rollouts reaches comparable flexibility through its own provider ecosystem rather than lacking the capability outright.

---

Automated Rollback — When Analysis Fails#

When failureLimit is exceeded, Argo Rollouts automatically aborts the rollout — setting the canary's traffic weight back to 0% and, depending on configuration, either holding at the last-known-good state or fully reverting to the previous version — with zero human intervention required.

Diagram

This is the single most important property distinguishing progressive delivery from the manual canary process this series' Part 1 first described — a manual canary requires a human actively watching dashboards and making a judgment call, which is both slow (human reaction time, and humans aren't watching 24/7) and inconsistent (different engineers may have different risk tolerances). Automated rollback via AnalysisTemplate makes the rollback decision as fast as the metric-check interval itself (commonly under a minute) and completely consistent, applying the exact same threshold every single time, regardless of what time of day or which engineer happens to be on call — directly extending this course's Incident Management series' argument for automating detection and response wherever a clear, reliable signal exists, applied here specifically to the deployment process itself.

A rollback event is itself worth treating as a real signal, not just an automatic self-correction to shrug off — even though no human had to intervene to stop the bad rollout from reaching full traffic, the fact that a change failed its own analysis is genuinely valuable information: it caught something real, worth a lightweight postmortem asking why the issue wasn't caught earlier (in the CI/CD pipeline's own testing, per Parts 4-11) rather than only in production canary analysis. Treating every automated rollback as a silent, fully-resolved non-event risks missing the same "why did this reach production at all" root-cause question this course's Incident Management series argues should follow any production-detected issue, automated recovery or not.

Argo Rollouts' abort behavior is also worth distinguishing from a genuinely full rollback: aborting sets traffic weight back to 0% and marks the Rollout Degraded, but the desired state (the new image tag in the Rollout spec) is left unchanged — a subsequent kubectl argo rollouts retry re-attempts the exact same rollout from the beginning, useful if the failure was itself transient (a flaky dependency, an unrelated infrastructure blip) rather than a genuine defect in the change itself, without requiring a fresh commit or redeploy to try again.

---

Blue-Green Strategy in Argo Rollouts#

Argo Rollouts' strategy.blueGreen implements the instant-traffic-switch pattern from Part 1, with its own analysis integration:

Copy
strategy:
  blueGreen:
    activeService: checkout-active      # the Service currently receiving real traffic
    previewService: checkout-preview    # the Service pointing at the NEW version, for pre-promotion testing
    autoPromotionEnabled: false         # require explicit promotion, don't switch automatically
    prePromotionAnalysis:
      templates: [{ templateName: smoke-test }]
    postPromotionAnalysis:
      templates: [{ templateName: success-rate }]

prePromotionAnalysis and postPromotionAnalysis are worth distinguishing precisely, since blue-green's instant-switch nature creates two genuinely different risk windows a canary's gradual rollout doesn't have as sharply: pre-promotion analysis validates the new (green) version before it ever receives real production traffic — smoke tests run against the previewService, which only test infrastructure/internal traffic can reach — while post-promotion analysis validates after the switch, watching real production metrics now flowing to the newly-active version, ready to trigger an automatic rollback (an instant switch back to the still-running blue environment) if something the pre-promotion smoke tests didn't catch shows up under real traffic.

Blue-green's rollback speed advantage over canary, already established conceptually in Part 1, is worth restating concretely in this chapter's terms: because the old (blue) environment's Pods are never scaled down until autoPromotionEnabled-gated cleanup explicitly happens, a post-promotion rollback is purely a Service-selector flip back to the still-fully-running blue environment — no new Pods to schedule, no image to pull, no application startup time to wait through, in genuine contrast to a canary rollback, which (absent the scaleDownDelaySeconds safety margin covered earlier) may need to scale the old version's Pods back up from zero. This is the same tradeoff Part 1 first named in the abstract — blue-green trades roughly double the standing infrastructure cost during a rollout for a meaningfully faster rollback — now visible as a concrete property of how Argo Rollouts' blue-green strategy actually manages its underlying Kubernetes resources.

scaleDownDelaySeconds applies to blue-green too, functioning identically to its canary-strategy counterpart — keeping the old (blue) environment's Pods alive for a configured window after promotion, protecting against exactly the same delayed-onset-failure scenario already covered for canary rollouts, just against blue-green's already-faster baseline rollback time rather than canary's comparatively slower one.

---

Traffic Management — Service Mesh and Ingress Integration#

Both setWeight (canary) and the active/preview Service switch (blue-green) need something that actually enforces traffic splitting at the network level — Argo Rollouts doesn't do this itself; it delegates to whichever traffic-management layer is already in the cluster.

The "basic, no mesh" row is worth flagging as a genuine limitation, not an equally-good alternative: without a real traffic-management layer, setWeight: 10 can only be approximated by running roughly 10% of total Pods on the new version — meaning the actual traffic split is coarse-grained and tied to replica count (achieving a genuinely precise 10% split requires a multiple-of-10 total replica count) rather than the smooth, request-level percentage a service mesh provides. A team seriously adopting progressive delivery, especially at lower traffic percentages (a cautious 1% or 5% initial canary step), very commonly needs a real service mesh or Gateway API implementation in place to make those fine percentages meaningful at all — this is frequently the actual adoption blocker for progressive delivery, more than the Rollout/Canary CRD configuration itself.

Gateway API deserves a specific forward-looking callout, since it's the direction both projects' own integration work has increasingly moved toward: rather than each tool needing bespoke, separately-maintained integration code for every individual service mesh and ingress controller, Gateway API is a single, mesh/ingress-agnostic Kubernetes-native standard for exactly this kind of weighted traffic routing — both Argo Rollouts and Flagger support it as a traffic-management backend, meaning a team standardizing on Gateway API gets progressive delivery support that isn't tied to a specific mesh vendor's own proprietary integration, a genuinely lower-lock-in path than committing to Istio- or Linkerd-specific configuration directly.

---

The Experiment CRD — Comparison Without Promotion#

Worth a dedicated mention as a genuinely distinctive Argo Rollouts capability with no direct Flagger equivalent: the Experiment CRD runs two (or more) versions side by side, gathering comparative metrics, with no intention of ever promoting either one — a genuinely different use case from a canary or blue-green rollout, which both exist specifically to eventually fully replace the old version with the new.

Copy
apiVersion: argoproj.io/v1alpha1
kind: Experiment
metadata:
  name: checkout-algorithm-comparison
spec:
  duration: 1h
  templates:
    - name: baseline
      replicas: 2
      template:
        spec:
          containers: [{ name: checkout, image: myregistry.io/checkout:current-algorithm }]
    - name: candidate
      replicas: 2
      template:
        spec:
          containers: [{ name: checkout, image: myregistry.io/checkout:new-algorithm }]
  analyses:
    - name: compare-conversion-rate
      templateName: conversion-rate-comparison

The distinction from a canary is worth stating with real precision, since conflating the two is an easy mistake: a canary's steps: list is fundamentally a path toward full promotion — every successful step moves closer to setWeight: 100. An Experiment has no such trajectory at all — both baseline and candidate run in parallel for a fixed duration, purely to gather comparative data (e.g. "does the new checkout algorithm's conversion rate genuinely differ from the current one"), and the Experiment resource itself is torn down afterward regardless of which performed better, with a completely separate, subsequent decision (a normal deploy, or nothing at all) determining what actually happens next. This is the concrete Argo Rollouts mechanism for a genuine controlled experiment — closer in spirit to product-analytics A/B testing infrastructure than to a deployment strategy, even though it's built on the same underlying traffic-splitting and metric-analysis machinery as this chapter's canary and blue-green coverage.

An Experiment can also be triggered by a canary rollout's own steps, rather than only run standalone — a steps: entry referencing an experiment: block runs a bounded, time-limited comparison as one step within an otherwise-ordinary canary, letting a single rollout combine "gather comparative data for a fixed window" with "then continue toward normal promotion" in one coherent process, rather than treating experimentation and deployment as two entirely separate, manually-sequenced activities.

---

SLO-Gated Rollouts — Tying Progressive Delivery to Error Budgets#

Worth a direct, explicit connection back to this course's SRE Fundamentals series: the successCondition threshold in an AnalysisTemplate (or Flagger's thresholdRange) doesn't have to be an arbitrary, deployment-specific number chosen in isolation — it can, and in a mature SRE practice commonly should, be derived directly from the service's own already-defined SLO and error budget.

Diagram

Why deriving the threshold from the existing SLO matters, beyond just convenience: a canary analysis threshold chosen independently of the service's actual SLO can be silently inconsistent with it — a canary might pass its own analysis (e.g. "success rate stayed above 95%") while quietly burning a disproportionate share of a much stricter 99.9% SLO's error budget, or conversely, reject a canary based on an overly strict threshold that doesn't match what the service's actual reliability target permits. Tying the analysis threshold directly to the already-established SLO closes this gap, making the automated rollout decision genuinely consistent with the same error-budget-driven risk tolerance this course's SRE Fundamentals series already established for every other category of risky change (a launch, a large migration) — progressive delivery becomes one more mechanism reading from and respecting the same single source of truth for "how much risk is this service currently allowed to take on," rather than an independently-tuned parallel system with its own, potentially inconsistent risk tolerance.

A more advanced version of this same idea, worth knowing exists even if less commonly implemented: rather than a fixed threshold derived once from the SLO, an organization can gate canary promotion on the remaining error budget itself, checked live — if a service has already burned most of its monthly error budget from unrelated incidents, even a canary that would ordinarily pass its own threshold check can be held back or given a stricter bar, since the service currently has less room to absorb additional risk than it would in a month with a fully intact budget. This directly operationalizes the SRE Fundamentals series' own argument that error budget policy should actively gate risky changes, not just retrospectively explain why one was or wasn't a good idea after the fact.

Implementing live error-budget gating concretely reuses the web or job analysis provider from earlier in this chapter — the analysis check queries an internal error-budget-tracking service's own API rather than a raw Prometheus metric, letting the canary's promotion decision read directly from whatever system already tracks the service's real-time budget consumption.

---

Ephemeral Preview Environments per Pull Request#

A genuinely popular pattern worth covering, extending blue-green's previewService concept from earlier in this chapter into something considerably more granular: spinning up a complete, isolated, temporary environment for every open pull request, torn down automatically when the PR closes.

Diagram

Where this connects directly to this chapter's core subject, rather than being a tangential feature: a PR preview environment is architecturally very close to Argo Rollouts' previewService from the blue-green section — a fully running instance of a candidate version, reachable and testable, before it's ever exposed to a single unit of real production traffic. The genuine difference is scope and audience: blue-green's preview service exists briefly, internally, purely for automated pre-promotion analysis; a PR preview environment exists for the full lifetime of the PR's review process, for human reviewers and stakeholders to interact with directly, commonly with its own distinct URL shared in the PR itself. The automatic-teardown discipline is worth emphasizing as the operationally critical part, not an afterthought: without reliable, automatic cleanup tied to PR close/merge, preview environments accumulate as orphaned, forgotten infrastructure — a genuine, recurring cost-management failure mode this pattern needs to guard against explicitly, directly connecting to the self-hosted-runner and infrastructure cost-consciousness theme Part 14 covers next in this series.

A common real-world refinement worth naming: rather than a full, independently-provisioned environment per PR (real infrastructure cost multiplied by however many PRs are simultaneously open), many teams implement preview environments as a lightweight namespace-per-PR within an already-running shared cluster, sharing common infrastructure (a shared database instance, scoped per-PR via schema or logical database isolation) while still giving each PR its own genuinely isolated, independently-testable application deployment — a middle ground between "one fully separate environment per PR" and "no preview environment at all," trading some isolation purity for meaningfully lower marginal cost per open PR.

---

Progressive Delivery for Non-HTTP Workloads#

Worth a direct, honest note on a real limitation implicit in nearly every example so far in this chapter: both Argo Rollouts' and Flagger's traffic-splitting mechanics (service mesh weighted routing, ingress-based percentage splits) are fundamentally built around HTTP request traffic — a genuine complication for a worker consuming from a message queue, processing batch jobs, or otherwise not serving discrete, individually-routable HTTP requests at all.

Diagram

The practical workaround, worth naming since it's a genuinely common real pattern rather than a dead end: rather than percentage-based traffic routing, a canary for a queue-consuming worker is more commonly implemented via partitioned or percentage-based queue consumption — running a small number of canary worker replicas alongside the stable fleet, all consuming from the same queue, with the canary's share of processed messages naturally proportional to its replica count relative to the stable fleet's (the same setCanaryScale mechanism from earlier in this chapter, repurposed for exactly this case) — and analysis then watches per-version processing metrics (error rate, processing latency, dead-letter-queue growth) tagged by which replica handled each message, rather than a request-routing-based split. This is a genuinely less precise mechanism than HTTP's per-request routing (a queue consumer's actual share of work is proportional to replica count and processing speed, not a cleanly dialed-in percentage), but preserves the same core progressive-delivery principle — a small, bounded fraction of real production work validates the new version before it fully replaces the old one, with the option of automated rollback if its metrics diverge from the stable fleet's.

Batch jobs (a scheduled data pipeline run, rather than a continuously-running consumer) present a related but distinct case worth a brief mention: since a batch job runs to completion rather than serving ongoing traffic, "canary" in this context more commonly means running the new version against a genuinely separate, smaller-scoped input (a sample dataset, a single partition) and comparing its output against the previous version's known-good result, before the new version is trusted to run against the full production dataset — closer in spirit to the Experiment CRD's side-by-side comparison than to a traffic-percentage canary, since there's no ongoing traffic to gradually shift at all.

---

Flagger — the Canary CRD Wrapping a Deployment#

Flagger's Canary resource references an existing, unmodified Deployment and drives the same fundamental canary process — staged traffic increase with metric-driven promotion — from outside it:

Copy
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: checkout-service
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service          # references an EXISTING, standard Deployment — unmodified
  service:
    port: 80
  analysis:
    interval: 1m
    threshold: 5                     # max consecutive failed checks before rollback
    maxWeight: 50
    stepWeight: 10                   # increase by 10% each successful interval
    metrics:
      - name: request-success-rate
        thresholdRange: { min: 99 }
        interval: 1m

Diagram

Flagger automatically creates a shadow "primary" and "canary" Deployment pair behind the scenes, orchestrating traffic between them, while the resource a developer actually edits (pushes a new image tag to) remains the single, original, standard Deployment — directly matching this chapter's opening architectural framing: Flagger's value is layered around existing Kubernetes primitives rather than replacing them, a meaningfully lower migration barrier for a team with a large number of existing Deployment-based workloads.

A practical consequence of this shadow-resource model worth knowing before debugging a Flagger rollout for the first time: kubectl get deployments in a namespace running Flagger shows three Deployments for what a developer thinks of as one service — the original (which Flagger's own reconciler keeps scaled to zero once a canary process is underway, functioning as the source template), plus the -primary and generated canary Deployments Flagger actually manages traffic between. This is worth knowing specifically so it doesn't read as confusing or broken the first time it's observed — kubectl describe canary checkout-service (querying Flagger's own Canary resource directly) is the correct way to check actual rollout status, rather than trying to reason about it from the underlying Deployment objects alone.

---

Flagger's Webhook System#

Flagger's most distinctive capability relative to Argo Rollouts' AnalysisTemplate model is its webhook system — arbitrary HTTP calls Flagger makes at specific lifecycle points during a canary rollout, extending analysis beyond pure metric-threshold checking into genuinely custom validation logic.

Copy
analysis:
  webhooks:
    - name: smoke-test
      type: pre-rollout
      url: http://smoke-tester.test/
      timeout: 30s
    - name: load-test
      type: rollout
      url: http://flagger-loadtester.test/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://checkout-canary.default/"
    - name: notify-team
      type: post-rollout
      url: http://slack-webhook.internal/
    - name: manual-gate
      type: confirm-promotion
      url: http://approval-service.internal/

The rollout-type load-test webhook deserves particular attention as a genuinely clever solution to a real, easy-to-overlook problem: metric-driven analysis needs enough actual traffic hitting the canary to produce a statistically meaningful signal — a low-traffic internal service might receive so few real requests during a short analysis window that "97% success rate" is based on only a handful of data points, barely more informative than a coin flip. Generating synthetic load specifically during the analysis window solves this directly, giving the success-rate metric enough real request volume to actually mean something.

The confirm-promotion webhook type is worth a further, specific note, since it's Flagger's most direct analogue to the manual-approval-gate concept covered across every CI/CD platform chapter in this series (Parts 4-11): unlike Argo Rollouts' indefinite pause: step (which blocks purely on a kubectl command), confirm-promotion blocks on an arbitrary external HTTP service's response — meaning the approval gate can be backed by literally any system capable of returning an HTTP response: a custom internal approval UI, a Slack-integrated bot awaiting a reaction, or even a check against an external change-freeze calendar (the same idea already covered as Azure DevOps's "Invoke REST API" environment check in Part 8), giving Flagger's manual gate a genuinely arbitrary, programmable backing rather than requiring a human to run a specific CLI command.

---

Flagger vs. Argo Rollouts — a Direct Comparison#

The honest recommendation, consistent with this series' pattern of naming a genuine "it depends" rather than picking a false winner: a team already using Argo CD for GitOps (Part 3) gains real, tight integration adopting Argo Rollouts specifically — the same project family, the same CLI ecosystem, the same UI showing both GitOps sync status and rollout progress together. A team using Flux for GitOps, or one with a large existing fleet of standard Deployments they'd strongly prefer not to migrate to a new CRD kind, gains more from Flagger's lower-migration-cost, webhook-centric model. Both are mature, production-grade, CNCF-adjacent projects — this is a genuine ecosystem-fit decision, not a maturity or capability gap between the two.

Neither choice is permanent or mutually exclusive at an organizational level, either — it's genuinely common for different teams within one organization to run different progressive-delivery tools for different workloads, particularly during a gradual migration between GitOps tools, or when a specific team's existing Deployment fleet makes Flagger's lower-migration-cost model the pragmatic near-term choice even while the broader organization standardizes on Argo CD elsewhere. Consistency across an entire organization is a nice-to-have, not a hard requirement the way, say, a shared secrets-management approach might be.

A platform team standardizing on one tool org-wide, once a choice is made, should treat that standardization itself as a deliberate governance decision worth documenting — the same "write down the decision and its rationale" discipline this course has argued for repeatedly, so a future team evaluating the "other" tool for a new workload has the original reasoning available rather than re-litigating the same comparison from scratch.

---

Canary Weight Curves — Choosing How Aggressively to Ramp#

Worth a dedicated, practical section on a design decision every real canary configuration has to make: how the actual sequence of setWeight values is chosen, since this chapter's examples so far have used a single illustrative curve without discussing the tradeoff explicitly.

Diagram

The exponential-style ramp is the more common real-world default, worth explaining why precisely: the very first, smallest traffic step (1-5%) is where a genuinely broken change is most likely to be caught, and it's also the stage carrying the least confidence that the change is safe at all — spending the most analysis time and the smallest steps here maximizes the chance of catching a real problem while its blast radius is still tiny. Once a change has already cleared several small-step analysis windows, the marginal value of yet another small, slow step diminishes — the change has already demonstrated real, sustained health at meaningful traffic levels, and larger, faster steps toward full promotion reflect that already-established confidence rather than starting the same cautious pace over at every single step. A linear ramp isn't wrong, but it's a less information-theoretically efficient use of the total rollout time an organization is willing to spend validating any single change.

The actual step values are also worth tuning per-service rather than treated as a single, org-wide constant — a high-traffic service reaches statistical confidence at a given percentage far faster (more absolute requests per minute at any given percentage) than a low-traffic one, meaning the same 5% → 20% → 50% → 100% curve that's comfortably fast for a high-traffic checkout service might still be statistically under-sampled at its very first step for a much lower-traffic internal tool — the load-testing webhook pattern from Flagger's webhook section is the direct mitigation for exactly that specific, per-service mismatch.

---

Multi-Cluster and Multi-Region Progressive Delivery#

Worth a brief, honest note on a real complication for any organization running production across multiple Kubernetes clusters or geographic regions: everything covered so far in this chapter — a single Rollout or Canary object, a single AnalysisTemplate — operates within the scope of one cluster, and neither Argo Rollouts nor Flagger natively coordinates a canary rollout across multiple independent clusters as a single, unified process.

Diagram

Option B — a CI/CD pipeline or custom controller sequencing region-by-region rollouts, each gated on the previous region's own successful promotion — is the more common real-world pattern for genuinely global production estates, and it composes directly with everything already covered across this series: a pipeline stage (Parts 4-11) polls one region's Rollout status via kubectl argo rollouts status, proceeding to trigger the next region's rollout only once the current one reports fully healthy — effectively treating each region as one more sequential stage in a larger, CI/CD-pipeline-orchestrated deployment, with this chapter's per-cluster progressive delivery tooling handling the actual safe rollout within each individual region. This is worth knowing as a genuine architectural gap in both tools' native scope, not a limitation unique to either one specifically — multi-cluster coordination is a layer an organization builds on top, using the exact same building blocks (a CI/CD pipeline checking a Rollout's status) already covered throughout this series.

Region ordering itself is worth choosing deliberately, not arbitrarily — a common, sensible default is starting with the region carrying the smallest fraction of total traffic (limiting blast radius even further than the canary's own traffic percentage already does, in case something region-specific and unrelated to the change itself goes wrong), and ending with the largest, highest-traffic region only once every smaller region has already validated the change successfully — layering region-level caution on top of the within-region traffic-percentage caution this chapter has covered throughout, rather than treating "canary within one region" and "which region goes first" as unrelated decisions.

---

Progressive Delivery + GitOps — Argo Rollouts and Argo CD Together#

Worth tying directly back to Part 3's GitOps coverage, since this is the single most common, most natural pairing for Argo Rollouts specifically: a Rollout manifest is just another Kubernetes resource, committed to the same Git repository Argo CD already watches and reconciles.

Diagram

This composition directly answers a question left implicit in Part 3's GitOps chapter: what actually happens the moment Argo CD applies a new manifest to the cluster? Without progressive delivery, a GitOps-managed Deployment update is a standard Kubernetes rolling update — reasonably safe, but with none of the metric-driven, automated analysis this chapter covers. Layering Argo Rollouts underneath Argo CD means the exact same Git-commit-triggers-reconciliation flow from Part 3 now drives a fully automated, metric-gated canary rollout — GitOps determines what the desired state is and reconciles toward it; Argo Rollouts determines how carefully and safely the cluster actually gets there.

Part 3's git revert-based rollback mechanism is worth revisiting specifically in light of this chapter's own automated rollback capability, since the two operate on genuinely different timescales and triggers. A git revert (Part 3) is a deliberate, human-initiated action — someone decides the deployed state is wrong and reverts the commit that caused it, with Argo CD's own reconciliation loop then bringing the cluster back to the reverted state, typically over the span of minutes. Argo Rollouts' automated rollback (this chapter) reacts within the analysis interval itself — commonly under a minute — entirely without a human deciding anything, reverting the live traffic weight immediately while the underlying Git state (and Argo CD's view of desired state) may still reflect the newer, failed version until a human separately reverts the commit. In practice, both matter and serve different purposes: Argo Rollouts' fast, automatic weight rollback stops the bleeding immediately; a subsequent git revert is still the correct, deliberate way to fully retire the failed change from the desired-state source of truth, closing the loop Part 3 established.

---

Progressive Delivery + CI/CD Pipelines#

Worth closing the loop with every platform-specific CI/CD chapter covered across Parts 4-11: where does a Rollout or Canary's new image tag actually come from? The same CI/CD pipelines this entire series has covered — a GitHub Actions workflow, a GitLab pipeline, a Jenkins Declarative Pipeline — remains exactly the mechanism that builds, tests, and (per Part 3's GitOps model) commits an updated image tag, which then triggers the progressive-delivery process this chapter covers.

Diagram

This is worth stating as the precise division of responsibility this entire series has been building toward: CI/CD pipelines (Parts 4-11) own validating and packaging a change; GitOps (Part 3) owns declaring and reconciling the desired state; progressive delivery (this chapter) owns safely, gradually exposing that new state to real traffic. Each of these three layers is independently replaceable — a team can swap CircleCI for GitHub Actions, or Argo CD for Flux, or Flagger for Argo Rollouts, without needing to change the other two layers at all — precisely because each addresses a genuinely distinct concern with a clean interface (a Git commit) between them.

A CI/CD pipeline can also actively participate in a rollout already underway, not just kick it off — a pipeline stage using kubectl argo rollouts status --watch (already shown in this chapter's multi-cluster section) to block on a rollout's completion before proceeding to a subsequent stage (a smoke test suite run once fully promoted, a notification, a downstream service's own deploy trigger) is a genuinely common pattern, treating the progressive-delivery process itself as one more gated stage within a larger CI/CD pipeline rather than a disconnected, fire-and-forget trigger.

---

A/B Testing — the Third Progressive Delivery Pattern#

Beyond canary and blue-green (both covered conceptually in Part 1), both Argo Rollouts and Flagger support a third pattern worth knowing: A/B testing, where traffic is split not by a raw percentage but by a specific, deliberate criterion — most commonly an HTTP header or cookie, routing a specific, identifiable set of users to the new version rather than a random percentage sample.

Copy
# Argo Rollouts — A/B testing via header-based routing
strategy:
  canary:
    steps:
      - setCanaryScale: { weight: 100 }   # canary gets 100% of MATCHING traffic, not a % of ALL traffic
        # ... combined with a matching traffic-routing rule that
        # only sends requests carrying a specific header to the canary

The distinction from a canary's random-percentage sampling is worth being precise about, since interviewers and real design discussions both treat this as a meaningfully different tool: a canary answers "is this new version safe for an arbitrary random slice of traffic" — useful for catching broad, general regressions. A/B testing answers a genuinely different question — "how does this specific, identifiable group of users (opted into a beta program, matching a particular cohort, or literally the internal engineering team dogfooding via a special header) actually experience the new version" — useful for validating a targeted change (a new UI variant, a pricing experiment) against a deliberately chosen population rather than a random one. Both tools in this chapter can implement either pattern; which one a given rollout needs depends entirely on whether the validation question is "is this broadly safe" (canary) or "does this specific group have the experience we intend" (A/B).

A genuinely important caveat worth stating explicitly, since it's easy to conflate deployment-layer A/B routing with true product-analytics experimentation: the header/cookie-based routing covered here determines which version of the running service a request reaches — it says nothing on its own about experiment design rigor (statistical power, sample-size planning, avoiding peeking bias) that a genuine product A/B test requires. A team running an actual business-metric experiment (does this pricing change increase conversion) needs real experimentation-platform discipline on top of this chapter's routing mechanics, not just the routing itself — this chapter's A/B pattern provides the deployment-layer plumbing; it is not, by itself, a complete experimentation methodology.

---

Fixed Thresholds vs. Statistical Analysis (Kayenta)#

Every AnalysisTemplate example in this chapter so far uses a fixed threshold — successCondition: result[0] >= 0.95, a single, static number a metric must clear. Worth knowing there's a genuinely more sophisticated alternative, since it's a specific, named metric provider Argo Rollouts supports: Kayenta (originated at Netflix, part of the broader Spinnaker ecosystem), which performs statistical comparison between the canary and a baseline, rather than checking the canary against one fixed number in isolation.

Diagram

The practical distinction worth understanding, since it's the actual reason a team would reach for the more complex statistical approach: a fixed threshold implicitly assumes "normal" is a stable, known constant — but real production metrics fluctuate for reasons entirely unrelated to the canary itself (a traffic spike, a downstream dependency having a slow day), and a fixed threshold can't distinguish "the canary made things worse" from "everything is a bit worse today, canary included." Kayenta's side-by-side statistical comparison sidesteps this by comparing the canary directly against a baseline experiencing the exact same real-world conditions at the exact same time, scoring the difference between them rather than either one's absolute value — a canary performing statistically indistinguishably from a currently-degraded baseline correctly passes, since it's demonstrably no worse than what's already happening, independent of the canary change itself.

The honest tradeoff, consistent with this series' pattern of not oversimplifying "more sophisticated is always better": Kayenta-style statistical analysis is genuinely more robust against baseline noise, but requires running an actual side-by-side baseline (real infrastructure cost, doubling at minimum the canary-phase resource footprint) and is meaningfully more complex to operate and reason about than a fixed threshold. Most teams' progressive delivery adoption starts with, and often stays with, fixed thresholds — they're simpler to configure, simpler to debug when they fire unexpectedly, and sufficient for the large majority of real use cases; statistical analysis earns its added complexity specifically for services with genuinely volatile, hard-to-threshold baseline metrics where fixed thresholds have proven to produce too many false positives or false negatives in practice.

Kayenta's approach traces its origins to Netflix's own internal Automated Canary Analysis (ACA) system, developed specifically because Netflix's own service metrics proved too volatile at their operating scale for fixed thresholds to be reliably actionable — worth knowing as context for why the technique exists at all: it's a solution born from a genuinely specific, large-scale operational pain point, not a generically "more advanced" default every team should aspire to regardless of whether they actually share that same pain point.

---

A Full Realistic Example: Canary with Automated Analysis and Rollback#

Tying every mechanism from this chapter together into one realistic Argo Rollouts canary, GitOps-managed:

Copy
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-service
spec:
  replicas: 10
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: checkout
          image: myregistry.io/checkout:1.2.4   # updated by CI/CD + GitOps, per this chapter's earlier section
  strategy:
    canary:
      analysis:
        templates: [{ templateName: success-rate }]
        startingStep: 1
      steps:
        - setWeight: 5
        - pause: { duration: 3m }
        - setWeight: 20
        - pause: { duration: 3m }
        - setWeight: 50
        - pause: { duration: 3m }
        - setWeight: 100
      scaleDownDelaySeconds: 300
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.98
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="checkout",status!~"5.."}[2m]))
            / sum(rate(http_requests_total{app="checkout"}[2m]))

Diagram

This single example demonstrates the complete arc this chapter has built up section by section: a GitOps-committed change (Part 3) triggers a staged rollout with automated, metric-driven promotion at every step, a defined failure threshold that triggers a fully automatic rollback with zero human involvement, and a safety margin (the scaledown delay) protecting against failures that only manifest after full promotion — the concrete, production-grade realization of the canary-deployment concept this series first introduced, in outline, all the way back in Part 1.

Notice, too, everything this specific example deliberately leaves out, each addressed in its own dedicated section elsewhere in this chapter: no explicit traffic-mesh configuration is shown (assumed already in place, per the traffic-management section), the successCondition threshold of 98% is presented as a given rather than derived from a specific SLO (a real team would tie this back to the service's actual error budget, per the SLO-gating section), and no schema-migration compatibility check is shown (a real deployment touching the database would need the expand/contract discipline covered later in this chapter). A genuinely complete production configuration layers all of these considerations together; this example isolates the core canary/analysis mechanics specifically for clarity.

---

Database Migrations and Progressive Delivery — a Genuine Complication#

Worth a dedicated, honest treatment, since it's one of the most commonly underestimated real-world complications when a team first adopts canary deployments: during a canary rollout, the OLD and NEW versions of a service are running simultaneously, against the SAME database — and any database schema change the new version depends on has to be compatible with the old version too, for the entire duration of the canary.

Diagram

The standard mitigation, worth naming explicitly since it's a real, non-optional discipline for any team seriously adopting progressive delivery alongside database-backed services: schema changes must be additive and backward-compatible for the duration any two versions might coexist — add a new column rather than renaming an existing one, and only remove or repurpose the old column in a later, separate deployment once every instance of the old application version is confirmed fully retired. This is commonly called the expand/contract pattern: "expand" the schema (add the new column, keep the old one working) ships and canaries safely; a separate, later "contract" change (actually removing the now-unused old column) ships only once no version depending on it remains running anywhere in the fleet.

Why this deserves emphasis specifically in a progressive delivery chapter, not just a general database-migrations note: the entire value proposition of a canary is running old and new versions simultaneously for the analysis window — a schema change that breaks this simultaneity doesn't just risk the new version; it actively breaks the old, previously-stable version too, for whatever fraction of traffic the canary hasn't yet promoted away from it. A team that skips the expand/contract discipline can find their carefully-designed, metric-gated canary process becomes the cause of an incident it was specifically built to prevent — the automated rollback machinery covered earlier in this chapter will correctly detect the resulting failure and roll back, but the "safe, gradual, metric-validated" promise of progressive delivery is only as good as the schema-compatibility discipline underneath it.

The same compatibility discipline extends beyond the database itself to any shared, stateful dependency two application versions might both touch during a canary window — a shared cache with a changed serialization format, a message queue whose message schema changed incompatibly, or a shared configuration store are all subject to the identical expand/contract reasoning, even though "database migration" is the most commonly cited example. The general principle, worth carrying forward past this chapter's specific database framing: any shared state two coexisting versions both read or write must remain mutually compatible for the full duration they might coexist, not just for the version that will eventually win.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A team's Argo Rollouts canary for a low-traffic internal admin tool (roughly 5 requests per minute) keeps triggering false-positive rollbacks — the success-rate metric swings wildly between 80% and 100% purely from small-sample noise, with no actual underlying problem. Diagnose the cause and propose two independent fixes.

Answer: The root cause is statistical, not a real regression — at 5 requests/minute, a single failed request already swings the success rate by 20 percentage points, making the metric far too noisy to reliably distinguish "real problem" from "normal small-sample variance." Two independent fixes, both legitimate and often combined: first, widen the metric query's time window (e.g. [10m] instead of [2m]) to aggregate more requests per data point, directly reducing sample-size noise; second, generate synthetic load specifically during the analysis window (Flagger's rollout-type load-test webhook, or an equivalent Argo Rollouts approach) so the metric is computed against enough real request volume to be statistically meaningful regardless of the tool's genuinely low organic traffic.

Problem 2: An organization already runs Flux for GitOps across their entire fleet (not Argo CD) and is evaluating Argo Rollouts vs. Flagger for progressive delivery. A team member argues "we should use Argo Rollouts since it's the most fully-featured option regardless of our GitOps tool." Evaluate this argument.

Answer: The argument overstates a real but secondary consideration (feature completeness) while understating a more directly relevant one (ecosystem fit): Argo Rollouts and Flagger are both mature, comparably capable tools for the core progressive-delivery mechanics this chapter covers (canary, blue-green, metric-driven analysis), and "most fully-featured" isn't a clean, uncontested claim between them — Flagger's webhook system is arguably more flexible for custom validation logic, while Argo Rollouts' analysis templates are more purpose-built for pure metric-threshold checking. The more decisive factor for this specific organization is that Flagger is part of the same broader Flux/CNCF GitOps lineage already in use, while Argo Rollouts' deepest integration value is specifically with Argo CD — an organization on Flux gains comparatively less of Argo Rollouts' "same project family" integration benefit than an Argo CD shop would, making Flagger the more directly justified choice here on ecosystem-fit grounds, independent of either tool's raw feature list.

Problem 3: A platform team wants to validate a risky database-query optimization change against only their own internal engineering team first, before exposing it to any real customer traffic at all — not a random percentage sample, but specifically their own team's requests. Which progressive delivery pattern fits this requirement, and how would it be implemented?

Answer: This is precisely the A/B testing pattern from this chapter, not a canary — the validation question is "does this specific, identifiable group (the engineering team) have the intended experience," not "is this broadly safe for an arbitrary random slice of traffic." Implementation: header-based (or cookie-based) routing, where the engineering team's requests carry a specific identifying header (set via an internal proxy, VPN configuration, or browser extension used only by the team), and the Rollout/Canary's traffic-routing rule sends only requests carrying that header to the new version — everyone else continues hitting the current, unchanged version entirely, with zero random-sample exposure to real customers until the team's own internal validation is complete and a decision is made to proceed to an ordinary percentage-based canary for the broader rollout.

Problem 4: A service has a well-established SLO of 99.9% success rate with a monthly error budget already tracked per this course's SRE Fundamentals series. The platform team is setting up Argo Rollouts for the first time and picks successCondition: result[0] >= 0.90 for the canary analysis, reasoning "we want the canary to be lenient so we don't get false-positive rollbacks." Evaluate this choice.

Answer: This threshold is inconsistent with the service's own established risk tolerance, and the reasoning behind it conflates two different problems. A 90% threshold permits the canary to run at up to 10% failure — one hundred times worse than the service's actual 99.9% SLO — meaning a canary could pass its own analysis while burning an enormous, wildly disproportionate share of the service's monthly error budget in the process, exactly the gap this chapter's SLO-gating section warns against. The "avoid false-positive rollbacks" goal the team is actually reaching for is a real, legitimate concern, but the correct fix is tuning failureLimit (tolerate a few transient failed checks before acting) or widening the metric's time window (reduce sample-noise sensitivity) — not loosening the threshold itself far below what the SLO actually permits. The threshold should be derived from the SLO (something meaningfully tighter, close to 99.9% or a deliberately-chosen fraction of the remaining error budget), with noise tolerance handled by the separate failureLimit/interval knobs this chapter covers, not by inflating the acceptable failure rate itself.

Problem 5: A team wants to validate a new recommendation algorithm specifically among users who've opted into a "early access" beta program (identified by a feature-flag-set cookie), while simultaneously running an ordinary, broader canary rollout of an unrelated infrastructure change to the same service. Can both progressive delivery patterns run at once, and how would you reason about it?

Answer: Yes, and reasoning through why clarifies the real distinction between the two patterns covered in this chapter: the A/B test (cookie-based routing to beta users) and the canary (percentage-based rollout of the infrastructure change) are answering genuinely independent questions — "does this specific cohort have the intended recommendation experience" versus "is this infrastructure change broadly safe for arbitrary traffic" — and neither one's routing logic needs to know about the other. In practice this is commonly implemented as two separate concerns layered together: the A/B cookie-based routing determines which version of the recommendation algorithm a given request sees, while the canary's percentage-based traffic split (potentially orthogonal, e.g. applied to the underlying service infrastructure both algorithm versions run on) determines what fraction of all requests — beta or not — hit the newly-updated infrastructure. The two mechanisms compose because they're solving genuinely different problems along different axes, not competing implementations of the same concern.

Problem 6: A team runs production across three regional Kubernetes clusters and wants a new version to roll out to eu-west only after us-east's own canary has fully, successfully promoted — never in parallel, and never skipping a region even if an engineer is in a hurry. Sketch how this would actually be implemented, given that neither Argo Rollouts nor Flagger natively coordinates across clusters.

Answer: Per this chapter's multi-cluster section, the coordination layer has to live above both tools, in a CI/CD pipeline (any platform from Parts 4-11) or a custom controller — a pipeline stage triggers (or lets GitOps trigger, per Part 3) the us-east cluster's Rollout, then polls its status via kubectl argo rollouts status checkout-service --context us-east --watch (or the cluster-appropriate equivalent), blocking the pipeline from proceeding until that command reports the rollout Healthy. Only once that gate passes does the pipeline proceed to trigger the eu-west cluster's Rollout the same way, repeating for each subsequent region. This is architecturally identical to any other multi-stage pipeline with sequential dependencies already covered across this series (Part 8's Azure dependsOn, Part 4's GitHub needs:) — each region is simply one more stage, gated on the previous stage's real, externally-observed success rather than a CI-platform-internal job result.

Problem 7: A team's Argo Rollouts AnalysisTemplate uses successCondition: result[0] >= 0.98 for HTTP success rate, and a canary rollout for their message-queue worker fleet keeps getting stuck — the Rollout never progresses past its first step, and no analysis result is ever reported at all, not even a failure. Diagnose the likely cause.

Answer: This is almost certainly a category-error configuration mistake, not a transient issue — the AnalysisTemplate's Prometheus query is very likely written against an HTTP-request metric (e.g. http_requests_total) that a queue-consuming worker, per this chapter's non-HTTP workloads section, simply never emits at all, since it doesn't serve HTTP requests. A query against a metric that never has any data doesn't return a clean "0% success" failure — depending on the exact PromQL, it more commonly returns no data at all, which Argo Rollouts can't evaluate against successCondition in the first place, leaving the analysis effectively stalled rather than explicitly failing. The fix is switching to metrics the worker actually emits (queue processing success/failure counts, dead-letter-queue depth) rather than an HTTP-shaped query copied from an unrelated HTTP service's own AnalysisTemplate — the underlying lesson being that analysis configuration has to match the actual workload's real observability surface, not be assumed generic across every workload type in the cluster.

---

Summary and What's Next#

Progressive delivery automates the promotion decision Part 1 first introduced conceptually — Argo Rollouts and Flagger are the two dominant Kubernetes-native tools that watch real production metrics during a canary or blue-green rollout and automatically promote or roll back, with no human required to watch a dashboard and make the call. The two tools differ architecturally at their core: Argo Rollouts replaces the Deployment kind with its own Rollout CRD, gaining deeper integration (especially with Argo CD) at the cost of a real migration; Flagger wraps an existing, unmodified Deployment via a separate Canary CRD, with a lower migration barrier and a genuinely flexible webhook system for custom validation logic beyond pure metric thresholds. Both depend on a real traffic-management layer (a service mesh or Gateway API) for precise, request-level percentage splitting rather than a coarse, Pod-count-based approximation, and both are deliberately Kubernetes-specific, requiring a different mechanism entirely for VM-based or serverless workloads. The Experiment CRD provides genuine side-by-side comparison without any promotion trajectory at all, distinct from both canary and blue-green; analysis thresholds should be derived from a service's existing SLO and error budget rather than chosen independently; and database schema changes require expand/contract discipline for the entire window two versions might coexist, since a canary can otherwise cause the very incident it exists to prevent. Fixed-threshold analysis is the right default, with Kayenta-style statistical comparison reserved for services whose baseline volatility genuinely demands it, and multi-cluster/multi-region coordination is a layer built above both tools using the same CI/CD-pipeline-orchestration patterns already established throughout this series.

Progressive delivery composes cleanly with every other layer this series has covered: CI/CD pipelines (Parts 4-11) validate and package a change, GitOps (Part 3) declares and reconciles the desired state, and progressive delivery (this chapter) safely, gradually exposes that state to real traffic — three independently replaceable layers connected by a clean Git-commit interface. A/B testing, the third pattern alongside canary and blue-green, answers a genuinely different validation question (a specific cohort's experience, not broad safety) using the same underlying tooling, and ephemeral PR preview environments extend the same "run the real thing before it reaches real traffic" principle all the way back to the code-review stage itself.

Part 12's monorepo affected-detection and this chapter's per-service progressive delivery compose the same way: a single monorepo commit touching several services correctly triggers several entirely independent, independently-paced canary rollouts — one per affected service — rather than one artificially synchronized rollout across all of them, directly consistent with Part 12's own independent-deploys principle.

Part 14, the final chapter in this series, covers Self-Hosted Runner Scaling & Cost Optimization — the Kubernetes-native autoscaling patterns referenced piecemeal across every platform chapter in this series (GitHub, GitLab, Bitbucket, Azure, Jenkins, CircleCI, Tekton's own execution models), now covered in full, unified depth. It closes this series' full arc: Part 1 through this chapter built the tool-agnostic model and its concrete platform realizations; Part 14 turns to the underlying compute infrastructure every one of those platforms ultimately depends on to actually execute a pipeline at all.

Carry forward from this chapter the same underlying discipline that will recur one final time in Part 14: automate the decision wherever a clear, reliable signal exists, and be explicit and honest about the real infrastructure cost of the tooling that makes that automation possible.