BuildConfigs, S2I, ImageStreams & OpenShift Pipelines/GitOps

Table of Contents#

1. From Traffic to Delivery: What This Chapter Covers
2. BuildConfigs — OpenShift's Built-In Build Mechanism
3. Source-to-Image (S2I) — How It Actually Works
4. A Worked Example: An S2I Build End to End
5. Docker Strategy and Custom Strategy Builds
6. ImageStreams — OpenShift's Image Abstraction
7. The Integrated Internal Registry
8. Image Change Triggers — Automatic Rebuilds and Rollouts
9. Build Strategies for CI: Where BuildConfigs Stop and Pipelines Start
10. OpenShift Pipelines — Tekton on OpenShift
11. Task, Pipeline, and PipelineRun — The Object Model
12. A Worked Example: A Tekton Pipeline Building and Deploying
13. Tekton Triggers — Reacting to Webhooks
14. OpenShift GitOps — Argo CD on OpenShift
15. The Application Object and Sync Policies
16. The App-of-Apps Pattern
17. Progressive Delivery: Argo Rollouts and Route Weighting Revisited
18. Putting It Together: A Full CI/CD Reference Architecture
19. Quick Reference: Key Terms From This Chapter
20. Common Mistakes and Interview Traps
21. Worked Practice Problems
22. Summary and What's Next

From Traffic to Delivery: What This Chapter Covers#

Part 3 closed the loop on how traffic reaches a running workload. This chapter goes one step further back: how does that workload's container image get built and land on the cluster in the first place? OpenShift's answer spans two generations of the same underlying goal — turning source code into a running, updated deployment — and both are still in active use across real production clusters today, which is why this chapter covers both rather than treating one as purely historical.

The first generation, BuildConfigs and Source-to-Image, is OpenShift-native, predates Kubernetes-standard CI/CD tooling, and remains genuinely useful for its original goal: letting a developer go from a Git repository to a running container without writing a Dockerfile or operating any CI infrastructure at all. The second generation, OpenShift Pipelines (Tekton) paired with OpenShift GitOps (Argo CD), is the Kubernetes-native, vendor-neutral approach most new production pipelines are actually built on today — CI as a series of Kubernetes-native objects, and CD as continuous reconciliation from a Git repository, rather than an imperative build-then-deploy script.

Neither replaces the other outright on a real cluster — many organizations use BuildConfigs for quick internal tools and prototypes while running Tekton/Argo CD for anything with real production delivery requirements, and this chapter covers both in the order a team is likely to actually encounter them.

From the Trenches: A platform team standardized every new service on Tekton/Argo CD from day one, including a handful of genuinely small internal tools (an admin dashboard, a one-off reporting script) that a single developer maintained alone. Each of those small tools ended up carrying its own multi-stage Pipeline, its own manifests repository, and its own Application object — infrastructure genuinely proportioned for a team's production service, not a single-developer internal tool nobody else touched. The fix wasn't abandoning Tekton/Argo CD for the team's real production services, where the investment was clearly justified — it was explicitly carving out oc new-app/BuildConfig as the sanctioned path for anything below a stated complexity threshold (single deployable, no multi-stage requirements, one owner), giving both tools a genuine, deliberately-scoped place in the same organization rather than forcing every workload through the heavier tool by default.

BuildConfigs — OpenShift's Built-In Build Mechanism#

A BuildConfig is OpenShift's declarative description of how to turn source into an image: where the source comes from (a Git repository, a binary artifact, or inline Dockerfile content), which strategy builds it (S2I, Docker, or Custom), and where the resulting image goes (almost always an ImageStream, covered later in this chapter).

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: web
  namespace: payments-dev
spec:
  source:
    type: Git
    git:
      uri: https://github.com/myorg/web.git
      ref: main
  strategy:
    type: Source
    sourceStrategy:
      from:
        kind: ImageStreamTag
        name: nodejs:20-ubi9
        namespace: openshift
  output:
    to:
      kind: ImageStreamTag
      name: web:latest
  triggers:
    - type: ConfigChange
    - type: ImageChange

Every Build (a single execution of a BuildConfig, conceptually parallel to a PipelineRun being one execution of a Pipeline) runs as an ordinary Pod on the cluster — subject to the same SCCs, quotas, and NetworkPolicy from Parts 2 and 3 as any other workload, a detail worth remembering the first time a build fails with a familiar-looking admission or resource-quota error rather than an application-specific one.

Triggers in Depth: Webhook, ConfigChange, and ImageChange#

A BuildConfig's triggers array can combine more than one trigger type, each firing independently:

spec:
  triggers:
    - type: GitHub
      github:
        secretReference: { name: web-github-webhook-secret }
    - type: Generic
      generic:
        secretReference: { name: web-generic-webhook-secret }
    - type: ConfigChange
    - type: ImageChange
      imageChange: {}

ImageChange in particular is worth internalizing as a supply-chain-relevant behavior: a BuildConfig referencing nodejs:20-ubi9 from the openshift namespace rebuilds automatically whenever Red Hat publishes a new patched version of that builder image, meaning a CVE fix in the base image propagates to every dependent application build without a developer needing to notice or manually trigger anything — the same "keep base images current automatically" discipline this catalog's Docker Container Fundamentals series recommends, provided here as a built-in platform behavior rather than something a team has to wire up itself.

Build Hooks: Running a Command Against the Built Image#

A post-commit hook runs a command against the newly-built image before it's pushed, most commonly a quick smoke test — and critically, a failing hook aborts the push entirely, so a broken build never reaches the ImageStream (and downstream deployments) at all:

spec:
  postCommit:
    script: "npm test"

This is a lightweight alternative to a full test stage in a Tekton pipeline (covered later in this chapter) for teams that want a minimal smoke-test gate without standing up separate pipeline infrastructure — a legitimate middle ground between "no verification at all" and "a full multi-stage pipeline," worth reaching for specifically when a team's actual need is that narrow.

Source-to-Image (S2I) — How It Actually Works#

S2I is the mechanism behind strategy.type: Source in the example above, and it's worth understanding the actual assemble process rather than treating it as a black box, since the failure modes make a lot more sense once the mechanism is visible.

Every S2I builder image (the nodejs:20-ubi9 reference above, or equivalents for Python, Java, Ruby, Go, PHP, .NET) implements two specific scripts as part of its own contract: assemble (how to turn source into a runnable artifact — npm install, mvn package, pip install -r requirements.txt, whatever fits that language ecosystem) and run (how to actually start the built application). This is the entire mechanism — no Dockerfile is written or needed, since the builder image already encodes the language-specific build knowledge, and the developer only supplies source code.

Chained and Incremental Builds#

A chained build links two BuildConfigs together — commonly, an S2I build producing a compiled artifact, feeding into a second build that copies just that artifact into a minimal runtime image, mirroring the multi-stage-Dockerfile pattern this catalog's Docker series recommends, but expressed as two cooperating BuildConfigs instead of one Dockerfile's stages:

spec:
  source:
    type: Image
    images:
      - from: { kind: ImageStreamTag, name: "web-builder:latest" }
        paths:
          - { sourcePath: "/opt/app/dist", destinationDir: "." }

An incremental build is a different, orthogonal optimization: it reuses specific artifacts (most commonly a language ecosystem's dependency cache — node_modules, .m2) from the previous successful build of the same BuildConfig, meaningfully speeding up repeated builds of the same application without needing a chained second BuildConfig at all:

spec:
  strategy:
    sourceStrategy:
      incremental: true

Both mechanisms solve different problems — chained builds separate build-time and runtime image concerns cleanly; incremental builds speed up repeated builds of the same application — and a team optimizing build performance should know which one actually addresses their specific bottleneck before reaching for either.

From the Trenches: A team's Node.js S2I build started failing intermittently with an out-of-memory error during npm install, despite the application itself being lightweight and running fine once built. The root cause was that the build Pod inherited the namespace's default LimitRange request/limit — sized for the running application's modest steady-state memory use, not for npm install's own memory-hungry dependency-resolution phase, which needed meaningfully more headroom than the application itself ever would. The fix was an explicit resources override on the BuildConfig itself (a field builds support independently of the namespace default), rather than raising the whole namespace's LimitRange default for every workload just to accommodate one build's transient resource spike.

A Worked Example: An S2I Build End to End#

oc new-app nodejs:20-ubi9~https://github.com/myorg/web.git --name=web

oc logs -f bc/web

oc get imagestreamtag web:latest -o jsonpath='{.image.dockerImageReference}'
oc start-build web --follow

oc new-app's ~ syntax (a builder ImageStreamTag, a tilde, then a Git URL) is the fastest path from zero to a running S2I build — it generates the BuildConfig, the destination ImageStream, a Deployment referencing that ImageStream, and a Service, wiring image-change triggers (covered later in this chapter) automatically so future pushes to main trigger a rebuild and redeploy without further configuration. oc logs -f bc/web streams the currently-running build's own logs directly — the fastest way to watch an assemble script's actual output as it happens, rather than waiting for the build to finish and only then discovering it failed.

A build that fails outright, rather than just running slowly, is worth triaging with the same layered instinct Part 3 taught for networking: oc get build (is this even the most recent build, or a stale one from before a fix), oc logs build/<build-name> (the actual assemble output, usually naming the failure directly), and — if the build never even starts — oc get events -n payments-dev --sort-by='.lastTimestamp' for an admission-level rejection (an SCC denial or exhausted quota from Part 2, not a code problem at all).

Docker Strategy and Custom Strategy Builds#

S2I isn't the only BuildConfig strategy, and knowing when to reach for the other two avoids forcing a Dockerfile-based build through a mechanism it wasn't designed for.

spec:
  strategy:
    type: Docker
    dockerStrategy:
      dockerfilePath: Dockerfile

A Docker strategy build still runs inside OpenShift's build infrastructure (subject to the same SCC/quota constraints as an S2I build) but interprets the repository's own Dockerfile directly, using this catalog's Docker Container Fundamentals series' multi-stage-build and layer-caching guidance unchanged — nothing about running on OpenShift changes how a Dockerfile itself is written or optimized. Choosing between Source and Docker strategy is largely a question of whether an existing Dockerfile already exists and is worth preserving, versus starting fresh with a supported S2I builder image's batteries-included convenience.

A Custom strategy build is worth treating as a genuine escape hatch rather than a default choice: it hands the build process a privileged-adjacent level of control (the custom builder image effectively drives the entire build itself, rather than filling in a well-defined assemble/run contract), which means it typically needs a broader SCC grant than either Source or Docker strategy builds do — the same "narrowest grant that satisfies the actual requirement" discipline from Part 2 applies here just as directly as it does to any other elevated-privilege workload.

ImageStreams — OpenShift's Image Abstraction#

An ImageStream is OpenShift's answer to a problem plain image tags handle poorly: a tag like web:latest is a mutable pointer that can silently repoint to a different image, with no built-in history of what it used to point to and no native way to trigger a downstream action when it changes. An ImageStream is a collection of ImageStreamTags, each an internal, versioned pointer to a specific image, decoupled from wherever that image is actually stored (the internal registry, or any external registry).

apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  name: web
  namespace: payments-dev
spec:
  tags:
    - name: latest
      from:
        kind: DockerImage
        name: quay.io/myorg/web:v3

oc tag quay.io/myorg/web:v4 web:latest -n payments-dev
oc get istag web:latest -o jsonpath='{.image.dockerImageReference}{"\n"}{.image.metadata.created}'
oc tag payments-dev/web:latest payments-dev/web:rollback-candidate

Every retagging operation is recorded — oc describe is web shows the full history of every image an ImageStream's tag has ever pointed to, giving a genuine audit trail plain Docker tags never provide, and letting a rollback target a specific prior ImageStreamTag reference directly rather than needing to remember or look up an external registry's own tag history.

Scheduled Import — Tracking an External Tag Automatically#

An ImageStream referencing an external image (quay.io/myorg/web:v3, as in the example above) can also be configured to periodically re-check that external reference and update its own ImageStreamTag automatically if the upstream tag has moved — useful for tracking a fast-moving external base image without a human running oc import-image by hand on a schedule:

spec:
  tags:
    - name: latest
      from: { kind: DockerImage, name: "registry.access.redhat.com/ubi9/nodejs-20:latest" }
      importPolicy:
        scheduled: true
      referencePolicy:
        type: Local

referencePolicy.type: Local is worth calling out specifically: it tells the internal registry to actually pull and cache a local copy of the external image rather than only storing a pointer to it — meaning a Deployment referencing this ImageStreamTag keeps working even if the external registry becomes temporarily unreachable, a small but genuine resilience gain for any base image sourced from a registry the platform team doesn't control the uptime of.

The Integrated Internal Registry#

Every OpenShift cluster ships an integrated internal registry (the image-registry Cluster Operator from Part 1's roster), reachable in-cluster at image-registry.openshift-image-registry.svc:5000 and, if exposed, externally through a Route. Pushing an image to this registry and creating a matching ImageStream are two sides of the same coin: pushing an image to the registry automatically creates (or updates) a matching ImageStreamTag, and a BuildConfig outputting to an ImageStreamTag automatically pushes the resulting image to this same registry — the two are deliberately, tightly coupled for images that live inside the cluster's own registry, though an ImageStream can equally reference an image hosted entirely externally (as the quay.io/myorg/web:v3 example above does), with no coupling to the internal registry required at all.

A cluster's image-registry Cluster Operator can be configured with managementState: Removed for clusters that deliberately don't want an internal registry at all (relying entirely on an external one) — a legitimate choice for organizations already standardized on Quay or a cloud-native registry, though it forgoes the ImageStream auto-coupling convenience described above for any image built inside the cluster.

Image Pruning — The Internal Registry Grows Without Bound Otherwise#

Every build and every oc tag operation adds another image (or another reference to one) into the internal registry's storage — with no automatic cleanup, a registry backing years of daily builds accumulates a large volume of images nothing references anymore, purely because nothing ever removed them. oc adm prune images is the built-in mechanism for reclaiming that space:

oc adm prune images --keep-tag-revisions=3 --keep-younger-than=48h --confirm
oc adm prune images --registry-url=image-registry.openshift-image-registry.svc:5000 \
  --keep-tag-revisions=3 --keep-younger-than=48h --confirm

--keep-tag-revisions=3 keeps the three most recent images behind each tag (enough for a quick rollback without keeping every historical build indefinitely), and --keep-younger-than=48h protects anything created recently regardless of tag history, avoiding a race where a build in progress gets pruned before it's even finished being referenced. Running this on a schedule (a CronJob, or as a step in the Day-2 maintenance routines Part 5 covers) is a genuinely necessary piece of cluster housekeeping that's easy to forget entirely until registry storage usage becomes a real, visible problem.

Image Change Triggers — Automatic Rebuilds and Rollouts#

The triggers field on the BuildConfig from earlier in this chapter, and an equivalent trigger on a DeploymentConfig (OpenShift's original, now largely superseded Deployment-equivalent object; a plain Kubernetes Deployment needs a small amount of extra wiring, covered below, to get the same behavior) is what makes "push new source, get a new running deployment" fully automatic, with no external CI system required at all.

A plain Kubernetes Deployment has no native concept of an ImageStream trigger — it references an image by a literal tag/digest string in its Pod template, with nothing watching for that reference to change. OpenShift closes this gap for Deployment objects the same way it does for DeploymentConfig, through a separate ImageChange trigger annotation the image-registry/build machinery understands:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  annotations:
    image.openshift.io/triggers: >-
      [{"from":{"kind":"ImageStreamTag","name":"web:latest"},
        "fieldPath":"spec.template.spec.containers[?(@.name==\"web\")].image"}]

This annotation is what lets a plain, portable Deployment still benefit from ImageStream-driven automatic rollout — the trigger machinery patches the Deployment's container image field directly whenever the named ImageStreamTag updates, without requiring the older, OpenShift-specific DeploymentConfig object at all.

Build Strategies for CI: Where BuildConfigs Stop and Pipelines Start#

BuildConfigs solve "turn source into an image" well, but they were never designed to express a genuinely multi-stage delivery pipeline: run unit tests, then a security scan, then build, then deploy to staging, then run integration tests, then promote to production — each stage potentially needing different tooling, different approval gates, and different failure-handling logic. A BuildConfig's own triggers and Custom strategy can be stretched to approximate parts of this, but it's the wrong tool for genuinely multi-stage orchestration, which is exactly the gap OpenShift Pipelines fills.

OpenShift Pipelines — Tekton on OpenShift#

OpenShift Pipelines is Red Hat's supported distribution of Tekton, installed via OLM (Part 1) like any other Operator. Every Tekton pipeline stage runs as an ordinary Kubernetes Pod — there is no separate CI server process at all, unlike a traditional Jenkins-style architecture where a central controller schedules work onto agent nodes it manages itself. This is the core architectural difference worth naming precisely in an interview: Tekton's "CI server" is the Kubernetes API server plus a set of CRDs and controllers, meaning pipeline execution inherits Kubernetes' own scheduling, RBAC, and resource-quota model automatically, rather than needing a parallel permission and capacity model bolted on.

Task, Pipeline, and PipelineRun — The Object Model#

Four objects compose the whole model, each a pure, declarative definition with no execution logic of its own — the actual work only happens once a Run object is created:

Workspaces are the mechanism Tasks use to share data — a "clone the repo" Task and a "build the image" Task both need access to the same checked-out source, and a shared Workspace (backed by a real PVC) is how that handoff happens, with the Pipeline's own runAfter ordering guaranteeing the clone completes before the build Task starts reading from it.

Reusing Tasks Instead of Reinventing Them#

The git-clone, buildah, and openshift-client Tasks referenced throughout this chapter aren't hand-authored from scratch — they're pulled from the Tekton Hub, a community catalog of pre-built, versioned, reusable Tasks covering the overwhelming majority of common CI needs (cloning from every major Git provider, building with every major tool, scanning with common security tools, notifying Slack). Installing one is as simple as:

tkn hub install task git-clone
tkn hub install task buildah
tkn hub install task openshift-client
tkn hub search --kinds task --tags security
tkn hub info task buildah

The practical guidance worth internalizing: writing a custom Task from scratch should be the exception, reserved for genuinely organization-specific steps (a proprietary internal tool, a bespoke compliance check) — reaching for the Hub first, the same "reuse before inventing" discipline this catalog applies to the topic registry and RBAC roles, avoids a team quietly re-implementing (and having to maintain) a worse version of something already solved and shared across the whole Tekton community.

Build Security: Buildah and SCCs, Revisited#

Part 2's SCC coverage flagged that granting a workload root or extra capabilities should always be scoped narrowly and justified concretely — a Tekton pipeline's buildah step is exactly the kind of case that needs a real, specific answer rather than a blanket anyuid grant reached for by habit. buildah can run fully rootless under the default restricted-v2 SCC in modern OpenShift Pipelines releases (via user namespaces and fuse-overlayfs), meaning a build pipeline's image-build step no longer requires the broader anyuid/privileged grants older Buildah/Docker-in-Docker setups needed — worth confirming against the specific OpenShift Pipelines version in use, since this is exactly the kind of capability that's improved release over release and is easy to carry an outdated assumption about from an older cluster or an older tutorial.

A Worked Example: A Tekton Pipeline Building and Deploying#

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: build-and-deploy
spec:
  workspaces:
    - name: shared-source
  params:
    - name: git-url
    - name: image-name
  tasks:
    - name: fetch-source
      taskRef: { name: git-clone }
      workspaces: [{ name: output, workspace: shared-source }]
      params: [{ name: url, value: $(params.git-url) }]
    - name: build-image
      taskRef: { name: buildah }
      runAfter: ["fetch-source"]
      workspaces: [{ name: source, workspace: shared-source }]
      params: [{ name: IMAGE, value: $(params.image-name) }]
    - name: deploy
      taskRef: { name: openshift-client }
      runAfter: ["build-image"]
      params:
        - name: SCRIPT
          value: "oc set image deployment/web web=$(params.image-name)"

tkn pipeline start build-and-deploy \
  -w name=shared-source,claimName=pipeline-source-pvc \
  -p git-url=https://github.com/myorg/web.git \
  -p image-name=image-registry.openshift-image-registry.svc:5000/payments-dev/web:latest

tkn pipelinerun logs -f --last
tkn pipelinerun list

Debugging a Failed PipelineRun#

tkn pipelinerun logs streams every TaskRun's output as it happens, but a PipelineRun that fails partway through needs a slightly more targeted diagnosis path than scrolling through combined logs from every stage:

tkn pipelinerun describe --last

oc get taskrun -l tekton.dev/pipelineRun=build-and-deploy-run-abc12

oc logs -f taskrun/build-and-deploy-run-abc12-build-image-pod --all-containers

oc get events -n payments-dev --sort-by='.lastTimestamp' | grep -i pipelinerun

tkn pipelinerun describe surfaces exactly which Task in the graph failed and why (a specific Step's non-zero exit code, most commonly), before diving into that one TaskRun's own Pod logs directly — since each Step within a Task runs as its own container within one Pod, --all-containers is worth including by default, since the actual failure can be in an earlier Step (a failed dependency install) whose own container has already exited by the time a later Step's failure is what's visible without it.

buildah here is worth calling out specifically: it's the standard, daemonless image-build tool Tekton pipelines use for the "build an image" step, since running a full Docker daemon inside a build Pod would require privileged access this series' Part 2 SCC coverage specifically works to avoid granting broadly — buildah builds OCI images without needing a persistent privileged daemon process at all, fitting cleanly within a restricted-v2-adjacent security posture (an appropriately-scoped SCC is still typically needed, but a materially narrower one than a full Docker-in-Docker setup would require).

Tekton Triggers — Reacting to Webhooks#

A Pipeline needs something to actually start it — Tekton Triggers (a separate, related project, also installed as part of OpenShift Pipelines) listens for incoming webhooks (a Git provider's push/PR event) and creates a PipelineRun in response, closing the same "push to Git, get a build" loop BuildConfig's own Git webhook trigger provided, but now for a full multi-stage Tekton pipeline instead of a single build step.

apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
  name: build-and-deploy-template
spec:
  params:
    - name: git-revision
  resourcetemplates:
    - apiVersion: tekton.dev/v1
      kind: PipelineRun
      metadata:
        generateName: build-and-deploy-run-
      spec:
        pipelineRef: { name: build-and-deploy }
        params:
          - name: git-url
            value: https://github.com/myorg/web.git

An EventListener object exposes an HTTP endpoint (typically fronted by a Route, per Part 3) that a Git provider's webhook configuration points at directly — every incoming push event creates a new PipelineRun from the TriggerTemplate, with the specific commit/branch information threaded through as parameters.

A TriggerBinding sits between the raw webhook payload and the TriggerTemplate above, extracting the specific fields (commit SHA, branch name, pusher identity) a pipeline actually needs out of a Git provider's own JSON payload shape — decoupling the TriggerTemplate's own parameters from any one provider's specific webhook format, so the same TriggerTemplate can be reused across GitHub, GitLab, or Bitbucket webhooks with only the TriggerBinding needing a provider-specific version.

OpenShift GitOps — Argo CD on OpenShift#

Everything up to this point covers CI — turning source into a built, tested artifact. OpenShift GitOps, Red Hat's supported distribution of Argo CD, covers CD — and it works on a genuinely different model worth being precise about: rather than a pipeline pushing a new deployment onto the cluster imperatively, Argo CD continuously reconciles the cluster's actual state toward whatever a Git repository declares it should be, the same reconciliation-loop philosophy Part 1 established for the CVO/MCO, now applied to application deployment manifests instead of platform components.

This model has a direct, practical consequence worth internalizing: a manual oc edit or oc scale against a resource Argo CD manages is not a permanent change — it's drift, and Argo CD (with selfHeal enabled, covered next) will revert it back to whatever Git declares, the exact same "the reconciler always wins" lesson this series has taught at every other layer (the MCO reverting a manual node change, default ClusterRoles reconciling, a route-controller-manager overwriting a hand-edited generated Route) — now at the application-deployment layer.

argocd app diff web (or the equivalent web console view) is the direct way to see exactly what's currently different between the cluster's live state and Git's declared state for a given Application, before deciding whether that difference is expected in-flight drift, an intentional pending change not yet synced, or a genuine problem worth investigating — reading the diff directly is consistently faster than guessing from symptoms alone which specific field changed.

The Application Object and Sync Policies#

An Argo CD Application is the core object: it names a Git repository, a path within it, and a destination cluster/namespace, and Argo CD's job is keeping that destination matching that Git path continuously.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web
  namespace: openshift-gitops
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/web-manifests.git
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: payments-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

prune: false (the implicit default if omitted) is a common, deliberate choice for a cautious rollout of GitOps to an existing cluster — it lets Argo CD apply and update resources from Git without ever deleting anything on its own, until a team is confident enough in the Git repository's own completeness to let pruning run unattended.

Manual Sync and Sync Windows#

Not every Application should auto-sync immediately — a production Application with no automated policy at all requires an explicit argocd app sync web (or a click in the Argo CD UI) before Git changes take effect, giving a team a deliberate approval gate between "merged to the manifests repository" and "actually live in production," a common, legitimate choice for a namespace where every production change genuinely needs a human's final go-ahead rather than immediate automatic application. Sync windows offer a middle ground for Applications that are auto-synced but only during specific, deliberate time windows (blocking automatic syncs outside a change-freeze period, for instance) — worth knowing as a name for "we want GitOps automation, but not during this specific blackout window," rather than disabling automation entirely to achieve the same effect.

From the Trenches: A team enabled selfHeal: true on an Application managing a namespace that still had several manually-created objects predating the GitOps migration, none of which existed in the Git repository yet. Argo CD didn't touch those objects directly (self-heal only reverts drift on resources it actually manages), but the very next prune: true sync deleted a manually-created ConfigMap an on-call engineer had added days earlier as a genuine emergency fix, because the running application's own manifest in Git had no matching entry — the object simply looked, from Argo CD's perspective, like something that used to be declared and was now removed. The lesson: enabling prune on an Application retroactively pointed at a namespace with pre-existing, undeclared objects requires an inventory pass first — reconcile what's actually running against what Git declares, and import anything still needed into Git, before turning on automatic pruning.

The App-of-Apps Pattern#

A single cluster typically runs dozens of applications, each with its own Application object — managing that many Application objects by hand quickly becomes its own maintenance burden. The app-of-apps pattern solves this by making the Application objects themselves GitOps-managed: one root Application points at a Git path containing nothing but other Application manifests, so adding a new application to the cluster is a single commit adding one more Application YAML file, rather than a manual oc apply a platform team has to remember to run.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: openshift-gitops
spec:
  source:
    repoURL: https://github.com/myorg/platform-gitops.git
    path: applications
  destination:
    server: https://kubernetes.default.svc
    namespace: openshift-gitops
  syncPolicy:
    automated: { selfHeal: true }

Every file under applications/ in that repository is itself an Application manifest — the root Application's own reconciliation loop is what creates and manages every child Application, which in turn manages its own actual workload manifests, giving a fully GitOps-managed hierarchy from the platform's own bootstrapping all the way down to individual application deployments, with a single Git repository as the one source of truth for the whole tree.

Generating Applications at Scale: ApplicationSet#

App-of-apps solves "many applications on one cluster" by making each one an explicit file; ApplicationSet solves a related but distinct scaling problem — the same application deployed across many clusters (a common pattern for an organization running the cluster-per-team or cluster-per-region topologies Part 2 discussed), without hand-writing one Application object per cluster:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: web-fleet
  namespace: openshift-gitops
spec:
  generators:
    - clusters: {}
  template:
    metadata:
      name: "web-{{name}}"
    spec:
      source:
        repoURL: https://github.com/myorg/web-manifests.git
        path: overlays/production
      destination:
        server: "{{server}}"
        namespace: payments-prod

The clusters: {} generator produces one Application per cluster Argo CD has registered — adding a new cluster to the fleet, then, automatically produces a new Application for web-fleet targeting it, with zero additional YAML to write per cluster. Other generator types (a Git directory listing, a plain list, a matrix combining two generators) cover the "many applications across many clusters" case ApplicationSet was specifically built to scale to.

Secrets in GitOps — the One Thing Git Genuinely Can't Hold in Plain Text#

Every mechanism this chapter covered assumes Git is the single source of truth — which raises an obvious, real problem: a Kubernetes Secret object's data is only base64-encoded, not encrypted, so committing one directly to a Git repository (even a private one) is a genuine credential leak, not just a style violation. Two complementary tools solve this without breaking the GitOps model's core promise:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: payments-prod
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEQ...

The SealedSecret object above is genuinely safe to commit to a public repository — only the cluster holding the matching private key can ever decrypt it back into a usable Secret, which is exactly what lets Argo CD manage secrets through the same Git-reconciliation model as every other resource in this chapter, without ever storing a plaintext credential in version control.

Both approaches share one operational consequence worth planning for explicitly: rotating the cluster's own Sealed Secrets private key, or losing access to the external secret manager, breaks decryption for every secret sealed against it — a disaster-recovery plan for a GitOps-managed cluster needs to account for this key/credential dependency specifically, not just for the Git repository and cluster state that most disaster-recovery planning already covers by default.

Progressive Delivery: Argo Rollouts and Route Weighting Revisited#

Part 3 introduced a Route's weighted alternateBackends as a simple, manual canary mechanism. Argo Rollouts (a related Argo project, often installed alongside OpenShift GitOps) automates exactly that same underlying mechanism — progressively shifting a Route's (or Service mesh's) traffic weight from a stable to a canary version, automatically analyzing success-rate/latency metrics at each step, and automatically rolling back if the canary's metrics regress, replacing a plain Deployment with a Rollout custom resource that expresses the same desired end state plus the strategy for getting there safely.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web
spec:
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

The direct line from Part 3's manual weighted Route to this object is worth keeping explicit: Rollout doesn't invent a new traffic-splitting mechanism — it drives the exact same Route-weight (or mesh VirtualService-weight) primitive automatically, on a schedule, with a pause-and-analyze step between each increment, turning a manual "watch dashboards, manually bump the weight" process into a codified, repeatable one.

Automated Analysis and Automatic Rollback#

An AnalysisTemplate is what turns a Rollout's pause steps from a fixed timer into a genuine, metrics-driven gate:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: error-rate
      interval: 1m
      successCondition: result < 0.05
      provider:
        prometheus:
          address: http://prometheus-k8s.openshift-monitoring.svc:9091
          query: |
            sum(rate(http_requests_total{status=~"5.."}[1m]))
            /
            sum(rate(http_requests_total[1m]))

Referenced from a Rollout's canary steps, this AnalysisTemplate queries the cluster's own built-in Prometheus (Part 5 covers this stack in depth) after each weight increment, and automatically aborts and rolls back the rollout if the canary's real error rate exceeds the threshold — closing the loop from "we shifted 10% of traffic to the new version" to "we shifted it back automatically the moment it looked worse," with no human needing to be watching a dashboard in real time for the rollback to happen promptly.

Putting It Together: A Full CI/CD Reference Architecture#

Every mechanism this chapter covered composes into one coherent, real production pipeline:

The separation between the application source repository and the manifests repository (the step where the pipeline updates an image tag in a separate Git repository, which Argo CD watches) is a deliberate, common GitOps pattern rather than an accident of this diagram — it keeps CI's fast-moving application-code changes decoupled from CD's slower, more deliberately-reviewed deployment-configuration changes, and gives a platform team a single, clean Git history of exactly what was deployed to production and when, independent of the application's own commit history.

Choosing the Right Depth of Tooling for the Actual Requirement#

Not every application needs the full reference architecture above, and over-provisioning delivery infrastructure for a workload that doesn't need it wastes real setup and maintenance effort without a corresponding benefit — the same opinionation-vs-overhead judgment call this series has applied at every other layer:

Reading this table top to bottom mirrors how a real organization's delivery tooling should actually grow over time — starting simple and adding a layer specifically when a concrete requirement demands it, rather than adopting the full stack for every workload from day one regardless of its actual complexity.

A Single-Pane Command Reference for This Chapter's Full Stack#

A quick reference worth keeping at hand while operating the reference architecture above, spanning every tool this chapter introduced:

# BuildConfig / S2I
oc get bc,builds -n payments-dev
oc start-build web --follow
oc logs -f bc/web

# ImageStreams
oc get is,istag -n payments-dev
oc describe is web
oc adm prune images --keep-tag-revisions=3 --confirm

# Tekton
tkn pipeline list
tkn pipelinerun logs -f --last
tkn taskrun describe --last
tkn task list

# Argo CD
argocd app list
argocd app diff web
argocd app sync web
argocd app history web
argocd app rollback web

Each command maps directly onto one of this chapter's own object models — worth treating this block as a starting checklist during an actual delivery-pipeline incident, working top to bottom until the layer that's actually broken is identified, the same layered-diagnosis discipline Part 3 established for networking incidents.

Quick Reference: Key Terms From This Chapter#

Common Mistakes and Interview Traps#

Worked Practice Problems#

1. A team's S2I build for a Java application fails during the assemble step with an out-of-memory error, despite the deployed application itself running well within its configured memory limit. What's the most likely explanation?#

The most likely explanation is that the build Pod's resource allocation — inherited from the namespace's LimitRange default unless the BuildConfig overrides it explicitly — is sized for the application's own modest runtime memory footprint, not for the assemble step's own resource-hungry phase (Maven's dependency resolution and compilation, in this case), which commonly needs meaningfully more memory than the finished, running application ever will. The fix is an explicit resources override on the BuildConfig itself, matched to the build's actual peak memory need, rather than raising the namespace's general LimitRange default for every workload to accommodate one build's transient spike.

2. A platform team wants developers to be able to deploy a new internal tool from source with minimal setup, but also wants any production-facing service to go through a multi-stage pipeline with security scanning and a manual approval gate. Should the team standardize on one mechanism for both?#

No — this is exactly the split this chapter's opening comparison table describes, and forcing one mechanism to cover both cases would be a worse fit for at least one of them. BuildConfig/S2I fits the internal-tool case well: oc new-app gets a developer from source to a running deployment in one command with no pipeline infrastructure to configure. The production-facing service's requirements — multiple sequential stages, a security scan, and a manual approval gate — are exactly what BuildConfigs are poorly suited to express and what Tekton Pipelines (with a manual-approval mechanism between stages) are purpose-built for. The right answer is using each mechanism for the case it actually fits, not standardizing on one for both.

3. An Application object managing a namespace shows OutOfSync after a platform engineer manually edited a ConfigMap during an incident to apply an emergency fix. selfHeal is enabled. What happens next, and what should the team have done instead?#

With selfHeal: true, Argo CD detects the manual edit as drift from Git's declared state and reverts the ConfigMap back to what Git specifies, typically within seconds of detecting the change — meaning the emergency fix is silently undone, likely before the engineer even confirms it resolved the incident. The correct approach during a genuine incident is either committing the fix to Git directly (letting Argo CD apply it through the normal reconciliation path) or explicitly pausing that Application's auto-sync first if an immediate, Git-bypassing change is truly necessary — treating Argo CD's reconciliation loop as an always-on constraint to work with deliberately, the same lesson this series has taught for the MCO, default ClusterRoles, and the route-controller-manager, now applied to application-level GitOps.

4. A team wants to add automated canary rollouts to an existing production Route that currently uses a hand-maintained weighted alternateBackends split. What does adopting Argo Rollouts actually change about their traffic-splitting mechanism itself?#

Nothing about the underlying mechanism changes — Argo Rollouts drives the exact same Route-weight primitive the team was already adjusting by hand; what changes is who adjusts it and on what basis. Instead of an engineer manually watching dashboards and editing the Route's weight fields at each step, a Rollout object codifies the sequence of weight increments, the pause duration between them, and (if configured with an AnalysisTemplate) an automated metrics check that can halt or roll back the rollout if the canary's error rate or latency regresses — turning a manual, judgment-dependent process into a repeatable, codified one, without introducing any new traffic-routing technology underneath it.

5. A team wants to deploy the same application across twelve regional clusters, each with minor per-region configuration differences, and is currently maintaining twelve hand-written Argo CD Application objects. What's the better pattern, and what does it actually change?#

ApplicationSet with a clusters (or list) generator is the better pattern — it generates one Application per registered cluster from a single template, so adding a thirteenth cluster to the fleet produces its Application automatically rather than requiring a hand-written thirteenth object. What it doesn't change is the underlying reconciliation model: each generated Application still independently reconciles its own cluster destination against Git exactly as a hand-written one would; ApplicationSet only removes the repetitive authoring burden, not the reconciliation mechanism itself. Per-region configuration differences are handled the same way they would be in twelve hand-written Applications — typically via a Kustomize overlay per region referenced in each generated Application's source.path, parameterized by the generator's own per-cluster values.

6. A security review flags that a Tekton pipeline's buildah Task is running under the privileged SCC, and asks whether that's actually necessary. What should the team check before answering?#

The team should check the specific OpenShift Pipelines version in use and whether rootless Buildah (via user namespaces and fuse-overlayfs) is both supported and actually configured for that Task — modern OpenShift Pipelines releases support building images under a much narrower SCC than privileged, and a privileged grant inherited from an older tutorial, an older cluster configuration, or simply never revisited since initial setup is a common, fixable case of exactly the "wider grant than the workload actually needs" pattern Part 2 warned against generally. If rootless building is confirmed supported and working, the fix is re-scoping the Task's Service Account to a narrower SCC and verifying the build still succeeds — not leaving privileged in place because "it's how the pipeline has always been configured."

Summary and What's Next#

This chapter covered two generations of the same underlying problem — turning source into a running, continuously-updated deployment — and the honest trade-off between them: BuildConfigs and Source-to-Image remain genuinely useful for their original goal of near-zero-setup build-from-source, encoding build knowledge in reusable, versioned builder images rather than requiring a hand-written Dockerfile; ImageStreams give OpenShift's image references a versioned history and native trigger support a plain tag never had; and OpenShift Pipelines (Tekton) paired with OpenShift GitOps (Argo CD) form the Kubernetes-native, fully portable approach most new production delivery pipelines are actually built on, with CI expressed as ordinary Kubernetes objects running as ordinary Pods, and CD expressed as continuous reconciliation from Git rather than an imperative deploy script — the same reconciliation philosophy this series has now shown operating at the node level (MCO), the RBAC level (default role reconciliation), the ingress level (route-controller-manager), and now the application-deployment level (Argo CD), with an identical practical consequence every time: a manual change outside the declared source of truth is drift, not a permanent fix, and the reconciler will eventually win.

The right-sized-tooling table from earlier in this chapter is worth carrying forward as a standing check against scope creep: every mechanism this chapter introduced — chained builds, ApplicationSet, Argo Rollouts' automated analysis — solves a genuinely real problem at a genuinely real scale, and adopting any of them before that scale is real trades a concrete, ongoing maintenance cost for a benefit that doesn't exist yet, the same trade this series has flagged at the platform level (Part 1), the multi-tenancy level (Part 2), and the service-mesh level (Part 3).

Part 5 closes the series with Day-2 operations: the Cluster Version Operator's actual upgrade mechanics in practice, the Machine API and cluster autoscaling, the built-in monitoring and logging stacks this chapter's own pipeline observability (and Part 3's Kiali/tracing) ultimately feed into, and the Operator Lifecycle Manager's deeper Day-2 patterns for a platform team running this cluster for years, not just standing it up once.

Cross-reference: this catalog's Docker Container Fundamentals series covers Dockerfile authoring, multi-stage builds, and layer caching referenced throughout this chapter's Docker-strategy coverage — worth revisiting in parallel for the parts of image-building that are genuinely unmodified between OpenShift and a plain docker build workflow.

Sources consulted for this chapter: Red Hat's OpenShift Container Platform Builds documentation (BuildConfig, Source-to-Image, strategies), the OpenShift Images documentation (ImageStreams, the integrated registry, image change triggers), Tekton's own Pipelines and Triggers documentation, and Red Hat's OpenShift Pipelines and OpenShift GitOps documentation and release notes.