GitOps

Table of Contents#

1. What GitOps Actually Is
2. The Four Principles of GitOps
3. Push-Based vs Pull-Based Deployment
4. Why Pull-Based Is Considered More Secure
5. The GitOps Reconciliation Loop
6. ArgoCD — Architecture and Core Concepts
7. A Full Worked ArgoCD Example
8. Flux — The Other Major GitOps Tool
9. Self-Healing — GitOps's Answer to Drift
10. Multi-Environment Promotion With GitOps
11. Secrets in GitOps — A Genuinely Hard Problem
12. Rollbacks in GitOps — Trivially Easy
13. GitOps and Disaster Recovery
14. Common Mistakes
15. Worked Practice Problems
16. Summary — The Complete Automation, CI/CD & GitOps Series

---

What GitOps Actually Is#

GitOps takes the "treat infrastructure as code" principle from Part 2 and adds one crucial, formalizing constraint: Git isn't just WHERE the code lives — it's the single, enforced source of truth for what should actually be running, continuously and automatically reconciled by a dedicated tool.

Diagram

Simple analogy: traditional CI/CD is like a chef who runs out to the walk-in fridge every time an order comes in and grabs whatever ingredients they think they need. GitOps is like a fully-stocked kitchen with a dedicated inventory manager who continuously checks the fridge against a master list and restocks/corrects it automatically the instant anything doesn't match — the chef (external systems) never needs a key to the fridge at all.

---

The Four Principles of GitOps#

Formalized by the GitOps Working Group (part of the CNCF, the same body that hosts Kubernetes and Prometheus) — worth knowing these four principles by name, since they're frequently cited directly in interviews.

Diagram

Notice principle 4 is exactly, precisely the Kubernetes reconciliation loop pattern from the Kubernetes Deep Dive series (Part 1), applied one layer up — from "keep this Deployment's replica count correct" to "keep this entire cluster's configuration correct, matching what's declared in Git."

---

Push-Based vs Pull-Based Deployment#

This distinction (principle 3, above) is genuinely the most important, most commonly-tested architectural difference between traditional CI/CD and GitOps — worth a full, dedicated comparison.

Diagram

Diagram

---

Why Pull-Based Is Considered More Secure#

Directly connecting to the CI/CD pipeline security discussion from the DevSecOps series (Part 5) — this is genuinely one of the strongest, most concrete arguments for GitOps, worth stating explicitly.

Diagram

A genuinely strong, senior-level interview line, directly tying this back to the DevSecOps series' supply chain security discussion: "GitOps directly reduces the blast radius of a compromised CI/CD pipeline — in a push model, a compromised CI system has a direct path to modify production, which is exactly the class of attack that compromised SolarWinds' build system. In a pull model, the CI pipeline never holds production credentials at all; it only ever writes to Git, and the cluster's own internal agent decides what to actually apply — collapsing an entire category of external attack surface."

---

The GitOps Reconciliation Loop#

Diagram

This is, one final time, the exact same observe-compare-act reconciliation loop pattern first introduced in the Kubernetes Deep Dive series — GitOps is best understood not as a brand-new idea, but as that exact pattern applied to an entire environment's configuration, with Git specifically as the desired-state source instead of a Kubernetes object's spec field.

---

ArgoCD — Architecture and Core Concepts#

ArgoCD is the most widely used GitOps tool for Kubernetes — genuinely worth real, hands-on familiarity.

Diagram

Copy
# An ArgoCD Application object — declares WHAT to sync and FROM WHERE
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-service
spec:
  source:
    repoURL: https://github.com/myorg/checkout-manifests.git
    targetRevision: main
    path: k8s/production
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  syncPolicy:
    automated:
      prune: true       # DELETE resources removed from Git
      selfHeal: true     # AUTOMATICALLY revert manual/drifted changes

Notice prune: true and selfHeal: true, worth calling out specifically — these two settings are the concrete, literal implementation of GitOps's core promise: prune means if something is deleted from the Git manifests, ArgoCD deletes it from the real cluster too (Git is the complete, authoritative truth, including what should NOT exist); selfHeal means if someone manually changes something in the cluster directly (the exact drift scenario from Part 2), ArgoCD automatically reverts it back to match Git, without waiting for the next scheduled deploy.

---

A Full Worked ArgoCD Example#

Copy
# Install ArgoCD into a cluster
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Register a new Application, pointing at a Git repo
argocd app create checkout-service \
  --repo https://github.com/myorg/checkout-manifests.git \
  --path k8s/production \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace checkout \
  --sync-policy automated

# Check sync status
argocd app get checkout-service
# Health Status: Healthy
# Sync Status:   Synced

Diagram

A genuinely important, practical detail worth naming: the developer's pull request review process becomes the deployment approval gate itself — this is exactly how a GitOps workflow implements Continuous Delivery's "human still decides when" principle from Part 1, without needing a separate CI pipeline approval step at all: merging the PR IS the approval, and ArgoCD's automated sync is the actual deployment.

---

Flux — The Other Major GitOps Tool#

Flux (also a CNCF project, and in fact one of the two original tools that co-authored the GitOps Working Group's formal principles alongside ArgoCD) is the other dominant GitOps tool, worth knowing by name and roughly how it differs.

Diagram

A balanced, senior-level interview line: "Both ArgoCD and Flux implement the same core GitOps principles and are both CNCF graduated projects — the choice often comes down to whether the team values ArgoCD's richer built-in UI, or prefers Flux's more modular, composable architecture and tighter native Helm/Kustomize integration. Functionally, for the core reconciliation guarantee, they're solving the exact same problem."

---

Self-Healing — GitOps's Answer to Drift#

This is the single most direct, concrete payoff of the entire GitOps model, closing the loop with Part 2's drift discussion.

Diagram

Why this is such a genuinely strong answer to "how does GitOps solve drift," directly contrasting with the manual, disciplined-only approach from Part 2's Terraform discussion: with plain Terraform, avoiding drift depends on TEAM DISCIPLINE — everyone consistently choosing to always go through the code, never the console. With GitOps's self-healing, drift is AUTOMATICALLY, CONTINUOUSLY corrected by the tool itself — the discipline is enforced by the system, not just relied upon from every individual engineer.

---

Multi-Environment Promotion With GitOps#

A practical, common pattern worth knowing — how staging-to-production promotion actually works in a GitOps workflow.

Diagram

The core insight worth stating explicitly: "promoting to production" in a GitOps workflow is nothing more than a Git commit/pull request updating one file — changing the image tag or config value in the production directory to match what was just validated in staging. This is fully auditable (every promotion is a reviewable, permanent git commit), fully reversible (revert the commit to instantly roll back — see below), and requires zero special tooling beyond Git itself and the GitOps agent already watching the repo.

---

Secrets in GitOps — A Genuinely Hard Problem#

A real, important tension worth understanding deeply — directly connecting back to the secrets management discussion from the DevSecOps series (Part 4).

Diagram

Copy
# External Secrets Operator — Git stores only a REFERENCE,
# never the actual secret value
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: checkout-db-credentials
spec:
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: checkout-db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: secret/checkout/db-credentials
        property: password

A strong, senior-level interview line, directly connecting two different tutorials in this course: "GitOps's 'everything in Git' principle creates a direct tension with the secrets-management best practices from the DevSecOps series — the resolution isn't to compromise on either principle, but to store only a REFERENCE to a secret in Git (via something like External Secrets Operator, pointing at Vault), so the actual sensitive value never touches version control at all, while the pointer to it still lives declaratively in Git like everything else."

---

Rollbacks in GitOps — Trivially Easy#

A genuinely strong, concrete payoff worth stating explicitly, tying together the deployment strategies from Part 1 with GitOps's git-native model.

Diagram

Why this is such a strong, memorable answer to "how does GitOps handle rollbacks": a rollback is just a git revert — using the exact same, already-trusted mechanism (Git history) as every other change, fully auditable, with no special "rollback tooling" needed at all. This directly complements (rather than replaces) the deployment-strategy-level rollback speed advantages from Part 1 (like blue-green's instant router switch) — GitOps rollback is about reverting the declared configuration; the deployment strategy determines how quickly the cluster converges to match that reverted configuration.

---

GitOps and Disaster Recovery#

A powerful, often-underappreciated benefit worth stating explicitly, directly previewing the Disaster Recovery topic (topic 12) in this course.

Diagram

A genuinely strong, senior-level interview line: "Because Git already IS the complete, declarative source of truth for everything that should exist, GitOps gives disaster recovery almost for free — recovering from a total cluster loss becomes 'point a new cluster's agent at the same repo,' and the exact same reconciliation loop that handles day-to-day drift correction does the entire rebuild automatically. This is a genuinely different, stronger guarantee than needing a separate, bespoke disaster-recovery runbook."

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A security team is comparing a traditional push-based CI/CD pipeline against a GitOps pull-based model, specifically asking "which one reduces our exposure to a compromised CI/CD provider, like the SolarWinds incident?" How would you answer, with specifics?

Answer: GitOps's pull-based model meaningfully reduces this exposure — in a push model, the CI/CD provider itself must hold standing, direct write credentials to production, meaning a compromise of that provider (exactly the SolarWinds attack pattern from the DevSecOps series) gives an attacker a direct path to modify production. In a pull model, the CI/CD provider never holds production credentials at all — it only ever writes to a Git repository, a much more limited, easier-to-scope permission — and the actual production-modifying credentials live only inside the target cluster's own GitOps agent, which never needs to be reachable from, or trust, the external CI system at all.

Problem 2: An engineer manually deletes a misbehaving pod directly via kubectl delete pod during an incident, in a cluster with ArgoCD self-healing enabled and pointing at a Deployment with replicas: 3. What happens next, and is this actually a problem?

Answer: Deleting an individual pod directly is actually fine and doesn't conflict with GitOps at all — the Deployment's own ReplicaSet controller (Kubernetes Deep Dive series, Part 2) will simply create a replacement pod to restore the desired replica count of 3, and ArgoCD sees the Deployment object itself is still perfectly in sync with Git (it never specified anything about individual, ephemeral pod identities). This would only become a genuine self-healing conflict if the engineer manually modified the Deployment SPEC itself (e.g., kubectl scale --replicas=1 or kubectl edit deployment) — that kind of change, unlike deleting a disposable pod, WOULD be detected as drift from the Git-declared replicas: 3 and automatically reverted on ArgoCD's next reconciliation pass.

Problem 3: A team using GitOps wants to store a database password needed by their application. A junior engineer suggests base64-encoding it and committing it directly to the manifests repo, arguing "GitOps says everything should be in Git." What's wrong with this reasoning, and what's the correct approach?

Answer: Base64 is encoding, not encryption — it's trivially, instantly reversible by anyone who can read the file, and (exactly as covered in the DevSecOps series) once committed, it's permanently present in Git history even if later "removed" from the latest commit. "Everything in Git" as a GitOps principle refers to the DECLARED DESIRED STATE being in Git — it doesn't mean every literal sensitive value has to be stored in plaintext (or trivially-reversible encoding) there. The correct approach: use either Sealed Secrets (genuinely encrypting the value before committing, decryptable only by the target cluster's private key) or an External Secrets Operator (storing only a reference/pointer in Git, with the actual secret value fetched live from a real secrets manager like Vault) — both keep the actual sensitive value out of Git entirely while still preserving the declarative, Git-driven model for everything else.

---

Summary — The Complete Automation, CI/CD & GitOps Series#

• GitOps formalizes "infrastructure as code" (Part 2) one step further: Git becomes the single, enforced source of truth, continuously and automatically reconciled by an agent — following the exact same reconciliation loop pattern first introduced for Kubernetes controllers.
• The pull-based model (an in-cluster agent pulls from Git, rather than an external CI pipeline pushing to the cluster) is genuinely more secure, since it eliminates the need for any external system to hold standing production write credentials — directly reducing the exact class of supply-chain risk covered in the DevSecOps series.
• ArgoCD and Flux are the two dominant GitOps tools for Kubernetes, both CNCF projects implementing the same core reconciliation principles.
• Self-healing (selfHeal: true in ArgoCD) is GitOps's concrete, automated answer to the drift problem from Part 2 — manual changes bypassing Git get automatically detected and reverted, enforced by the system rather than relying purely on team discipline.
• Multi-environment promotion becomes nothing more than a reviewable Git commit/PR updating a value in a target environment's directory — fully auditable with zero special tooling.
• Secrets genuinely conflict with GitOps's "everything in Git" principle, resolved via Sealed Secrets (encrypted-in-Git) or External Secrets Operator (only a reference lives in Git, the real value stays in a dedicated secrets manager).
• Rollbacks become a simple git revert — using the same trusted, auditable mechanism as every other change, with the reconciliation loop automatically converging the cluster back to the previous known-good state.
• GitOps provides a genuinely strong disaster recovery story almost for free: since Git is already the complete declarative source of truth, rebuilding a lost environment is as simple as pointing a fresh agent at the same repository.

This completes the Automation, CI/CD & GitOps series. See questions.md in this folder for the full interview question bank covering all three parts.