# Kubernetes Deep Dive — Part 19: Building Custom Controllers and Operators

> **Series:** Kubernetes Deep Dive (19 of 19)
> **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane
> **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects
> **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI)
> **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators
> **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE
> **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes
> **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth
> **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway
> **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem
> **Part 10:** `10-troubleshooting-kubernetes.md` — Troubleshooting Kubernetes, Systematically
> **Part 11:** `11-kubernetes-security-deep-dive.md` — Kubernetes Security Deep Dive (CKS-Aligned)
> **Part 12:** `12-autoscaling-hpa-vpa-keda.md` — Autoscaling: HPA, VPA, KEDA & Cluster Autoscaling
> **Part 13:** `13-multi-tenancy-and-cluster-sharing.md` — Multi-Tenancy & Cluster Sharing at Scale
> **Part 14:** `14-ai-ml-workloads-on-kubernetes.md` — Running AI/ML Workloads on Kubernetes
> **Part 15:** `15-cluster-upgrades-and-lifecycle.md` — Cluster Upgrades & Lifecycle Management
> **Part 16:** `16-capacity-planning.md` — Capacity Planning for Kubernetes Clusters
> **Part 17:** `17-cilium-ebpf-networking.md` — Cilium & eBPF Networking Deep Dive
> **Part 18:** `18-backup-and-disaster-recovery.md` — Backup & Disaster Recovery for Workloads
> **Part 19:** This file — Building Custom Controllers and Operators
> **Questions:** `questions.md`

Assumes you're comfortable with the reconciliation loop and the Operator pattern's conceptual shape from
Part 4, and RBAC/ServiceAccount mechanics from Part 11 — this chapter goes deep on the actual client-go and
controller-runtime mechanics Part 4 only sketched, rather than re-introducing the Operator pattern itself.

## Table of Contents

1. [Why This Part Exists](#why-this-part-exists)
2. [Recap: The Reconciliation Loop, and Where Part 4 Left Off](#recap-the-reconciliation-loop-and-where-part-4-left-off)
3. [client-go Architecture: Informers, Listers, and the Local Cache](#client-go-architecture-informers-listers-and-the-local-cache)
4. [Inside the SharedInformer: Reflector, DeltaFIFO, and the Indexer](#inside-the-sharedinformer-reflector-deltafifo-and-the-indexer)
5. [The Workqueue: Deduplication, Rate Limiting, and Backoff](#the-workqueue-deduplication-rate-limiting-and-backoff)
6. [A Minimal Raw client-go Controller, End to End](#a-minimal-raw-client-go-controller-end-to-end)
7. [controller-runtime's Manager, Cache, and Client](#controller-runtimes-manager-cache-and-client)
8. [Kubebuilder Project Scaffolding, Walked Through](#kubebuilder-project-scaffolding-walked-through)
9. [Designing the `TenantOnboarding` Operator](#designing-the-tenantonboarding-operator)
10. [Writing the Reconciler](#writing-the-reconciler)
11. [Owner References and Garbage Collection Between Custom Resources](#owner-references-and-garbage-collection-between-custom-resources)
12. [Finalizers in Practice — Safe Cleanup Before Deletion](#finalizers-in-practice--safe-cleanup-before-deletion)
13. [CRD Versioning and Conversion Webhooks in Practice](#crd-versioning-and-conversion-webhooks-in-practice)
14. [Validating and Mutating Admission Webhooks for Custom Resources](#validating-and-mutating-admission-webhooks-for-custom-resources)
15. [Status Subresources and Conditions](#status-subresources-and-conditions)
16. [Testing Controllers with envtest](#testing-controllers-with-envtest)
17. [Leader Election and Running Operators Highly Available](#leader-election-and-running-operators-highly-available)
18. [Upgrading an Operator Safely — CRD and Controller Rollout Order](#upgrading-an-operator-safely--crd-and-controller-rollout-order)
19. [Observability for Operators — Metrics, Events, and Logs](#observability-for-operators--metrics-events-and-logs)
20. [Common Controller Anti-Patterns, Explained Two Levels Deep](#common-controller-anti-patterns-explained-two-levels-deep)
21. [A Full Worked Scenario: Debugging a Runaway Reconcile Loop in Production](#a-full-worked-scenario-debugging-a-runaway-reconcile-loop-in-production)
22. [A Full Worked Scenario: A Botched CRD Migration Without a Conversion Webhook](#a-full-worked-scenario-a-botched-crd-migration-without-a-conversion-webhook)
23. [Part 19 CLI Cheat Sheet](#part-19-cli-cheat-sheet)
24. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
25. [Worked Practice Problems](#worked-practice-problems)
26. [Summary and What's Next](#summary-and-whats-next)
27. [Where to Go From Here](#where-to-go-from-here)
28. [Closing the Series](#closing-the-series)

---

## Why This Part Exists

**Part 4 introduced the Operator pattern conceptually — a `Reconcile` function sketch, the "controller-runtime
handles the boilerplate" framing, finalizers and idempotency named as principles — and left every mechanism
underneath that sketch unexplained: how does a controller actually learn that an object changed, without
polling the API server in a tight loop? What exactly queues a reconcile request, and what stops the same
object from being reconciled twice concurrently? This chapter answers those questions at the actual
`client-go`/`controller-runtime` mechanics level, then builds a real, working Operator from scaffolding
through tests.**

The throughline gains its most code-heavy addition yet: a `TenantOnboarding` custom resource and its
`tenant-operator`, automating exactly the manual checklist Part 13 introduced for bringing a new tenant
namespace onto the shared cluster — a `ResourceQuota`, `NetworkPolicy`, and baseline `RoleBinding`, applied
consistently instead of by hand. Building this operator over the course of the chapter is a deliberate
illustration that Part 13's onboarding checklist and this chapter's reconciler are the same operational
knowledge, expressed two different ways — a human-followed runbook versus continuously-running code.

> [!NOTE]
> Every code example in this chapter is Go, since that's the language client-go, controller-runtime,
> Kubebuilder, and the overwhelming majority of real-world Operators are written in — there's no serious
> alternative ecosystem for this specific job today, unlike, say, choosing a scripting language for a CI
> pipeline.

## Recap: The Reconciliation Loop, and Where Part 4 Left Off

**Every controller in Kubernetes, built-in or custom, follows the same three-step loop Part 1 introduced and
Part 4 applied to Operators specifically: observe the current state, compare it to the desired state, act to
close any gap.** What Part 4 didn't cover is *how* a controller efficiently observes state for millions of
objects across a live cluster without either polling constantly or missing changes — that's the specific gap
the next three sections close, in the order a request for a new object actually flows through the machinery.

| Part 4 covered | This chapter covers |
|---|---|
| The reconciliation loop, conceptually | The exact mechanism that triggers each `Reconcile` call |
| A representative `Reconcile` function sketch | A complete, working `Reconciler` type, scaffolded and tested |
| "Finalizers handle cleanup," named as a principle | A worked finalizer implementation, including the failure mode where it goes wrong |
| CRD versioning's hub-and-spoke idea | A working conversion webhook, generated and wired up |

## client-go Architecture: Informers, Listers, and the Local Cache

**A naive controller could `kubectl get` every object it cares about on every reconcile — this would work
correctness-wise and destroy the API server's performance the moment more than a handful of controllers did
it simultaneously. The Informer pattern exists specifically to solve this: watch once, cache locally, read
from the cache everywhere.**

```mermaid
flowchart LR
    APIServer[("API Server")] -->|"one long-lived<br/>watch connection"| Reflector["Reflector"]
    Reflector -->|"add/update/delete<br/>events"| DeltaFIFO["DeltaFIFO<br/>(ordered event queue)"]
    DeltaFIFO --> Indexer[["Indexer<br/>(local, in-memory cache)"]]
    Indexer --> Lister["Lister<br/>(typed read API<br/>over the Indexer)"]
    DeltaFIFO --> Handlers{"Registered<br/>event handlers"}
    Handlers -->|"enqueue a key"| Queue[["Workqueue"]]

    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef accent fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    class APIServer info
    class Reflector,DeltaFIFO accent
    class Indexer,Lister,Queue ok
```

*Every controller watching the same resource type shares this exact same cache — a `SharedInformer` means
one watch connection to the API server serves every controller in the process, not one watch per controller.*

**The critical performance property worth stating precisely: a `Lister`'s `Get`/`List` calls never touch the
API server at all** — they read directly from the local `Indexer`, an in-memory cache kept current by the
`Reflector`'s ongoing watch. A controller's `Reconcile` function calling `Get` a thousand times across a busy
reconcile cycle costs the API server exactly zero additional requests, which is precisely what makes it safe
for dozens of controllers to run in the same cluster without coordinating their read patterns.

> [!IMPORTANT]
> `SharedInformerFactory` is the mechanism that makes the "shared" part real — creating informers for the
> same GroupVersionResource through the same factory returns the *same* underlying informer and cache to
> every caller, rather than each controller standing up its own independent watch. Two controllers
> independently calling `client-go`'s bare `NewSharedInformer` (bypassing the factory) each open their own
> separate watch connection to the API server — a real, avoidable load multiplier that the factory pattern
> exists specifically to prevent.

## Inside the SharedInformer: Reflector, DeltaFIFO, and the Indexer

**Tracing exactly what happens between "an object changed on the API server" and "the local cache reflects
it" is worth doing once precisely, since a surprising number of subtle controller bugs trace back to a
misunderstanding of this exact sequence.**

```mermaid
sequenceDiagram
    participant API as API Server
    participant Reflector
    participant FIFO as DeltaFIFO
    participant Indexer
    participant Handler as Event handler

    API->>Reflector: Watch event: TenantOnboarding<br/>"acme-corp" updated
    Reflector->>FIFO: Push a Delta<br/>(type=Updated, object)
    FIFO->>Indexer: Pop() applies the delta,<br/>updating the cached object
    FIFO->>Handler: Deliver the same delta<br/>to every registered handler
    Handler->>Handler: OnUpdate callback fires -<br/>typically just enqueues<br/>the object's key
    Note over Indexer: The local cache is now<br/>consistent BEFORE any<br/>handler callback runs
```

*The cache update and the handler notification happen from the same `Pop()` call — an event handler can
always trust that the `Indexer` already reflects the change it was just notified about, never a stale prior
version.*

**`DeltaFIFO`'s specific job is worth naming precisely: it's a FIFO queue of *deltas* (add/update/delete),
not a queue of full objects, and it compresses redundant deltas for the same key** — if an object is updated
three times before a consumer processes it, `DeltaFIFO` collapses those into the object's current state
rather than replaying three separate stale intermediate versions. This compression is exactly why a
well-written `Reconcile` must never assume it's being told "what changed" — by the time it runs, several
raw watch events for the same object may have already been folded into one.

**The `Indexer` adds one further capability beyond a plain cache: secondary indexes**, letting a controller
efficiently look up objects by something other than name — e.g., "every `TenantOnboarding` whose
`spec.namespace` matches X" — without a linear scan across every cached object, the same underlying idea as
a database index applied to an in-memory object cache.

## The Workqueue: Deduplication, Rate Limiting, and Backoff

**A raw stream of watch events, delivered straight to `Reconcile` one at a time, has a real problem: if the
same object changes five times in a second, a naive controller would run `Reconcile` five times, when one
reconcile against the object's *current* state would have been enough — the workqueue exists specifically to
collapse this down and add retry discipline on top.**

```go
queue := workqueue.NewTypedRateLimitingQueue(
    workqueue.DefaultTypedControllerRateLimiter[string](),
)

// Event handler: never pushes the object itself, only its key
informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: func(obj interface{}) {
        key, _ := cache.MetaNamespaceKeyFunc(obj)
        queue.Add(key)
    },
    UpdateFunc: func(old, new interface{}) {
        key, _ := cache.MetaNamespaceKeyFunc(new)
        queue.Add(key)
    },
})
```

| Workqueue property | What it guarantees |
|---|---|
| Deduplication | Adding the same key while it's already queued (or already being processed) is a no-op — the key is only ever processed once per "batch" of changes, not once per individual event |
| `AddRateLimited(key)` | Requeues a key after a failed reconcile, with exponential backoff — a handful of retries in the first second, widening out to roughly every 16-17 minutes for a persistently failing key |
| `Forget(key)` | Clears a key's backoff history after a successful reconcile — the next unrelated failure starts backoff fresh, rather than inheriting an old failure streak |
| Keys, never objects | The queue holds only `namespace/name` strings — the actual current object is always re-fetched from the `Lister`'s cache at processing time, never carried through the queue itself |

> [!IMPORTANT]
> **Queueing keys instead of objects is the single design decision that makes the "level-based, not
> edge-triggered" principle from Part 4 actually work in practice.** By the time a queued key is popped and
> processed, the object may have changed again — re-fetching it fresh from the `Lister` at that moment,
> rather than processing whatever version triggered the original enqueue, guarantees `Reconcile` always acts
> on genuinely current state, never a stale snapshot from whenever the event first fired.

## A Minimal Raw client-go Controller, End to End

**Seeing the informer, workqueue, and a bare reconcile function wired together directly — without
controller-runtime's abstraction over them — is worth doing once, purely so the abstraction in the next
section reads as "the same thing, with the boilerplate factored out" rather than as unfamiliar magic.**

```go
func Run(ctx context.Context, client kubernetes.Interface) error {
    factory := informers.NewSharedInformerFactory(client, 30*time.Second)
    podInformer := factory.Core().V1().Pods()
    queue := workqueue.NewTypedRateLimitingQueue(
        workqueue.DefaultTypedControllerRateLimiter[string](),
    )

    podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc:    func(o interface{}) { enqueue(queue, o) },
        UpdateFunc: func(_, o interface{}) { enqueue(queue, o) },
        DeleteFunc: func(o interface{}) { enqueue(queue, o) },
    })

    factory.Start(ctx.Done())
    factory.WaitForCacheSync(ctx.Done())   // block until the initial List completes

    for processNextItem(queue, podInformer.Lister()) {
    }
    return nil
}

func processNextItem(q workqueue.TypedRateLimitingInterface[string], lister v1.PodLister) bool {
    key, shutdown := q.Get()
    if shutdown {
        return false
    }
    defer q.Done(key)

    if err := reconcile(key, lister); err != nil {
        q.AddRateLimited(key)   // retry with backoff
        return true
    }
    q.Forget(key)
    return true
}
```

**`WaitForCacheSync` deserves its own emphasis: reconciling before the informer's initial `List` has
finished populating the cache means every `Get`/`List` call sees a partially-populated, misleadingly-empty
view of the cluster** — a controller that skips this call can make genuinely wrong decisions on startup
(e.g., concluding an object doesn't exist yet, when it simply hasn't synced into the cache), a real and
easy-to-introduce bug in a hand-rolled controller that `controller-runtime`'s `Manager` handles automatically,
covered next.

## controller-runtime's Manager, Cache, and Client

**Almost no real-world Operator is written directly against raw client-go anymore — `controller-runtime`
wraps informers, listers, and the workqueue behind a small set of well-designed abstractions, and Kubebuilder
and Operator SDK both scaffold projects on top of it.**

```mermaid
flowchart TD
    Mgr["Manager<br/>(top-level process)"] --> Cache[["Cache<br/>(wraps SharedInformers<br/>for every watched type)"]]
    Mgr --> Client["Client<br/>(reads from Cache,<br/>writes directly to<br/>the API Server)"]
    Mgr --> Ctrl["Controller<br/>(owns ONE workqueue,<br/>watches one or more types)"]
    Ctrl -->|"on any watched<br/>event"| Recon(("Reconciler.Reconcile()"))
    Recon -->|"reads via"| Client
    Recon -->|"writes via"| Client

    classDef accent fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    class Mgr accent
    class Cache,Client info
    class Ctrl,Recon ok
```

*The `Client` your `Reconciler` calls looks like a single, ordinary Kubernetes client — reads are
transparently served from the `Cache` (the same Indexer mechanism from earlier), writes go straight to the
API server, and the `Reconciler` code never has to know or care about that split.*

| Component | Role |
|---|---|
| `Manager` | The top-level object every controller-runtime binary runs — owns the shared `Cache`, the `Client`, leader election, health/metrics servers, and the lifecycle of every registered `Controller` |
| `Cache` | A collection of `SharedInformer`s, one per watched GroupVersionKind — the same Reflector/DeltaFIFO/Indexer machinery from earlier sections, just no longer hand-wired |
| `Client` | A read-through-cache, write-through-API-server client — the only thing a `Reconciler` typically ever touches directly |
| `Controller` | Owns exactly one workqueue and watches one primary resource type (plus any number of secondary "owned" or "watched" types that also trigger reconciles) |

## Kubebuilder Project Scaffolding, Walked Through

**Kubebuilder generates a complete, working project skeleton from a single command — worth understanding
what each generated piece actually is, rather than treating the scaffold as opaque boilerplate to work
around.**

```bash
kubebuilder init --domain platform.example.com --repo github.com/example/tenant-operator
kubebuilder create api --group platform --version v1alpha1 --kind TenantOnboarding --resource --controller
```

```mermaid
mindmap
  root((tenant-operator))
    api/v1alpha1
      tenantonboarding_types.go
      Spec and Status structs
    internal/controller
      tenantonboarding_controller.go
      The actual Reconciler
    config/crd
      Generated CRD YAML
    config/rbac
      Generated Role/ClusterRole for<br/>whatever the Reconciler touches
    config/samples
      Example TenantOnboarding CR
    cmd/main.go
      Manager setup and startup
```

**Two of these generated pieces are worth calling out specifically, since they're easy to under-appreciate
as "just generated boilerplate":**

- `config/rbac/role.yaml` is generated from `//+kubebuilder:rbac` marker comments directly above the
  `Reconcile` function — the exact least-privilege discipline Part 11 covered manually for application
  ServiceAccounts is here derived automatically from the code that actually needs each permission, rather
  than hand-maintained separately from what the controller does.
- `config/crd/bases/*.yaml` is generated from Go struct tags on the `Spec`/`Status` types — the CRD's
  OpenAPI schema (required fields, enums, validation) lives as Go type annotations, not as hand-written YAML
  kept in sync with the code by hand.

```go
//+kubebuilder:rbac:groups=platform.example.com,resources=tenantonboardings,verbs=get;list;watch;update
//+kubebuilder:rbac:groups="",resources=resourcequotas;namespaces,verbs=get;list;watch;create;update
//+kubebuilder:rbac:groups=networking.k8s.io,resources=networkpolicies,verbs=get;list;watch;create;update
func (r *TenantOnboardingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // ...
}
```

## Designing the `TenantOnboarding` Operator

**Before writing the `Reconcile` function itself, the CRD's schema is the actual design decision — what a
platform team writes to onboard a tenant, and what the operator is responsible for turning that into.**

```yaml
apiVersion: platform.example.com/v1alpha1
kind: TenantOnboarding
metadata:
  name: recommendations-team
spec:
  namespace: recommendations
  resourceQuota:
    cpu: "20"
    memory: 64Gi
    gpuCount: 4
  networkPolicy:
    allowEgressTo: ["catalog", "kube-system"]
```

**This is a direct, literal translation of Part 13's onboarding checklist and Part 14's GPU quota discussion
into a declarative spec** — a platform engineer applies one `TenantOnboarding` object instead of manually
running through a multi-step checklist, and the operator is responsible for making the `Namespace`,
`ResourceQuota`, and `NetworkPolicy` objects match what that spec declares, continuously, the same
"encode the runbook as code" principle Part 4 introduced generally.

| Object the operator manages | Sourced from |
|---|---|
| `Namespace` (created if missing) | `spec.namespace` |
| `ResourceQuota` (including `requests.nvidia.com/gpu`, Part 14) | `spec.resourceQuota` |
| `NetworkPolicy` (default-deny plus explicit allows, Part 11) | `spec.networkPolicy.allowEgressTo` |

## Writing the Reconciler

**The `Reconcile` function itself, written the correct, idempotent, level-based way Part 4 named as a
principle — every call re-derives the full desired state and applies it, regardless of what specifically
triggered this particular invocation.**

```go
func (r *TenantOnboardingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var onboarding platformv1alpha1.TenantOnboarding
    if err := r.Get(ctx, req.NamespacedName, &onboarding); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: onboarding.Spec.Namespace}}
    if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, ns, func() error {
        return nil // Namespace has no spec fields this operator manages beyond existence
    }); err != nil {
        return ctrl.Result{}, err
    }

    quota := buildResourceQuota(&onboarding)
    if err := controllerutil.SetControllerReference(&onboarding, quota, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }
    if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, quota, func() error {
        quota.Spec = buildResourceQuotaSpec(&onboarding)
        return nil
    }); err != nil {
        return ctrl.Result{}, err
    }

    onboarding.Status.Ready = true
    if err := r.Status().Update(ctx, &onboarding); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}
```

**`controllerutil.CreateOrUpdate` is doing the actual "observe, compare, act" work in three lines**: it reads
the current object if one exists, applies the mutation function, and only issues a write if something
actually changed — this is the concrete mechanism behind "idempotent, level-based reconciliation," not an
abstract principle floating separate from real code.

> **From the Trenches:** An early version of this reconciler set the `ResourceQuota`'s spec unconditionally
> on every reconcile, without wrapping the mutation in `CreateOrUpdate`'s diff-and-only-write-if-changed
> behavior. Under normal operation this was invisible — but during a control-plane slowdown that caused the
> `TenantOnboarding` informer to redeliver a backlog of stale-looking update events, the operator issued
> thousands of no-op `ResourceQuota` writes in a few minutes, each one triggering its own watch event to
> every *other* controller watching `ResourceQuota` objects cluster-wide. The immediate cause was skipping
> the diff-before-write step; the underlying condition was that "idempotent" was treated as "safe to call
> repeatedly" without also asking whether repeated calls were cheap for the rest of the cluster watching the
> same object type — a distinction that only matters at exactly the moment a backlog of events actually
> arrives.

## Owner References and Garbage Collection Between Custom Resources

**Every object the `TenantOnboarding` reconciler creates should carry an owner reference back to the
`TenantOnboarding` object itself — `controllerutil.SetControllerReference`, used in the reconciler above, is
what actually establishes this, and it's what makes deleting the parent object clean up everything it
created automatically, without the reconciler ever writing explicit delete logic for most of its children.**

```mermaid
flowchart TD
    Parent(("TenantOnboarding:<br/>recommendations-team")) -->|"ownerReference"| Quota[("ResourceQuota")]
    Parent -->|"ownerReference"| NetPol[("NetworkPolicy")]
    Delete["kubectl delete<br/>tenantonboarding<br/>recommendations-team"] --> GC["Kubernetes's built-in<br/>garbage collector"]
    GC -->|"cascading delete,<br/>NO controller code<br/>needed for this part"| Quota
    GC -->|"cascading delete"| NetPol

    classDef accent fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c
    class Parent accent
    class Quota,NetPol info
    class Delete,GC crit
```

*Owner references get this specific cleanup case (namespaced children of a namespaced parent) for free from
Kubernetes's built-in garbage collector — no reconciler code is needed to delete the `ResourceQuota`/
`NetworkPolicy` when the `TenantOnboarding` object is deleted.*

> [!NOTE]
> Owner references only support **cascading deletion within the same namespace** for namespaced owners — a
> cluster-scoped resource (like the `Namespace` object itself, created earlier in the reconciler) cannot be
> owned by a namespaced `TenantOnboarding`, since owner references require the owner and the owned object to
> share a namespace unless the owner is itself cluster-scoped. This is exactly why the `Namespace` object in
> this chapter's reconciler is created directly rather than owner-referenced — it needs its own explicit
> finalizer-based cleanup logic instead, covered next.

## Finalizers in Practice — Safe Cleanup Before Deletion

**The `Namespace` object the previous section flagged as un-owner-referenceable is exactly the case a
finalizer solves: cleanup logic the operator must run *before* Kubernetes considers the `TenantOnboarding`
object actually gone, for anything owner references can't reach automatically.**

```go
const tenantFinalizer = "platform.example.com/tenant-cleanup"

func (r *TenantOnboardingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var onboarding platformv1alpha1.TenantOnboarding
    if err := r.Get(ctx, req.NamespacedName, &onboarding); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    if onboarding.DeletionTimestamp.IsZero() {
        if !controllerutil.ContainsFinalizer(&onboarding, tenantFinalizer) {
            controllerutil.AddFinalizer(&onboarding, tenantFinalizer)
            return ctrl.Result{}, r.Update(ctx, &onboarding)
        }
    } else {
        // Object is being deleted — run cleanup BEFORE removing the finalizer
        if controllerutil.ContainsFinalizer(&onboarding, tenantFinalizer) {
            if err := r.deleteNamespace(ctx, onboarding.Spec.Namespace); err != nil {
                return ctrl.Result{}, err   // retry — finalizer stays, object stays Terminating
            }
            controllerutil.RemoveFinalizer(&onboarding, tenantFinalizer)
            return ctrl.Result{}, r.Update(ctx, &onboarding)
        }
        return ctrl.Result{}, nil
    }
    // ... normal reconciliation continues below
    return ctrl.Result{}, nil
}
```

```mermaid
stateDiagram-v2
    [*] --> Active: Created, finalizer added
    Active --> Terminating: kubectl delete —<br/>DeletionTimestamp set,<br/>object NOT yet removed
    Terminating --> CleanupRunning: Reconciler sees<br/>DeletionTimestamp set
    CleanupRunning --> CleanupRunning: Cleanup fails —<br/>finalizer stays,<br/>object stays Terminating
    CleanupRunning --> Removed: Cleanup succeeds,<br/>finalizer removed
    Removed --> [*]

    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c
    classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c
    class Active ok
    class Terminating,CleanupRunning warn
    class Removed ok
```

*An object stuck in `Terminating` forever is almost always this exact diagram, stuck on the self-loop —
cleanup keeps failing, or the controller that owns the finalizer isn't running at all to ever attempt it.*

> [!WARNING]
> **The most common real-world cause of an object "stuck in Terminating" is a finalizer whose owning
> controller no longer exists** — the operator was uninstalled, its Deployment scaled to zero, or its CRD
> was deleted without first removing finalizers from every instance. Kubernetes has no timeout or fallback
> for an unmet finalizer; it holds the object in `Terminating` indefinitely. Before uninstalling any
> Operator, confirm no live custom resources still carry its finalizers — `kubectl patch` to manually strip
> the finalizer is the correct emergency recovery, but it means whatever external cleanup that finalizer was
> supposed to guarantee (here, actually deleting the tenant's `Namespace`) never happens, and needs to be
> done by hand afterward.

## CRD Versioning and Conversion Webhooks in Practice

**Part 4 covered the hub-and-spoke concept abstractly — here's what actually changes when
`TenantOnboarding` evolves from `v1alpha1` to `v1`, renaming `spec.resourceQuota` to a more general
`spec.quota` that can eventually cover non-GPU resource types uniformly.**

```bash
kubebuilder create api --group platform --version v1 --kind TenantOnboarding --resource --controller=false
```

```go
// v1 is the hub — v1alpha1 implements Convertible against it
func (src *TenantOnboarding) ConvertTo(dstRaw conversion.Hub) error {
    dst := dstRaw.(*v1.TenantOnboarding)
    dst.Spec.Quota = src.Spec.ResourceQuota   // field rename happens here
    dst.Spec.Namespace = src.Spec.Namespace
    return nil
}

func (dst *TenantOnboarding) ConvertFrom(srcRaw conversion.Hub) error {
    src := srcRaw.(*v1.TenantOnboarding)
    dst.Spec.ResourceQuota = src.Spec.Quota
    dst.Spec.Namespace = src.Spec.Namespace
    return nil
}
```

```yaml
# config/crd/bases — kubebuilder-generated, wiring the webhook in
versions:
  - name: v1alpha1
    served: true
    storage: false
  - name: v1
    served: true
    storage: true
conversion:
  strategy: Webhook
  webhook:
    conversionReviewVersions: ["v1alpha1", "v1"]
```

**controller-runtime generates and serves the `/convert` HTTP endpoint automatically from the
`ConvertTo`/`ConvertFrom` methods above** — inside the same manager binary as the reconciler itself, with no
separate webhook deployment to run. The only code an operator author writes is the two conversion functions;
the HTTP handling, TLS, and API server integration are all generated scaffolding.

> [!TIP]
> Migrate the storage version deliberately and explicitly rather than assuming it happens automatically —
> flipping `storage: true` to the new version only changes what *new* writes get stored as. Every existing
> `v1alpha1`-shaped object already in etcd stays stored in its original form until it's next written, unless
> a one-time `kube-storage-version-migrator` job (or an equivalent forced rewrite, the same pattern Part 11
> used for retroactive Secret encryption) explicitly re-persists every existing object under the new
> storage version.

## Validating and Mutating Admission Webhooks for Custom Resources

**Part 1's admission chain and Part 4's sidecar-injection mutating webhook both applied to built-in
resources — the exact same admission mechanism is available for a custom resource too, and Kubebuilder
scaffolds it the same way it scaffolds a conversion webhook.**

```mermaid
sequenceDiagram
    participant User
    participant API as API Server
    participant Mutate as Mutating webhook<br/>(tenant-operator)
    participant Validate as Validating webhook<br/>(tenant-operator)
    participant Etcd

    User->>API: kubectl apply -f tenantonboarding.yaml
    API->>Mutate: Mutating admission phase
    Mutate-->>API: Defaults spec.resourceQuota.cpu<br/>to "4" if unset
    API->>Validate: Validating admission phase
    Validate->>Validate: Reject if requested GPU count<br/>exceeds cluster's total advertised<br/>nvidia.com/gpu capacity
    Validate-->>API: Admit or reject
    API->>Etcd: Store the (possibly defaulted)<br/>object, only if admitted
```

*Validation runs strictly after mutation — a validating webhook always sees the final, already-defaulted
object, never the raw request a user actually submitted.*

```go
//+kubebuilder:webhook:path=/validate-platform-example-com-v1-tenantonboarding,mutating=false,failurePolicy=fail,groups=platform.example.com,resources=tenantonboardings,verbs=create;update,versions=v1,name=vtenantonboarding.kb.io

func (v *TenantOnboardingValidator) ValidateCreate(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {
    onboarding := obj.(*platformv1alpha1.TenantOnboarding)
    if onboarding.Spec.ResourceQuota.GPUCount > clusterAvailableGPUs(ctx) {
        return nil, fmt.Errorf(
            "requested GPU count %d exceeds cluster capacity", onboarding.Spec.ResourceQuota.GPUCount,
        )
    }
    return nil, nil
}
```

| Webhook kind | Runs when | Typical use in a custom Operator |
|---|---|---|
| Mutating | Before validation | Fill in sensible defaults (`spec.resourceQuota.cpu` if a platform team leaves it unset), inject standard labels |
| Validating | After mutation, before persistence | Reject a spec that's structurally valid YAML but semantically wrong (a GPU request exceeding real cluster capacity, per Part 14's supply-constrained GPU quota problem) |

> [!TIP]
> Prefer expressing simple, static validation rules (a field's allowed enum values, a required-together
> field pair) directly in the CRD's OpenAPI schema via Kubebuilder's `//+kubebuilder:validation` markers,
> and reserve an actual validating webhook for rules that need to check against *live cluster state* —
> like this section's GPU capacity check — which a static schema can't express at all. Reaching for a
> webhook when a schema marker would do adds an extra network hop and a new availability dependency
> (the webhook service itself) to every single admission of that resource type, for no benefit over the
> simpler, static option.

> [!WARNING]
> `failurePolicy: fail` (the default, and generally the correct choice for a validating webhook enforcing a
> real invariant) means the API server rejects the request outright if the webhook itself is unreachable —
> which also means a bug or outage in the webhook's own Deployment can block *every* create/update of that
> resource type cluster-wide, including from the Operator's own reconciler if it ever needs to update the
> object. Run the webhook server with the same leader-election-independent high availability as the main
> controller (multiple replicas behind a Service, never a single pod) precisely because its blast radius on
> failure is the entire resource type, not just this one Operator's own reconciliation.

## Status Subresources and Conditions

**The `Status().Update()` call in this chapter's reconciler writes to a genuinely separate API endpoint from
a plain `Update()` — the status subresource — and understanding why that split exists prevents a specific,
common source of unnecessary reconcile churn.**

```go
type TenantOnboardingStatus struct {
    Ready      bool               `json:"ready"`
    Conditions []metav1.Condition `json:"conditions,omitempty"`
}
```

| Update kind | Endpoint | Triggers a reconcile in controllers watching this type? |
|---|---|---|
| `r.Update(ctx, obj)` | Main resource endpoint (`spec` and metadata) | Yes — any watcher of this type sees the change |
| `r.Status().Update(ctx, obj)` | The `/status` subresource | Yes, but *only* the status portion changed — a controller can filter on this to avoid reconciling on its own status writes |

**`Conditions` (a standard, structured slice of `type`/`status`/`reason`/`message`/`lastTransitionTime`)
is the conventional way an Operator communicates *why* it's in a given state, not just whether it is** —
`kubectl describe tenantonboarding` renders these directly, giving a platform engineer the same kind of
human-readable status a built-in `Deployment`'s `kubectl describe` output already provides.

```go
meta.SetStatusCondition(&onboarding.Status.Conditions, metav1.Condition{
    Type:    "QuotaProvisioned",
    Status:  metav1.ConditionTrue,
    Reason:  "ResourceQuotaCreated",
    Message: "ResourceQuota successfully applied to the tenant namespace",
})
```

> [!IMPORTANT]
> A `Reconciler` that watches its own primary resource type and updates that same object's `status` on every
> reconcile must guard against re-triggering itself in an infinite loop — this is precisely why the split
> above exists. Filtering out status-only self-triggered events (controller-runtime's `predicate` package
> supports this directly) is the standard fix, covered again in this chapter's anti-patterns section under
> "reconcile loop explosions."

## Testing Controllers with envtest

**`envtest` starts a real `kube-apiserver` and `etcd` binary locally — no mocking of the Kubernetes API at
all — making it the right tool for testing what a `Reconciler` actually does against real API server
behavior (admission, defaulting, status subresource semantics) rather than a hand-built fake that might not
match.**

```go
var _ = Describe("TenantOnboarding controller", func() {
    It("creates a ResourceQuota matching the spec", func() {
        onboarding := &platformv1alpha1.TenantOnboarding{
            ObjectMeta: metav1.ObjectMeta{Name: "test-tenant"},
            Spec: platformv1alpha1.TenantOnboardingSpec{
                Namespace:     "test-tenant-ns",
                ResourceQuota: platformv1alpha1.QuotaSpec{CPU: "4", Memory: "8Gi"},
            },
        }
        Expect(k8sClient.Create(ctx, onboarding)).To(Succeed())

        Eventually(func() error {
            var quota corev1.ResourceQuota
            return k8sClient.Get(ctx, types.NamespacedName{
                Name: "test-tenant-quota", Namespace: "test-tenant-ns",
            }, &quota)
        }, "10s", "250ms").Should(Succeed())
    })
})
```

| Test layer | Tool | Catches |
|---|---|---|
| Pure logic (e.g., `buildResourceQuotaSpec`) | Plain Go unit tests, or the fake client | Business-logic bugs with no real API server interaction needed |
| Reconciler behavior end to end | `envtest` (real API server + etcd, no scheduler/kubelet) | Whether the reconciler's actual API calls produce the right objects, including status subresource and finalizer behavior |
| Full runtime behavior (pod scheduling, actual container startup) | A real cluster (kind, a cloud dev cluster) | Anything `envtest` explicitly doesn't simulate — pods never actually run under `envtest` |

> [!TIP]
> Use `Eventually()`, never a bare synchronous assertion immediately after a `Create`/`Update` call, for
> anything a controller-runtime reconciler is expected to react to — the reconcile loop runs asynchronously
> relative to the test's own `Create` call, exactly as it would against a real cluster, and a test that
> doesn't account for that timing will be flaky rather than reliably wrong or reliably right.

## Leader Election and Running Operators Highly Available

**Running two replicas of an Operator for availability creates an obvious risk if both are allowed to
reconcile simultaneously: two concurrent, possibly conflicting writes to the same object from two processes
that don't know about each other — leader election exists specifically to prevent this.**

```go
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    LeaderElection:   true,
    LeaderElectionID: "tenant-operator-leader",
})
```

**Only the elected leader replica actually runs any `Reconcile` calls — every standby replica's `Manager`
sits idle, ready to acquire leadership (via a `Lease` object, the same primitive Part 1's own control-plane
leader election covers for `kube-controller-manager`/`kube-scheduler`) the moment the current leader stops
renewing it**, whether from a graceful shutdown or a crash. This gives an Operator deployment genuine
high availability — a second replica ready to take over within the lease's configured renewal window — 
without ever risking two active reconcilers stepping on each other's writes.

## Upgrading an Operator Safely — CRD and Controller Rollout Order

**Part 15 covered cluster upgrade sequencing in general — an Operator upgrade has its own, narrower version
of the same "order matters" lesson, specifically around the relationship between a CRD's schema and the
controller code that reads it.**

```mermaid
flowchart TD
    Start["Rolling out tenant-operator v2,<br/>which adds a new required<br/>spec.quota.priorityClass field"] --> Wrong{"Deploy the new<br/>controller FIRST?"}
    Wrong -->|"Yes — wrong order"| Crash["New controller code expects<br/>spec.quota.priorityClass on<br/>every object — crashes or<br/>errors reconciling every<br/>EXISTING object that predates<br/>the new field"]
    Wrong -->|"No — CRD first"| Safe["1. Apply updated CRD<br/>(new field optional/defaulted)"]
    Safe --> Safe2["2. Confirm existing objects<br/>still validate against<br/>the updated schema"]
    Safe2 --> Safe3["3. Roll out the new<br/>controller version"]
    Safe3 --> Safe4["4. Only once stable,<br/>consider tightening the<br/>new field to required"]

    classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c
    class Wrong warn
    class Crash crit
    class Safe,Safe2,Safe3,Safe4 ok
```

*The same "expand, then contract" migration discipline a relational-database schema change uses applies
directly here — add the new field as optional first, roll out code that can handle both its presence and
absence, and only make it mandatory once every object in the cluster is confirmed to already have it.*

| Rollout order | Risk |
|---|---|
| New controller before new CRD schema | The controller's Go struct expects fields the CRD doesn't yet define/allow — the API server rejects writes the controller tries to make |
| New CRD schema (with a new *required* field) before any existing objects have it | Every existing object instantly fails validation on its next write, including the controller's own routine status updates |
| New CRD schema (new field *optional*, sensibly defaulted) before the new controller | Existing objects remain valid; the old controller simply ignores the new field it doesn't know about yet — genuinely safe |

> [!IMPORTANT]
> This is exactly why this chapter's CRD versioning section treated adding a new field as something to
> introduce as **optional first, tightened to required only later** — the "expand, then contract" pattern
> above is the general form of that same specific lesson, and it applies to plain in-place schema additions
> just as much as to a full version bump with a conversion webhook. A required field added directly, with no
> expand phase, breaks every object that predates it the instant the new schema is applied.

## Observability for Operators — Metrics, Events, and Logs

**controller-runtime exposes a Prometheus metrics endpoint automatically, without any extra code, giving
every custom controller the same observability baseline Part 10 established for cluster-wide monitoring
generally.**

| Metric | What it reveals |
|---|---|
| `controller_runtime_reconcile_total{result="success"\|"error"}` | Overall reconcile success/error rate, per controller |
| `controller_runtime_reconcile_time_seconds` | Reconcile latency distribution — a rising p99 often precedes a full reconcile-loop explosion |
| `workqueue_depth` | How many keys are currently queued but not yet processed — a sustained rise means the controller can't keep up with incoming events |
| `workqueue_retries_total` | How often keys are being requeued after failure — a leading indicator for the anti-patterns in the next section |

**Kubernetes `Event` objects are the other half of Operator observability, and the half most often
skipped** — emitting a normal, human-readable `Event` (`r.Recorder.Event(&onboarding, corev1.EventTypeNormal,
"QuotaProvisioned", "ResourceQuota created for tenant namespace")`) surfaces directly in
`kubectl describe tenantonboarding`, giving a platform engineer immediate, in-context troubleshooting
information without needing to correlate a separate log stream or metrics dashboard at all.

## Common Controller Anti-Patterns, Explained Two Levels Deep

**Every pattern below has caused a real, recurring class of production incident across the Operator
ecosystem — worth internalizing the "why," not just memorizing the list.**

- **Edge-triggered logic instead of level-based.** *Symptom*: a controller behaves correctly under normal
  operation but silently drifts from desired state after a missed event (a controller restart during a busy
  window, a dropped watch reconnect). *Immediate cause*: `Reconcile` branches on "what just changed" instead
  of re-deriving the full desired state from scratch every time. *Underlying condition*: `client-go`/
  `controller-runtime` explicitly do not guarantee every individual event is delivered — restarts, resyncs,
  and `DeltaFIFO` compression (covered earlier) can all legitimately collapse or skip intermediate states,
  and only a reconciler that treats every invocation as "figure out what's needed right now" is actually
  correct under that guarantee.
- **Reconcile loop explosions from self-triggered status updates.** *Symptom*: a controller's CPU and API
  server request volume climb steadily with no corresponding change in real cluster activity. *Immediate
  cause*: the reconciler watches its own resource type and unconditionally writes to `status` on every
  reconcile, and that status write itself re-triggers a watch event, creating a tight self-sustaining loop.
  *Underlying condition*: the reconciler never applied the predicate-based status-change filtering covered
  earlier, so it can't distinguish "something meaningful changed" from "I just wrote my own status update a
  moment ago."
- **Missing or dangling finalizers.** *Symptom*: objects stuck in `Terminating` indefinitely, or (the
  opposite failure) external resources silently orphaned after a CR is deleted. *Immediate cause*: either a
  finalizer whose owning controller isn't running, or a controller that manages external state but never
  registered a finalizer to guarantee cleanup runs at all. *Underlying condition*: finalizer lifecycle wasn't
  treated as part of the Operator's own uninstall/upgrade runbook — this chapter's earlier warning about
  checking for live finalizers before uninstalling an Operator exists precisely because this gets missed.
- **No rate limiting or backoff on a persistently failing reconcile.** *Symptom*: a single misconfigured
  custom resource generates a continuous, high-volume stream of API server requests. *Immediate cause*: a
  hand-rolled retry loop that calls `Reconcile` again immediately on error, bypassing the workqueue's
  built-in `AddRateLimited` backoff entirely. *Underlying condition*: treating the workqueue as optional
  plumbing rather than the actual mechanism responsible for retry discipline — this is exactly why the raw
  client-go example earlier in this chapter routes every retry through `q.AddRateLimited`, never a bespoke
  loop.
- **Overly broad RBAC "to keep things working."** *Symptom*: an Operator's ServiceAccount can read/write
  far more than its actual reconciler logic touches. *Immediate cause*: a wildcard grant added once to make
  an early RBAC error go away, never revisited. *Underlying condition*: the `//+kubebuilder:rbac` marker
  discipline from the scaffolding section exists specifically to keep generated RBAC tied to what the code
  actually does — bypassing it with a hand-edited wildcard `Role` breaks that guarantee the same way Part 11
  flagged for application ServiceAccounts generally.

## A Full Worked Scenario: Debugging a Runaway Reconcile Loop in Production

**`tenant-operator`'s `workqueue_depth` metric alarms at 3 a.m. — the queue is growing continuously, and the
operator's Deployment is being OOMKilled (Part 10) every few minutes and restarting.**

```mermaid
flowchart TD
    Alert["workqueue_depth alarm,<br/>repeated OOMKills"] --> Check1{"controller_runtime_reconcile_total<br/>error rate elevated?"}
    Check1 -->|"No — mostly success"| Check2{"Is ONE object being<br/>reconciled far more<br/>often than others?"}
    Check1 -->|"Yes"| Check3["Read Reconcile error logs —<br/>likely a persistent, unhandled<br/>error looping via AddRateLimited"]
    Check2 -->|"Yes"| SelfTrigger["Check: does Reconcile write<br/>status unconditionally on<br/>every pass? (self-trigger loop)"]
    Check2 -->|"No — broadly<br/>distributed"| VolumeIssue["Genuine event volume spike —<br/>investigate upstream cause,<br/>not the controller itself"]
    SelfTrigger --> Fix["Fix: apply a status-change<br/>predicate so status writes<br/>don't re-trigger reconciles"]

    classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c
    classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    class Alert crit
    class Check1,Check2,Check3 info
    class SelfTrigger,VolumeIssue warn
    class Fix ok
```

**The investigation finds `recommendations-team`'s `TenantOnboarding` object being reconciled roughly 40
times per second, while every other tenant reconciles normally — pointing straight at the "single object,
disproportionate volume" branch, not a general event storm.** Reading the reconciler code confirms this
chapter's own earlier "From the Trenches" mistake was reintroduced during a recent refactor: a new
`onboarding.Status.LastSyncTime = time.Now()` line was added to the reconciler unconditionally, on every
pass, with no predicate filtering it out — every reconcile now writes a status change that immediately
re-triggers the next reconcile. The fix is exactly the predicate-based status filtering from the earlier
Status Subresources section, applied in code review going forward as a specific thing to check for on any
diff touching a `Reconcile` function's status-writing path.

## A Full Worked Scenario: A Botched CRD Migration Without a Conversion Webhook

**A different team, maintaining a separate `CachePolicy` CRD, renames `spec.ttlSeconds` to `spec.ttl` (a
`string` with unit suffixes, like `"5m"`) directly in the existing CRD's schema — without adding a second
served version or a conversion webhook — and rolls the new controller version out.**

1. **Every existing `CachePolicy` object already in etcd was written under the old schema** — `spec.ttlSeconds:
   300`, an integer field the new schema no longer defines.
2. **The new controller's Go struct expects `spec.ttl` as a string** — reading an existing object either
   silently drops the now-unrecognized `ttlSeconds` field (if using an unstructured/loose decode path) or
   fails validation outright on the next write, depending on exactly how the schema change was applied.
3. **Every pre-existing `CachePolicy` object is now either broken or has quietly lost its configured TTL**,
   discovered only when several caches stop expiring entries on schedule days later.
4. **The correct approach, using this chapter's tools**: add `v1` alongside the existing `v1alpha1` version
   rather than mutating the single existing version's schema in place, implement `ConvertTo`/`ConvertFrom`
   translating `ttlSeconds` (int) to `ttl` (string) and back, keep `v1alpha1` as `storage: true` until every
   client and controller is confirmed running the new code, and only then flip the storage version and run
   the migrator job to rewrite existing objects.

> [!CAUTION]
> **Changing a served CRD version's schema in place, rather than introducing a new version, is safe only if
> every existing stored object either doesn't use the changed field or is guaranteed to be rewritten before
> anything reads it under the new schema — a guarantee that's much harder to actually verify than it sounds,
> especially across a cluster with objects created months or years earlier.** The hub-and-spoke conversion
> pattern this chapter covered exists specifically to make this kind of change safe without needing that
> guarantee at all — treat "just edit the existing version's schema" as the anti-pattern this scenario shows
> it to be, not a shortcut available when a change "seems small."

## Part 19 CLI Cheat Sheet

| Command | Purpose |
|---|---|
| `kubebuilder create api --group <g> --version <v> --kind <K>` | Scaffold a new CRD type and its controller |
| `make manifests` | Regenerate CRD YAML and RBAC from `//+kubebuilder` markers |
| `make generate` | Regenerate `DeepCopy` methods and other generated Go code |
| `make test` | Run `envtest`-based controller tests |
| `kubectl get <crd> -o yaml \| grep -A5 conditions` | Inspect an object's `status.conditions` directly |
| `kubectl get events --field-selector involvedObject.name=<name>` | See every `Event` a controller emitted for a specific object |
| `kubectl patch <crd> <name> -p '{"metadata":{"finalizers":[]}}' --type=merge` | Emergency-strip a stuck finalizer (Part 19's own warning applies — external cleanup won't run) |
| `curl localhost:8080/metrics \| grep controller_runtime_reconcile` | Inspect a controller's own reconcile metrics directly, e.g. via `kubectl port-forward` |

## Common Mistakes and Interview Traps

| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Branching `Reconcile` logic on "what changed" rather than re-deriving full desired state | `client-go`/`controller-runtime` never guarantee every individual event is delivered exactly once | Always compute desired state fresh and apply it, regardless of what triggered this reconcile |
| Skipping `WaitForCacheSync` before processing events | Reconciling against a partially-populated cache can produce wrong decisions on startup | Always wait for the initial cache sync before processing the workqueue |
| Writing `status` unconditionally on every reconcile with no change filtering | Self-triggers an immediate re-reconcile, causing a tight loop | Use a status-change predicate, or only write status when it actually changed |
| Editing an existing CRD version's schema in place instead of adding a new version | Existing stored objects under the old schema silently break or lose data | Add a new served version with a conversion webhook, per the hub-and-spoke pattern |
| Uninstalling an Operator without checking for live finalizers on its CRDs | Every remaining object with that finalizer gets stuck in `Terminating` forever | Confirm zero live custom resources still carry the Operator's finalizer before removing it |
| Hand-rolling retry loops instead of using the workqueue's `AddRateLimited` | Bypasses built-in exponential backoff, risking a request-volume storm against the API server | Always route retries through the workqueue, never a bespoke immediate-retry loop |
| Granting an Operator's ServiceAccount broad, hand-edited RBAC "to be safe" | Breaks the least-privilege guarantee the `//+kubebuilder:rbac` marker discipline is designed to enforce | Keep RBAC generated from markers tied to actual reconciler code, per Part 11's least-privilege discipline |

## Worked Practice Problems

**Problem 1**: A `Reconciler`'s workqueue depth grows steadily even though `controller_runtime_reconcile_total`
shows a healthy, low error rate. What does this combination of signals suggest, and what should the next
diagnostic step be?

*Answer*: A low error rate rules out reconciles failing and re-queuing via `AddRateLimited` as the cause —
the queue is growing because keys are being *added* faster than they're being processed successfully, not
because of retries. The next step is checking whether one specific object is being enqueued at a
disproportionate rate compared to others (this chapter's runaway-reconcile worked scenario), which usually
points to a self-triggering status write or an external system generating an abnormal volume of legitimate
update events, rather than a broad, uniform event-volume increase across every watched object.

**Problem 2**: A team adds a finalizer to their CRD's objects to guarantee an external cloud resource gets
deprovisioned on deletion, tests it successfully, and later uninstalls the Operator entirely as part of a
platform migration — without first checking for any remaining live custom resources. What happens to any
object that still had the finalizer set, and what's the correct recovery?

*Answer*: Every remaining object with that finalizer becomes permanently stuck in `Terminating` the moment
someone tries to delete it, since nothing is left running to observe the deletion, perform the external
cleanup, and remove the finalizer — Kubernetes itself has no timeout or override for an unmet finalizer.
The correct recovery is manually patching the finalizer list to empty (`kubectl patch ... --type=merge`) to
unblock the object's actual deletion, while explicitly accepting that the external cloud resource the
finalizer was supposed to guarantee gets cleaned up will now need to be found and deprovisioned by hand,
since the code that would have done it no longer exists.

**Problem 3**: A CRD's `v1alpha1` schema has `spec.replicas` as an integer. A new `v1` version changes this
to `spec.scaling.replicas`, nested one level deeper, with a conversion webhook correctly implemented and
`v1` set as the storage version. Six months later, an old client that has never been updated still submits
objects using the flat `v1alpha1` shape. Does this client's request succeed, and why?

*Answer*: Yes — this is exactly the scenario conversion webhooks exist to support. The API server accepts
the `v1alpha1`-shaped request, and because `v1alpha1` is still a *served* version with a working conversion
webhook, the server calls the webhook to convert the request into the `v1` (storage) shape before
persisting it, and can convert it back to `v1alpha1` shape whenever that old client subsequently reads it
back. The old client can keep working indefinitely without ever knowing `v1` exists, as long as
`v1alpha1` remains both served and correctly convertible — the entire point of the hub-and-spoke pattern is
letting exactly this kind of client keep working through an API evolution it's never been updated to know
about.

## Summary and What's Next

Building a real Kubernetes controller comes down to a small number of well-defined layers stacked on each
other: a `SharedInformer` maintaining a live local cache so `Reconcile` never has to hit the API server just
to read state, a workqueue turning a noisy stream of watch events into deduplicated, rate-limited,
retryable work items, and a `Reconcile` function that's correct precisely because it re-derives full desired
state on every call rather than trusting what specifically triggered it. `controller-runtime` and Kubebuilder
don't replace any of that mechanism — they generate the wiring around it, so an Operator author's real job is
writing the domain-specific reconciliation logic (this chapter's `TenantOnboarding` operator, turning Part
13's manual onboarding checklist into continuously-enforced code) plus getting owner references, finalizers,
and CRD versioning right for the specific object graph and schema evolution a real Operator eventually needs.

The anti-patterns covered in this chapter — edge-triggered logic, self-triggering status loops, dangling
finalizers, bypassed rate limiting, and unsafe in-place schema changes — account for the overwhelming
majority of real Operator incidents in production, and every one of them traces back to skipping a mechanism
this chapter walked through deliberately: the workqueue's deduplication and backoff, the status-subresource
split, the hub-and-spoke conversion pattern, and `envtest`'s ability to catch these classes of bug against a
real API server before they ever reach a live cluster. These same foundations — client-go's informer
architecture, controller-runtime's reconciliation model, and the operational discipline this chapter's
worked scenarios walked through — are exactly what every Operator this series has referenced throughout
(the Prometheus Operator, KServe, the Kubeflow Training Operator, cert-manager, and this chapter's own
`tenant-operator`) is built on, and they continue to be the right lens for evaluating any new Operator this
series' throughline system adopts as its platform requirements keep growing.

## Where to Go From Here

**This series was scoped, from Part 10 onward, against the certification landscape that maps most directly
onto real production Kubernetes ownership — worth naming explicitly now that all nineteen parts are
complete.**

| Certification | This series' coverage |
|---|---|
| CKA (Certified Kubernetes Administrator) | Parts 1-3 (architecture, workloads, networking/storage), Part 10 (troubleshooting — CKA's single largest domain), Part 15 (cluster lifecycle), Part 16 (capacity planning) |
| CKAD (Certified Kubernetes Application Developer) | Parts 2-3 (workload/application objects), Part 12 (autoscaling from an application-owner's perspective) |
| CKS (Certified Kubernetes Security Specialist) | Part 11, mapped domain-by-domain against the official CKS curriculum, plus Part 17's Cilium network-policy depth |
| KCNA/KCSA (Cloud Native Associate tracks) | Parts 1, 4, 8-9 (fundamentals, service mesh, Gateway API) |
| CNCF-adjacent (Cilium/eBPF, Backstage-style platform engineering) | Part 17 (Cilium & eBPF), Part 13 (multi-tenancy/platform engineering), Part 19 (this chapter — Operator authorship) |

For hands-on practice beyond this series' worked examples, the companion repository
`devops-msrashed-com-handson` holds real, clonable exercises for several of the patterns covered across
these nineteen parts — check it for `init`/`start`/`complete` progressions matching a given part's topic
before building a lab environment from scratch. A local `kind` or `k3d` cluster is enough to practice most
of this series' cluster-level exercises (Part 15's kubeadm upgrade sequence, this chapter's `envtest`-based
Operator testing) safely and repeatedly, without any of the cost or blast-radius concerns of practicing
against real cloud infrastructure first — reserve a real cloud cluster for the parts that are genuinely
provider-specific (Parts 5, 7, 9, 14's GPU node pools).

## Closing the Series

Nineteen parts ago, this series opened with the single question every infrastructure engineer eventually
has to answer for themselves: what problem does Kubernetes actually solve? The answer — continuously
reconciling declared intent against real-world state — turned out to be the thread running through every
part since: the Scheduler reconciling pod placement against node capacity, a Deployment reconciling replica
count against a rolling update, an HPA reconciling replica count against a load metric, a Capsule Tenant
reconciling policy against a growing set of namespaces, an operator (Part 15) reconciling a cluster's
actual running version against the version it needs to be at, and finally, in this chapter, the exact
`SharedInformer`/workqueue/`Reconcile` machinery that makes every one of those reconciliation loops
possible in the first place — the mechanism this whole series kept pointing at was, this whole time, simply
a controller like the one built in this chapter.

**No single part of this series stands alone.** Troubleshooting (Part 10) is architecture knowledge run in
reverse. Security (Part 11) is the admission chain from Part 1 taken to its full depth. Autoscaling (Part
12) depends on the resource model from Part 2. Multi-tenancy (Part 13) composes RBAC, NetworkPolicy, and
quotas from across the whole series. AI/ML workloads (Part 14) apply every one of those same primitives
under GPU-shaped constraints. Cluster upgrades (Part 15) depend on nearly everything that came before them.
Capacity planning (Part 16) turns Part 12's reactive autoscaling into a deliberately-sized ceiling. Cilium
and eBPF (Part 17) replace Part 3's foundational networking model with a faster, identity-based one without
changing any of the concepts built on top of it. Backup and disaster recovery (Part 18) is Part 4's etcd
backup story extended to the workload data etcd alone never covered. And this closing chapter is the
`controller-runtime` machinery every Operator referenced across all eighteen prior parts was quietly built
on.

The throughline system — `checkout`, `catalog`, `inventory`, `recommendations` — was never really the
point; it was a fixed, familiar reference point to keep every new mechanism grounded in something concrete
rather than abstract. The actual point was always the reconciliation loop itself, showing up again and again
at every layer, from a single Pod's container state to an entire fleet of clusters staying current across
their supported version window, to the custom controller code in this very chapter. That's the idea worth
carrying forward past the last page of this series: Kubernetes is not a fixed set of features to memorize,
it's one small idea — desired state, continuously reconciled — applied consistently at every layer of a
genuinely large, genuinely production system, all the way down to the Go code that makes the idea real.

That consistency is also why this series was worth writing as nineteen connected parts rather than
nineteen independent ones. A reader who works through the whole thing shouldn't come away having memorized
nineteen separate tool inventories — HPA's formula, Capsule's `Tenant` CRD, vLLM's queue-depth metric,
kubeadm's exact upgrade command sequence, Cilium's identity-based policy model, a workqueue's backoff
behavior. They should come away recognizing the same handful of ideas wearing different clothes each time:
a controller watching for drift and correcting it, a staged rollout that proves safety before committing to
it, and a boundary — RBAC, a NetworkPolicy, a ResourceQuota, a version skew tolerance, a CRD schema —
drawn deliberately around exactly the blast radius that boundary is meant to contain. Recognizing that
pattern is what makes the *next* Kubernetes feature, the one this series didn't cover because it doesn't
exist yet, learnable in an afternoon instead of a fresh multi-week research project.