Building Custom Controllers and Operators

Assumes you're comfortable with the reconciliation loop and the Operator pattern's conceptual shape from Part 4, and RBAC/ServiceAccount mechanics from Part 11 — this chapter goes deep on the actual client-go and controller-runtime mechanics Part 4 only sketched, rather than re-introducing the Operator pattern itself.

Table of Contents#

1. Why This Part Exists
2. Recap: The Reconciliation Loop, and Where Part 4 Left Off
3. client-go Architecture: Informers, Listers, and the Local Cache
4. Inside the SharedInformer: Reflector, DeltaFIFO, and the Indexer
5. The Workqueue: Deduplication, Rate Limiting, and Backoff
6. A Minimal Raw client-go Controller, End to End
7. controller-runtime's Manager, Cache, and Client
8. Kubebuilder Project Scaffolding, Walked Through
9. Designing the TenantOnboarding Operator
10. Writing the Reconciler
11. Owner References and Garbage Collection Between Custom Resources
12. Finalizers in Practice — Safe Cleanup Before Deletion
13. CRD Versioning and Conversion Webhooks in Practice
14. Validating and Mutating Admission Webhooks for Custom Resources
15. Status Subresources and Conditions
16. Testing Controllers with envtest
17. Leader Election and Running Operators Highly Available
18. Upgrading an Operator Safely — CRD and Controller Rollout Order
19. Observability for Operators — Metrics, Events, and Logs
20. Common Controller Anti-Patterns, Explained Two Levels Deep
21. A Full Worked Scenario: Debugging a Runaway Reconcile Loop in Production
22. A Full Worked Scenario: A Botched CRD Migration Without a Conversion Webhook
23. Part 19 CLI Cheat Sheet
24. Common Mistakes and Interview Traps
25. Worked Practice Problems
26. Summary and What's Next
27. Where to Go From Here
28. Closing the Series

Why This Part Exists#

Part 4 introduced the Operator pattern conceptually — a Reconcile function sketch, the "controller-runtime handles the boilerplate" framing, finalizers and idempotency named as principles — and left every mechanism underneath that sketch unexplained: how does a controller actually learn that an object changed, without polling the API server in a tight loop? What exactly queues a reconcile request, and what stops the same object from being reconciled twice concurrently? This chapter answers those questions at the actual client-go/controller-runtime mechanics level, then builds a real, working Operator from scaffolding through tests.

The throughline gains its most code-heavy addition yet: a TenantOnboarding custom resource and its tenant-operator, automating exactly the manual checklist Part 13 introduced for bringing a new tenant namespace onto the shared cluster — a ResourceQuota, NetworkPolicy, and baseline RoleBinding, applied consistently instead of by hand. Building this operator over the course of the chapter is a deliberate illustration that Part 13's onboarding checklist and this chapter's reconciler are the same operational knowledge, expressed two different ways — a human-followed runbook versus continuously-running code.

Recap: The Reconciliation Loop, and Where Part 4 Left Off#

Every controller in Kubernetes, built-in or custom, follows the same three-step loop Part 1 introduced and Part 4 applied to Operators specifically: observe the current state, compare it to the desired state, act to close any gap. What Part 4 didn't cover is how a controller efficiently observes state for millions of objects across a live cluster without either polling constantly or missing changes — that's the specific gap the next three sections close, in the order a request for a new object actually flows through the machinery.

client-go Architecture: Informers, Listers, and the Local Cache#

A naive controller could kubectl get every object it cares about on every reconcile — this would work correctness-wise and destroy the API server's performance the moment more than a handful of controllers did it simultaneously. The Informer pattern exists specifically to solve this: watch once, cache locally, read from the cache everywhere.

Every controller watching the same resource type shares this exact same cache — a SharedInformer means one watch connection to the API server serves every controller in the process, not one watch per controller.

The critical performance property worth stating precisely: a Lister's Get/List calls never touch the API server at all — they read directly from the local Indexer, an in-memory cache kept current by the Reflector's ongoing watch. A controller's Reconcile function calling Get a thousand times across a busy reconcile cycle costs the API server exactly zero additional requests, which is precisely what makes it safe for dozens of controllers to run in the same cluster without coordinating their read patterns.

Inside the SharedInformer: Reflector, DeltaFIFO, and the Indexer#

Tracing exactly what happens between "an object changed on the API server" and "the local cache reflects it" is worth doing once precisely, since a surprising number of subtle controller bugs trace back to a misunderstanding of this exact sequence.

The cache update and the handler notification happen from the same Pop() call — an event handler can always trust that the Indexer already reflects the change it was just notified about, never a stale prior version.

DeltaFIFO's specific job is worth naming precisely: it's a FIFO queue of deltas (add/update/delete), not a queue of full objects, and it compresses redundant deltas for the same key — if an object is updated three times before a consumer processes it, DeltaFIFO collapses those into the object's current state rather than replaying three separate stale intermediate versions. This compression is exactly why a well-written Reconcile must never assume it's being told "what changed" — by the time it runs, several raw watch events for the same object may have already been folded into one.

The Indexer adds one further capability beyond a plain cache: secondary indexes, letting a controller efficiently look up objects by something other than name — e.g., "every TenantOnboarding whose spec.namespace matches X" — without a linear scan across every cached object, the same underlying idea as a database index applied to an in-memory object cache.

The Workqueue: Deduplication, Rate Limiting, and Backoff#

A raw stream of watch events, delivered straight to Reconcile one at a time, has a real problem: if the same object changes five times in a second, a naive controller would run Reconcile five times, when one reconcile against the object's current state would have been enough — the workqueue exists specifically to collapse this down and add retry discipline on top.

queue := workqueue.NewTypedRateLimitingQueue(
    workqueue.DefaultTypedControllerRateLimiter[string](),
)

// Event handler: never pushes the object itself, only its key
informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: func(obj interface{}) {
        key, _ := cache.MetaNamespaceKeyFunc(obj)
        queue.Add(key)
    },
    UpdateFunc: func(old, new interface{}) {
        key, _ := cache.MetaNamespaceKeyFunc(new)
        queue.Add(key)
    },
})

A Minimal Raw client-go Controller, End to End#

Seeing the informer, workqueue, and a bare reconcile function wired together directly — without controller-runtime's abstraction over them — is worth doing once, purely so the abstraction in the next section reads as "the same thing, with the boilerplate factored out" rather than as unfamiliar magic.

func Run(ctx context.Context, client kubernetes.Interface) error {
    factory := informers.NewSharedInformerFactory(client, 30*time.Second)
    podInformer := factory.Core().V1().Pods()
    queue := workqueue.NewTypedRateLimitingQueue(
        workqueue.DefaultTypedControllerRateLimiter[string](),
    )

    podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc:    func(o interface{}) { enqueue(queue, o) },
        UpdateFunc: func(_, o interface{}) { enqueue(queue, o) },
        DeleteFunc: func(o interface{}) { enqueue(queue, o) },
    })

    factory.Start(ctx.Done())
    factory.WaitForCacheSync(ctx.Done())   // block until the initial List completes

    for processNextItem(queue, podInformer.Lister()) {
    }
    return nil
}

func processNextItem(q workqueue.TypedRateLimitingInterface[string], lister v1.PodLister) bool {
    key, shutdown := q.Get()
    if shutdown {
        return false
    }
    defer q.Done(key)

    if err := reconcile(key, lister); err != nil {
        q.AddRateLimited(key)   // retry with backoff
        return true
    }
    q.Forget(key)
    return true
}

WaitForCacheSync deserves its own emphasis: reconciling before the informer's initial List has finished populating the cache means every Get/List call sees a partially-populated, misleadingly-empty view of the cluster — a controller that skips this call can make genuinely wrong decisions on startup (e.g., concluding an object doesn't exist yet, when it simply hasn't synced into the cache), a real and easy-to-introduce bug in a hand-rolled controller that controller-runtime's Manager handles automatically, covered next.

controller-runtime's Manager, Cache, and Client#

Almost no real-world Operator is written directly against raw client-go anymore — controller-runtime wraps informers, listers, and the workqueue behind a small set of well-designed abstractions, and Kubebuilder and Operator SDK both scaffold projects on top of it.

The Client your Reconciler calls looks like a single, ordinary Kubernetes client — reads are transparently served from the Cache (the same Indexer mechanism from earlier), writes go straight to the API server, and the Reconciler code never has to know or care about that split.

Kubebuilder Project Scaffolding, Walked Through#

Kubebuilder generates a complete, working project skeleton from a single command — worth understanding what each generated piece actually is, rather than treating the scaffold as opaque boilerplate to work around.

kubebuilder init --domain platform.example.com --repo github.com/example/tenant-operator
kubebuilder create api --group platform --version v1alpha1 --kind TenantOnboarding --resource --controller

Two of these generated pieces are worth calling out specifically, since they're easy to under-appreciate as "just generated boilerplate":

• config/rbac/role.yaml is generated from //+kubebuilder:rbac marker comments directly above the Reconcile function — the exact least-privilege discipline Part 11 covered manually for application ServiceAccounts is here derived automatically from the code that actually needs each permission, rather than hand-maintained separately from what the controller does.
• config/crd/bases/*.yaml is generated from Go struct tags on the Spec/Status types — the CRD's OpenAPI schema (required fields, enums, validation) lives as Go type annotations, not as hand-written YAML kept in sync with the code by hand.

//+kubebuilder:rbac:groups=platform.example.com,resources=tenantonboardings,verbs=get;list;watch;update
//+kubebuilder:rbac:groups="",resources=resourcequotas;namespaces,verbs=get;list;watch;create;update
//+kubebuilder:rbac:groups=networking.k8s.io,resources=networkpolicies,verbs=get;list;watch;create;update
func (r *TenantOnboardingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // ...
}

Designing the TenantOnboarding Operator#

Before writing the Reconcile function itself, the CRD's schema is the actual design decision — what a platform team writes to onboard a tenant, and what the operator is responsible for turning that into.

apiVersion: platform.example.com/v1alpha1
kind: TenantOnboarding
metadata:
  name: recommendations-team
spec:
  namespace: recommendations
  resourceQuota:
    cpu: "20"
    memory: 64Gi
    gpuCount: 4
  networkPolicy:
    allowEgressTo: ["catalog", "kube-system"]

This is a direct, literal translation of Part 13's onboarding checklist and Part 14's GPU quota discussion into a declarative spec — a platform engineer applies one TenantOnboarding object instead of manually running through a multi-step checklist, and the operator is responsible for making the Namespace, ResourceQuota, and NetworkPolicy objects match what that spec declares, continuously, the same "encode the runbook as code" principle Part 4 introduced generally.

Writing the Reconciler#

The Reconcile function itself, written the correct, idempotent, level-based way Part 4 named as a principle — every call re-derives the full desired state and applies it, regardless of what specifically triggered this particular invocation.

func (r *TenantOnboardingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var onboarding platformv1alpha1.TenantOnboarding
    if err := r.Get(ctx, req.NamespacedName, &onboarding); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: onboarding.Spec.Namespace}}
    if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, ns, func() error {
        return nil // Namespace has no spec fields this operator manages beyond existence
    }); err != nil {
        return ctrl.Result{}, err
    }

    quota := buildResourceQuota(&onboarding)
    if err := controllerutil.SetControllerReference(&onboarding, quota, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }
    if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, quota, func() error {
        quota.Spec = buildResourceQuotaSpec(&onboarding)
        return nil
    }); err != nil {
        return ctrl.Result{}, err
    }

    onboarding.Status.Ready = true
    if err := r.Status().Update(ctx, &onboarding); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

controllerutil.CreateOrUpdate is doing the actual "observe, compare, act" work in three lines: it reads the current object if one exists, applies the mutation function, and only issues a write if something actually changed — this is the concrete mechanism behind "idempotent, level-based reconciliation," not an abstract principle floating separate from real code.

From the Trenches: An early version of this reconciler set the ResourceQuota's spec unconditionally on every reconcile, without wrapping the mutation in CreateOrUpdate's diff-and-only-write-if-changed behavior. Under normal operation this was invisible — but during a control-plane slowdown that caused the TenantOnboarding informer to redeliver a backlog of stale-looking update events, the operator issued thousands of no-op ResourceQuota writes in a few minutes, each one triggering its own watch event to every other controller watching ResourceQuota objects cluster-wide. The immediate cause was skipping the diff-before-write step; the underlying condition was that "idempotent" was treated as "safe to call repeatedly" without also asking whether repeated calls were cheap for the rest of the cluster watching the same object type — a distinction that only matters at exactly the moment a backlog of events actually arrives.

Owner References and Garbage Collection Between Custom Resources#

Every object the TenantOnboarding reconciler creates should carry an owner reference back to the TenantOnboarding object itself — controllerutil.SetControllerReference, used in the reconciler above, is what actually establishes this, and it's what makes deleting the parent object clean up everything it created automatically, without the reconciler ever writing explicit delete logic for most of its children.

Owner references get this specific cleanup case (namespaced children of a namespaced parent) for free from Kubernetes's built-in garbage collector — no reconciler code is needed to delete the ResourceQuota/ NetworkPolicy when the TenantOnboarding object is deleted.

Finalizers in Practice — Safe Cleanup Before Deletion#

The Namespace object the previous section flagged as un-owner-referenceable is exactly the case a finalizer solves: cleanup logic the operator must run before Kubernetes considers the TenantOnboarding object actually gone, for anything owner references can't reach automatically.

const tenantFinalizer = "platform.example.com/tenant-cleanup"

func (r *TenantOnboardingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var onboarding platformv1alpha1.TenantOnboarding
    if err := r.Get(ctx, req.NamespacedName, &onboarding); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    if onboarding.DeletionTimestamp.IsZero() {
        if !controllerutil.ContainsFinalizer(&onboarding, tenantFinalizer) {
            controllerutil.AddFinalizer(&onboarding, tenantFinalizer)
            return ctrl.Result{}, r.Update(ctx, &onboarding)
        }
    } else {
        // Object is being deleted — run cleanup BEFORE removing the finalizer
        if controllerutil.ContainsFinalizer(&onboarding, tenantFinalizer) {
            if err := r.deleteNamespace(ctx, onboarding.Spec.Namespace); err != nil {
                return ctrl.Result{}, err   // retry — finalizer stays, object stays Terminating
            }
            controllerutil.RemoveFinalizer(&onboarding, tenantFinalizer)
            return ctrl.Result{}, r.Update(ctx, &onboarding)
        }
        return ctrl.Result{}, nil
    }
    // ... normal reconciliation continues below
    return ctrl.Result{}, nil
}

An object stuck in Terminating forever is almost always this exact diagram, stuck on the self-loop — cleanup keeps failing, or the controller that owns the finalizer isn't running at all to ever attempt it.

CRD Versioning and Conversion Webhooks in Practice#

Part 4 covered the hub-and-spoke concept abstractly — here's what actually changes when TenantOnboarding evolves from v1alpha1 to v1, renaming spec.resourceQuota to a more general spec.quota that can eventually cover non-GPU resource types uniformly.

kubebuilder create api --group platform --version v1 --kind TenantOnboarding --resource --controller=false

// v1 is the hub — v1alpha1 implements Convertible against it
func (src *TenantOnboarding) ConvertTo(dstRaw conversion.Hub) error {
    dst := dstRaw.(*v1.TenantOnboarding)
    dst.Spec.Quota = src.Spec.ResourceQuota   // field rename happens here
    dst.Spec.Namespace = src.Spec.Namespace
    return nil
}

func (dst *TenantOnboarding) ConvertFrom(srcRaw conversion.Hub) error {
    src := srcRaw.(*v1.TenantOnboarding)
    dst.Spec.ResourceQuota = src.Spec.Quota
    dst.Spec.Namespace = src.Spec.Namespace
    return nil
}

# config/crd/bases — kubebuilder-generated, wiring the webhook in
versions:
  - name: v1alpha1
    served: true
    storage: false
  - name: v1
    served: true
    storage: true
conversion:
  strategy: Webhook
  webhook:
    conversionReviewVersions: ["v1alpha1", "v1"]

controller-runtime generates and serves the /convert HTTP endpoint automatically from the ConvertTo/ConvertFrom methods above — inside the same manager binary as the reconciler itself, with no separate webhook deployment to run. The only code an operator author writes is the two conversion functions; the HTTP handling, TLS, and API server integration are all generated scaffolding.

Validating and Mutating Admission Webhooks for Custom Resources#

Part 1's admission chain and Part 4's sidecar-injection mutating webhook both applied to built-in resources — the exact same admission mechanism is available for a custom resource too, and Kubebuilder scaffolds it the same way it scaffolds a conversion webhook.

Validation runs strictly after mutation — a validating webhook always sees the final, already-defaulted object, never the raw request a user actually submitted.

//+kubebuilder:webhook:path=/validate-platform-example-com-v1-tenantonboarding,mutating=false,failurePolicy=fail,groups=platform.example.com,resources=tenantonboardings,verbs=create;update,versions=v1,name=vtenantonboarding.kb.io

func (v *TenantOnboardingValidator) ValidateCreate(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {
    onboarding := obj.(*platformv1alpha1.TenantOnboarding)
    if onboarding.Spec.ResourceQuota.GPUCount > clusterAvailableGPUs(ctx) {
        return nil, fmt.Errorf(
            "requested GPU count %d exceeds cluster capacity", onboarding.Spec.ResourceQuota.GPUCount,
        )
    }
    return nil, nil
}

Status Subresources and Conditions#

The Status().Update() call in this chapter's reconciler writes to a genuinely separate API endpoint from a plain Update() — the status subresource — and understanding why that split exists prevents a specific, common source of unnecessary reconcile churn.

type TenantOnboardingStatus struct {
    Ready      bool               `json:"ready"`
    Conditions []metav1.Condition `json:"conditions,omitempty"`
}

Conditions (a standard, structured slice of type/status/reason/message/lastTransitionTime) is the conventional way an Operator communicates why it's in a given state, not just whether it is — kubectl describe tenantonboarding renders these directly, giving a platform engineer the same kind of human-readable status a built-in Deployment's kubectl describe output already provides.

meta.SetStatusCondition(&onboarding.Status.Conditions, metav1.Condition{
    Type:    "QuotaProvisioned",
    Status:  metav1.ConditionTrue,
    Reason:  "ResourceQuotaCreated",
    Message: "ResourceQuota successfully applied to the tenant namespace",
})

Testing Controllers with envtest#

envtest starts a real kube-apiserver and etcd binary locally — no mocking of the Kubernetes API at all — making it the right tool for testing what a Reconciler actually does against real API server behavior (admission, defaulting, status subresource semantics) rather than a hand-built fake that might not match.

var _ = Describe("TenantOnboarding controller", func() {
    It("creates a ResourceQuota matching the spec", func() {
        onboarding := &platformv1alpha1.TenantOnboarding{
            ObjectMeta: metav1.ObjectMeta{Name: "test-tenant"},
            Spec: platformv1alpha1.TenantOnboardingSpec{
                Namespace:     "test-tenant-ns",
                ResourceQuota: platformv1alpha1.QuotaSpec{CPU: "4", Memory: "8Gi"},
            },
        }
        Expect(k8sClient.Create(ctx, onboarding)).To(Succeed())

        Eventually(func() error {
            var quota corev1.ResourceQuota
            return k8sClient.Get(ctx, types.NamespacedName{
                Name: "test-tenant-quota", Namespace: "test-tenant-ns",
            }, &quota)
        }, "10s", "250ms").Should(Succeed())
    })
})

Leader Election and Running Operators Highly Available#

Running two replicas of an Operator for availability creates an obvious risk if both are allowed to reconcile simultaneously: two concurrent, possibly conflicting writes to the same object from two processes that don't know about each other — leader election exists specifically to prevent this.

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    LeaderElection:   true,
    LeaderElectionID: "tenant-operator-leader",
})

Only the elected leader replica actually runs any Reconcile calls — every standby replica's Manager sits idle, ready to acquire leadership (via a Lease object, the same primitive Part 1's own control-plane leader election covers for kube-controller-manager/kube-scheduler) the moment the current leader stops renewing it, whether from a graceful shutdown or a crash. This gives an Operator deployment genuine high availability — a second replica ready to take over within the lease's configured renewal window — without ever risking two active reconcilers stepping on each other's writes.

Upgrading an Operator Safely — CRD and Controller Rollout Order#

Part 15 covered cluster upgrade sequencing in general — an Operator upgrade has its own, narrower version of the same "order matters" lesson, specifically around the relationship between a CRD's schema and the controller code that reads it.

The same "expand, then contract" migration discipline a relational-database schema change uses applies directly here — add the new field as optional first, roll out code that can handle both its presence and absence, and only make it mandatory once every object in the cluster is confirmed to already have it.

Observability for Operators — Metrics, Events, and Logs#

controller-runtime exposes a Prometheus metrics endpoint automatically, without any extra code, giving every custom controller the same observability baseline Part 10 established for cluster-wide monitoring generally.

Kubernetes Event objects are the other half of Operator observability, and the half most often skipped — emitting a normal, human-readable Event (r.Recorder.Event(&onboarding, corev1.EventTypeNormal, "QuotaProvisioned", "ResourceQuota created for tenant namespace")) surfaces directly in kubectl describe tenantonboarding, giving a platform engineer immediate, in-context troubleshooting information without needing to correlate a separate log stream or metrics dashboard at all.

Common Controller Anti-Patterns, Explained Two Levels Deep#

Every pattern below has caused a real, recurring class of production incident across the Operator ecosystem — worth internalizing the "why," not just memorizing the list.

• Edge-triggered logic instead of level-based. Symptom: a controller behaves correctly under normal operation but silently drifts from desired state after a missed event (a controller restart during a busy window, a dropped watch reconnect). Immediate cause: Reconcile branches on "what just changed" instead of re-deriving the full desired state from scratch every time. Underlying condition: client-go/ controller-runtime explicitly do not guarantee every individual event is delivered — restarts, resyncs, and DeltaFIFO compression (covered earlier) can all legitimately collapse or skip intermediate states, and only a reconciler that treats every invocation as "figure out what's needed right now" is actually correct under that guarantee.
• Reconcile loop explosions from self-triggered status updates. Symptom: a controller's CPU and API server request volume climb steadily with no corresponding change in real cluster activity. Immediate cause: the reconciler watches its own resource type and unconditionally writes to status on every reconcile, and that status write itself re-triggers a watch event, creating a tight self-sustaining loop. Underlying condition: the reconciler never applied the predicate-based status-change filtering covered earlier, so it can't distinguish "something meaningful changed" from "I just wrote my own status update a moment ago."
• Missing or dangling finalizers. Symptom: objects stuck in Terminating indefinitely, or (the opposite failure) external resources silently orphaned after a CR is deleted. Immediate cause: either a finalizer whose owning controller isn't running, or a controller that manages external state but never registered a finalizer to guarantee cleanup runs at all. Underlying condition: finalizer lifecycle wasn't treated as part of the Operator's own uninstall/upgrade runbook — this chapter's earlier warning about checking for live finalizers before uninstalling an Operator exists precisely because this gets missed.
• No rate limiting or backoff on a persistently failing reconcile. Symptom: a single misconfigured custom resource generates a continuous, high-volume stream of API server requests. Immediate cause: a hand-rolled retry loop that calls Reconcile again immediately on error, bypassing the workqueue's built-in AddRateLimited backoff entirely. Underlying condition: treating the workqueue as optional plumbing rather than the actual mechanism responsible for retry discipline — this is exactly why the raw client-go example earlier in this chapter routes every retry through q.AddRateLimited, never a bespoke loop.
• Overly broad RBAC "to keep things working." Symptom: an Operator's ServiceAccount can read/write far more than its actual reconciler logic touches. Immediate cause: a wildcard grant added once to make an early RBAC error go away, never revisited. Underlying condition: the //+kubebuilder:rbac marker discipline from the scaffolding section exists specifically to keep generated RBAC tied to what the code actually does — bypassing it with a hand-edited wildcard Role breaks that guarantee the same way Part 11 flagged for application ServiceAccounts generally.

A Full Worked Scenario: Debugging a Runaway Reconcile Loop in Production#

tenant-operator's workqueue_depth metric alarms at 3 a.m. — the queue is growing continuously, and the operator's Deployment is being OOMKilled (Part 10) every few minutes and restarting.

The investigation finds recommendations-team's TenantOnboarding object being reconciled roughly 40 times per second, while every other tenant reconciles normally — pointing straight at the "single object, disproportionate volume" branch, not a general event storm. Reading the reconciler code confirms this chapter's own earlier "From the Trenches" mistake was reintroduced during a recent refactor: a new onboarding.Status.LastSyncTime = time.Now() line was added to the reconciler unconditionally, on every pass, with no predicate filtering it out — every reconcile now writes a status change that immediately re-triggers the next reconcile. The fix is exactly the predicate-based status filtering from the earlier Status Subresources section, applied in code review going forward as a specific thing to check for on any diff touching a Reconcile function's status-writing path.

A Full Worked Scenario: A Botched CRD Migration Without a Conversion Webhook#

A different team, maintaining a separate CachePolicy CRD, renames spec.ttlSeconds to spec.ttl (a string with unit suffixes, like "5m") directly in the existing CRD's schema — without adding a second served version or a conversion webhook — and rolls the new controller version out.

1. Every existing CachePolicy object already in etcd was written under the old schema — spec.ttlSeconds: 300, an integer field the new schema no longer defines.
2. The new controller's Go struct expects spec.ttl as a string — reading an existing object either silently drops the now-unrecognized ttlSeconds field (if using an unstructured/loose decode path) or fails validation outright on the next write, depending on exactly how the schema change was applied.
3. Every pre-existing CachePolicy object is now either broken or has quietly lost its configured TTL, discovered only when several caches stop expiring entries on schedule days later.
4. The correct approach, using this chapter's tools: add v1 alongside the existing v1alpha1 version rather than mutating the single existing version's schema in place, implement ConvertTo/ConvertFrom translating ttlSeconds (int) to ttl (string) and back, keep v1alpha1 as storage: true until every client and controller is confirmed running the new code, and only then flip the storage version and run the migrator job to rewrite existing objects.

Part 19 CLI Cheat Sheet#

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A Reconciler's workqueue depth grows steadily even though controller_runtime_reconcile_total shows a healthy, low error rate. What does this combination of signals suggest, and what should the next diagnostic step be?

Answer: A low error rate rules out reconciles failing and re-queuing via AddRateLimited as the cause — the queue is growing because keys are being added faster than they're being processed successfully, not because of retries. The next step is checking whether one specific object is being enqueued at a disproportionate rate compared to others (this chapter's runaway-reconcile worked scenario), which usually points to a self-triggering status write or an external system generating an abnormal volume of legitimate update events, rather than a broad, uniform event-volume increase across every watched object.

Problem 2: A team adds a finalizer to their CRD's objects to guarantee an external cloud resource gets deprovisioned on deletion, tests it successfully, and later uninstalls the Operator entirely as part of a platform migration — without first checking for any remaining live custom resources. What happens to any object that still had the finalizer set, and what's the correct recovery?

Answer: Every remaining object with that finalizer becomes permanently stuck in Terminating the moment someone tries to delete it, since nothing is left running to observe the deletion, perform the external cleanup, and remove the finalizer — Kubernetes itself has no timeout or override for an unmet finalizer. The correct recovery is manually patching the finalizer list to empty (kubectl patch ... --type=merge) to unblock the object's actual deletion, while explicitly accepting that the external cloud resource the finalizer was supposed to guarantee gets cleaned up will now need to be found and deprovisioned by hand, since the code that would have done it no longer exists.

Problem 3: A CRD's v1alpha1 schema has spec.replicas as an integer. A new v1 version changes this to spec.scaling.replicas, nested one level deeper, with a conversion webhook correctly implemented and v1 set as the storage version. Six months later, an old client that has never been updated still submits objects using the flat v1alpha1 shape. Does this client's request succeed, and why?

Answer: Yes — this is exactly the scenario conversion webhooks exist to support. The API server accepts the v1alpha1-shaped request, and because v1alpha1 is still a served version with a working conversion webhook, the server calls the webhook to convert the request into the v1 (storage) shape before persisting it, and can convert it back to v1alpha1 shape whenever that old client subsequently reads it back. The old client can keep working indefinitely without ever knowing v1 exists, as long as v1alpha1 remains both served and correctly convertible — the entire point of the hub-and-spoke pattern is letting exactly this kind of client keep working through an API evolution it's never been updated to know about.

Summary and What's Next#

Building a real Kubernetes controller comes down to a small number of well-defined layers stacked on each other: a SharedInformer maintaining a live local cache so Reconcile never has to hit the API server just to read state, a workqueue turning a noisy stream of watch events into deduplicated, rate-limited, retryable work items, and a Reconcile function that's correct precisely because it re-derives full desired state on every call rather than trusting what specifically triggered it. controller-runtime and Kubebuilder don't replace any of that mechanism — they generate the wiring around it, so an Operator author's real job is writing the domain-specific reconciliation logic (this chapter's TenantOnboarding operator, turning Part 13's manual onboarding checklist into continuously-enforced code) plus getting owner references, finalizers, and CRD versioning right for the specific object graph and schema evolution a real Operator eventually needs.

The anti-patterns covered in this chapter — edge-triggered logic, self-triggering status loops, dangling finalizers, bypassed rate limiting, and unsafe in-place schema changes — account for the overwhelming majority of real Operator incidents in production, and every one of them traces back to skipping a mechanism this chapter walked through deliberately: the workqueue's deduplication and backoff, the status-subresource split, the hub-and-spoke conversion pattern, and envtest's ability to catch these classes of bug against a real API server before they ever reach a live cluster. These same foundations — client-go's informer architecture, controller-runtime's reconciliation model, and the operational discipline this chapter's worked scenarios walked through — are exactly what every Operator this series has referenced throughout (the Prometheus Operator, KServe, the Kubeflow Training Operator, cert-manager, and this chapter's own tenant-operator) is built on, and they continue to be the right lens for evaluating any new Operator this series' throughline system adopts as its platform requirements keep growing.

Where to Go From Here#

This series was scoped, from Part 10 onward, against the certification landscape that maps most directly onto real production Kubernetes ownership — worth naming explicitly now that all nineteen parts are complete.

For hands-on practice beyond this series' worked examples, the companion repository devops-msrashed-com-handson holds real, clonable exercises for several of the patterns covered across these nineteen parts — check it for init/start/complete progressions matching a given part's topic before building a lab environment from scratch. A local kind or k3d cluster is enough to practice most of this series' cluster-level exercises (Part 15's kubeadm upgrade sequence, this chapter's envtest-based Operator testing) safely and repeatedly, without any of the cost or blast-radius concerns of practicing against real cloud infrastructure first — reserve a real cloud cluster for the parts that are genuinely provider-specific (Parts 5, 7, 9, 14's GPU node pools).

Closing the Series#

Nineteen parts ago, this series opened with the single question every infrastructure engineer eventually has to answer for themselves: what problem does Kubernetes actually solve? The answer — continuously reconciling declared intent against real-world state — turned out to be the thread running through every part since: the Scheduler reconciling pod placement against node capacity, a Deployment reconciling replica count against a rolling update, an HPA reconciling replica count against a load metric, a Capsule Tenant reconciling policy against a growing set of namespaces, an operator (Part 15) reconciling a cluster's actual running version against the version it needs to be at, and finally, in this chapter, the exact SharedInformer/workqueue/Reconcile machinery that makes every one of those reconciliation loops possible in the first place — the mechanism this whole series kept pointing at was, this whole time, simply a controller like the one built in this chapter.

No single part of this series stands alone. Troubleshooting (Part 10) is architecture knowledge run in reverse. Security (Part 11) is the admission chain from Part 1 taken to its full depth. Autoscaling (Part 12) depends on the resource model from Part 2. Multi-tenancy (Part 13) composes RBAC, NetworkPolicy, and quotas from across the whole series. AI/ML workloads (Part 14) apply every one of those same primitives under GPU-shaped constraints. Cluster upgrades (Part 15) depend on nearly everything that came before them. Capacity planning (Part 16) turns Part 12's reactive autoscaling into a deliberately-sized ceiling. Cilium and eBPF (Part 17) replace Part 3's foundational networking model with a faster, identity-based one without changing any of the concepts built on top of it. Backup and disaster recovery (Part 18) is Part 4's etcd backup story extended to the workload data etcd alone never covered. And this closing chapter is the controller-runtime machinery every Operator referenced across all eighteen prior parts was quietly built on.

The throughline system — checkout, catalog, inventory, recommendations — was never really the point; it was a fixed, familiar reference point to keep every new mechanism grounded in something concrete rather than abstract. The actual point was always the reconciliation loop itself, showing up again and again at every layer, from a single Pod's container state to an entire fleet of clusters staying current across their supported version window, to the custom controller code in this very chapter. That's the idea worth carrying forward past the last page of this series: Kubernetes is not a fixed set of features to memorize, it's one small idea — desired state, continuously reconciled — applied consistently at every layer of a genuinely large, genuinely production system, all the way down to the Go code that makes the idea real.

That consistency is also why this series was worth writing as nineteen connected parts rather than nineteen independent ones. A reader who works through the whole thing shouldn't come away having memorized nineteen separate tool inventories — HPA's formula, Capsule's Tenant CRD, vLLM's queue-depth metric, kubeadm's exact upgrade command sequence, Cilium's identity-based policy model, a workqueue's backoff behavior. They should come away recognizing the same handful of ideas wearing different clothes each time: a controller watching for drift and correcting it, a staged rollout that proves safety before committing to it, and a boundary — RBAC, a NetworkPolicy, a ResourceQuota, a version skew tolerance, a CRD schema — drawn deliberately around exactly the blast radius that boundary is meant to contain. Recognizing that pattern is what makes the next Kubernetes feature, the one this series didn't cover because it doesn't exist yet, learnable in an afternoon instead of a fresh multi-week research project.