Service Mesh, etcd & Operators

Table of Contents#

1. The Problem a Service Mesh Actually Solves
2. The Sidecar Pattern, Fully Explained
3. Sidecar Injection Mechanics
4. Service Mesh Architecture: Data Plane vs Control Plane
5. mTLS — Automatic, Zero-Code Encryption Everywhere
6. mTLS Modes: STRICT vs PERMISSIVE
7. Traffic Management — Canary Releases Without Application Changes
8. Circuit Breaking and Retry Policies
9. Fault Injection — Chaos Engineering at the Mesh Layer
10. GAMMA — Gateway API for Service Mesh
11. Observability for Free — Revisited
12. The Real Cost of a Service Mesh
13. Multi-Cluster Mesh Federation
14. etcd — Operational Depth
15. etcd Compaction and Defragmentation
16. A Real etcd Disaster Recovery Drill
17. Custom Resource Definitions (CRDs) — Extending the API
18. CRD Versioning and Conversion Webhooks
19. The Operator Pattern
20. A Worked Operator Example
21. Building an Operator: The controller-runtime Pattern
22. Helm — Packaging Kubernetes Applications
23. Helm Chart Structure and Hooks
24. Part 4 CLI Cheat Sheet
25. Common Mistakes
26. Worked Practice Problems
27. Summary and What's Next

The Problem a Service Mesh Actually Solves#

Part 3 covered Services and how basic request routing works inside a cluster. But real production microservice architectures need more than "can this reach that" — they need encryption, retries, fine-grained traffic control, and observability, consistently, across every single service, without every single application team having to reimplement the same logic in their own code.

The Sidecar Pattern, Fully Explained#

This is the exact mechanical foundation a service mesh is built on — and it's a direct, concrete application of the "containers in the same pod share a network namespace" fact from Part 3.

The core trick, worth stating explicitly: iptables rules are configured (usually automatically, via an init container) inside the pod's network namespace to transparently redirect ALL inbound and outbound traffic through the sidecar proxy — the application code doesn't need to know this is happening at all; it just makes what looks like a completely normal network call, and the sidecar silently intercepts it, applies mTLS/retries/metrics, and forwards it on.

Sidecar Injection Mechanics#

How does a sidecar container actually end up in every pod, without every team manually adding it to their own manifests? This is a direct, concrete application of the mutating admission webhook mechanism from Part 1 — worth tracing through precisely.

Why namespace-level labeling (istio-injection: enabled), not per-pod configuration, is the standard pattern, worth stating explicitly: it means an entire team's namespace gets automatic injection for every workload deployed there, with zero per-manifest changes needed — a developer writes a completely normal pod spec, never mentioning a sidecar at all, and the mutating webhook transparently adds it during admission, exactly the mechanism covered in Part 1's admission-control chain deep dive.

A genuinely important, real architectural shift worth knowing about: "sidecarless" or "ambient" mesh modes (Istio's Ambient Mode being the most prominent current example) move mTLS/traffic-management enforcement OUT of a per-pod sidecar and into a shared, per-NODE proxy instead — trading some of the sidecar model's per-pod isolation for meaningfully lower resource overhead (no extra container per pod) and no need to restart existing pods to add/remove mesh membership. This is a genuinely active, evolving area of the service mesh ecosystem worth being aware of, not settled orthodoxy.

Service Mesh Architecture: Data Plane vs Control Plane#

Exactly the same conceptual split as Kubernetes's own architecture from Part 1, applied one layer up.

Simple analogy: the control plane is like a company's central policy office, writing the actual rules ("all deliveries must be insured, all packages logged"). The data plane is every individual delivery driver (sidecar) actually following those rules on every single delivery, in real time — the office doesn't personally handle any package; it just tells every driver what the rules are.

A commonly-cited real name worth knowing: Envoy is the dominant sidecar proxy implementation, used as the actual data plane by both Istio and (in some configurations) other mesh products — genuinely worth recognizing by name, since it comes up frequently.

mTLS — Automatic, Zero-Code Encryption Everywhere#

mTLS (mutual TLS) means both sides of a connection prove their identity to each other via certificates, not just the server proving its identity to the client (as in standard, one-way TLS from the Linux & Networking Fundamentals series).

Why this is such a genuinely strong, concrete "what does a service mesh give you for free" answer, directly extending the STRIDE threat modeling from the DevSecOps series: mTLS between every service directly defends against Spoofing (a rogue pod can't successfully impersonate a legitimate service without a valid certificate) and Information Disclosure (all inter-service traffic is encrypted in transit, even inside the cluster's own internal network) — and it's applied automatically, uniformly, to every single service in the mesh, with zero application code ever needing to implement TLS logic itself.

mTLS Modes: STRICT vs PERMISSIVE#

A genuinely important, real operational detail for anyone actually rolling out mTLS on an existing cluster, not a new one: you can't simply flip mTLS on cluster-wide instantly without breaking things.

Why this staged rollout matters concretely, a real production incident this specifically prevents: a cluster mid-migration to a service mesh commonly has some namespaces/services already mesh-enabled and others not yet. Flipping straight to STRICT mTLS cluster-wide the moment the mesh control plane is installed would immediately break every still-unmigrated service trying to call a migrated one (or vice versa) with plaintext traffic — PERMISSIVE is the deliberately-designed transitional state, letting migration happen incrementally, namespace by namespace, with STRICT only applied once every relevant caller is confirmed to be mesh-enabled.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: team-checkout
spec:
  mtls:
    mode: STRICT

Traffic Management — Canary Releases Without Application Changes#

A genuinely powerful, concrete capability worth demonstrating with an actual example — this directly ties into deployment strategies covered in the Automation/CI-CD topic.

# An Istio VirtualService splitting traffic between two versions
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: checkout
spec:
  hosts:
    - checkout
  http:
    - route:
        - destination:
            host: checkout
            subset: v1
          weight: 90
        - destination:
            host: checkout
            subset: v2
          weight: 10

Why this is such a strong capability, worth stating explicitly: this canary rollout happens entirely at the infrastructure/mesh layer, with ZERO changes to the application code itself — the application has no idea it's v1 or v2 of a canary split; the mesh is silently, transparently splitting traffic based purely on this declarative configuration, which can be adjusted live (10% -> 50% -> 100%) without redeploying anything.

Circuit Breaking and Retry Policies#

Beyond simple traffic splitting, a mesh's data plane can implement genuinely sophisticated resilience patterns — directly, mechanically implementing concepts from the Reliability & Architecture Patterns series, without any application code.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: checkout-circuit-breaker
spec:
  host: checkout
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Why outlierDetection (the mesh's implementation of the Circuit Breaker pattern from the Reliability & Architecture Patterns series) is such a strong, concrete capability worth demonstrating precisely: it operates per-INDIVIDUAL-pod, not per-service — if one specific replica out of ten is failing (a bad node, a stuck process) while the other nine are healthy, the mesh automatically stops routing to just that one struggling pod, without needing any custom application-level health-check logic, and without taking the whole service down or requiring human intervention. maxEjectionPercent is a genuinely important safety bound worth naming explicitly: it caps how much of the total pool can be ejected simultaneously, preventing a cascading-failure scenario where a systemic issue (not an individual-pod issue) ejects the entire backend pool at once and leaves nothing to serve traffic at all.

Worth distinguishing precisely from a readiness probe (Part 1): a readiness probe is the APPLICATION's own, proactive self-assessment ("am I ready for traffic"), evaluated by the kubelet; outlier detection is the MESH's reactive, observed-behavior assessment ("has this specific pod actually been returning errors to real requests"), evaluated by the sidecar proxies. Both remove a pod from active traffic, but for genuinely different reasons and via genuinely different mechanisms — a strong answer names both and explains why having both layers is more robust than relying on either alone.

Retry policies, worth knowing the specific dangerous interaction:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: checkout
spec:
  hosts: ["checkout"]
  http:
    - retries:
        attempts: 3
        perTryTimeout: 2s
        retryOn: 5xx,reset,connect-failure
      route:
        - destination: {host: checkout}

A genuinely important, real-world danger worth naming explicitly: automatic retries can amplify an outage rather than mitigate it, a "retry storm." If checkout is failing because it's genuinely overloaded, 3 automatic retries per failed request roughly QUADRUPLES the load hitting an already-overloaded service — directly making the underlying problem worse, not better. This is exactly why retry budgets, sensible perTryTimeout values, and combining retries with circuit breaking (so a genuinely unhealthy pod gets ejected rather than endlessly retried against) are all real, necessary companion practices, not optional refinements — a strong interview answer names this tension explicitly rather than presenting retries as an unconditionally good thing to enable everywhere.

Fault Injection — Chaos Engineering at the Mesh Layer#

A genuinely powerful, mesh-native capability worth knowing about explicitly, directly extending the chaos engineering discussion from the Incident Management series: a mesh can deliberately inject failures into live traffic, on purpose, without touching any application code.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: checkout
spec:
  hosts: ["checkout"]
  http:
    - fault:
        delay:
          percentage:
            value: 10
          fixedDelay: 3s
        abort:
          percentage:
            value: 5
          httpStatus: 500
      route:
        - destination: {host: checkout}

Why this is genuinely more valuable than it might first sound, worth stating precisely: it lets a team verify resilience patterns (timeouts, retries, circuit breakers — everything covered earlier in this Part and in the Reliability & Architecture Patterns series) actually work correctly under real degraded conditions, WITHOUT needing to actually break a real downstream dependency to test it. This is a direct, concrete, lower-risk implementation of the exact chaos engineering philosophy from the Incident Management series — deliberately injecting controlled failure to validate assumptions before reality validates them the hard way, but scoped precisely (an exact percentage of traffic, a specific fixed delay/error code) and instantly reversible by removing the fault injection rule, rather than needing to actually kill a real process or sever a real network link.

A genuinely important safety practice worth naming explicitly, given this operates on REAL traffic, not synthetic test traffic: fault injection rules should be scoped as narrowly as possible (a specific low percentage, a specific route, ideally gated behind a header only test traffic carries) and treated with the same change-management rigor as any other production configuration change — accidentally leaving a 100%-abort fault rule active, or applying it to the wrong VirtualService, is a fully real, self-inflicted outage, not a hypothetical risk.

GAMMA — Gateway API for Service Mesh#

Gateway API (Part 3) wasn't designed only for north-south (external-to-cluster) traffic — the GAMMA (Gateway API for Mesh Management and Administration) initiative extends the same standard API to configure east-west (service-to-service, inside-the-mesh) traffic too.

Why this matters as a real, worth-naming ecosystem direction, not a minor technical footnote: historically, every service mesh product (Istio, Linkerd, Consul) defined its own, incompatible traffic-management API (Istio's VirtualService/DestinationRule, shown throughout this Part, is a real example) — meaning mesh-specific YAML that doesn't transfer if you ever change mesh implementations. GAMMA's goal is a single, standard, portable way to configure traffic splitting, retries, and routing that works the same way regardless of which underlying mesh implementation a cluster runs — directly mirroring the same "standard interface over vendor-specific implementation" philosophy already seen repeatedly across CRI, CNI, and CSI in this series. As of current adoption, this is real and growing but not yet universal — worth knowing the direction of travel, while recognizing most production meshes today still primarily use their own native APIs (as shown in the examples throughout this Part).

Observability for Free — Revisited#

This tutorial series already introduced this idea in the Monitoring Methodologies series (Part 1): because every single request flows through a sidecar proxy, RED metrics (Rate, Errors, Duration) can be emitted automatically for every service, with zero application instrumentation required.

Why this is worth restating here, one more time, tied to the full architecture just explained: this isn't magic — it's a direct, mechanical consequence of the sidecar pattern: since literally 100% of network traffic to and from a pod already passes through its sidecar, that sidecar is perfectly positioned to observe and report on every single request, uniformly, across every language and framework a team might use, without a single line of application-level instrumentation code.

The Real Cost of a Service Mesh#

A balanced, senior-level answer names the tradeoffs, not just the benefits — a genuinely common, important interview follow-up.

A genuinely strong, balanced interview answer: "I'd adopt a service mesh when the organization has enough services that consistent mTLS, retries, and observability genuinely can't be reasonably maintained per-team in application code anymore — but I wouldn't reach for it by default on a small handful of services, where the added latency, resource overhead, and operational complexity of running the mesh itself can outweigh the benefit."

A consolidated comparison of the major service mesh implementations, worth having as a reference:

Why Linkerd's non-Envoy proxy is worth knowing as a specific, real differentiator: Istio and Consul both default to Envoy, a general-purpose, highly configurable but also genuinely complex proxy; Linkerd deliberately built its own minimal, purpose-specific proxy, trading some of Envoy's broader feature surface for meaningfully lower resource consumption and operational simplicity — a real, legitimate reason some teams choose Linkerd specifically when the full breadth of Istio's feature set isn't needed.

A concrete decision framework worth stating explicitly, rather than defaulting to "everyone uses Istio": choose Linkerd when the primary need is mTLS + basic observability with minimal operational overhead; choose Istio when the team genuinely needs its broader feature set (fine-grained traffic management, fault injection, the largest plugin ecosystem) and has the operational capacity to run it; choose Consul Connect specifically when the mesh needs to span Kubernetes AND non-Kubernetes infrastructure (VMs, bare metal) in one unified mesh — a real, common requirement during a gradual containerization migration.

Multi-Cluster Mesh Federation#

A genuinely important extension worth knowing about, directly connecting to the multi-region resilience patterns in the Disaster Recovery and Reliability & Architecture Patterns topics: a service mesh doesn't have to be confined to a single cluster.

Why this matters concretely, worth stating as a real capability, not just a theoretical extension: a federated multi-cluster mesh lets a service in Cluster A transparently call a service in Cluster B using the exact same service name and mTLS guarantees as an in-cluster call — this is a genuine, production-grade building block for multi-region active-active or active-passive architectures (directly extending the multi-region patterns from the Disaster Recovery topic), letting traffic failover across entire clusters/regions without the calling application needing any region-awareness logic of its own. The mesh's control plane handles cross-cluster service discovery and certificate trust the same way it handles intra-cluster discovery — extending, not replacing, everything already covered in this Part.

A real, worth-naming operational cost of multi-cluster federation: it requires a shared root of trust for certificates across every federated cluster (so mTLS verification works across the cluster boundary) and genuinely reliable, low-latency network connectivity between clusters — federating clusters across regions with poor connectivity between them can introduce exactly the kind of cross-region latency and reliability risk the architecture was meant to mitigate, so this is a deliberate infrastructure investment, not a free extension of single-cluster mesh capability.

etcd — Operational Depth#

Part 1 introduced etcd as Kubernetes's source of truth, using Raft consensus and requiring a quorum. This section goes deeper into the operational realities of actually running it well.

Why the watch mechanism deserves its own callout: it's the actual, concrete technical reason Kubernetes controllers can react to changes near-instantly rather than on a slow polling cycle — every controller and kubelet maintains an open watch connection (via the API Server, which itself watches etcd) for exactly the objects it cares about, and gets pushed an update the moment anything changes.

etcd's Write-Ahead Log (WAL), worth knowing as the actual durability mechanism underneath the Raft consensus already covered in Part 1: before any write is considered durable, it's first appended to an on-disk WAL file — a genuinely important, real performance-vs-durability consideration: disk I/O latency for the WAL write is directly on etcd's write critical path, which is precisely why etcd's own official hardware guidance strongly recommends fast, low-latency SSD storage specifically for the WAL, and why running etcd on genuinely slow or shared/contended disk I/O is a real, common root cause of a struggling, high-latency control plane — even when CPU and memory look completely fine. A concrete, worth-remembering interview fact: etcd's own documentation explicitly calls out disk latency, not CPU or memory, as the most common real-world etcd performance bottleneck.

etcd Compaction and Defragmentation#

A genuinely practical, sometimes-overlooked operational concern — a real interview differentiator if you know it.

# Manually compact etcd's history (usually automated on a schedule
# in a well-configured cluster, but worth knowing the raw command)
etcdctl compact $(etcdctl endpoint status --write-out="json" | \
  python3 -c "import sys,json; print(json.load(sys.stdin)[0]['Status']['header']['revision'])")

# Defragment to actually reclaim disk space
etcdctl defrag --endpoints=https://127.0.0.1:2379

Why an unmaintained etcd database is a genuinely real operational risk worth knowing about, not just trivia: without regular compaction/defragmentation, etcd's on-disk database file can grow until it hits etcd's own configured storage quota (--quota-backend-bytes), at which point etcd stops accepting writes entirely — meaning the entire cluster loses the ability to create or update any object at all, a genuinely severe, cluster-wide outage caused purely by operational neglect of a component most teams rarely think about directly.

A Real etcd Disaster Recovery Drill#

Extending the backup command from Part 1 into a full, practiced recovery procedure — directly connecting to the Disaster Recovery topic (topic 11) in this course.

# 1. Take a regular, scheduled snapshot (as shown in Part 1)
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db

# 2. In a real disaster, RESTORE from that snapshot to a fresh
#    data directory
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db \
  --data-dir /var/lib/etcd-restored \
  --name etcd-restored \
  --initial-cluster etcd-restored=https://127.0.0.1:2380 \
  --initial-advertise-peer-urls https://127.0.0.1:2380

# 3. Update the etcd service/static pod manifest to point at the
#    NEW data directory, then restart etcd

The single most important, most-often-skipped practice worth stating explicitly, directly reusing the chaos engineering philosophy from the Incident Management series: a backup that's never been test-restored is not a verified backup — it's a hypothesis. Just like the "chaos engineering" principle of deliberately testing assumptions before reality tests them, teams should periodically practice a full etcd restore drill in a non-production environment, confirming the backup actually works and the team actually knows the procedure, rather than discovering a broken backup or an unfamiliar procedure for the first time during a genuine control-plane disaster.

Custom Resource Definitions (CRDs) — Extending the API#

Everything covered so far (Pods, Deployments, Services) are built-in Kubernetes API objects. A CRD lets you define your own, entirely custom object type, which then behaves exactly like a native Kubernetes object — kubectl get, kubectl apply, RBAC, everything, works on it identically.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresqlclusters.db.example.com
spec:
  group: db.example.com
  scope: Namespaced
  names:
    plural: postgresqlclusters
    singular: postgresqlcluster
    kind: PostgreSQLCluster
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                version:
                  type: string

# Once the CRD exists, you can create objects of this NEW type,
# just like any built-in Kubernetes object:
apiVersion: db.example.com/v1
kind: PostgreSQLCluster
metadata:
  name: my-database
spec:
  replicas: 3
  version: "15.2"

Why this matters, and it's a genuinely important architectural insight worth stating explicitly: by itself, a CRD just defines a shape of data Kubernetes will accept and store — creating a PostgreSQLCluster object above, with no more, does absolutely nothing on its own. Something still has to actually watch for these objects and take real action — which is exactly the job of an Operator, covered next.

CRD Versioning and Conversion Webhooks#

Real CRDs, maintained over time, need to evolve their schema — a genuinely important, often-overlooked operational reality worth understanding, directly mirroring the API versioning discussion in Part 1.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresqlclusters.db.example.com
spec:
  group: db.example.com
  versions:
    - name: v1
      served: true
      storage: false    # v1 objects are STORED as v2 internally
    - name: v2
      served: true
      storage: true     # v2 is the CURRENT storage version
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service:
          name: crd-conversion-webhook
          namespace: platform
      conversionReviewVersions: ["v1", "v2"]

Why exactly one version can have storage: true at any given time, a precise, worth-knowing rule: etcd only ever stores ONE actual representation of each object internally — the version marked storage: true. When an older client requests the object using the v1 API, the API Server calls the conversion webhook to translate the internally-stored v2 representation back into v1 shape on the fly, and vice versa for writes. This is exactly the same underlying pattern as a database schema migration with a compatibility view layered on top — old and new clients can keep working simultaneously against a single, evolving underlying representation, without a disruptive, coordinated "everyone must switch to v2 at the exact same moment" cutover.

The Operator Pattern#

An Operator is simply a custom controller, following the exact same reconciliation loop pattern from Part 1, but built specifically to manage a CRD instead of a built-in Kubernetes object.

Why Operators are such a powerful, genuinely important pattern, worth explaining the "why" clearly: a plain Deployment or StatefulSet only knows generic Kubernetes concepts (pods, replicas, storage) — it has zero built-in understanding of, say, "how do I safely fail over a PostgreSQL primary to a replica" or "how do I take an application-consistent database backup." An Operator encodes exactly that kind of domain-specific operational knowledge as actual, executable code, running continuously inside the cluster — effectively automating what would otherwise be a human database administrator's runbook (directly connecting to the runbook automation discussion from the Incident Management series), but triggered automatically and continuously rather than manually, by a human, after being paged.

A Worked Operator Example#

A concrete, narrated example showing an Operator handling something a generic StatefulSet fundamentally cannot.

This is precisely why widely-used, real-world Operators exist for complex stateful systems — the Prometheus Operator (managing Prometheus/Alertmanager configuration declaratively), various PostgreSQL/MySQL Operators, the etcd Operator, and many more — each one encoding deep, product-specific operational expertise as automated, continuously-running code rather than a manual runbook a human has to execute under pressure.

Building an Operator: The controller-runtime Pattern#

Worth knowing what actually building an Operator looks like at the code level, even briefly — genuinely useful context for evaluating whether a team should build one, not just consume existing ones.

// A genuinely minimal, illustrative sketch of a Reconcile function —
// the actual unit of work every real Operator is built around
func (r *PostgreSQLClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var cluster dbv1.PostgreSQLCluster
    if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Desired: cluster.Spec.Replicas StatefulSet replicas
    // Actual: query the real StatefulSet's current replica count
    // If they don't match, update the StatefulSet to reconcile
    desired := buildStatefulSetFor(&cluster)
    if err := r.Patch(ctx, desired, client.Apply, client.ForceOwnership); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

Why this framework-handles-the-boilerplate design is worth calling out precisely: the reconciliation loop pattern itself (Part 1) is exactly the same for every controller, built-in or custom — controller-runtime (the library underlying both Kubebuilder and the Operator SDK) implements the generic "watch, queue, retry, requeue" machinery ONCE, so an Operator author only needs to write the actual domain-specific logic (what StatefulSets/Services/PVCs should exist, how to detect and handle a failed primary) inside a single Reconcile function — not reimplement the entire watch-and-retry infrastructure from scratch for every new Operator.

A genuinely important, real design principle worth naming: Reconcile should be idempotent and level-based, never edge-triggered. It receives just an object's name, not "what specifically changed" — a well-written Reconcile re-derives the full desired state and re-applies it every single time it runs, regardless of why it was triggered, exactly mirroring the "observe, compare, act" loop from Part 1 rather than trying to process a sequence of discrete "diffs." This is precisely why calling Reconcile redundantly (which happens constantly in practice — on a timer, on any watched-object change, on startup) is always safe: applying the same desired state twice in a row should always be a no-op the second time.

Finalizers, worth knowing as the standard mechanism for cleanup-before-deletion logic: an Operator that needs to perform real external cleanup before a CRD object is actually deleted (deprovisioning a cloud resource, for instance) adds a finalizer string to the object's metadata — Kubernetes then holds the object in a Terminating state (rather than actually removing it) until the Operator's Reconcile loop observes the deletion timestamp, performs its cleanup, and explicitly removes its own finalizer, at which point the object is finally, actually deleted. Without a finalizer, an Operator has no reliable way to guarantee its external cleanup logic runs before Kubernetes considers the object gone.

Helm — Packaging Kubernetes Applications#

A real, deployed application is rarely just one YAML file — it's typically a Deployment, a Service, a ConfigMap, an Ingress, maybe a PVC, all needing to work together and often needing slightly different values per environment (staging vs. production). Helm is the most widely used Kubernetes package manager, solving exactly this problem.

# Install an application from a Helm chart
helm install my-checkout ./checkout-chart --values production-values.yaml

# Upgrade to a new version/configuration
helm upgrade my-checkout ./checkout-chart --values production-values.yaml

# Roll back to the previous release, if something goes wrong
helm rollback my-checkout

Why this matters practically, tying it back to the reproducibility principle from the DevSecOps series: Helm turns "apply these 8 YAML files, in the right order, with the right environment-specific substitutions, remembered correctly every time" into one versioned, repeatable, auditable command — directly reducing the exact kind of manual, error-prone toil the SRE Fundamentals series identifies as worth automating away.

Helm Chart Structure and Hooks#

Worth knowing the actual anatomy of a chart, and one genuinely powerful, easy-to-misuse feature: Helm Hooks.

# templates/deployment.yaml — {{ }} syntax pulls from values.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-checkout
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

# A Helm Hook — runs a Job BEFORE the main install/upgrade proceeds
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: checkout-migrations:1.2.3
      restartPolicy: Never

Why pre-install/pre-upgrade hooks are worth knowing precisely, and it's a genuinely real, common pattern: this is exactly the standard way to run a database migration Job (Part 2) automatically as part of a Helm-managed deployment, guaranteed to complete before the new application version's pods actually start — directly connecting to Part 2's warning about init containers vs. hooks solving different problems: init containers gate a single pod's own startup, while a Helm hook gates the entire release's rollout, running once per install/upgrade rather than once per pod.

A genuinely real, worth-naming caution about hooks: they run OUTSIDE Helm's normal templated-resource tracking and rollback machinery — a failed hook can leave a release in a genuinely awkward, partially-applied state that helm rollback doesn't cleanly undo the way it does for normal templated resources, since the hook's side effects (a partially-run migration, for instance) already happened outside Kubernetes's own object lifecycle. Hooks are powerful but deserve real testing, not blind trust, precisely because they sit outside Helm's normal safety net.

Part 4 CLI Cheat Sheet#

# Service mesh diagnostics (Istio examples, similar for other meshes)
istioctl proxy-status
istioctl proxy-config cluster <pod-name>
kubectl get peerauthentication -A
kubectl get virtualservice,destinationrule -A

# etcd operations
etcdctl endpoint status --write-out=table
etcdctl endpoint health
etcdctl alarm list
etcdctl member list -w table

# CRDs and Operators
kubectl get crd
kubectl explain postgresqlcluster.spec
kubectl get postgresqlclusters -A
kubectl logs -n platform deployment/postgresql-operator -f

# Helm
helm list -A
helm status my-checkout
helm get values my-checkout
helm template ./checkout-chart --values production-values.yaml   # render WITHOUT installing
helm diff upgrade my-checkout ./checkout-chart --values production-values.yaml   # requires helm-diff plugin

Common Mistakes#

Worked Practice Problems#

Problem 1: A team is deciding whether to adopt a service mesh for their 6-service application. What questions would you ask before recommending for or against it?

Answer: I'd ask whether they currently have inconsistent or duplicated implementations of retries/circuit breakers/mTLS across their 6 services (a sign a mesh would genuinely help), whether they're already struggling with cross-service observability that RED-metric auto-instrumentation would solve, and whether the team has the operational capacity to run and maintain an entirely separate control-plane system. At only 6 services, I'd lean toward recommending against a full mesh initially — the added latency, sidecar resource overhead, and operational complexity are real costs that may not be justified yet, and I'd suggest revisiting the decision as the service count and cross-cutting-concern pain genuinely grow.

Problem 2: A cluster's control plane suddenly stops accepting any writes — kubectl apply and kubectl create all fail, though reads still work. What's a plausible root cause specific to etcd operations, and how would you confirm it?

Answer: A plausible cause is etcd hitting its configured storage quota (--quota-backend-bytes) due to a database file that's grown unbounded from a lack of regular compaction/defragmentation — etcd deliberately stops accepting new writes once its backend database exceeds this quota, while still being able to serve existing reads. I'd confirm by checking etcd's own logs/metrics for quota-related alarm messages (etcdctl alarm list shows an active NOSPACE alarm in exactly this scenario), and the fix would involve compacting old history and defragmenting to reclaim space, then clearing the alarm — while also fixing the underlying gap by scheduling regular compaction going forward.

Problem 3: A team builds a CRD for PostgreSQLCluster objects but doesn't deploy any accompanying controller. They create a PostgreSQLCluster object and are confused when no actual database appears. What's the misunderstanding?

Answer: A CRD only teaches the Kubernetes API server to accept and store a new shape of object — it defines the schema, nothing more. It has no inherent behavior of its own; creating a PostgreSQLCluster object just stores that data in etcd, exactly like creating any object type. For anything to actually happen as a result (creating StatefulSets, PVCs, Services, running database-specific logic), a separate controller — an Operator — needs to be deployed, watching for PostgreSQLCluster objects and reconciling real Kubernetes resources to match. Without that Operator running, the CRD is just an inert data definition with nothing acting on it.

Problem 4: During a service mesh rollout, a team enables mTLS STRICT mode cluster-wide immediately after installing the mesh control plane. Several services that haven't been migrated yet start failing with connection errors. What went wrong, and what's the correct rollout sequence?

Answer: STRICT mode rejects any plaintext connection outright — but not every service in the cluster has an injected sidecar yet this early in a migration, so those still-unmigrated services are making (or receiving) genuinely plaintext connections, which STRICT mode now refuses. The correct sequence: start in PERMISSIVE mode, which accepts both mTLS and plaintext simultaneously, letting migrated and not-yet-migrated services keep working together during the transition. Migrate services into the mesh incrementally, namespace by namespace, and only switch to STRICT once every caller and callee that needs to reach a given service is confirmed to already be mesh-enabled — STRICT is the correct END state, not the correct starting configuration for a rollout in progress.

Problem 5: A Reconcile function in a custom Operator is written to only handle the specific field that changed in the triggering event (e.g., "if spec.replicas changed, adjust replica count; otherwise do nothing"). During a period of high API server load, some watch events are coalesced/dropped, and the Operator ends up in a state inconsistent with the actual desired spec. What's the underlying design flaw?

Answer: The Reconcile function violates the level-based, idempotent reconciliation model that Kubernetes controllers are built around — it's written as if reconciliation were edge-triggered (react only to what specifically changed in one event), when in fact controller-runtime's design assumes and requires that Reconcile re-derive and re-apply the FULL desired state every time it runs, using only the object's name (not event-specific diff data) as input. Under real-world conditions, watch events can legitimately be coalesced, dropped, or delivered out of order — a correctly-written Reconcile function is safe against all of that because it always recomputes everything from scratch and converges to the same correct state regardless of how many times or in what order it's called; an edge-triggered implementation instead accumulates drift exactly as described here. The fix: rewrite Reconcile to always fetch the full current object, compute the full desired state, and reconcile every relevant resource against it on every single invocation — never branch on "what specifically changed."

Problem 6: A self-managed cluster's control plane shows healthy CPU and memory on every control-plane node, yet kubectl apply and other write operations have become noticeably slow — taking multiple seconds where they used to be near-instant. Reads remain fast. What component would you investigate first, and what specific resource is the most likely bottleneck?

Answer: etcd, and specifically disk I/O latency for its Write-Ahead Log — this exact symptom pattern (writes slow, reads fine, CPU/memory both healthy) is the textbook signature etcd's own documentation identifies as its most common real-world performance bottleneck. Every write must be durably appended to the WAL before etcd considers it committed, meaning WAL disk latency sits directly on the write critical path — CPU and memory being healthy rules out compute contention, while reads staying fast is consistent with the watch-cache-driven read path (Part 1) being largely unaffected by WAL write latency specifically. The investigation: check the actual disk I/O metrics (latency, not just utilization) on the volume backing etcd's data directory — a noisy neighbor on shared storage, an underprovisioned IOPS tier, or genuinely failing/degrading storage hardware are all realistic root causes, and the fix is provisioning etcd onto genuinely fast, dedicated, low-latency storage per etcd's own official hardware recommendations.

Summary and What's Next#

• A service mesh solves the problem of consistent cross-cutting concerns (mTLS, retries, observability) across many services without duplicating logic in every team's application code — built entirely on the sidecar pattern, which works because containers in a pod share a network namespace (Part 3).
• The mesh's control plane (central policy) and data plane (every sidecar proxy actually enforcing it) mirror Kubernetes's own control-plane/worker-node split from Part 1.
• mTLS provides automatic, zero-code mutual authentication and encryption for every service-to-service call — a direct, concrete defense against Spoofing and Information Disclosure from the DevSecOps series' STRIDE framework.
• Traffic splitting for canary releases happens entirely at the mesh layer, with zero application code changes — and RED-metric observability comes "for free" for exactly the same mechanical reason.
• A service mesh has real costs (latency, resource overhead, operational complexity) — it's a deliberate tradeoff, not a default best practice for every scale.
• etcd needs ongoing operational care — compaction and defragmentation prevent its database file from growing unbounded and eventually blocking all writes cluster-wide — and a backup that's never been test-restored is not a verified recovery path.
• CRDs extend the Kubernetes API with custom object types, but by themselves do nothing — an Operator (a custom controller following the same reconciliation loop pattern as every built-in controller) is what actually encodes domain-specific operational knowledge and takes real action.
• Helm packages multi-file, multi-environment Kubernetes applications into versioned, repeatable, auditable installs — directly reducing manual deployment toil.
• Sidecar injection is mutating-webhook-driven and namespace-scoped by convention (Part 1's admission chain applied concretely); newer ambient/sidecarless mesh modes trade per-pod isolation for lower resource overhead.
• mTLS rollout is staged — PERMISSIVE during migration, STRICT only once every caller is confirmed mesh-enabled — and circuit breaking/outlier detection ejects individual unhealthy pods automatically, while unchecked retries risk amplifying an outage into a full one.
• GAMMA extends Gateway API to east-west mesh traffic, aiming for the same portable-standard-interface pattern already seen in CRI/CNI/CSI, though most production meshes today still primarily use native, mesh-specific APIs.
• CRD versioning uses a single storage: true version internally, with a conversion webhook translating between API versions on the fly — the same schema-evolution pattern as a database migration with a compatibility view.
• A well-written Operator's Reconcile function is idempotent and level-based — it always re-derives and re-applies the full desired state, never branches on "what specifically changed," which is exactly what makes redundant or out-of-order triggers safe.
• Helm Hooks (pre-install/pre-upgrade) are the standard way to run migrations as part of a release, but operate outside Helm's normal rollback safety net and deserve real testing.
• Multi-cluster mesh federation extends mTLS and service discovery across cluster/region boundaries — a real building block for active-active or active-passive multi-region architectures, letting applications call cross-cluster services with zero region-awareness logic, at the real cost of needing a shared certificate trust root and reliable inter-cluster connectivity.
• Finalizers are the standard mechanism for guaranteeing an Operator's external cleanup logic runs before Kubernetes actually deletes an object — without one, cleanup can be skipped entirely and leak the underlying external resource.
• Outlier detection and readiness probes are complementary, not redundant — one is the mesh's reactive, observed-behavior assessment; the other is the application's own proactive self-assessment, and robust systems benefit from both layers together.
• CRD/Operator/Helm together form the standard extension toolkit for anything Kubernetes doesn't understand natively — a CRD defines the shape, an Operator supplies the domain-specific behavior, and Helm packages the whole bundle for repeatable, versioned installation across environments.
• etcd's Write-Ahead Log sits directly on the write critical path — disk I/O latency, not CPU or memory, is etcd's own documented most common real-world performance bottleneck, making fast dedicated storage a genuine, non-optional operational requirement.
• Mesh-native fault injection (artificial delays/aborts on a precise percentage of traffic) is a lower-risk, instantly-reversible implementation of chaos engineering — validating timeout/retry/circuit-breaker logic under real degraded conditions without breaking an actual dependency.

Continue to Part 5 (05-managed-kubernetes-eks-aks-gke.md) to see how everything covered so far actually gets deployed in practice — the major managed Kubernetes platforms (EKS, AKS, GKE) and how much of this architecture each one manages on your behalf.