Networking (CNI) & Storage (CSI)

Table of Contents#

1. The Kubernetes Networking Model — The Ground Rules
2. CNI — The Plugin That Makes the Model Real
3. Why Pods Get IPs and Why That's a Big Deal
4. Services — Solving the "Pods Are Disposable" Problem
5. The Four Service Types
6. Session Affinity
7. Headless Services
8. Endpoints and EndpointSlices
9. NetworkPolicy — Full Depth
10. Topology Aware Routing
11. DNS Policies and Custom DNS Configuration
12. Ingress — Getting Traffic In From Outside
13. Gateway API — The Modern Ingress Successor
14. CoreDNS — Service Discovery Inside the Cluster
15. A Full Worked Request Journey
16. Volumes — The Basic Storage Building Block
17. PersistentVolumes and PersistentVolumeClaims
18. StorageClass and Dynamic Provisioning
19. StorageClass Topology Awareness
20. CSI — The Storage Plugin Interface
21. CSI Volume Snapshots, Cloning, and Expansion
22. Access Modes — A Genuinely Common Gotcha
23. StatefulSets and Storage, Tied Together
24. Part 3 CLI Cheat Sheet
25. Common Mistakes
26. Worked Practice Problems
27. Summary and What's Next

The Kubernetes Networking Model — The Ground Rules#

Kubernetes networking is built on a small number of simple, strict rules — and nearly everything more complex (Services, Ingress, NetworkPolicies) is built as a layer on top of these ground rules.

Why this "flat network" model is such a deliberate, important design choice: it means container-to-container networking in Kubernetes works essentially like normal networking between separate physical machines — no special-case NAT traversal logic, no "which port did this get mapped to" complexity that plagued earlier container networking approaches (like classic Docker's default bridge networking). Every pod is a full, first-class citizen on the network, directly addressable by its own IP.

CNI — The Plugin That Makes the Model Real#

Kubernetes itself doesn't implement this networking model — it defines the rules and delegates the actual implementation to a CNI (Container Network Interface) plugin, exactly the same delegation pattern as the CRI for container runtimes from Part 1.

A directly important connection back to the DevSecOps series' container security tutorial, worth restating explicitly here: NetworkPolicy objects are a Kubernetes API concept, but enforcing them is entirely up to the CNI plugin — some plugins (Calico, Cilium) fully support and enforce them; others (some simpler/older CNI setups) don't enforce them at all, meaning a NetworkPolicy YAML could apply successfully with zero actual effect. Always verify which CNI plugin a cluster actually runs before assuming NetworkPolicies are doing anything.

# Check which CNI plugin a cluster is running (commonly, look at
# the CNI-related DaemonSet running in kube-system)
kubectl get pods -n kube-system | grep -Ei "calico|cilium|flannel|weave"

Why Pods Get IPs and Why That's a Big Deal#

A directly practical consequence worth spelling out: because every pod gets its own real IP, containers within the same pod share that single IP and its network namespace (from the Linux & Networking Fundamentals series, Part 1) — they talk to each other over localhost, exactly like processes on the same machine, while still each having their own separate IP address relative to every other pod in the cluster.

This is precisely the mechanical foundation of the "sidecar" pattern referenced throughout this course (service mesh proxies, log-shipping sidecars) — a sidecar container works because it shares its pod's network namespace and can transparently intercept traffic to/from localhost without the main application container needing to know or care.

Services — Solving the "Pods Are Disposable" Problem#

Pods are disposable — they get created and destroyed constantly (deploys, crashes, scaling, rescheduling), and each new pod gets a brand-new IP address. This creates an obvious problem: how does anything reliably talk to "the checkout service" if the actual IPs behind that name keep changing?

Simple analogy: a Service is like a company's general customer support phone number — the number itself never changes, even though the specific employee who actually answers any given call is different every time (and employees come and go). Callers only need to remember the one stable number.

The Four Service Types#

apiVersion: v1
kind: Service
metadata:
  name: checkout-svc
spec:
  type: ClusterIP
  selector:
    app: checkout-service   # matches pods with THIS label
  ports:
    - port: 80
      targetPort: 8080

A genuinely important detail, worth being explicit about: a Service finds its backing pods purely via a label selector — this is a loose, dynamic coupling (not a fixed list of pod names), which is exactly what lets a Service automatically pick up new pods and drop terminated ones, continuously, with zero manual reconfiguration.

Session Affinity#

By default, a Service load-balances every individual request independently, round-robin-ish, across its healthy backend pods — with no concept of "this specific client should keep hitting the same pod." Session Affinity changes that, worth knowing precisely since it's a real, commonly-needed setting for a specific class of application.

apiVersion: v1
kind: Service
metadata:
  name: legacy-app-svc
spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800   # 3 hours
  selector:
    app: legacy-app

Why ClientIP affinity is worth treating as a real, worth-questioning tradeoff rather than a convenient default, worth stating explicitly: it's genuinely useful for legacy applications that store session state in local process memory instead of an external store (Redis, a database), but it directly works against even load distribution — a small number of high-traffic clients (common behind a corporate NAT, where many real users share one apparent source IP) can end up disproportionately pinned to the same backend pods. The architecturally preferred fix, worth naming explicitly, is moving session state to an external store (making the app genuinely stateless at the pod level) rather than leaning on ClientIP affinity as a permanent solution — session affinity is a legitimate stopgap for legacy apps, not a best practice to design new services around.

Worth noting: sessionAffinity is a Service-level, ClientIP-based mechanism, distinct from — and much coarser than — application-level session cookies or an Ingress Controller's own cookie-based sticky-session features, which operate at Layer 7 with far more precision (tied to an actual session identity, not just a source IP that may represent many different real users behind shared infrastructure). A strong interview answer distinguishes these two layers explicitly rather than treating "sticky sessions" as one single, undifferentiated concept.

Headless Services#

A special, genuinely important fifth variant worth its own section, since it directly explains the StatefulSet DNS behavior referenced in Part 2: setting clusterIP: None on a Service creates a Headless Service — one with NO stable virtual IP at all.

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
    - port: 5432

Why this is precisely what makes StatefulSet DNS names work, closing a loop left open in Part 2: a headless Service paired with a StatefulSet gives each individual pod its own resolvable DNS name (postgres-0.postgres-headless.default.svc.cluster.local), rather than only a single load-balanced virtual IP for the whole set. This is exactly why a client that needs to specifically reach postgres-0 (the primary in a replicated database setup, for instance) can do so directly by name — a regular ClusterIP Service could only ever route to some healthy pod, with no way to target one specific replica.

Endpoints and EndpointSlices#

Underneath a Service, Kubernetes maintains the actual, current list of healthy pod IPs backing it — historically via an Endpoints object, now more commonly via the newer, more scalable EndpointSlice API.

This is the exact, concrete mechanism tying together the readiness probe discussion from Part 2 and the kube-proxy discussion from Part 1: a failed readiness probe doesn't just log a status — it actively, automatically removes that pod from the real, live list of addresses traffic gets routed to, cluster-wide, within moments.

NetworkPolicy — Full Depth#

Part 3's earlier CNI section flagged that NetworkPolicy enforcement depends entirely on the CNI plugin — here's the actual policy model in depth, since this is genuinely one of the most important, most commonly under-configured Kubernetes security mechanisms.

This default-open posture is worth stating explicitly as a real security gap in any cluster without NetworkPolicies applied: a compromised pod (from a vulnerable dependency, a supply-chain issue per the DevSecOps series) can, by default, reach every other pod in the entire cluster — including services it has no legitimate business talking to. NetworkPolicy is the mechanism that closes this.

# A default-deny-all policy for a namespace — the recommended
# starting point for any genuinely security-conscious namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: team-checkout
spec:
  podSelector: {}      # matches ALL pods in this namespace
  policyTypes:
    - Ingress
    - Egress

# Then EXPLICITLY allow only what's actually needed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-checkout-to-db
  namespace: team-checkout
spec:
  podSelector:
    matchLabels:
      app: checkout-service
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432

The podSelector: {} idiom (an empty selector) matching ALL pods is a genuinely important, easy-to-miss detail — worth stating precisely: an empty {} selector is NOT "no policy" — it's "this policy applies to every pod in the namespace." Combined with policyTypes: [Ingress, Egress] and no ingress/egress rules listed at all, this produces the default-deny-everything baseline shown above — the recommended starting point, with specific allow rules layered on top.

Combining namespaceSelector and podSelector in one rule, a genuinely important, precise mechanic worth knowing exactly:

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            team: platform
        podSelector:
          matchLabels:
            app: monitoring-agent

A frequently-tested precision point: when both namespaceSelector AND podSelector appear together in the SAME list item (as above, both under one - entry), they're ANDed — this rule allows traffic ONLY from pods matching app: monitoring-agent that are ALSO in a namespace matching team: platform. If they were instead two SEPARATE list items (two different - entries), they'd be ORed — matching either condition independently. This exact YAML-structure distinction (same list item vs. separate list items) is a real, common source of accidentally-too-permissive or accidentally-too-restrictive policies, and worth double-checking carefully rather than assuming.

A quick reference for the full NetworkPolicy selector vocabulary, worth having in one place:

A final, worth-stating precision point on how multiple NetworkPolicies combine: they are strictly ADDITIVE, never subtractive. If two policies both select the same pod, the pod's actual allowed traffic is the UNION of everything either policy permits — there's no "more specific policy wins" or "deny takes precedence" resolution logic the way some other systems work. This means a single overly-permissive policy anywhere in a namespace can silently undermine an otherwise-tight default-deny baseline, which is exactly why reviewing the FULL set of policies selecting a given pod (kubectl get networkpolicy -n <namespace>, checked against a pod's actual labels) is the correct way to reason about its real effective access — never just one policy in isolation.

Topology Aware Routing#

A genuinely important, cost-relevant Service feature worth knowing, connecting directly to the multi-zone topology theme running throughout this Part: by default, kube-proxy load-balances a Service's traffic across ALL healthy backend pods cluster-wide, with zero regard for which zone the traffic originated from — Topology Aware Routing changes this.

apiVersion: v1
kind: Service
metadata:
  name: checkout-svc
  annotations:
    service.kubernetes.io/topology-mode: Auto
spec:
  selector:
    app: checkout-service
  ports:
    - port: 80

Why this matters concretely, worth stating as a real, quantifiable cost lever, not just a nice-to-have: cross-AZ data transfer is a genuinely real, often-underestimated line item in cloud bills for chatty microservice architectures — a service making thousands of requests per second to a backend, with roughly two-thirds of that traffic randomly crossing zone boundaries under default round-robin routing, pays real, ongoing cross-zone transfer fees for traffic that topology-aware routing could largely keep same-zone. The tradeoff worth naming honestly: this optimizes for cost/latency at a small, deliberate cost to routing evenness — if a zone has disproportionately more caller pods than backend pods, same-zone preference can mean some backend pods handle more traffic than others; the feature includes safeguards (falling back to cross-zone routing when a zone would otherwise be overwhelmed) precisely to bound this tradeoff.

DNS Policies and Custom DNS Configuration#

A pod's own dnsPolicy field, worth knowing since the CoreDNS discussion above assumes the common default without stating it explicitly.

Why the Default policy's name is a genuinely real, worth-warning-about footgun: despite being named "Default," it is NOT the actual default dnsPolicy value (ClusterFirst is) — Default means "use the node's own /etc/resolv.conf," which bypasses CoreDNS and cluster-internal Service DNS resolution entirely. A pod accidentally configured with dnsPolicy: Default (perhaps copy-pasted from an example without understanding the naming) will mysteriously fail to resolve any .svc.cluster.local name, while external DNS lookups work completely normally — a genuinely confusing failure mode to debug without knowing this exact naming trap.

spec:
  dnsPolicy: ClusterFirst
  dnsConfig:
    nameservers:
      - 1.1.1.1
    searches:
      - custom.internal
    options:
      - name: ndots
        value: "2"

ndots, worth knowing as a real, concrete performance-relevant setting: it controls how many dots a name needs before the resolver tries it as a fully-qualified external name FIRST, versus appending the cluster's search domains and trying those first. Kubernetes's default ndots: 5 means any hostname with fewer than 5 dots (which is nearly every hostname anyone writes, including plain external domains like api.stripe.com) gets tried against every internal search domain FIRST before falling back to a direct external lookup — a genuinely real, measurable source of unnecessary DNS latency for pods that make heavy use of external API calls, and a real, well-known production tuning target for latency-sensitive workloads.

Ingress — Getting Traffic In From Outside#

A LoadBalancer Service works, but provisioning a full, separate cloud load balancer for every single service in a cluster gets expensive and unwieldy fast. Ingress solves this by providing a single entry point that can route to many different Services based on the request's hostname/path — directly reusing the Layer 7 load balancing concepts from the Reliability & Architecture Patterns series.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
spec:
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: checkout-svc
                port:
                  number: 80

A worthwhile, genuinely important nuance: the Ingress object is just a set of routing rules — it does nothing on its own without an Ingress Controller (like NGINX Ingress Controller, or a cloud-managed one) actually running in the cluster to read those rules and configure real routing. This is exactly the same "the API object is a declaration; something else has to actually implement it" pattern as CNI and CSI throughout this whole series.

Gateway API — The Modern Ingress Successor#

Ingress's design has real, well-known limitations worth naming precisely, and the Gateway API (a separate, more expressive API, now GA and increasingly the direction the ecosystem is moving) exists specifically to address them.

This three-way role split is the single biggest, most deliberate architectural difference from Ingress, worth stating explicitly: Ingress mixes cluster-infrastructure concerns and application-routing concerns into one flat object, commonly requiring a platform team to hand-edit or approve every application team's Ingress changes. Gateway API cleanly separates these: a platform team owns the GatewayClass/Gateway (the shared, cluster-level infrastructure), while individual application teams independently own their own HTTPRoute objects, attaching to a shared Gateway without needing platform-team involvement for every routing change — a genuinely better fit for how real, multi-team organizations actually operate.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout-route
spec:
  parentRefs:
    - name: shared-gateway
  hostnames:
    - "shop.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /checkout
      backendRefs:
        - name: checkout-svc
          port: 80
          weight: 90
        - name: checkout-svc-canary
          port: 80
          weight: 10

Native, standard traffic-splitting (the weight fields above) is worth calling out specifically, since it's genuinely not possible with plain Ingress at all — achieving a 90/10 canary split with Ingress requires vendor-specific annotations (different syntax per Ingress Controller, not portable), while Gateway API's HTTPRoute expresses it as a first-class, standard field, portable across any Gateway API-compliant implementation — directly connecting to the progressive-delivery patterns covered in the Automation, CI/CD & GitOps series.

A realistic, honest assessment worth giving if asked "should we migrate today": Ingress remains extremely widely deployed and is not going away imminently — most production clusters today still run Ingress, and it remains entirely valid for straightforward host/path routing. Gateway API is the clear direction of travel for anything needing traffic splitting, more expressive matching, or genuine platform/application team separation of concerns, but a wholesale migration is a real, deliberate project, not a drop-in replacement to reach for by default.

CoreDNS — Service Discovery Inside the Cluster#

CoreDNS runs as pods inside the cluster (itself, notably, deployed as a Deployment) and provides DNS resolution for Kubernetes objects — this is exactly what lets application code simply call http://checkout-svc (or the fuller checkout-svc.default.svc.cluster.local) instead of ever needing to know a Service's actual virtual IP.

This directly builds on the full DNS resolution journey covered in the Linux & Networking Fundamentals series (Part 2) — CoreDNS is simply a Kubernetes-specific authoritative DNS server for the cluster's internal .svc.cluster.local domain.

A Full Worked Request Journey#

Tying networking concepts together into one complete story: a user's browser reaching a specific backend pod.

Volumes — The Basic Storage Building Block#

By default, a container's filesystem is ephemeral — anything written to it disappears the moment the container restarts or the pod is deleted, since it's just the top writable layer of the container image. A Volume attaches durable (or at least longer-lived) storage to a pod.

PersistentVolumes and PersistentVolumeClaims#

This is the genuinely important storage pattern, and its two-sided design is a very commonly asked interview topic.

Simple analogy: a PersistentVolume is like an actual apartment unit that exists in a building. A PersistentVolumeClaim is like a tenant's application/lease request ("I need a 2-bedroom unit") — the system matches ("binds") the request to an available unit meeting those requirements. The application developer writing a pod spec never has to know or care about the underlying real storage implementation (which cloud disk type, which specific NFS server) — they just declare a PVC, and Kubernetes handles the matching.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: fast-ssd

StorageClass and Dynamic Provisioning#

In modern Kubernetes, you almost never manually pre-create PersistentVolumes one by one — a StorageClass defines a "template" for automatically creating a new, real storage volume on demand, the moment a matching PVC is created.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Retain

Why reclaimPolicy is worth knowing specifically, and it's a genuinely important, sometimes costly-to-get-wrong setting: Delete (the common default) means the underlying real storage is destroyed the moment its PVC is deleted; Retain keeps the underlying storage around even after the PVC is gone, requiring manual cleanup. For genuinely critical data, Retain is often the safer choice — accidentally deleting a PVC with Delete reclaim policy means the actual data is gone, immediately, with no recovery path.

StorageClass Topology Awareness#

A genuinely important, easy-to-miss interaction between storage provisioning and the scheduling model from Part 2, worth understanding precisely for anyone running multi-zone clusters.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3

Why this is a genuinely real, commonly-hit production issue worth naming precisely, not a theoretical edge case: most cloud block storage (AWS EBS being the canonical example) is zone-scoped — an EBS volume created in us-east-1a simply cannot be attached to an EC2 instance running in us-east-1b. With Immediate binding, the provisioner has no idea yet which zone the Scheduler will eventually choose, so it's genuinely possible (and does happen in real multi-AZ clusters) for a pod to get permanently stuck Pending because its already-provisioned volume lives in the wrong zone relative to where the Scheduler wants to place it. WaitForFirstConsumer is the correct default for essentially all zone-scoped block storage, and is exactly why it became the modern default for most cloud CSI drivers' StorageClasses.

CSI — The Storage Plugin Interface#

Exactly the same delegation pattern as CRI (Part 1) and CNI (earlier in this Part): Kubernetes doesn't hardcode support for every possible storage backend — it defines a standard CSI (Container Storage Interface), and any storage vendor can write a CSI driver implementing it.

Why this three-times-repeated pattern (CRI, CNI, CSI) is genuinely worth calling out as a unifying theme, not three unrelated facts to memorize separately: Kubernetes's core design philosophy is to define stable, standard interfaces for pluggable concerns (how to run a container, how to network it, how to give it storage) rather than hardcoding any specific vendor's implementation — this is exactly what lets the same Kubernetes YAML manifest work essentially unchanged across AWS, GCP, on-prem, or any other CSI/CNI/CRI-compliant environment.

CSI Volume Snapshots, Cloning, and Expansion#

Modern CSI drivers support three genuinely important operational capabilities beyond basic provisioning — each solves a real, concrete production need.

# Take a snapshot of an existing PVC
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-data-snapshot
spec:
  volumeSnapshotClassName: csi-aws-vsc
  source:
    persistentVolumeClaimName: postgres-data-postgres-0
---
# Restore a NEW PVC from that snapshot
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data-restored
spec:
  dataSource:
    name: postgres-data-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 20Gi

# Volume expansion is just editing the PVC's requested size directly —
# no separate "expand" command, ASSUMING the StorageClass allows it
kubectl patch pvc postgres-data-postgres-0 -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

Volume expansion's one real prerequisite, worth stating precisely: the StorageClass must have allowVolumeExpansion: true set — it's not universally on by default. Without it, attempting to grow a PVC is simply rejected. A second, genuinely important nuance for some CSI drivers and filesystems: the underlying block device may expand immediately, but the filesystem on top of it sometimes requires the pod to restart (or, for some drivers, an online filesystem resize happens automatically) before the additional space is actually usable inside the container — worth verifying the specific CSI driver's documented behavior rather than assuming either way.

Why the snapshot-then-restore-as-a-new-PVC pattern matters concretely, directly connecting to the Disaster Recovery topic in this course: this is exactly the mechanism a real backup/restore strategy for stateful Kubernetes workloads relies on — combined with a scheduled CronJob (Part 2) triggering periodic VolumeSnapshot creation, this gives genuine, native point-in-time recovery capability without needing a separate, external backup agent for the storage layer itself.

Access Modes — A Genuinely Common Gotcha#

A PersistentVolume's access mode determines how many pods (and nodes) can use it simultaneously — a real, frequently-encountered source of confusion.

A genuinely common, real production mistake this explains: trying to scale a Deployment using a ReadWriteOnce PVC across multiple nodes fails, or pods get stuck Pending, because the underlying storage (typically block storage like AWS EBS) simply cannot be attached read-write to more than one node at once — this is precisely why StatefulSets (each replica gets its own, separate PVC, not one shared PVC) are the standard pattern for multi-replica stateful workloads on RWO storage, rather than trying to share a single volume.

StatefulSets and Storage, Tied Together#

Closing the loop with Part 2's StatefulSet discussion: this is exactly why StatefulSets use a volumeClaimTemplates field instead of a single shared volumes reference.

apiVersion: apps/v1
kind: StatefulSet
spec:
  volumeClaimTemplates:
    - metadata:
        name: postgres-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi

Each replica automatically gets its own dedicated PVC, generated from this template, and — critically — that specific PVC stays bound to that specific replica's identity (postgres-0 always gets postgres-data-postgres-0) even across restarts and rescheduling — exactly the stable identity + stable storage guarantee that makes StatefulSets suitable for real databases, where each replica has genuinely different, non-interchangeable data.

Part 3 CLI Cheat Sheet#

# Networking diagnostics
kubectl get svc,endpointslices -l app=checkout-service
kubectl describe svc checkout-svc | grep -A 5 Endpoints
kubectl get networkpolicy -A
kubectl exec -it debug-pod -- nslookup checkout-svc.default.svc.cluster.local
kubectl run tmp-shell --rm -it --image=nicolaka/netshoot -- /bin/bash   # ad hoc network debugging

# Ingress / Gateway API
kubectl get ingress -A
kubectl describe ingress main-ingress
kubectl get gatewayclass,gateway,httproute -A

# Storage diagnostics
kubectl get pv,pvc -A
kubectl describe pvc postgres-data-postgres-0 | grep -A 10 Events
kubectl get storageclass
kubectl get volumesnapshot -A

# Common troubleshooting one-liners
kubectl get pods -o json | jq '.items[] | select(.status.phase=="Pending") | .metadata.name'
kubectl get events --sort-by='.lastTimestamp' -A | tail -30

Common Mistakes#

Worked Practice Problems#

Problem 1: A pod fails its readiness probe, but a teammate insists "the pod is still running fine, why did traffic stop?" Walk through the exact mechanism that explains this.

Answer: A failed readiness probe doesn't stop the pod from running — the container keeps executing normally. What actually happens: the EndpointSlice controller, watching pods matching the Service's label selector, notices this specific pod is no longer marked Ready and removes its IP from the EndpointSlice backing that Service. kube-proxy, watching EndpointSlices on every node, updates its local routing rules accordingly — so traffic simply stops being routed to that pod's IP, even though the pod (and its container) is technically still alive and running. This is precisely the mechanism tying together readiness probes, EndpointSlices, and kube-proxy from across this tutorial.

Problem 2: A team deploys a 3-replica StatefulSet running a database, expecting each replica to have its own independent storage. Instead, they configured a single PersistentVolumeClaim referenced directly in the pod template (not volumeClaimTemplates), and now all 3 replicas appear to share/conflict over the same data. What's the root cause?

Answer: Using a single, directly-referenced PVC means all 3 replicas are attempting to mount the exact same underlying volume — which, for typical ReadWriteOnce block storage, either fails outright for pods on different nodes, or (if it happens to work, e.g. all replicas landed on the same node) results in multiple independent database processes writing to the exact same files, causing corruption or conflicts, since each replica's database engine expects to own its own separate data directory. The fix is using volumeClaimTemplates in the StatefulSet spec instead, which automatically generates a separate, dedicated PVC per replica (postgres-data-postgres-0, -1, -2), each bound to its own real storage volume.

Problem 3: A cluster admin sets a StorageClass's reclaimPolicy to Delete for a StorageClass used by a critical production database's PVCs. Someone accidentally runs kubectl delete pvc postgres-data-postgres-0. What happens, and how could this have been prevented?

Answer: With reclaimPolicy: Delete, deleting the PVC immediately triggers deletion of the underlying real storage volume (e.g., the actual AWS EBS disk) as well — the database's actual data is gone, immediately, with no recovery path unless a separate backup exists entirely outside Kubernetes's own storage lifecycle. This could have been prevented by setting reclaimPolicy: Retain on the StorageClass for anything holding genuinely critical data — with Retain, deleting the PVC leaves the underlying volume intact (just unbound), requiring a deliberate, separate manual step to actually destroy the real data, adding a meaningful safety buffer against exactly this kind of accidental deletion.

Problem 4: A newly-deployed pod remains stuck in Pending in a 3-AZ EKS cluster. kubectl describe pod shows a volume attach error referencing an availability zone that doesn't match any node the pod could run on. What's the root cause, and what StorageClass setting fixes it?

Answer: The StorageClass used by this pod's PVC almost certainly has volumeBindingMode: Immediate — the CSI provisioner created the actual EBS volume immediately upon PVC creation, before the Scheduler had decided which node (and therefore which zone) the pod would actually run on, and the volume ended up in a different zone than where the Scheduler subsequently placed the pod. Since EBS volumes are zone-scoped and can't attach across zones, the pod is permanently stuck. The fix: change the StorageClass's volumeBindingMode to WaitForFirstConsumer, which delays actual volume provisioning until a pod requiring it is being scheduled, guaranteeing the volume is created in the same zone the Scheduler has already selected.

Problem 5: A security audit finds that a compromised, low-privilege pod in one namespace was able to directly reach a database pod in a completely different namespace, despite the two teams believing their namespaces were isolated from each other. What's the most likely root cause, and what's the concrete fix?

Answer: Kubernetes's actual default networking behavior is fully open — every pod can reach every other pod cluster-wide unless a NetworkPolicy explicitly restricts it. The teams' belief that separate namespaces implied network isolation was simply incorrect; namespaces provide RBAC and resource-quota boundaries (Part 1, Part 2), but NOT network isolation on their own. The concrete fix: apply a default-deny-all NetworkPolicy (matching all pods via podSelector: {}, covering both Ingress and Egress in policyTypes) to every namespace as a baseline, then add specific, narrow allow rules (using namespaceSelector/podSelector combinations) only for the traffic flows that are actually legitimate — closing off the default-open posture entirely rather than assuming namespace boundaries provide network security they don't actually provide.

Problem 6: A pod's application logs show it successfully resolving google.com and other external domains, but every single request to checkout-svc.default.svc.cluster.local fails with an unresolvable-hostname error, while every other pod in the same namespace resolves that exact name fine. What field on this specific pod is almost certainly misconfigured, and why does the symptom pattern point there so precisely?

Answer: This is a strong signature of dnsPolicy: Default being set on this one pod specifically (likely copy-pasted from an example, given the misleading name) — Default bypasses CoreDNS entirely and resolves DNS using the node's own /etc/resolv.conf, which correctly resolves genuine external names (explaining why google.com works) but has no knowledge whatsoever of the cluster's internal .svc.cluster.local domain (explaining why cluster-internal Service names fail specifically and consistently). The fact that every other pod in the same namespace resolves the same name fine strongly rules out a CoreDNS or NetworkPolicy issue (which would typically affect all pods needing that name, not just one) and points specifically at a per-pod dnsPolicy misconfiguration. The fix: change this pod's dnsPolicy to ClusterFirst (the actual default), restoring CoreDNS-based resolution for cluster-internal names.

Summary and What's Next#

• Kubernetes's networking model gives every pod its own real IP, directly reachable from every other pod, cluster-wide, with no NAT — a deliberately simple, flat model, actually implemented by a pluggable CNI plugin.
• Services solve the "pods are disposable, their IPs keep changing" problem by providing a stable virtual IP/DNS name, load-balancing across whichever pods currently match a label selector — the four types (ClusterIP, NodePort, LoadBalancer, ExternalName) serve genuinely different purposes.
• EndpointSlices are the live, continuously-updated list of healthy pod IPs backing a Service — this is the exact, concrete mechanism connecting readiness probes to actual traffic routing.
• Ingress (with an Ingress Controller actually running) provides a single, shared entry point routing to many Services by host/path, avoiding a separate cloud load balancer per service.
• CoreDNS provides internal service discovery, letting application code use stable names instead of ever needing to know a Service's virtual IP directly.
• PersistentVolumes/PersistentVolumeClaims decouple "what storage actually exists" from "what an application asked for," matched via binding — and StorageClass enables automatic, on-demand (dynamic) provisioning of new real storage.
• CSI, like CRI and CNI, is a standard plugin interface — Kubernetes doesn't hardcode any specific storage vendor, which is exactly what makes the same manifests portable across environments.
• Access modes (especially the very common ReadWriteOnce limitation) directly explain why StatefulSets use volumeClaimTemplates to give each replica its own dedicated storage, rather than sharing one volume.
• Headless Services (clusterIP: None) resolve DNS directly to individual pod IPs instead of a shared virtual IP — exactly what gives StatefulSet pods their own resolvable, stable DNS names.
• NetworkPolicy's real default is fully open — network isolation is opt-in, not opt-out, and a default-deny-all policy per namespace is the recommended security baseline; namespaceSelector/podSelector combined in one list item AND, in separate items OR.
• NetworkPolicies combine additively across a namespace — never subtract from each other — so a single overly-permissive policy can silently undermine an otherwise-tight default-deny baseline; always review the full set selecting a given pod, not just one policy in isolation.
• Gateway API cleanly separates infrastructure ownership (GatewayClass/Gateway, platform team) from routing ownership (HTTPRoute, application teams), and natively supports traffic splitting without vendor-specific annotations — the clear direction of travel, though Ingress remains widely deployed today.
• volumeBindingMode: WaitForFirstConsumer is the correct default for zone-scoped block storage in multi-AZ clusters, avoiding a real, common "volume provisioned in the wrong zone" stuck-Pending failure mode.
• CSI snapshots, cloning, and expansion give native backup/restore and in-place growth capability, provided the StorageClass has allowVolumeExpansion: true where growth is needed.
• Topology Aware Routing and dnsPolicy both directly affect real, everyday production behavior — the first is a genuine cross-zone cost/latency lever most default configurations leave on the table, the second's misleadingly-named Default option is a real, confusing footgun worth knowing by name before it costs debugging time.
• Session affinity (sessionAffinity: ClientIP) is a legitimate stopgap for legacy, stateful applications, but works against even load distribution and horizontal scaling — new services should be designed stateless with session data in an external store instead of leaning on it.

Continue to Part 4 (04-service-mesh-and-advanced-topics.md) for a deeper look at service meshes (building on the sidecar pattern from this Part), etcd operational depth, and how Custom Resources and Operators extend Kubernetes itself.