Networking (CNI) & Storage (CSI)

Table of Contents#

1. The Kubernetes Networking Model — The Ground Rules
2. CNI — The Plugin That Makes the Model Real
3. Why Pods Get IPs and Why That's a Big Deal
4. Services — Solving the "Pods Are Disposable" Problem
5. The Four Service Types
6. Endpoints and EndpointSlices
7. Ingress — Getting Traffic In From Outside
8. CoreDNS — Service Discovery Inside the Cluster
9. A Full Worked Request Journey
10. Volumes — The Basic Storage Building Block
11. PersistentVolumes and PersistentVolumeClaims
12. StorageClass and Dynamic Provisioning
13. CSI — The Storage Plugin Interface
14. Access Modes — A Genuinely Common Gotcha
15. StatefulSets and Storage, Tied Together
16. Common Mistakes
17. Worked Practice Problems
18. Summary and What's Next

---

The Kubernetes Networking Model — The Ground Rules#

Kubernetes networking is built on a small number of simple, strict rules — and nearly everything more complex (Services, Ingress, NetworkPolicies) is built as a layer on top of these ground rules.

Diagram

Why this "flat network" model is such a deliberate, important design choice: it means container-to-container networking in Kubernetes works essentially like normal networking between separate physical machines — no special-case NAT traversal logic, no "which port did this get mapped to" complexity that plagued earlier container networking approaches (like classic Docker's default bridge networking). Every pod is a full, first-class citizen on the network, directly addressable by its own IP.

---

CNI — The Plugin That Makes the Model Real#

Kubernetes itself doesn't implement this networking model — it defines the rules and delegates the actual implementation to a CNI (Container Network Interface) plugin, exactly the same delegation pattern as the CRI for container runtimes from Part 1.

Diagram

A directly important connection back to the DevSecOps series' container security tutorial, worth restating explicitly here: NetworkPolicy objects are a Kubernetes API concept, but enforcing them is entirely up to the CNI plugin — some plugins (Calico, Cilium) fully support and enforce them; others (some simpler/older CNI setups) don't enforce them at all, meaning a NetworkPolicy YAML could apply successfully with zero actual effect. Always verify which CNI plugin a cluster actually runs before assuming NetworkPolicies are doing anything.

Copy
# Check which CNI plugin a cluster is running (commonly, look at
# the CNI-related DaemonSet running in kube-system)
kubectl get pods -n kube-system | grep -Ei "calico|cilium|flannel|weave"

---

Why Pods Get IPs and Why That's a Big Deal#

A directly practical consequence worth spelling out: because every pod gets its own real IP, containers within the same pod share that single IP and its network namespace (from the Linux & Networking Fundamentals series, Part 1) — they talk to each other over localhost, exactly like processes on the same machine, while still each having their own separate IP address relative to every other pod in the cluster.

Diagram

This is precisely the mechanical foundation of the "sidecar" pattern referenced throughout this course (service mesh proxies, log-shipping sidecars) — a sidecar container works because it shares its pod's network namespace and can transparently intercept traffic to/from localhost without the main application container needing to know or care.

---

Services — Solving the "Pods Are Disposable" Problem#

Pods are disposable — they get created and destroyed constantly (deploys, crashes, scaling, rescheduling), and each new pod gets a brand-new IP address. This creates an obvious problem: how does anything reliably talk to "the checkout service" if the actual IPs behind that name keep changing?

Diagram

Simple analogy: a Service is like a company's general customer support phone number — the number itself never changes, even though the specific employee who actually answers any given call is different every time (and employees come and go). Callers only need to remember the one stable number.

---

The Four Service Types#

Diagram

Copy
apiVersion: v1
kind: Service
metadata:
  name: checkout-svc
spec:
  type: ClusterIP
  selector:
    app: checkout-service   # matches pods with THIS label
  ports:
    - port: 80
      targetPort: 8080

A genuinely important detail, worth being explicit about: a Service finds its backing pods purely via a label selector — this is a loose, dynamic coupling (not a fixed list of pod names), which is exactly what lets a Service automatically pick up new pods and drop terminated ones, continuously, with zero manual reconfiguration.

---

Endpoints and EndpointSlices#

Underneath a Service, Kubernetes maintains the actual, current list of healthy pod IPs backing it — historically via an Endpoints object, now more commonly via the newer, more scalable EndpointSlice API.

Diagram

This is the exact, concrete mechanism tying together the readiness probe discussion from Part 2 and the kube-proxy discussion from Part 1: a failed readiness probe doesn't just log a status — it actively, automatically removes that pod from the real, live list of addresses traffic gets routed to, cluster-wide, within moments.

---

Ingress — Getting Traffic In From Outside#

A LoadBalancer Service works, but provisioning a full, separate cloud load balancer for every single service in a cluster gets expensive and unwieldy fast. Ingress solves this by providing a single entry point that can route to many different Services based on the request's hostname/path — directly reusing the Layer 7 load balancing concepts from the Reliability & Architecture Patterns series.

Diagram

Copy
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
spec:
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: checkout-svc
                port:
                  number: 80

A worthwhile, genuinely important nuance: the Ingress object is just a set of routing rules — it does nothing on its own without an Ingress Controller (like NGINX Ingress Controller, or a cloud-managed one) actually running in the cluster to read those rules and configure real routing. This is exactly the same "the API object is a declaration; something else has to actually implement it" pattern as CNI and CSI throughout this whole series.

Worth knowing exists, even briefly: the newer Gateway API is gradually superseding Ingress as the more expressive, more standardized way to configure L7 routing in Kubernetes — Ingress remains extremely widely deployed today, but Gateway API is increasingly the direction the ecosystem is moving, worth mentioning if asked about the current state of the art.

---

CoreDNS — Service Discovery Inside the Cluster#

CoreDNS runs as pods inside the cluster (itself, notably, deployed as a Deployment) and provides DNS resolution for Kubernetes objects — this is exactly what lets application code simply call http://checkout-svc (or the fuller checkout-svc.default.svc.cluster.local) instead of ever needing to know a Service's actual virtual IP.

Diagram

This directly builds on the full DNS resolution journey covered in the Linux & Networking Fundamentals series (Part 2) — CoreDNS is simply a Kubernetes-specific authoritative DNS server for the cluster's internal .svc.cluster.local domain.

---

A Full Worked Request Journey#

Tying networking concepts together into one complete story: a user's browser reaching a specific backend pod.

Diagram

---

Volumes — The Basic Storage Building Block#

By default, a container's filesystem is ephemeral — anything written to it disappears the moment the container restarts or the pod is deleted, since it's just the top writable layer of the container image. A Volume attaches durable (or at least longer-lived) storage to a pod.

Diagram

---

PersistentVolumes and PersistentVolumeClaims#

This is the genuinely important storage pattern, and its two-sided design is a very commonly asked interview topic.

Diagram

Simple analogy: a PersistentVolume is like an actual apartment unit that exists in a building. A PersistentVolumeClaim is like a tenant's application/lease request ("I need a 2-bedroom unit") — the system matches ("binds") the request to an available unit meeting those requirements. The application developer writing a pod spec never has to know or care about the underlying real storage implementation (which cloud disk type, which specific NFS server) — they just declare a PVC, and Kubernetes handles the matching.

Copy
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: fast-ssd

---

StorageClass and Dynamic Provisioning#

In modern Kubernetes, you almost never manually pre-create PersistentVolumes one by one — a StorageClass defines a "template" for automatically creating a new, real storage volume on demand, the moment a matching PVC is created.

Diagram

Copy
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Retain

Why reclaimPolicy is worth knowing specifically, and it's a genuinely important, sometimes costly-to-get-wrong setting: Delete (the common default) means the underlying real storage is destroyed the moment its PVC is deleted; Retain keeps the underlying storage around even after the PVC is gone, requiring manual cleanup. For genuinely critical data, Retain is often the safer choice — accidentally deleting a PVC with Delete reclaim policy means the actual data is gone, immediately, with no recovery path.

---

CSI — The Storage Plugin Interface#

Exactly the same delegation pattern as CRI (Part 1) and CNI (earlier in this Part): Kubernetes doesn't hardcode support for every possible storage backend — it defines a standard CSI (Container Storage Interface), and any storage vendor can write a CSI driver implementing it.

Diagram

Why this three-times-repeated pattern (CRI, CNI, CSI) is genuinely worth calling out as a unifying theme, not three unrelated facts to memorize separately: Kubernetes's core design philosophy is to define stable, standard interfaces for pluggable concerns (how to run a container, how to network it, how to give it storage) rather than hardcoding any specific vendor's implementation — this is exactly what lets the same Kubernetes YAML manifest work essentially unchanged across AWS, GCP, on-prem, or any other CSI/CNI/CRI-compliant environment.

---

Access Modes — A Genuinely Common Gotcha#

A PersistentVolume's access mode determines how many pods (and nodes) can use it simultaneously — a real, frequently-encountered source of confusion.

Diagram

A genuinely common, real production mistake this explains: trying to scale a Deployment using a ReadWriteOnce PVC across multiple nodes fails, or pods get stuck Pending, because the underlying storage (typically block storage like AWS EBS) simply cannot be attached read-write to more than one node at once — this is precisely why StatefulSets (each replica gets its own, separate PVC, not one shared PVC) are the standard pattern for multi-replica stateful workloads on RWO storage, rather than trying to share a single volume.

---

StatefulSets and Storage, Tied Together#

Closing the loop with Part 2's StatefulSet discussion: this is exactly why StatefulSets use a volumeClaimTemplates field instead of a single shared volumes reference.

Diagram

Copy
apiVersion: apps/v1
kind: StatefulSet
spec:
  volumeClaimTemplates:
    - metadata:
        name: postgres-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi

Each replica automatically gets its own dedicated PVC, generated from this template, and — critically — that specific PVC stays bound to that specific replica's identity (postgres-0 always gets postgres-data-postgres-0) even across restarts and rescheduling — exactly the stable identity + stable storage guarantee that makes StatefulSets suitable for real databases, where each replica has genuinely different, non-interchangeable data.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A pod fails its readiness probe, but a teammate insists "the pod is still running fine, why did traffic stop?" Walk through the exact mechanism that explains this.

Answer: A failed readiness probe doesn't stop the pod from running — the container keeps executing normally. What actually happens: the EndpointSlice controller, watching pods matching the Service's label selector, notices this specific pod is no longer marked Ready and removes its IP from the EndpointSlice backing that Service. kube-proxy, watching EndpointSlices on every node, updates its local routing rules accordingly — so traffic simply stops being routed to that pod's IP, even though the pod (and its container) is technically still alive and running. This is precisely the mechanism tying together readiness probes, EndpointSlices, and kube-proxy from across this tutorial.

Problem 2: A team deploys a 3-replica StatefulSet running a database, expecting each replica to have its own independent storage. Instead, they configured a single PersistentVolumeClaim referenced directly in the pod template (not volumeClaimTemplates), and now all 3 replicas appear to share/conflict over the same data. What's the root cause?

Answer: Using a single, directly-referenced PVC means all 3 replicas are attempting to mount the exact same underlying volume — which, for typical ReadWriteOnce block storage, either fails outright for pods on different nodes, or (if it happens to work, e.g. all replicas landed on the same node) results in multiple independent database processes writing to the exact same files, causing corruption or conflicts, since each replica's database engine expects to own its own separate data directory. The fix is using volumeClaimTemplates in the StatefulSet spec instead, which automatically generates a separate, dedicated PVC per replica (postgres-data-postgres-0, -1, -2), each bound to its own real storage volume.

Problem 3: A cluster admin sets a StorageClass's reclaimPolicy to Delete for a StorageClass used by a critical production database's PVCs. Someone accidentally runs kubectl delete pvc postgres-data-postgres-0. What happens, and how could this have been prevented?

Answer: With reclaimPolicy: Delete, deleting the PVC immediately triggers deletion of the underlying real storage volume (e.g., the actual AWS EBS disk) as well — the database's actual data is gone, immediately, with no recovery path unless a separate backup exists entirely outside Kubernetes's own storage lifecycle. This could have been prevented by setting reclaimPolicy: Retain on the StorageClass for anything holding genuinely critical data — with Retain, deleting the PVC leaves the underlying volume intact (just unbound), requiring a deliberate, separate manual step to actually destroy the real data, adding a meaningful safety buffer against exactly this kind of accidental deletion.

---

Summary and What's Next#

• Kubernetes's networking model gives every pod its own real IP, directly reachable from every other pod, cluster-wide, with no NAT — a deliberately simple, flat model, actually implemented by a pluggable CNI plugin.
• Services solve the "pods are disposable, their IPs keep changing" problem by providing a stable virtual IP/DNS name, load-balancing across whichever pods currently match a label selector — the four types (ClusterIP, NodePort, LoadBalancer, ExternalName) serve genuinely different purposes.
• EndpointSlices are the live, continuously-updated list of healthy pod IPs backing a Service — this is the exact, concrete mechanism connecting readiness probes to actual traffic routing.
• Ingress (with an Ingress Controller actually running) provides a single, shared entry point routing to many Services by host/path, avoiding a separate cloud load balancer per service.
• CoreDNS provides internal service discovery, letting application code use stable names instead of ever needing to know a Service's virtual IP directly.
• PersistentVolumes/PersistentVolumeClaims decouple "what storage actually exists" from "what an application asked for," matched via binding — and StorageClass enables automatic, on-demand (dynamic) provisioning of new real storage.
• CSI, like CRI and CNI, is a standard plugin interface — Kubernetes doesn't hardcode any specific storage vendor, which is exactly what makes the same manifests portable across environments.
• Access modes (especially the very common ReadWriteOnce limitation) directly explain why StatefulSets use volumeClaimTemplates to give each replica its own dedicated storage, rather than sharing one volume.

Continue to Part 4 (04-service-mesh-and-advanced-topics.md) for a deeper look at service meshes (building on the sidecar pattern from this Part), etcd operational depth, and how Custom Resources and Operators extend Kubernetes itself.