# Kubernetes Deep Dive — Part 3: Networking (CNI) & Storage (CSI)
> **Series:** Kubernetes Deep Dive (3 of 9)
> **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane
> **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects
> **Part 3:** This file — Networking (CNI) & Storage (CSI)
> **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators
> **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE
> **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes
> **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth
> **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway
> **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem
> **Questions:** `questions.md`
## Table of Contents
1. [The Kubernetes Networking Model — The Ground Rules](#the-kubernetes-networking-model--the-ground-rules)
2. [CNI — The Plugin That Makes the Model Real](#cni--the-plugin-that-makes-the-model-real)
3. [Why Pods Get IPs and Why That's a Big Deal](#why-pods-get-ips-and-why-thats-a-big-deal)
4. [Services — Solving the "Pods Are Disposable" Problem](#services--solving-the-pods-are-disposable-problem)
5. [The Four Service Types](#the-four-service-types)
6. [Session Affinity](#session-affinity)
7. [Headless Services](#headless-services)
8. [Endpoints and EndpointSlices](#endpoints-and-endpointslices)
9. [NetworkPolicy — Full Depth](#networkpolicy--full-depth)
10. [Topology Aware Routing](#topology-aware-routing)
11. [DNS Policies and Custom DNS Configuration](#dns-policies-and-custom-dns-configuration)
12. [Ingress — Getting Traffic In From Outside](#ingress--getting-traffic-in-from-outside)
13. [Gateway API — The Modern Ingress Successor](#gateway-api--the-modern-ingress-successor)
14. [CoreDNS — Service Discovery Inside the Cluster](#coredns--service-discovery-inside-the-cluster)
15. [A Full Worked Request Journey](#a-full-worked-request-journey)
16. [Volumes — The Basic Storage Building Block](#volumes--the-basic-storage-building-block)
17. [PersistentVolumes and PersistentVolumeClaims](#persistentvolumes-and-persistentvolumeclaims)
18. [StorageClass and Dynamic Provisioning](#storageclass-and-dynamic-provisioning)
19. [StorageClass Topology Awareness](#storageclass-topology-awareness)
20. [CSI — The Storage Plugin Interface](#csi--the-storage-plugin-interface)
21. [CSI Volume Snapshots, Cloning, and Expansion](#csi-volume-snapshots-cloning-and-expansion)
22. [Access Modes — A Genuinely Common Gotcha](#access-modes--a-genuinely-common-gotcha)
23. [StatefulSets and Storage, Tied Together](#statefulsets-and-storage-tied-together)
24. [Part 3 CLI Cheat Sheet](#part-3-cli-cheat-sheet)
25. [Common Mistakes](#common-mistakes)
26. [Worked Practice Problems](#worked-practice-problems)
27. [Summary and What's Next](#summary-and-whats-next)
---
## The Kubernetes Networking Model — The Ground Rules
Kubernetes networking is built on a small number of simple, strict rules — and nearly everything more complex (Services, Ingress, NetworkPolicies) is built as a layer on top of these ground rules.
```mermaid
graph TD
Rules["The Kubernetes Networking
Model's Ground Rules"] --> R1["Every Pod gets its OWN
unique IP address"]
Rules --> R2["Every Pod can reach EVERY
other Pod's IP directly,
WITHOUT NAT (Network
Address Translation),
cluster-wide"]
Rules --> R3["A Pod sees its OWN IP the
SAME way everyone else
sees it (no confusing
internal-vs-external
address difference)"]
```
**Why this "flat network" model is such a deliberate, important design choice:** it means container-to-container networking in Kubernetes works essentially like normal networking between separate physical machines — no special-case NAT traversal logic, no "which port did this get mapped to" complexity that plagued earlier container networking approaches (like classic Docker's default bridge networking). Every pod is a full, first-class citizen on the network, directly addressable by its own IP.
---
## CNI — The Plugin That Makes the Model Real
Kubernetes itself doesn't implement this networking model — it defines the *rules* and delegates the actual implementation to a **CNI (Container Network Interface)** plugin, exactly the same delegation pattern as the CRI for container runtimes from Part 1.
```mermaid
graph TD
Kubelet["kubelet"] -->|"CNI - a STANDARD
plugin interface"| Plugin["CNI Plugin
(Calico, Cilium, Flannel,
AWS VPC CNI, etc.)"]
Plugin --> Job["Actually sets up: pod IP
assignment, routing
between nodes, and
(for some plugins)
NetworkPolicy enforcement"]
```
**A directly important connection back to the DevSecOps series' container security tutorial, worth restating explicitly here:** NetworkPolicy objects are a *Kubernetes API* concept, but **enforcing** them is entirely up to the CNI plugin — some plugins (Calico, Cilium) fully support and enforce them; others (some simpler/older CNI setups) don't enforce them at all, meaning a NetworkPolicy YAML could apply successfully with zero actual effect. **Always verify which CNI plugin a cluster actually runs before assuming NetworkPolicies are doing anything.**
```bash
# Check which CNI plugin a cluster is running (commonly, look at
# the CNI-related DaemonSet running in kube-system)
kubectl get pods -n kube-system | grep -Ei "calico|cilium|flannel|weave"
```
---
## Why Pods Get IPs and Why That's a Big Deal
A directly practical consequence worth spelling out: because every pod gets its own real IP, **containers within the same pod share that single IP and its network namespace** (from the Linux & Networking Fundamentals series, Part 1) — they talk to each other over `localhost`, exactly like processes on the same machine, while still each having their own separate IP address relative to every *other* pod in the cluster.
```mermaid
graph TD
Pod["One Pod, IP: 10.244.1.5"] --> C1["Container A
(app)"]
Pod --> C2["Container B
(sidecar, e.g. a service
mesh proxy - Part 4)"]
C1 <-->|"talk via localhost -
SAME network namespace"| C2
```
**This is precisely the mechanical foundation of the "sidecar" pattern** referenced throughout this course (service mesh proxies, log-shipping sidecars) — a sidecar container works because it shares its pod's network namespace and can transparently intercept traffic to/from `localhost` without the main application container needing to know or care.
---
## Services — Solving the "Pods Are Disposable" Problem
Pods are disposable — they get created and destroyed constantly (deploys, crashes, scaling, rescheduling), and **each new pod gets a brand-new IP address.** This creates an obvious problem: how does anything reliably talk to "the checkout service" if the actual IPs behind that name keep changing?
```mermaid
graph TD
Problem["Pod IPs constantly change
as pods are created/
destroyed"] --> Solution["A Service provides a
STABLE virtual IP and DNS
name that NEVER changes,
automatically load-balancing
across whichever pods are
CURRENTLY healthy"]
```
**Simple analogy:** a Service is like a company's general customer support phone number — the number itself never changes, even though the specific employee who actually answers any given call is different every time (and employees come and go). Callers only need to remember the one stable number.
---
## The Four Service Types
```mermaid
graph TD
Types[Service Types] --> ClusterIP["ClusterIP (default):
a stable, INTERNAL-only
virtual IP - reachable
only from WITHIN the
cluster"]
Types --> NodePort["NodePort: exposes the
Service on the SAME
static port on EVERY
node's own IP"]
Types --> LB["LoadBalancer: provisions
an ACTUAL external cloud
load balancer (AWS ELB,
GCP LB, etc.), pointing
at the Service"]
Types --> External["ExternalName: a pure DNS
alias, pointing to an
EXTERNAL name outside
the cluster entirely"]
```
| Type | Reachable From | Common Use |
|---|---|---|
| **ClusterIP** | Only inside the cluster | Internal service-to-service communication (the vast majority of Services) |
| **NodePort** | Any node's IP, on a fixed port (30000-32767 range) | Rarely used directly in production; often a building block underneath LoadBalancer |
| **LoadBalancer** | The public internet (via a real cloud load balancer) | Exposing a service externally, in a cloud environment |
| **ExternalName** | Internally, but just as a DNS alias to something outside | Referencing an external database/API by a consistent internal name |
```yaml
apiVersion: v1
kind: Service
metadata:
name: checkout-svc
spec:
type: ClusterIP
selector:
app: checkout-service # matches pods with THIS label
ports:
- port: 80
targetPort: 8080
```
**A genuinely important detail, worth being explicit about: a Service finds its backing pods purely via a label selector** — this is a loose, dynamic coupling (not a fixed list of pod names), which is exactly what lets a Service automatically pick up new pods and drop terminated ones, continuously, with zero manual reconfiguration.
---
## Session Affinity
By default, a Service load-balances every individual request independently, round-robin-ish, across its healthy backend pods — with no concept of "this specific client should keep hitting the same pod." **Session Affinity** changes that, worth knowing precisely since it's a real, commonly-needed setting for a specific class of application.
```yaml
apiVersion: v1
kind: Service
metadata:
name: legacy-app-svc
spec:
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800 # 3 hours
selector:
app: legacy-app
```
```mermaid
graph TD
None2["sessionAffinity: None
(default): every request
independently load-balanced
- any healthy pod can
answer any request"] --> NoneUse2["Correct for STATELESS
apps - the overwhelming
majority of services"]
ClientIP["sessionAffinity: ClientIP:
requests from the SAME
source IP are routed to
the SAME backend pod, for
the configured timeout"] --> ClientIPUse["For apps that keep
IN-MEMORY session state
and haven't been re-
architected to be truly
stateless (e.g. legacy
in-memory session apps)"]
```
**Why `ClientIP` affinity is worth treating as a real, worth-questioning tradeoff rather than a convenient default, worth stating explicitly:** it's genuinely useful for legacy applications that store session state in local process memory instead of an external store (Redis, a database), but it directly works against even load distribution — a small number of high-traffic clients (common behind a corporate NAT, where many real users share one apparent source IP) can end up disproportionately pinned to the same backend pods. **The architecturally preferred fix, worth naming explicitly, is moving session state to an external store** (making the app genuinely stateless at the pod level) rather than leaning on `ClientIP` affinity as a permanent solution — session affinity is a legitimate stopgap for legacy apps, not a best practice to design new services around.
**Worth noting: `sessionAffinity` is a Service-level, `ClientIP`-based mechanism, distinct from — and much coarser than — application-level session cookies or an Ingress Controller's own cookie-based sticky-session features**, which operate at Layer 7 with far more precision (tied to an actual session identity, not just a source IP that may represent many different real users behind shared infrastructure). A strong interview answer distinguishes these two layers explicitly rather than treating "sticky sessions" as one single, undifferentiated concept.
| Layer | Mechanism | Precision |
|---|---|---|
| Service (L4) | `sessionAffinity: ClientIP` | Coarse — keyed on source IP, which may represent many real users |
| Ingress/L7 proxy | Cookie-based sticky sessions | Precise — keyed on an actual per-session identity |
| Application | Server-side session store (Redis, a database) | Most precise, and the architecturally preferred long-term fix |
---
## Headless Services
A special, genuinely important fifth variant worth its own section, since it directly explains the StatefulSet DNS behavior referenced in Part 2: setting `clusterIP: None` on a Service creates a **Headless Service** — one with NO stable virtual IP at all.
```mermaid
graph TD
Normal["Normal ClusterIP Service:
DNS resolves to ONE stable
virtual IP, kube-proxy
load-balances behind it"] --> NormalUse["Right for STATELESS
backends - any pod is
interchangeable"]
Headless["Headless Service
(clusterIP: None): DNS
resolves DIRECTLY to the
INDIVIDUAL pod IPs
themselves - no virtual IP,
no load-balancing layer
at all"] --> HeadlessUse["Right when the CLIENT
needs to know about and
choose BETWEEN individual
backend instances -
StatefulSets, above all"]
```
```yaml
apiVersion: v1
kind: Service
metadata:
name: postgres-headless
spec:
clusterIP: None
selector:
app: postgres
ports:
- port: 5432
```
**Why this is precisely what makes StatefulSet DNS names work, closing a loop left open in Part 2:** a headless Service paired with a StatefulSet gives each individual pod its own resolvable DNS name (`postgres-0.postgres-headless.default.svc.cluster.local`), rather than only a single load-balanced virtual IP for the whole set. This is exactly why a client that needs to specifically reach `postgres-0` (the primary in a replicated database setup, for instance) can do so directly by name — a regular ClusterIP Service could only ever route to *some* healthy pod, with no way to target one specific replica.
---
## Endpoints and EndpointSlices
Underneath a Service, Kubernetes maintains the *actual, current list* of healthy pod IPs backing it — historically via an **Endpoints** object, now more commonly via the newer, more scalable **EndpointSlice** API.
```mermaid
sequenceDiagram
participant Kubelet as kubelet
participant API as API Server
participant EPC as EndpointSlice Controller
participant Proxy as kube-proxy (every node)
Kubelet->>API: Pod fails its readiness
probe (Part 2)
EPC->>API: Watching pods matching
the Service's selector...
notices this pod is
no longer Ready
EPC->>API: Removes it from the
EndpointSlice
Proxy->>API: Watching EndpointSlices...
updates its local routing
rules
Note over Proxy: Traffic STOPS routing
to that pod, immediately
```
**This is the exact, concrete mechanism tying together the readiness probe discussion from Part 2 and the kube-proxy discussion from Part 1**: a failed readiness probe doesn't just log a status — it actively, automatically removes that pod from the real, live list of addresses traffic gets routed to, cluster-wide, within moments.
---
## NetworkPolicy — Full Depth
Part 3's earlier CNI section flagged that NetworkPolicy enforcement depends entirely on the CNI plugin — here's the actual policy model in depth, since this is genuinely one of the most important, most commonly under-configured Kubernetes security mechanisms.
```mermaid
graph TD
Default["DEFAULT Kubernetes
behavior, NO NetworkPolicy
applied at all: EVERY pod
can reach EVERY other pod,
cluster-wide, with ZERO
restriction"] --> DefaultNote["A genuinely important,
often-surprising fact: this
is Kubernetes's actual
default - network isolation
is OPT-IN, not opt-out"]
```
**This default-open posture is worth stating explicitly as a real security gap in any cluster without NetworkPolicies applied:** a compromised pod (from a vulnerable dependency, a supply-chain issue per the DevSecOps series) can, by default, reach every other pod in the entire cluster — including services it has no legitimate business talking to. NetworkPolicy is the mechanism that closes this.
```yaml
# A default-deny-all policy for a namespace — the recommended
# starting point for any genuinely security-conscious namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: team-checkout
spec:
podSelector: {} # matches ALL pods in this namespace
policyTypes:
- Ingress
- Egress
```
```yaml
# Then EXPLICITLY allow only what's actually needed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-checkout-to-db
namespace: team-checkout
spec:
podSelector:
matchLabels:
app: checkout-service
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
```
**The `podSelector: {}` idiom (an empty selector) matching ALL pods is a genuinely important, easy-to-miss detail — worth stating precisely:** an empty `{}` selector is NOT "no policy" — it's "this policy applies to every pod in the namespace." Combined with `policyTypes: [Ingress, Egress]` and no `ingress`/`egress` rules listed at all, this produces the default-deny-everything baseline shown above — the recommended starting point, with specific allow rules layered on top.
**Combining `namespaceSelector` and `podSelector` in one rule, a genuinely important, precise mechanic worth knowing exactly:**
```yaml
ingress:
- from:
- namespaceSelector:
matchLabels:
team: platform
podSelector:
matchLabels:
app: monitoring-agent
```
**A frequently-tested precision point: when both `namespaceSelector` AND `podSelector` appear together in the SAME list item (as above, both under one `-` entry), they're ANDed** — this rule allows traffic ONLY from pods matching `app: monitoring-agent` that are ALSO in a namespace matching `team: platform`. **If they were instead two SEPARATE list items (two different `-` entries), they'd be ORed** — matching either condition independently. This exact YAML-structure distinction (same list item vs. separate list items) is a real, common source of accidentally-too-permissive or accidentally-too-restrictive policies, and worth double-checking carefully rather than assuming.
**A quick reference for the full NetworkPolicy selector vocabulary, worth having in one place:**
| Selector | Matches |
|---|---|
| `podSelector` (alone) | Pods within the SAME namespace as the policy |
| `namespaceSelector` (alone) | ALL pods in namespaces matching the label, regardless of pod labels |
| `namespaceSelector` + `podSelector` (same item) | Only pods matching BOTH conditions (AND) |
| `ipBlock` | Traffic from/to a specific CIDR range — for traffic outside the cluster entirely (e.g. a known external partner IP range) |
| `ipBlock` with `except` | The CIDR range MINUS specific excluded sub-ranges — a carve-out within a broader allow |
| Empty `podSelector: {}` | ALL pods in the policy's own namespace — the building block of a default-deny baseline |
| No `ingress`/`egress` rules at all, with matching `policyTypes` | Zero traffic allowed in that direction — the strictest possible baseline for the selected pods |
| No `NetworkPolicy` selecting a pod at all | Fully open — the pod is unaffected by any policy in the cluster |
| Egress rule allowing DNS (port 53 to CoreDNS) | Almost always required alongside a default-deny-all — otherwise even legitimate name resolution breaks |
| `port`/`protocol` fields on an `ingress`/`egress` rule | Narrows an otherwise pod/namespace-scoped rule to specific ports — omit to allow all ports on the matched pods |
| `policyTypes: [Egress]` alone, no `Ingress` | The pod's inbound traffic stays fully open — only outbound is restricted, a genuinely common half-applied-policy mistake |
| Testing a policy change directly in production without a dry run | A too-strict rule can silently sever legitimate traffic paths cluster-wide before anyone notices | Validate against a staging namespace, or use `kubectl auth can-i`-style dry-run tooling where available before rolling out broadly |
**A final, worth-stating precision point on how multiple NetworkPolicies combine: they are strictly ADDITIVE, never subtractive.** If two policies both select the same pod, the pod's actual allowed traffic is the UNION of everything either policy permits — there's no "more specific policy wins" or "deny takes precedence" resolution logic the way some other systems work. This means a single overly-permissive policy anywhere in a namespace can silently undermine an otherwise-tight default-deny baseline, which is exactly why reviewing the FULL set of policies selecting a given pod (`kubectl get networkpolicy -n `, checked against a pod's actual labels) is the correct way to reason about its real effective access — never just one policy in isolation.
---
## Topology Aware Routing
A genuinely important, cost-relevant Service feature worth knowing, connecting directly to the multi-zone topology theme running throughout this Part: by default, kube-proxy load-balances a Service's traffic across ALL healthy backend pods cluster-wide, with zero regard for which zone the traffic originated from — **Topology Aware Routing** changes this.
```mermaid
graph TD
Default["Default kube-proxy
behavior: traffic from a
pod in zone A can get
routed to a backend pod in
zone B, C, or anywhere else
- pure round-robin,
zone-blind"] --> DefaultCost["Real cost consequence:
CROSS-ZONE data transfer
fees on every cloud
provider, for traffic that
often didn't NEED to leave
the zone at all"]
Topology["Topology Aware Routing
enabled: EndpointSlices
carry ZONE hints, kube-
proxy PREFERS routing
traffic to a backend in the
SAME zone as the caller,
when a healthy same-zone
backend exists"] --> TopologyBenefit["Reduces cross-zone
data transfer costs AND
often reduces latency -
same-zone hops are faster"]
```
```yaml
apiVersion: v1
kind: Service
metadata:
name: checkout-svc
annotations:
service.kubernetes.io/topology-mode: Auto
spec:
selector:
app: checkout-service
ports:
- port: 80
```
**Why this matters concretely, worth stating as a real, quantifiable cost lever, not just a nice-to-have:** cross-AZ data transfer is a genuinely real, often-underestimated line item in cloud bills for chatty microservice architectures — a service making thousands of requests per second to a backend, with roughly two-thirds of that traffic randomly crossing zone boundaries under default round-robin routing, pays real, ongoing cross-zone transfer fees for traffic that topology-aware routing could largely keep same-zone. **The tradeoff worth naming honestly:** this optimizes for cost/latency at a small, deliberate cost to routing evenness — if a zone has disproportionately more caller pods than backend pods, same-zone preference can mean some backend pods handle more traffic than others; the feature includes safeguards (falling back to cross-zone routing when a zone would otherwise be overwhelmed) precisely to bound this tradeoff.
---
## DNS Policies and Custom DNS Configuration
A pod's own `dnsPolicy` field, worth knowing since the CoreDNS discussion above assumes the common default without stating it explicitly.
```mermaid
graph TD
ClusterFirst["ClusterFirst (the default):
cluster-internal names
(.svc.cluster.local) resolve
via CoreDNS; anything else
FORWARDS to the node's
own upstream DNS"] --> ClusterFirstUse["Correct for the VAST
majority of pods"]
Default2["Default (a misleadingly-
named option): uses the
NODE's own DNS resolution
directly, bypassing CoreDNS
ENTIRELY - cluster-internal
Service names WON'T resolve"] --> DefaultUse2["Rare - a real footgun if
picked by mistake due to
the confusing name"]
None["None: fully custom -
YOU supply nameservers/
search domains explicitly
via dnsConfig"] --> NoneUse["Rare, specialized use
cases needing total control"]
```
**Why the `Default` policy's name is a genuinely real, worth-warning-about footgun:** despite being named "Default," it is NOT the actual default `dnsPolicy` value (`ClusterFirst` is) — `Default` means "use the node's own `/etc/resolv.conf`," which bypasses CoreDNS and cluster-internal Service DNS resolution entirely. A pod accidentally configured with `dnsPolicy: Default` (perhaps copy-pasted from an example without understanding the naming) will mysteriously fail to resolve any `.svc.cluster.local` name, while external DNS lookups work completely normally — a genuinely confusing failure mode to debug without knowing this exact naming trap.
```yaml
spec:
dnsPolicy: ClusterFirst
dnsConfig:
nameservers:
- 1.1.1.1
searches:
- custom.internal
options:
- name: ndots
value: "2"
```
**`ndots`, worth knowing as a real, concrete performance-relevant setting:** it controls how many dots a name needs before the resolver tries it as a fully-qualified external name FIRST, versus appending the cluster's search domains and trying those first. Kubernetes's default `ndots: 5` means any hostname with fewer than 5 dots (which is nearly every hostname anyone writes, including plain external domains like `api.stripe.com`) gets tried against every internal search domain FIRST before falling back to a direct external lookup — a genuinely real, measurable source of unnecessary DNS latency for pods that make heavy use of external API calls, and a real, well-known production tuning target for latency-sensitive workloads.
---
## Ingress — Getting Traffic In From Outside
A **LoadBalancer** Service works, but provisioning a full, separate cloud load balancer for every single service in a cluster gets expensive and unwieldy fast. **Ingress** solves this by providing a single entry point that can route to *many* different Services based on the request's hostname/path — directly reusing the Layer 7 load balancing concepts from the Reliability & Architecture Patterns series.
```mermaid
graph TD
Internet["Internet"] --> LB["ONE LoadBalancer /
Ingress Controller"]
LB -->|"Host: shop.example.com"| SVC1["checkout-svc"]
LB -->|"Host: api.example.com"| SVC2["api-svc"]
LB -->|"Path: /admin"| SVC3["admin-svc"]
```
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: main-ingress
spec:
rules:
- host: shop.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: checkout-svc
port:
number: 80
```
**A worthwhile, genuinely important nuance: the Ingress *object* is just a set of routing rules — it does nothing on its own without an Ingress Controller** (like NGINX Ingress Controller, or a cloud-managed one) actually running in the cluster to read those rules and configure real routing. This is exactly the same "the API object is a declaration; something else has to actually implement it" pattern as CNI and CSI throughout this whole series.
---
## Gateway API — The Modern Ingress Successor
Ingress's design has real, well-known limitations worth naming precisely, and the **Gateway API** (a separate, more expressive API, now GA and increasingly the direction the ecosystem is moving) exists specifically to address them.
```mermaid
graph TD
IngressLimit["Ingress limitations:
ONE flat annotation-based
config format, vendor-
specific extensions live
ONLY as annotations (no
standard way to express
traffic splitting, header-
based routing, etc.)"] --> GWSolve["Gateway API: a role-
oriented, STRUCTURED API
with real, standard fields
for traffic splitting,
header matching, and more
- no more vendor-specific
annotation soup"]
```
```mermaid
graph TD
GWClass["GatewayClass (cluster-admin
owned): defines WHICH
controller implements this
class - e.g. 'nginx',
'istio', 'aws-alb'"] --> GW["Gateway (platform-team
owned): a specific LISTENER
- which ports/protocols/
hostnames this gateway
accepts"]
GW --> Route["HTTPRoute (application-
team owned): the ACTUAL
routing rules - which
backend Service gets
which requests"]
```
**This three-way role split is the single biggest, most deliberate architectural difference from Ingress, worth stating explicitly:** Ingress mixes cluster-infrastructure concerns and application-routing concerns into one flat object, commonly requiring a platform team to hand-edit or approve every application team's Ingress changes. **Gateway API cleanly separates these**: a platform team owns the `GatewayClass`/`Gateway` (the shared, cluster-level infrastructure), while individual application teams independently own their own `HTTPRoute` objects, attaching to a shared `Gateway` without needing platform-team involvement for every routing change — a genuinely better fit for how real, multi-team organizations actually operate.
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: checkout-route
spec:
parentRefs:
- name: shared-gateway
hostnames:
- "shop.example.com"
rules:
- matches:
- path:
type: PathPrefix
value: /checkout
backendRefs:
- name: checkout-svc
port: 80
weight: 90
- name: checkout-svc-canary
port: 80
weight: 10
```
**Native, standard traffic-splitting (the `weight` fields above) is worth calling out specifically, since it's genuinely not possible with plain Ingress at all** — achieving a 90/10 canary split with Ingress requires vendor-specific annotations (different syntax per Ingress Controller, not portable), while Gateway API's `HTTPRoute` expresses it as a first-class, standard field, portable across any Gateway API-compliant implementation — directly connecting to the progressive-delivery patterns covered in the Automation, CI/CD & GitOps series.
**A realistic, honest assessment worth giving if asked "should we migrate today":** Ingress remains extremely widely deployed and is not going away imminently — most production clusters today still run Ingress, and it remains entirely valid for straightforward host/path routing. Gateway API is the clear direction of travel for anything needing traffic splitting, more expressive matching, or genuine platform/application team separation of concerns, but a wholesale migration is a real, deliberate project, not a drop-in replacement to reach for by default.
---
## CoreDNS — Service Discovery Inside the Cluster
**CoreDNS** runs as pods inside the cluster (itself, notably, deployed as a Deployment) and provides DNS resolution for Kubernetes objects — this is exactly what lets application code simply call `http://checkout-svc` (or the fuller `checkout-svc.default.svc.cluster.local`) instead of ever needing to know a Service's actual virtual IP.
```mermaid
graph TD
App["Application code:
connects to
'checkout-svc'"] --> DNS["CoreDNS resolves this to
the Service's stable
ClusterIP"] --> Proxy["kube-proxy's rules then
route THAT to an actual,
currently-healthy pod IP"]
```
This directly builds on the full DNS resolution journey covered in the Linux & Networking Fundamentals series (Part 2) — CoreDNS is simply a Kubernetes-specific authoritative DNS server for the cluster's internal `.svc.cluster.local` domain.
---
## A Full Worked Request Journey
Tying networking concepts together into one complete story: a user's browser reaching a specific backend pod.
```mermaid
sequenceDiagram
participant User as User's Browser
participant DNS as Public DNS
participant Ingress as Ingress Controller
participant CoreDNS as CoreDNS
participant Proxy as kube-proxy rules
participant Pod as Backend Pod
User->>DNS: Resolve shop.example.com
DNS-->>User: IP of the cloud
Load Balancer
User->>Ingress: HTTPS request
Ingress->>Ingress: Match host/path rule ->
route to checkout-svc
Ingress->>CoreDNS: (Internally) resolve
checkout-svc
CoreDNS-->>Ingress: checkout-svc's
ClusterIP
Ingress->>Proxy: Send to that ClusterIP
Proxy->>Proxy: Rules translate ClusterIP
to a REAL, currently-
healthy pod IP
Proxy->>Pod: Forwards the actual
request
Pod-->>User: Response flows back
the same path
```
---
## Volumes — The Basic Storage Building Block
By default, a container's filesystem is **ephemeral** — anything written to it disappears the moment the container restarts or the pod is deleted, since it's just the top writable layer of the container image. A **Volume** attaches durable (or at least longer-lived) storage to a pod.
```mermaid
graph TD
Types[Volume Types] --> Empty["emptyDir: exists ONLY as
long as the POD does -
useful for scratch space
shared BETWEEN containers
in the same pod"]
Types --> ConfigMap["configMap / secret:
mounts CONFIGURATION or
SECRET data as files -
NOT for general storage"]
Types --> Persistent["persistentVolumeClaim:
REAL, durable storage that
OUTLIVES the pod itself
(covered next)"]
```
---
## PersistentVolumes and PersistentVolumeClaims
This is the genuinely important storage pattern, and its two-sided design is a very commonly asked interview topic.
```mermaid
graph TD
Admin["Cluster admin (or dynamic
provisioning, below) makes
REAL storage available"] --> PV["PersistentVolume (PV):
represents an ACTUAL piece
of storage (an AWS EBS
volume, an NFS share, etc.)"]
App["Application developer
REQUESTS storage"] --> PVC["PersistentVolumeClaim (PVC):
'I need 10Gi of storage,
with THESE characteristics'"]
PVC -->|"gets BOUND to a
matching PV"| PV
```
**Simple analogy:** a PersistentVolume is like an actual apartment unit that exists in a building. A PersistentVolumeClaim is like a tenant's application/lease request ("I need a 2-bedroom unit") — the system matches ("binds") the request to an available unit meeting those requirements. **The application developer writing a pod spec never has to know or care about the underlying real storage implementation** (which cloud disk type, which specific NFS server) — they just declare a PVC, and Kubernetes handles the matching.
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: fast-ssd
```
---
## StorageClass and Dynamic Provisioning
In modern Kubernetes, you almost never manually pre-create PersistentVolumes one by one — a **StorageClass** defines a "template" for automatically creating a new, real storage volume on demand, the moment a matching PVC is created.
```mermaid
sequenceDiagram
participant Dev as Developer
participant API as API Server
participant Provisioner as CSI Provisioner
(cloud-specific)
participant Cloud as Cloud Storage API
Dev->>API: Creates a PVC requesting
20Gi, storageClassName:
fast-ssd
API->>Provisioner: Sees a PVC with no
matching PV available
Provisioner->>Cloud: Dynamically creates a
REAL 20Gi disk
(e.g. an AWS EBS volume)
Cloud-->>Provisioner: New volume created
Provisioner->>API: Creates a matching
PersistentVolume object,
BINDS it to the PVC
```
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
type: gp3
reclaimPolicy: Retain
```
**Why `reclaimPolicy` is worth knowing specifically, and it's a genuinely important, sometimes costly-to-get-wrong setting:** `Delete` (the common default) means the underlying real storage is destroyed the moment its PVC is deleted; `Retain` keeps the underlying storage around even after the PVC is gone, requiring manual cleanup. **For genuinely critical data, `Retain` is often the safer choice** — accidentally deleting a PVC with `Delete` reclaim policy means the actual data is gone, immediately, with no recovery path.
---
## StorageClass Topology Awareness
A genuinely important, easy-to-miss interaction between storage provisioning and the scheduling model from Part 2, worth understanding precisely for anyone running multi-zone clusters.
```mermaid
flowchart TD
Immediate["volumeBindingMode:
Immediate (older default):
volume provisioned
IMMEDIATELY when the PVC
is created - BEFORE the
Scheduler has even decided
which node/zone the pod
will run on"] --> ImmediateProblem["REAL problem: the volume
might get provisioned in
zone A, but the Scheduler
then picks a node in zone
B for other reasons - the
pod gets stuck, unable to
attach a volume from a
DIFFERENT zone"]
WFFC["volumeBindingMode:
WaitForFirstConsumer
(the modern, recommended
default): volume
provisioning WAITS until a
pod actually needs it,
THEN provisions in the
SAME zone the Scheduler
already chose"] --> WFFCGood["Scheduling and provisioning
are correctly SEQUENCED -
no cross-zone attachment
mismatch possible"]
```
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
type: gp3
```
**Why this is a genuinely real, commonly-hit production issue worth naming precisely, not a theoretical edge case:** most cloud block storage (AWS EBS being the canonical example) is zone-scoped — an EBS volume created in `us-east-1a` simply cannot be attached to an EC2 instance running in `us-east-1b`. With `Immediate` binding, the provisioner has no idea yet which zone the Scheduler will eventually choose, so it's genuinely possible (and does happen in real multi-AZ clusters) for a pod to get permanently stuck `Pending` because its already-provisioned volume lives in the wrong zone relative to where the Scheduler wants to place it. **`WaitForFirstConsumer` is the correct default for essentially all zone-scoped block storage**, and is exactly why it became the modern default for most cloud CSI drivers' StorageClasses.
---
## CSI — The Storage Plugin Interface
Exactly the same delegation pattern as CRI (Part 1) and CNI (earlier in this Part): Kubernetes doesn't hardcode support for every possible storage backend — it defines a standard **CSI (Container Storage Interface)**, and any storage vendor can write a CSI driver implementing it.
```mermaid
graph TD
K8s["Kubernetes Storage API
(PV, PVC, StorageClass)"] -->|"CSI - a STANDARD
interface"| Driver["CSI Driver
(AWS EBS CSI, GCP PD CSI,
Ceph CSI, Portworx, etc.)"]
Driver --> Real["Actually provisions,
attaches, and mounts the
REAL underlying storage"]
```
**Why this three-times-repeated pattern (CRI, CNI, CSI) is genuinely worth calling out as a unifying theme, not three unrelated facts to memorize separately:** Kubernetes's core design philosophy is to define stable, standard *interfaces* for pluggable concerns (how to run a container, how to network it, how to give it storage) rather than hardcoding any specific vendor's implementation — this is exactly what lets the same Kubernetes YAML manifest work essentially unchanged across AWS, GCP, on-prem, or any other CSI/CNI/CRI-compliant environment.
---
## CSI Volume Snapshots, Cloning, and Expansion
Modern CSI drivers support three genuinely important operational capabilities beyond basic provisioning — each solves a real, concrete production need.
```mermaid
graph TD
Snapshot["VolumeSnapshot: a
POINT-IN-TIME copy of a
PVC's data, using the
underlying storage's
native snapshot mechanism
(e.g. an EBS snapshot)"] --> SnapUse["Backup, or a starting
point to restore/clone
FROM later"]
Clone["PVC Cloning: creates a
NEW, independent PVC
PRE-POPULATED with an
existing PVC's data"] --> CloneUse["Spin up a fresh copy of
production data for
testing/debugging, without
touching the original"]
Expand["Volume Expansion: GROW an
EXISTING PVC's size
in-place, without
data loss or downtime
(for most CSI drivers)"] --> ExpandUse["A database outgrows its
original size allocation -
no need to provision a
NEW volume and migrate data"]
```
```yaml
# Take a snapshot of an existing PVC
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: postgres-data-snapshot
spec:
volumeSnapshotClassName: csi-aws-vsc
source:
persistentVolumeClaimName: postgres-data-postgres-0
---
# Restore a NEW PVC from that snapshot
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data-restored
spec:
dataSource:
name: postgres-data-snapshot
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
```
```bash
# Volume expansion is just editing the PVC's requested size directly —
# no separate "expand" command, ASSUMING the StorageClass allows it
kubectl patch pvc postgres-data-postgres-0 -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
```
**Volume expansion's one real prerequisite, worth stating precisely: the StorageClass must have `allowVolumeExpansion: true` set — it's not universally on by default.** Without it, attempting to grow a PVC is simply rejected. **A second, genuinely important nuance for some CSI drivers and filesystems:** the underlying block device may expand immediately, but the *filesystem* on top of it sometimes requires the pod to restart (or, for some drivers, an online filesystem resize happens automatically) before the additional space is actually usable inside the container — worth verifying the specific CSI driver's documented behavior rather than assuming either way.
**Why the snapshot-then-restore-as-a-new-PVC pattern matters concretely, directly connecting to the Disaster Recovery topic in this course:** this is exactly the mechanism a real backup/restore strategy for stateful Kubernetes workloads relies on — combined with a scheduled CronJob (Part 2) triggering periodic `VolumeSnapshot` creation, this gives genuine, native point-in-time recovery capability without needing a separate, external backup agent for the storage layer itself.
---
## Access Modes — A Genuinely Common Gotcha
A PersistentVolume's **access mode** determines how many pods (and nodes) can use it simultaneously — a real, frequently-encountered source of confusion.
```mermaid
graph TD
RWO["ReadWriteOnce (RWO):
mountable as READ-WRITE
by a SINGLE node at a time
(the MOST common mode -
most block storage, like
AWS EBS, only supports this)"] --> RWONote["Multiple PODS on the SAME
node CAN share it, but
pods on DIFFERENT nodes
cannot, simultaneously"]
ROX["ReadOnlyMany (ROX):
read-only, from MANY
nodes simultaneously"]
RWX["ReadWriteMany (RWX):
read-write, from MANY
nodes simultaneously -
requires storage that
SUPPORTS this (NFS, EFS,
etc.) - NOT most block
storage"]
```
**A genuinely common, real production mistake this explains: trying to scale a Deployment using a ReadWriteOnce PVC across multiple nodes fails, or pods get stuck `Pending`**, because the underlying storage (typically block storage like AWS EBS) simply cannot be attached read-write to more than one node at once — this is precisely why StatefulSets (each replica gets its *own*, separate PVC, not one shared PVC) are the standard pattern for multi-replica stateful workloads on RWO storage, rather than trying to share a single volume.
---
## StatefulSets and Storage, Tied Together
Closing the loop with Part 2's StatefulSet discussion: this is exactly *why* StatefulSets use a `volumeClaimTemplates` field instead of a single shared `volumes` reference.
```mermaid
graph TD
STS["StatefulSet: postgres"] --> P0["postgres-0"]
STS --> P1["postgres-1"]
STS --> P2["postgres-2"]
P0 --> PVC0["postgres-data-postgres-0
(its OWN PVC)"]
P1 --> PVC1["postgres-data-postgres-1
(its OWN PVC)"]
P2 --> PVC2["postgres-data-postgres-2
(its OWN PVC)"]
```
```yaml
apiVersion: apps/v1
kind: StatefulSet
spec:
volumeClaimTemplates:
- metadata:
name: postgres-data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
```
**Each replica automatically gets its own dedicated PVC, generated from this template, and — critically — that specific PVC stays bound to that specific replica's identity (`postgres-0` always gets `postgres-data-postgres-0`) even across restarts and rescheduling** — exactly the stable identity + stable storage guarantee that makes StatefulSets suitable for real databases, where each replica has genuinely different, non-interchangeable data.
---
## Part 3 CLI Cheat Sheet
```bash
# Networking diagnostics
kubectl get svc,endpointslices -l app=checkout-service
kubectl describe svc checkout-svc | grep -A 5 Endpoints
kubectl get networkpolicy -A
kubectl exec -it debug-pod -- nslookup checkout-svc.default.svc.cluster.local
kubectl run tmp-shell --rm -it --image=nicolaka/netshoot -- /bin/bash # ad hoc network debugging
# Ingress / Gateway API
kubectl get ingress -A
kubectl describe ingress main-ingress
kubectl get gatewayclass,gateway,httproute -A
# Storage diagnostics
kubectl get pv,pvc -A
kubectl describe pvc postgres-data-postgres-0 | grep -A 10 Events
kubectl get storageclass
kubectl get volumesnapshot -A
# Common troubleshooting one-liners
kubectl get pods -o json | jq '.items[] | select(.status.phase=="Pending") | .metadata.name'
kubectl get events --sort-by='.lastTimestamp' -A | tail -30
```
---
## Common Mistakes
| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Assuming NetworkPolicy is enforced regardless of the CNI plugin | Some CNI plugins don't implement enforcement at all — the policy silently does nothing | Verify the cluster's specific CNI plugin actually supports NetworkPolicy enforcement |
| Trying to scale a stateful workload with a single shared ReadWriteOnce PVC | RWO storage can't be attached read-write to multiple nodes simultaneously — pods get stuck Pending | Use a StatefulSet with `volumeClaimTemplates` so each replica gets its own dedicated PVC |
| Using `reclaimPolicy: Delete` on storage holding genuinely critical data | Deleting the PVC immediately destroys the underlying real data with no recovery path | Use `reclaimPolicy: Retain` for critical data, requiring a deliberate manual cleanup step |
| Assuming an Ingress object alone does anything | It's just a set of routing rules — nothing happens without an Ingress Controller actually running to implement them | Confirm an Ingress Controller is deployed and watching Ingress objects |
| Provisioning a separate LoadBalancer Service per application | Expensive and unwieldy at any real scale — each one provisions a full, separate cloud load balancer | Use a single Ingress (with an Ingress Controller) to route many services through one entry point |
| Forgetting that a Service selects pods purely by label | A pod that loses its matching label (e.g., a typo, a bad template change) silently stops receiving any traffic, with no obvious error | Double-check label selectors match exactly, especially after template/spec changes |
| Assuming a cluster is network-isolated by default | Kubernetes's actual default is fully open — every pod can reach every other pod with zero restriction until a NetworkPolicy is applied | Apply a default-deny-all NetworkPolicy per namespace as a baseline, then layer explicit allow rules on top |
| Using `volumeBindingMode: Immediate` on a StorageClass in a multi-zone cluster | The volume can be provisioned in a different zone than the Scheduler later chooses, permanently stranding the pod as Pending | Use `WaitForFirstConsumer` so provisioning waits until the Scheduler has already picked a node/zone |
| Attempting to expand a PVC without `allowVolumeExpansion: true` on its StorageClass | The resize request is simply rejected — expansion isn't universally enabled by default | Set `allowVolumeExpansion: true` explicitly on any StorageClass where growth is a realistic future need |
| Setting `dnsPolicy: Default` expecting it to mean "the standard, default behavior" | It's a misleadingly-named option that bypasses CoreDNS entirely, using the node's own DNS — cluster-internal Service names silently fail to resolve | Use `ClusterFirst` (the actual default) unless you specifically need to bypass CoreDNS |
| Ignoring cross-zone data transfer costs in a chatty, multi-AZ microservice architecture | Default kube-proxy routing is zone-blind, sending a large fraction of traffic across zone boundaries unnecessarily | Enable Topology Aware Routing so same-zone traffic is preferred when a healthy same-zone backend exists |
| Relying on `sessionAffinity: ClientIP` as a permanent architecture choice for new services | Pins traffic unevenly (especially behind shared NATs) and works against the whole point of horizontal scaling | Treat it as a legacy stopgap only; design new services to be genuinely stateless with session state in an external store |
---
## Worked Practice Problems
**Problem 1:** A pod fails its readiness probe, but a teammate insists "the pod is still running fine, why did traffic stop?" Walk through the exact mechanism that explains this.
*Answer:* A failed readiness probe doesn't stop the pod from running — the container keeps executing normally. What actually happens: the EndpointSlice controller, watching pods matching the Service's label selector, notices this specific pod is no longer marked Ready and removes its IP from the EndpointSlice backing that Service. kube-proxy, watching EndpointSlices on every node, updates its local routing rules accordingly — so traffic simply stops being routed to that pod's IP, even though the pod (and its container) is technically still alive and running. This is precisely the mechanism tying together readiness probes, EndpointSlices, and kube-proxy from across this tutorial.
**Problem 2:** A team deploys a 3-replica StatefulSet running a database, expecting each replica to have its own independent storage. Instead, they configured a single PersistentVolumeClaim referenced directly in the pod template (not `volumeClaimTemplates`), and now all 3 replicas appear to share/conflict over the same data. What's the root cause?
*Answer:* Using a single, directly-referenced PVC means all 3 replicas are attempting to mount the exact same underlying volume — which, for typical ReadWriteOnce block storage, either fails outright for pods on different nodes, or (if it happens to work, e.g. all replicas landed on the same node) results in multiple independent database processes writing to the exact same files, causing corruption or conflicts, since each replica's database engine expects to own its own separate data directory. The fix is using `volumeClaimTemplates` in the StatefulSet spec instead, which automatically generates a separate, dedicated PVC per replica (postgres-data-postgres-0, -1, -2), each bound to its own real storage volume.
**Problem 3:** A cluster admin sets a StorageClass's `reclaimPolicy` to `Delete` for a StorageClass used by a critical production database's PVCs. Someone accidentally runs `kubectl delete pvc postgres-data-postgres-0`. What happens, and how could this have been prevented?
*Answer:* With `reclaimPolicy: Delete`, deleting the PVC immediately triggers deletion of the underlying real storage volume (e.g., the actual AWS EBS disk) as well — the database's actual data is gone, immediately, with no recovery path unless a separate backup exists entirely outside Kubernetes's own storage lifecycle. This could have been prevented by setting `reclaimPolicy: Retain` on the StorageClass for anything holding genuinely critical data — with `Retain`, deleting the PVC leaves the underlying volume intact (just unbound), requiring a deliberate, separate manual step to actually destroy the real data, adding a meaningful safety buffer against exactly this kind of accidental deletion.
**Problem 4:** A newly-deployed pod remains stuck in `Pending` in a 3-AZ EKS cluster. `kubectl describe pod` shows a volume attach error referencing an availability zone that doesn't match any node the pod could run on. What's the root cause, and what StorageClass setting fixes it?
*Answer:* The StorageClass used by this pod's PVC almost certainly has `volumeBindingMode: Immediate` — the CSI provisioner created the actual EBS volume immediately upon PVC creation, before the Scheduler had decided which node (and therefore which zone) the pod would actually run on, and the volume ended up in a different zone than where the Scheduler subsequently placed the pod. Since EBS volumes are zone-scoped and can't attach across zones, the pod is permanently stuck. The fix: change the StorageClass's `volumeBindingMode` to `WaitForFirstConsumer`, which delays actual volume provisioning until a pod requiring it is being scheduled, guaranteeing the volume is created in the same zone the Scheduler has already selected.
**Problem 5:** A security audit finds that a compromised, low-privilege pod in one namespace was able to directly reach a database pod in a completely different namespace, despite the two teams believing their namespaces were isolated from each other. What's the most likely root cause, and what's the concrete fix?
*Answer:* Kubernetes's actual default networking behavior is fully open — every pod can reach every other pod cluster-wide unless a NetworkPolicy explicitly restricts it. The teams' belief that separate namespaces implied network isolation was simply incorrect; namespaces provide RBAC and resource-quota boundaries (Part 1, Part 2), but NOT network isolation on their own. The concrete fix: apply a default-deny-all NetworkPolicy (matching all pods via `podSelector: {}`, covering both `Ingress` and `Egress` in `policyTypes`) to every namespace as a baseline, then add specific, narrow allow rules (using `namespaceSelector`/`podSelector` combinations) only for the traffic flows that are actually legitimate — closing off the default-open posture entirely rather than assuming namespace boundaries provide network security they don't actually provide.
**Problem 6:** A pod's application logs show it successfully resolving `google.com` and other external domains, but every single request to `checkout-svc.default.svc.cluster.local` fails with an unresolvable-hostname error, while every other pod in the same namespace resolves that exact name fine. What field on this specific pod is almost certainly misconfigured, and why does the symptom pattern point there so precisely?
*Answer:* This is a strong signature of `dnsPolicy: Default` being set on this one pod specifically (likely copy-pasted from an example, given the misleading name) — `Default` bypasses CoreDNS entirely and resolves DNS using the node's own `/etc/resolv.conf`, which correctly resolves genuine external names (explaining why `google.com` works) but has no knowledge whatsoever of the cluster's internal `.svc.cluster.local` domain (explaining why cluster-internal Service names fail specifically and consistently). The fact that every other pod in the same namespace resolves the same name fine strongly rules out a CoreDNS or NetworkPolicy issue (which would typically affect all pods needing that name, not just one) and points specifically at a per-pod `dnsPolicy` misconfiguration. The fix: change this pod's `dnsPolicy` to `ClusterFirst` (the actual default), restoring CoreDNS-based resolution for cluster-internal names.
---
## Summary and What's Next
- Kubernetes's networking model gives **every pod its own real IP**, directly reachable from every other pod, cluster-wide, with no NAT — a deliberately simple, flat model, actually implemented by a pluggable **CNI** plugin.
- **Services** solve the "pods are disposable, their IPs keep changing" problem by providing a stable virtual IP/DNS name, load-balancing across whichever pods currently match a label selector — the four types (ClusterIP, NodePort, LoadBalancer, ExternalName) serve genuinely different purposes.
- **EndpointSlices** are the live, continuously-updated list of healthy pod IPs backing a Service — this is the exact, concrete mechanism connecting readiness probes to actual traffic routing.
- **Ingress** (with an Ingress Controller actually running) provides a single, shared entry point routing to many Services by host/path, avoiding a separate cloud load balancer per service.
- **CoreDNS** provides internal service discovery, letting application code use stable names instead of ever needing to know a Service's virtual IP directly.
- **PersistentVolumes/PersistentVolumeClaims** decouple "what storage actually exists" from "what an application asked for," matched via binding — and **StorageClass** enables automatic, on-demand (dynamic) provisioning of new real storage.
- **CSI**, like CRI and CNI, is a standard plugin interface — Kubernetes doesn't hardcode any specific storage vendor, which is exactly what makes the same manifests portable across environments.
- **Access modes** (especially the very common ReadWriteOnce limitation) directly explain why StatefulSets use `volumeClaimTemplates` to give each replica its own dedicated storage, rather than sharing one volume.
- **Headless Services** (`clusterIP: None`) resolve DNS directly to individual pod IPs instead of a shared virtual IP — exactly what gives StatefulSet pods their own resolvable, stable DNS names.
- **NetworkPolicy's real default is fully open** — network isolation is opt-in, not opt-out, and a default-deny-all policy per namespace is the recommended security baseline; `namespaceSelector`/`podSelector` combined in one list item AND, in separate items OR.
- **NetworkPolicies combine additively across a namespace** — never subtract from each other — so a single overly-permissive policy can silently undermine an otherwise-tight default-deny baseline; always review the full set selecting a given pod, not just one policy in isolation.
- **Gateway API** cleanly separates infrastructure ownership (`GatewayClass`/`Gateway`, platform team) from routing ownership (`HTTPRoute`, application teams), and natively supports traffic splitting without vendor-specific annotations — the clear direction of travel, though Ingress remains widely deployed today.
- **`volumeBindingMode: WaitForFirstConsumer`** is the correct default for zone-scoped block storage in multi-AZ clusters, avoiding a real, common "volume provisioned in the wrong zone" stuck-Pending failure mode.
- **CSI snapshots, cloning, and expansion** give native backup/restore and in-place growth capability, provided the StorageClass has `allowVolumeExpansion: true` where growth is needed.
- **Topology Aware Routing** and **`dnsPolicy`** both directly affect real, everyday production behavior — the first is a genuine cross-zone cost/latency lever most default configurations leave on the table, the second's misleadingly-named `Default` option is a real, confusing footgun worth knowing by name before it costs debugging time.
- **Session affinity** (`sessionAffinity: ClientIP`) is a legitimate stopgap for legacy, stateful applications, but works against even load distribution and horizontal scaling — new services should be designed stateless with session data in an external store instead of leaning on it.
**Continue to Part 4** (`04-service-mesh-and-advanced-topics.md`) for a deeper look at service meshes (building on the sidecar pattern from this Part), etcd operational depth, and how Custom Resources and Operators extend Kubernetes itself.