# Kubernetes Deep Dive — Part 17: Cilium & eBPF Networking Deep Dive > **Series:** Kubernetes Deep Dive (17 of 19) > **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane > **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects > **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI) > **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators > **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE > **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes > **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth > **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway > **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem > **Part 10:** `10-troubleshooting-kubernetes.md` — Troubleshooting Kubernetes, Systematically > **Part 11:** `11-kubernetes-security-deep-dive.md` — Kubernetes Security Deep Dive (CKS-Aligned) > **Part 12:** `12-autoscaling-hpa-vpa-keda.md` — Autoscaling: HPA, VPA, KEDA & Cluster Autoscaling > **Part 13:** `13-multi-tenancy-and-cluster-sharing.md` — Multi-Tenancy & Cluster Sharing at Scale > **Part 14:** `14-ai-ml-workloads-on-kubernetes.md` — Running AI/ML Workloads on Kubernetes > **Part 15:** `15-cluster-upgrades-and-lifecycle.md` — Cluster Upgrades & Lifecycle Management > **Part 16:** `16-capacity-planning.md` — Capacity Planning for Kubernetes Clusters > **Part 17:** This file — Cilium & eBPF Networking Deep Dive > **Part 18:** `18-backup-and-disaster-recovery.md` — Backup & Disaster Recovery for Workloads > **Part 19:** `19-building-custom-controllers-and-operators.md` — Building Custom Controllers and Operators > **Questions:** `questions.md` Assumes you're comfortable with the base networking model, CNI, Services, and NetworkPolicy from Part 3, the sidecar-based service mesh architecture and mTLS from Part 4, and the Gateway API resource model from Part 8 (including that chapter's brief mention of Cilium Gateway) — this chapter goes deep on Cilium's own eBPF data plane, kube-proxy replacement, identity-based policy, Hubble, and Cilium's sidecar-less service mesh, rather than re-explaining Kubernetes networking fundamentals or the Gateway API resource model itself. ## Table of Contents 1. [Why This Part Exists](#why-this-part-exists) 2. [eBPF Fundamentals — What It Actually Is and Why It Replaces iptables](#ebpf-fundamentals--what-it-actually-is-and-why-it-replaces-iptables) 3. [Where eBPF Programs Actually Run: Hooks and Program Types](#where-ebpf-programs-actually-run-hooks-and-program-types) 4. [Cilium's Architecture — Agent, Operator, and the eBPF Maps](#ciliums-architecture--agent-operator-and-the-ebpf-maps) 5. [Identity-Based Networking — Cilium's Core Architectural Idea](#identity-based-networking--ciliums-core-architectural-idea) 6. [Installing Cilium and Confirming It's Actually Running](#installing-cilium-and-confirming-its-actually-running) 7. [kube-proxy Replacement — eBPF-Based Service Load Balancing](#kube-proxy-replacement--ebpf-based-service-load-balancing) 8. [Why the eBPF Datapath Outperforms iptables at Scale](#why-the-ebpf-datapath-outperforms-iptables-at-scale) 9. [CiliumNetworkPolicy — Beyond Standard NetworkPolicy](#ciliumnetworkpolicy--beyond-standard-networkpolicy) 10. [L7-Aware Policy — HTTP-, gRPC-, and Kafka-Level Rules](#l7-aware-policy--http--grpc--and-kafka-level-rules) 11. [DNS-Based Egress Policy — Securing Calls to the Outside World](#dns-based-egress-policy--securing-calls-to-the-outside-world) 12. [Hubble — Cilium's Observability Layer](#hubble--ciliums-observability-layer) 13. [Using Hubble to Debug a Real Connectivity Problem](#using-hubble-to-debug-a-real-connectivity-problem) 14. [Cilium Service Mesh — the Sidecar-less Architecture](#cilium-service-mesh--the-sidecar-less-architecture) 15. [mTLS Without a Sidecar — How It Actually Works](#mtls-without-a-sidecar--how-it-actually-works) 16. [Cilium vs. a Sidecar Mesh — a Direct Comparison](#cilium-vs-a-sidecar-mesh--a-direct-comparison) 17. [Cilium Gateway API Support — Building on Part 8](#cilium-gateway-api-support--building-on-part-8) 18. [Cluster Mesh — Cilium Across Multiple Clusters](#cluster-mesh--cilium-across-multiple-clusters) 19. [Bandwidth Management and Quality of Service](#bandwidth-management-and-quality-of-service) 20. [Migrating an Existing Cluster From Another CNI to Cilium](#migrating-an-existing-cluster-from-another-cni-to-cilium) 21. [Troubleshooting Cilium — When the eBPF Layer Itself Is the Problem](#troubleshooting-cilium--when-the-ebpf-layer-itself-is-the-problem) 22. [Security Beyond NetworkPolicy: Tetragon and eBPF-Based Runtime Enforcement](#security-beyond-networkpolicy-tetragon-and-ebpf-based-runtime-enforcement) 23. [A Full Worked Scenario: Migrating `checkout` to Cilium With Zero-Downtime](#a-full-worked-scenario-migrating-checkout-to-cilium-with-zero-downtime) 24. [A Full Worked Scenario: Catching a Data Exfiltration Attempt With DNS Policy and Hubble](#a-full-worked-scenario-catching-a-data-exfiltration-attempt-with-dns-policy-and-hubble) 25. [A Full Worked Scenario: Replacing an Istio Sidecar Mesh With Cilium Service Mesh](#a-full-worked-scenario-replacing-an-istio-sidecar-mesh-with-cilium-service-mesh) 26. [Part 17 CLI Cheat Sheet](#part-17-cli-cheat-sheet) 27. [A Cilium Adoption Checklist](#a-cilium-adoption-checklist) 28. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps) 29. [Worked Practice Problems](#worked-practice-problems) 30. [Summary and What's Next](#summary-and-whats-next) --- ## Why This Part Exists **Part 3 taught the CNI contract and the Kubernetes networking model in general terms; this chapter goes one specific implementation deep, because Cilium has moved from "one CNI option among many" to the default or recommended CNI on every major managed Kubernetes offering** — GKE Autopilot defaults to it, it's Amazon EKS's recommended CNI for clusters wanting kube-proxy replacement and L7 policy, and it ships as AKS's built-in eBPF dataplane option. Understanding Cilium specifically, not just "a CNI" abstractly, is now a practical requirement for operating a large fraction of real production clusters, not an optional deep cut. The throughline system continues here: `checkout-service` (namespace `checkout`), `catalog-service` (namespace `catalog`), and `inventory-service` (namespace `inventory`) are the running examples for every policy, and this chapter treats them as a cluster mid-migration from a standard CNI plus `kube-proxy` to Cilium's eBPF datapath — a realistic scenario for a platform team adopting Cilium on an already-running cluster rather than starting from a green field. > [!NOTE] > This chapter is about Cilium the CNI/networking/security/mesh platform, not a general eBPF tutorial for > application developers. eBPF fundamentals are covered only as deep as needed to understand *why* Cilium's > architecture works the way it does — the goal is operating Cilium confidently, not writing your own eBPF > programs from scratch. ## eBPF Fundamentals — What It Actually Is and Why It Replaces iptables **eBPF (extended Berkeley Packet Filter) lets a sandboxed, verified program run directly inside the Linux kernel, triggered by specific kernel events — network packets arriving, a socket being created, a syscall being made — without needing a kernel module, a context switch to userspace, or a kernel recompile.** This single capability is what every Cilium feature in this chapter builds on: instead of asking the kernel to consult a long, sequentially-evaluated list of `iptables` rules for every packet (`iptables`'s actual mechanism, still the default in most non-Cilium clusters), Cilium loads compiled eBPF programs that make routing, load-balancing, and policy decisions directly in the kernel's own packet-processing path. ```mermaid flowchart TD subgraph IPT["iptables-based dataplane"] P1["Packet arrives"] --> Chain1["Traverses a sequential
chain of rules —
O(n) with rule count"] Chain1 --> Match1["First matching rule wins"] end subgraph EBPF["eBPF-based dataplane (Cilium)"] P2["Packet arrives"] --> Map["Kernel hash-map lookup —
O(1) regardless of
rule/endpoint count"] Map --> Decision["Verified eBPF program
makes the decision inline,
no context switch"] end classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c class Chain1,Match1 warn class Map,Decision ok ``` **The complexity difference is the entire performance story.** `iptables` rules for Service routing and NetworkPolicy grow roughly linearly with the number of Services and endpoints in a cluster, and every packet pays the cost of walking that chain from the top — a cluster with thousands of Services can measurably slow down every single packet, including ones with no logical relationship to most of those rules. eBPF's kernel-resident hash maps look up the exact entry a packet needs directly, in constant time, regardless of how many other Services or endpoints exist elsewhere in the cluster — this is the concrete mechanism behind every "Cilium scales better than iptables-based CNIs at scale" claim in this chapter. > [!IMPORTANT] > eBPF programs are **verified before the kernel ever runs them** — the kernel's verifier statically proves > a program terminates, never accesses unauthorized memory, and stays within a bounded instruction count, > rejecting anything that can't be proven safe. This verification step is precisely why eBPF programs can run > with kernel privilege without the same risk profile as a hand-written kernel module — a genuinely different > security and stability trust model than the older way of extending kernel behavior. ## Where eBPF Programs Actually Run: Hooks and Program Types **eBPF programs attach to specific kernel "hook points," each seeing a packet or event at a different stage of its journey — Cilium composes several of these together, and which hook a given feature uses explains a lot about what that feature can and can't see.** ```mermaid sequenceDiagram participant NIC as Network interface participant XDP as XDP hook
(earliest, pre-socket-buffer) participant TC as TC hook
(ingress/egress, per-interface) participant Sock as Socket-layer hook
(sendmsg/recvmsg, connect) participant App as Application NIC->>XDP: Raw packet arrives Note over XDP: Fastest possible point —
used for DDoS drop,
load-balancing at line rate XDP->>TC: Packet continues (if not dropped) Note over TC: Full packet metadata available —
most Cilium policy enforcement
and routing happens here TC->>Sock: Delivered toward socket Note over Sock: Sees the socket/connection
itself — used for some
service-mesh and kube-proxy
replacement optimizations Sock->>App: Data delivered ``` | Hook | Sees | Cilium uses it for | |---|---|---| | XDP (eXpress Data Path) | Raw packet, before the kernel allocates a full socket buffer for it | The earliest possible drop point — DDoS mitigation, some load-balancing fast paths | | TC (Traffic Control), ingress/egress | Full packet with kernel networking-stack metadata | The bulk of Cilium's routing, NetworkPolicy enforcement, and service load-balancing | | Socket-layer (`connect`, `sendmsg`/`recvmsg`) | The socket/connection itself, not just individual packets | Some kube-proxy-replacement and service-mesh optimizations that operate before a packet is even fully constructed | **Cilium doesn't pick one hook exclusively — it composes several deliberately**, choosing the earliest hook that still has enough information to make a given decision correctly, which is exactly what lets it make some decisions (an early DDoS drop) far cheaper than others (a decision that genuinely needs full L7 payload visibility, which requires cooperation with a userspace proxy — covered later in this chapter's L7 policy section). ## Cilium's Architecture — Agent, Operator, and the eBPF Maps **Cilium runs as a DaemonSet (`cilium-agent`, one pod per node) plus a small cluster-wide `cilium-operator` Deployment — no sidecar, no per-pod proxy process for basic CNI/policy functionality, which is the architectural root of the "sidecar-less" claim covered fully in this chapter's service-mesh section.** ```mermaid flowchart TD subgraph Node1["Node A"] Agent1["cilium-agent
(DaemonSet pod)"] --> Maps1["Kernel eBPF maps
(per-node, in-kernel)"] Agent1 --> Prog1["Loaded eBPF programs
on this node's interfaces"] end subgraph Node2["Node B"] Agent2["cilium-agent
(DaemonSet pod)"] --> Maps2["Kernel eBPF maps
(per-node, in-kernel)"] Agent2 --> Prog2["Loaded eBPF programs
on this node's interfaces"] end Operator["cilium-operator
(cluster-wide, single
leader-elected Deployment)"] --> Agent1 Operator --> Agent2 Operator --> APIServer["Kubernetes API Server —
watches Pods, Services,
NetworkPolicies, CRDs"] classDef agent fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef op fill:#f0e9fb,stroke:#6d43c0,color:#10161c class Agent1,Agent2,Maps1,Maps2,Prog1,Prog2 agent class Operator,APIServer op ``` | Component | Role | |---|---| | `cilium-agent` (DaemonSet) | Watches the API Server for local relevance, compiles and loads eBPF programs onto its node's network interfaces, and maintains that node's own kernel maps | | `cilium-operator` | Cluster-wide housekeeping that shouldn't run per-node — IP address management (IPAM) coordination, garbage-collecting stale identities, some CRD status reconciliation | | eBPF maps (in-kernel, per-node) | The actual constant-time lookup tables (endpoint identities, policy verdicts, service backends) every packet decision reads from | | `cilium-envoy` (optional, per-node) | A userspace Envoy proxy Cilium manages itself, invoked only when a policy genuinely needs L7 visibility (this chapter's L7 policy section) | **The absence of a control-plane-wide single point of decision is deliberate** — each node's `cilium-agent` makes its own local packet-forwarding and policy decisions from its own kernel maps, so an API Server or `cilium-operator` outage degrades to "no new policy/endpoint updates are propagated" rather than "packets stop being forwarded correctly," a resilience property directly comparable to how kube-proxy's own iptables rules keep working on a node even if the control plane is briefly unreachable (Part 10's control-plane failure diagnostics). ## Identity-Based Networking — Cilium's Core Architectural Idea **Standard Kubernetes NetworkPolicy and `iptables`-based CNIs enforce rules against IP addresses — Cilium's core architectural departure is enforcing policy against a numeric *security identity* derived from a pod's Kubernetes labels instead, and this single idea is what makes several of this chapter's later features (fast policy updates at scale, cluster mesh, L7 policy) actually work the way they do.** ```mermaid flowchart LR Pod["Pod: checkout-service-7d8f9c
labels: app=checkout-service,
tier=backend"] --> Identity["Cilium computes a numeric
Security Identity from
the label set"] Identity --> Cache["Identity cached cluster-wide —
every pod with the SAME
label set shares the SAME
identity, regardless of IP"] Cache --> Policy["Policy enforcement checks
identity-to-identity rules,
never IP-to-IP rules"] classDef step fill:#e5f0fa,stroke:#1d6fb8,color:#10161c class Pod,Identity,Cache,Policy step ``` **Why this matters in practice, not just architecturally**: when `checkout-service` scales from 3 replicas to 30, or gets rescheduled onto entirely different nodes with entirely different pod IPs, its security identity — derived from its stable labels, not its ephemeral IP — never changes. An `iptables`-based CNI has to update rules referencing the new IPs every time a pod is rescheduled; Cilium's policy rules reference the identity once and never need to change as pods churn underneath it, which is a direct, measurable win for policy-update latency in a cluster with high pod churn (a CI/CD-heavy namespace redeploying frequently, or an HPA scaling aggressively under load — Part 12). | Concept | IP-based enforcement (traditional) | Identity-based enforcement (Cilium) | |---|---|---| | What a policy rule references | A specific IP or CIDR range | A label-derived identity, stable across pod rescheduling | | Cost of a pod being rescheduled | Rules referencing its old IP must be updated | No change — the identity is unchanged | | Cost of scaling replicas up/down | Rules must track the changing set of IPs | No change — new replicas with the same labels get the same identity automatically | | Cross-node/cross-cluster consistency | Requires the same IP to mean the same thing everywhere (hard across clusters with independent IPAM) | Identity is a logical construct independent of any one cluster's IP space — the direct enabler of Cluster Mesh later in this chapter | > [!TIP] > **Best Practice**: Design label schemes with policy in mind from the start — since Cilium's entire > enforcement model keys off labels, a workload whose labels change frequently (a label carrying a build > hash or a timestamp, for instance) inadvertently creates a new security identity on every change, which is > both wasteful and defeats the "identity is stable across churn" benefit this section describes. Keep > policy-relevant labels (`app`, `tier`, `team`) stable, and put anything that changes per-deployment (a > version tag, a build ID) in annotations instead, which Cilium's identity computation ignores. ## Installing Cilium and Confirming It's Actually Running **Cilium installs via its own CLI or a Helm chart — verifying it's actually healthy afterward matters more than the install command itself, since a Cilium agent that fails to load its eBPF programs can leave a node with silently broken networking that looks fine at a glance.** ```bash cilium install --version 1.20.1 --set kubeProxyReplacement=true cilium status --wait ``` ```bash # The single most useful post-install command — a full connectivity # self-test across the cluster, covering pod-to-pod, pod-to-service, # and DNS in one pass cilium connectivity test ``` | Check | Command | What a failure means | |---|---|---| | Agent health per node | `cilium status` | An unhealthy agent on any node means that node's pods have degraded or broken networking | | eBPF program load | `cilium status --verbose \| grep "BPF"` | Programs failing to load usually means a kernel version too old for the requested feature set | | Full connectivity | `cilium connectivity test` | Runs real pod-to-pod, pod-to-service, and DNS traffic — catches issues a status check alone misses | > [!WARNING] > Enabling `kubeProxyReplacement=true` on an already-running cluster that still has `kube-proxy`'s own > iptables rules in place from before the migration can create two competing sets of Service-routing logic — > always remove `kube-proxy` (or confirm it was never installed, on a fresh cluster) as an explicit step in > the same change that enables Cilium's replacement, never as an afterthought. This chapter's migration > scenario near the end walks the safe sequencing for an already-running cluster. ## kube-proxy Replacement — eBPF-Based Service Load Balancing **`kube-proxy`'s traditional job — translating a Service's stable ClusterIP into one of its backing pods — is exactly the kind of per-packet lookup eBPF's kernel-resident hash maps are built for, and Cilium's replacement eliminates `kube-proxy` and its iptables rules entirely, not just supplementing them.** ```mermaid sequenceDiagram participant Client as checkout-service pod participant EBPF as eBPF program
(socket-layer hook) participant Map as Kernel Service map
(ClusterIP -> backend set) participant Backend as catalog-service backend pod Client->>EBPF: connect() to catalog-svc ClusterIP EBPF->>Map: O(1) lookup: ClusterIP -> healthy backends Map-->>EBPF: Backend pod IP selected EBPF->>Backend: Connection established directly
to the real backend IP Note over Client,Backend: No iptables DNAT chain traversed —
the socket-layer hook rewrites
the destination before the packet
is even fully constructed ``` **The socket-layer hook placement matters specifically here** — by intercepting at `connect()` time rather than per-packet, Cilium's kube-proxy replacement can perform the ClusterIP-to-backend translation once per connection instead of on every single packet in that connection's lifetime, which is a meaningfully different (and cheaper) cost model than an iptables DNAT rule re-evaluated on every packet. ```bash cilium status | grep KubeProxyReplacement # KubeProxyReplacement: True [eth0 (Direct Routing)] kubectl get pods -n kube-system -l k8s-app=kube-proxy # No resources found — confirms kube-proxy itself is genuinely absent, not just idle ``` | Aspect | kube-proxy (iptables mode) | Cilium eBPF replacement | |---|---|---| | Service-to-backend lookup cost | O(n) chain traversal, scales with total Service/endpoint count | O(1) hash map lookup, independent of cluster size | | Where translation happens | Per-packet, in the netfilter DNAT chain | Once per connection, at the socket layer | | Session affinity implementation | iptables `--nth`/probability-based rule matching | Native eBPF map tracking per-client backend assignment | | Failure mode if the daemon is down | Existing iptables rules keep working; only *updates* stop | Existing eBPF programs (already loaded into the kernel) keep working identically; only *updates* stop — the same resilience property as iptables mode | ## Why the eBPF Datapath Outperforms iptables at Scale **The performance argument isn't abstract — it's specifically about how each mechanism's cost scales as a cluster grows, and the crossover point is exactly where most organizations start to feel Kubernetes networking as a real operational cost rather than a solved problem.** ```mermaid xychart-beta title "Per-packet Service-routing latency vs. cluster size (illustrative)" x-axis ["100 Services", "1,000 Services", "5,000 Services", "10,000 Services"] y-axis "Relative latency" 0 --> 10 line "iptables (O(n) chain traversal)" [1, 3, 8, 10] line "eBPF hash maps (O(1) lookup)" [1, 1, 1, 1] ``` **Per-CPU eBPF maps are the second half of the scaling story, beyond the O(1)-vs-O(n) lookup cost alone**: Cilium structures many of its kernel maps per-CPU specifically to eliminate lock contention between cores processing packets concurrently — a design choice that lets throughput scale close to linearly with core count, rather than flattening out as more cores contend for the same global lock a naive shared-map design would need. > [!NOTE] > This performance gap is real but conditional — a small cluster with a few dozen Services will not > perceive a measurable difference, and choosing Cilium purely for this reason on a small, stable cluster is > optimizing for a problem that doesn't yet exist there. The gap becomes operationally significant at the > scale of thousands of Services and high pod churn, which is precisely the regime large managed-Kubernetes > fleets and platform teams running many tenants (Part 13) actually operate in — matching why Cilium has > become the default at that tier specifically. ## CiliumNetworkPolicy — Beyond Standard NetworkPolicy **Standard Kubernetes `NetworkPolicy` (Part 3, Part 11) only expresses L3/L4 rules — pod/namespace selectors and ports. `CiliumNetworkPolicy`, Cilium's own CRD, is a strict superset: every standard NetworkPolicy is still fully honored (Cilium enforces both formats simultaneously), and `CiliumNetworkPolicy` additionally expresses L7 rules and DNS-based egress this chapter covers in the next two sections.** ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: checkout-allow-catalog namespace: checkout spec: endpointSelector: matchLabels: app: checkout-service egress: - toEndpoints: - matchLabels: app: catalog-service k8s:io.kubernetes.pod.namespace: catalog toPorts: - ports: - port: "8080" protocol: TCP ``` **The `toEndpoints` selector above is enforced against the identity model from earlier in this chapter, not against IPs** — this rule stays correct through any number of `catalog-service` pod reschedulings or replica count changes, for exactly the reason the identity-based networking section explained. | Capability | Standard `NetworkPolicy` | `CiliumNetworkPolicy` | |---|---|---| | L3/L4 pod/namespace/port selectors | Yes | Yes (superset, same semantics honored) | | CIDR-based rules | Yes (`ipBlock`) | Yes, plus richer `toCIDRSet`/`toEntities` options | | L7 (HTTP path/method, gRPC, Kafka) rules | No | Yes — this chapter's next section | | DNS/FQDN-based egress rules | No | Yes — this chapter's DNS policy section | | Cluster-wide (non-namespaced) policies | No (NetworkPolicy is always namespaced) | Yes, via the separate `CiliumClusterwideNetworkPolicy` kind | > [!TIP] > **Best Practice**: Write baseline L3/L4 isolation as standard `NetworkPolicy` wherever it's sufficient, and > reach for `CiliumNetworkPolicy` specifically for the L7/DNS capabilities it uniquely provides — this keeps > the majority of a cluster's policy portable to a future CNI migration, and confines the CNI-specific CRD > usage to exactly the rules that genuinely need Cilium's extended capabilities. ## L7-Aware Policy — HTTP-, gRPC-, and Kafka-Level Rules **An L7 rule lets a policy say "this caller may only issue a `GET` to `/catalog/items`," not just "this caller may reach port 8080 at all" — a meaningfully stronger boundary than L3/L4 alone, and one standard NetworkPolicy has no vocabulary for expressing.** ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: catalog-l7-read-only-for-checkout namespace: catalog spec: endpointSelector: matchLabels: app: catalog-service ingress: - fromEndpoints: - matchLabels: app: checkout-service k8s:io.kubernetes.pod.namespace: checkout toPorts: - ports: - port: "8080" protocol: TCP rules: http: - method: "GET" path: "/catalog/items.*" ``` **L7 visibility requires a userspace proxy, which is the one place Cilium deliberately steps outside pure in-kernel eBPF processing** — enforcing an HTTP method/path match needs to actually parse the HTTP request, which eBPF's kernel-verifier constraints make impractical to do safely and fully in-kernel. Cilium transparently redirects only the specific traffic an L7 rule applies to through its own managed Envoy instance (`cilium-envoy`, mentioned in the architecture section) for that parsing, then hands the already-decided verdict back to the eBPF datapath — every other packet Envoy is not implicated in still takes the pure eBPF fast path. > [!IMPORTANT] > L7 policy has a real, measurable cost the pure L3/L4 eBPF fast path does not — traffic matched by an L7 > rule is redirected through Envoy for parsing, which reintroduces a userspace hop for that specific traffic. > Scope L7 rules to the specific ingress/egress pairs that genuinely need method/path-level enforcement, > rather than applying an L7 rule broadly "for extra safety" on traffic that only ever needed L3/L4 — the > unnecessary Envoy hop is a real latency cost paid on every matched request. ## DNS-Based Egress Policy — Securing Calls to the Outside World **Standard Kubernetes `NetworkPolicy` can restrict egress to a CIDR block, but most external dependencies (a third-party API, a SaaS webhook target) don't have a stable IP a CIDR rule can pin to — DNS-based egress policy lets a rule reference the hostname directly, with Cilium resolving and tracking the IPs behind it continuously.** ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: checkout-allow-stripe-egress namespace: checkout spec: endpointSelector: matchLabels: app: checkout-service egress: - toFQDNs: - matchName: "api.stripe.com" - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s-app: kube-dns toPorts: - ports: - port: "53" protocol: UDP ``` **The second rule block above — explicitly allowing egress to CoreDNS — is not optional boilerplate.** `toFQDNs` policy only works because Cilium can observe and correlate the pod's own DNS lookups; if a default-deny egress policy blocks port 53 to CoreDNS, the FQDN policy has nothing to resolve against and the "allowed" hostname's traffic fails anyway — the exact same DNS-egress trap Part 10's NetworkPolicy troubleshooting chapter covers for standard NetworkPolicy, reappearing here in FQDN-policy form. > [!WARNING] > `toFQDNs` policy is enforced based on IPs Cilium has actually observed being resolved for that hostname — > a DNS response with a very short TTL that resolves to a new IP the policy hasn't yet seen can produce a > brief window of unexpected denial immediately after a legitimate DNS change (a third-party API providential > failing over to new infrastructure, for instance). This is a real, if narrow, operational edge case worth > knowing before treating `toFQDNs` policy as instantaneously perfect at every DNS change. ## Hubble — Cilium's Observability Layer **Hubble observes every flow eBPF already sees passing through Cilium's datapath and turns it into structured, Kubernetes-aware flow events — pod names, namespaces, labels, Service names, and the actual policy verdict — without packet sampling, a separate packet-capture pipeline, or any sidecar.** ```mermaid flowchart TD subgraph Node["Every node"] Agent["cilium-agent"] --> HubbleAgent["Hubble server
(embedded in the agent —
no separate pod)"] end HubbleAgent --> Relay["Hubble Relay
(aggregates every node's
flow stream into one API)"] Relay --> CLI["hubble CLI"] Relay --> UI["Hubble UI
(web-based flow visualization)"] classDef agent fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef relay fill:#f0e9fb,stroke:#6d43c0,color:#10161c class Agent,HubbleAgent agent class Relay,CLI,UI relay ``` ```bash hubble observe --namespace checkout --to-namespace catalog -f # TIMESTAMP SOURCE DESTINATION VERDICT SUMMARY # 10:42:03 checkout/checkout-svc-x catalog/catalog-svc-y FORWARDED TCP Flags: SYN # 10:42:03 checkout/checkout-svc-x catalog/catalog-svc-y FORWARDED HTTP/1.1 GET /catalog/items -> 200 ``` **Because the Hubble agent is embedded directly in `cilium-agent` — no additional pod, no sidecar injection — it adds essentially zero deployment overhead beyond Cilium itself, which is a real architectural difference from bolting a separate observability agent onto an existing CNI.** This is the concrete mechanism behind "network observability built into Cilium" rather than "a network observability tool that also happens to run alongside Cilium." | Hubble capability | What it shows | |---|---| | L3/L4 flow visibility | Every connection, its verdict (forwarded/dropped), and which policy (if any) made the decision | | L7 flow visibility (for traffic matched by an L7 policy) | The actual HTTP method/path, gRPC method, or Kafka topic — not just the fact that a connection happened | | Policy verdict attribution | Exactly which `CiliumNetworkPolicy`/`NetworkPolicy` allowed or denied a specific flow — a direct answer to the Part 10 "is this NetworkPolicy actually the cause" question | | Cluster-wide or per-namespace scoping | Hubble Relay aggregates across every node, so a single `hubble observe` command can span the whole cluster | ## Using Hubble to Debug a Real Connectivity Problem **Part 10's NetworkPolicy troubleshooting chapter used a pod-IP-then-ClusterIP-then-DNS diagnostic ladder built from generic `kubectl`/`curl` commands — on a Cilium cluster, Hubble replaces most of that ladder with one direct query that names the actual policy verdict, instead of inferring it indirectly.** ```bash hubble observe --pod checkout/checkout-service-7d8f9c --to-pod catalog/catalog-service-x2k9p -f # TIMESTAMP SOURCE DESTINATION VERDICT SUMMARY # 10:51:02 ... ... DROPPED Policy denied by # (CiliumNetworkPolicy) catalog-l7-read-only-for-checkout ``` **Compare this to the Part 10 approach**, which required curling the pod IP directly, then the ClusterIP, then checking `kubectl get endpoints`, inferring the failing layer from which step succeeded — Hubble's output above names the exact policy object responsible in a single command, collapsing several manual diagnostic steps into one structured, already-correlated answer. > [!TIP] > **Best Practice**: Reach for `hubble observe` as the *first* troubleshooting step on any Cilium cluster > for a connectivity problem, before falling back to Part 10's generic `kubectl`-based diagnostic ladder — > Hubble's policy-verdict attribution answers the "is a NetworkPolicy the cause, and if so which one" question > directly, which is usually the most time-consuming part of the generic ladder to work out manually. ## Cilium Service Mesh — the Sidecar-less Architecture **Part 4 covered the sidecar pattern in depth — a proxy container injected into every mesh-participating pod — as the traditional service mesh architecture. Cilium Service Mesh implements the same core mesh capabilities (mTLS, L7 traffic management, observability) without injecting anything into application pods at all, by pushing that functionality into the eBPF datapath every pod already traverses.** ```mermaid flowchart TD subgraph Sidecar["Sidecar mesh (Part 4's Istio model)"] AppA["App container"] <--> ProxyA["Injected sidecar
proxy container"] ProxyA -->|"proxy-to-proxy,
mTLS handled
in userspace"| ProxyB["Injected sidecar
proxy container"] ProxyB <--> AppB["App container"] end subgraph Sidecarless["Cilium Service Mesh"] AppC["App container
(unmodified — no
injected proxy)"] -->|"traffic intercepted
at the eBPF/kernel
layer, transparently"| KernelC["eBPF datapath
on the node"] KernelC -->|"mTLS, routing,
policy applied
in-kernel or via
per-node Envoy"| KernelD["eBPF datapath
on the destination node"] KernelD --> AppD["App container
(unmodified)"] end classDef sidecar fill:#fbeee0,stroke:#b8650f,color:#10161c classDef ebpf fill:#e5f5ea,stroke:#1f8a4c,color:#10161c class AppA,ProxyA,ProxyB,AppB sidecar class AppC,KernelC,KernelD,AppD ebpf ``` **The resource-overhead argument is the most concrete, easiest-to-measure difference**: a sidecar mesh adds one proxy container's CPU/memory footprint (Part 4's resource-cost coverage) *per pod*, which scales linearly with replica count — a Deployment with 50 replicas carries 50 sidecar processes. Cilium's mesh functionality runs once per *node* in the shared `cilium-agent`/`cilium-envoy` processes, regardless of how many pods that node hosts, which is a fundamentally different (and, at high pod density, much cheaper) resource-scaling curve. | Aspect | Sidecar mesh (Part 4) | Cilium Service Mesh | |---|---|---| | Where mesh logic runs | One injected proxy container per pod | Shared per-node `cilium-agent`/eBPF datapath, plus per-node Envoy for L7 | | Resource cost scaling | Linear with pod/replica count | Roughly flat per node, regardless of pod density | | Pod restart required to join the mesh | Yes — sidecar injection happens at pod creation | No — Cilium already intercepts all pod traffic at the node level once installed | | L7 traffic management maturity | Very mature (Istio's traffic-splitting, fault injection — Part 4) | Present, though the ecosystem and feature breadth is younger than Istio's | ## mTLS Without a Sidecar — How It Actually Works **Part 4 explained sidecar mTLS as two Envoy proxies negotiating a TLS handshake on the application's behalf — Cilium achieves the same outcome (encrypted, mutually-authenticated pod-to-pod traffic) by having eBPF programs intercept traffic and hand it to per-node processes (WireGuard or IPsec, depending on configuration) that perform the encryption, again without any per-pod proxy.** ```bash cilium config view | grep -i encryption # encryption: wireguard cilium status | grep Encryption # Encryption: Wireguard [NodeEncryption: Disabled] ``` | Encryption mode | Mechanism | Trade-off | |---|---|---| | WireGuard | Kernel-native, per-node-pair encrypted tunnels; Cilium configures WireGuard automatically | Generally lower CPU overhead than IPsec; requires a Linux kernel with WireGuard support | | IPsec | Traditional kernel IPsec, Cilium manages key rotation and tunnel setup | Broader kernel/platform compatibility; historically higher CPU cost per encrypted packet than WireGuard | | mTLS via SPIFFE/SPIRE integration | Cilium can integrate with a separate identity provider for application-layer mTLS semantics closer to a traditional service mesh's certificate model | Adds an external dependency (SPIRE) but provides workload identity semantics some compliance frameworks specifically require | **The practical takeaway for a team migrating off a sidecar mesh (this chapter's closing worked scenario)**: Cilium's node-to-node encryption modes (WireGuard/IPsec) protect traffic *between nodes* transparently and without application changes, but map onto a different trust boundary than per-pod mTLS certificates — verify which specific compliance or security requirement drove the original sidecar-mesh mTLS adoption before assuming Cilium's encryption modes satisfy it as a drop-in equivalent. ## Cilium vs. a Sidecar Mesh — a Direct Comparison **Bringing Part 4's sidecar-mesh coverage and this chapter's sidecar-less coverage together into one decision framework, since "which mesh architecture" is a real, consequential choice a platform team makes once and lives with for years.** | Choose... | When | |---|---| | A sidecar mesh (Istio, Part 4) | The team needs the most mature, broadest L7 traffic-management feature set (fine-grained canary rollouts, fault injection, mesh-wide observability tooling with the longest production track record) | | Cilium Service Mesh | Pod density is high enough that per-pod sidecar resource overhead is a real, measured cost; the team already runs Cilium as its CNI and wants to avoid operating two separate networking/security stacks; L7 needs are more modest than Istio's full feature set covers | | Cilium as CNI + a sidecar mesh on top | A team wants Cilium's eBPF performance/policy benefits at the CNI layer specifically, while keeping Istio's mature L7 traffic management — Istio's ambient mode (Part 8) with Cilium as the underlying CNI is a real, supported combination for teams not ready to give up sidecar-mesh L7 maturity entirely | > [!NOTE] > These options aren't mutually exclusive in the way this table might suggest at a glance — Part 8 already > covered Istio's own ambient mode (sidecar-less, using ztunnel) as a middle ground, and Cilium can serve as > the CNI underneath either a traditional sidecar Istio deployment or Istio ambient mode. "Cilium vs. Istio" > is really "Cilium vs. Istio" only when comparing Cilium Service Mesh specifically against Istio's own mesh > layer — the two projects are not mutually exclusive at the CNI layer. ## Cilium Gateway API Support — Building on Part 8 **Part 8 introduced Cilium Gateway as one of several Gateway API implementations, briefly — this section goes one layer deeper into what makes it specifically an eBPF-native implementation rather than "yet another Envoy-based Gateway controller with a different name."** ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: checkout-gateway namespace: checkout spec: gatewayClassName: cilium listeners: - name: https protocol: HTTPS port: 443 ``` **Cilium's Gateway API implementation reuses the same eBPF datapath and per-node Envoy instances this chapter has already covered for L7 policy and service mesh — a `Gateway`/`HTTPRoute` on a Cilium-managed cluster is implemented by the same `cilium-envoy` processes handling L7 CiliumNetworkPolicy rules, not a separate, independently-deployed ingress controller.** This is the practical advantage of adopting Cilium Gateway specifically on an already-Cilium cluster: one less independent networking component to operate, patch, and reason about compared to running a different vendor's Gateway API implementation (Envoy Gateway, NGINX Gateway Fabric — both covered in Part 8) alongside Cilium as the CNI. | Consideration | Cilium Gateway | A separate Gateway API implementation (Part 8) | |---|---|---| | Operational surface | Reuses Cilium's own agent/Envoy — one less component | An independent controller and data plane to operate, upgrade, and monitor separately | | Feature parity with Envoy Gateway/Istio Gateway | Solid core Gateway API conformance; narrower advanced-feature set than dedicated Gateway-focused projects as of this writing | Broader advanced-routing feature depth, since Gateway API support is those projects' primary focus | | Best fit | A cluster already standardized on Cilium wanting to minimize the number of distinct networking components | A cluster prioritizing the deepest Gateway API feature set regardless of CNI choice | ## Cluster Mesh — Cilium Across Multiple Clusters **Cluster Mesh extends Cilium's identity-based networking model (this chapter, earlier) across multiple independent Kubernetes clusters, letting a pod in one cluster reach a Service in another using the exact same identity-based policy model as within a single cluster — directly building on why identity, not IP, is Cilium's foundational idea.** ```mermaid flowchart LR subgraph ClusterA["Cluster A (us-east)"] CheckoutA["checkout-service"] end subgraph ClusterB["Cluster B (us-west)"] CatalogB["catalog-service"] end CheckoutA <-->|"Cluster Mesh: identity-aware
cross-cluster connectivity,
same CiliumNetworkPolicy
semantics apply"| CatalogB classDef cluster fill:#e9eafb,stroke:#4550c4,color:#10161c class CheckoutA,CatalogB cluster ``` **Because identity is a logical construct derived from labels rather than tied to any one cluster's IP address space, a `CiliumNetworkPolicy` written once continues to apply correctly whether the traffic it governs stays within one cluster or crosses into another** — this is precisely the property Part 13's separate-physical-clusters isolation tier and its fleet-management escape hatch would benefit from if the tenants involved needed *connectivity*, not just consistent configuration, across clusters. Cluster Mesh and Part 13's fleet-management tooling (Cluster API, Rancher Fleet) solve adjacent but distinct problems: fleet tools keep many clusters configured consistently; Cluster Mesh lets them actually talk to each other securely. > [!CAUTION] > Cluster Mesh requires every participating cluster's pod and Service CIDR ranges to be non-overlapping — > retrofitting this onto clusters that were provisioned independently, without cross-cluster IP planning from > the start, can require re-provisioning a cluster's pod CIDR entirely, which is a disruptive, not incremental, > change. Plan non-overlapping CIDRs from the start for any cluster that might eventually join a Cluster Mesh, > even if cross-cluster connectivity isn't needed on day one. ## Bandwidth Management and Quality of Service **Beyond policy and observability, Cilium's eBPF datapath also enforces bandwidth limits directly at the kernel level — a capability that has no equivalent in standard Kubernetes NetworkPolicy at all, since NetworkPolicy only ever expresses allow/deny, never rate.** ```yaml apiVersion: v1 kind: Pod metadata: name: recommendation-model-batch namespace: recommendations annotations: kubernetes.io/egress-bandwidth: "50M" # Cilium-enforced egress cap ``` **This is directly relevant to the noisy-neighbor concerns Part 13 raised about shared node-level resources that CPU/memory limits alone can't hard-partition** — Part 13 explicitly named network throughput as a gap plain ResourceQuota/limits leave open; Cilium's bandwidth-manager feature is one concrete way to close that specific gap, capping a tenant's egress bandwidth the same way a CPU limit caps its compute consumption. > [!TIP] > **Best Practice**: Apply egress bandwidth limits specifically to batch/bulk-transfer workloads (a nightly > data export, a large model-weight download for a `recommendations` team pod — Part 14) that could otherwise > saturate a shared node's network interface and degrade latency-sensitive request-serving pods co-located on > the same node, rather than applying bandwidth limits uniformly across every workload. ## Migrating an Existing Cluster From Another CNI to Cilium **Migrating a live, already-running cluster's CNI is one of the riskiest single operations covered in this whole series, precisely because networking is the one layer nearly everything else depends on — this section sequences it deliberately rather than treating it as a simple `helm upgrade`.** ```mermaid flowchart TD Start(["Existing cluster,
non-Cilium CNI"]) --> Parallel["Install Cilium in
chaining/hybrid mode
alongside the existing CNI"] Parallel --> Verify1["Confirm Cilium's own
connectivity test passes
while old CNI still active"] Verify1 --> Drain["Cordon and drain nodes
one at a time (Part 15's
rolling node pattern)"] Drain --> Reboot["Reboot/replace each drained
node so it re-joins with
Cilium as the sole CNI"] Reboot --> Verify2["Re-run connectivity test
after each node — do not
batch multiple nodes
before verifying"] Verify2 --> Remaining{"More nodes
remaining?"} Remaining -->|"Yes"| Drain Remaining -->|"No"| RemoveOld["Remove old CNI's
components and kube-proxy
(if replacing it too)"] RemoveOld --> Done(["Cluster fully on Cilium"]) classDef risky fill:#fbe8e6,stroke:#b3261e,color:#10161c classDef safe fill:#e5f5ea,stroke:#1f8a4c,color:#10161c class Drain,Reboot risky class Verify1,Verify2,RemoveOld safe ``` **Node-by-node, verify-after-each-step is the discipline that makes this safe, and it's the same discipline Part 15 already established for cluster upgrades generally** — a CNI migration is, in effect, a rolling node-replacement operation with an unusually high blast radius per mistake, so the same "one node, verify, next node" cadence that Part 15 uses for version upgrades applies here with even less tolerance for skipping the verification step. > [!CAUTION] > Never migrate every node's CNI simultaneously, even on a cluster small enough that it seems fast to do all > at once. A subtle Cilium misconfiguration (an incorrect IPAM mode, a missing kernel feature on some but not > all nodes) that only manifests under real traffic can take down the entire cluster's networking at once if > applied everywhere simultaneously, versus being caught and rolled back after a single node during a > node-by-node rollout. ## Troubleshooting Cilium — When the eBPF Layer Itself Is the Problem **Part 10's general troubleshooting chapter covers the standard pod/Service/DNS diagnostic ladder — Cilium adds one more layer beneath all of that worth checking specifically: is the eBPF datapath itself healthy on this node.** ```bash cilium status # per-node agent health summary cilium-dbg bpf endpoint list # every endpoint (pod) this node's agent knows about cilium-dbg monitor --type drop # live stream of every packet eBPF is actively dropping, and why ``` | Symptom | Likely cause | Where to look | |---|---|---| | A specific pod has no network connectivity at all, others on the same node are fine | The pod's own eBPF endpoint program failed to attach — often a transient issue during pod creation | `cilium-dbg bpf endpoint list` — confirm the pod's endpoint is present and in `ready` state | | Cluster-wide connectivity degrades right after a Cilium upgrade | An eBPF program failed to load on some nodes due to a kernel-version/feature mismatch not caught pre-upgrade | `cilium status --verbose` per node, checking BPF program load status | | Traffic between two specific pods is silently dropped with no obvious policy match | An unexpected `CiliumNetworkPolicy` (or the combination of several) is denying it — Hubble's verdict attribution is the fastest path | `hubble observe`, per this chapter's earlier debugging section, before manually re-reading every policy | | `cilium-dbg monitor --type drop` shows a high rate of drops with no corresponding policy | Possible resource exhaustion in a kernel eBPF map (a map size limit reached) rather than an intentional policy decision | Check Cilium's own metrics for map utilization; consider raising map size limits documented for the specific map involved | > [!WARNING] > A Cilium upgrade that changes eBPF program behavior in a way a specific node's older kernel doesn't > support can fail silently rather than loudly — the agent pod itself might report `Running` and even > `cilium status` might look superficially healthy while a specific feature quietly falls back to a degraded > mode. Always run `cilium connectivity test` after any Cilium version upgrade, not just a status check, > exactly for the same "verify what you actually shipped, not what you meant to ship" reason this series > applies everywhere else. ## Security Beyond NetworkPolicy: Tetragon and eBPF-Based Runtime Enforcement **Part 11 covered Falco for runtime security — Tetragon is Cilium's own sibling project applying the same eBPF foundation to runtime security observability and enforcement, worth naming here specifically because of how closely its architecture parallels everything else in this chapter.** ```mermaid flowchart TD Syscall["A process inside a
container makes a syscall
(e.g. execve, a file open)"] --> TetragonHook["Tetragon's eBPF programs,
attached to kernel
tracepoints/kprobes"] TetragonHook --> Observe["Structured event emitted:
process, container, Pod,
and the exact syscall/args"] TetragonHook --> Enforce{"A configured policy
matches this event?"} Enforce -->|"Yes, block configured"| Kill["Enforcement happens
in-kernel — the syscall
itself can be blocked,
not just reported after the fact"] Enforce -->|"No match"| Allow["Syscall proceeds normally"] classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c class Observe,Allow ok class Kill crit ``` | Aspect | Falco (Part 11) | Tetragon | |---|---|---| | Detection mechanism | eBPF or a kernel module, generating alerts from a rule engine | eBPF only, generating structured events with full Kubernetes context | | Enforcement (not just detection) | Primarily an alerting/detection tool | Can actively block a matched syscall in-kernel, not just alert after the fact | | Project lineage | CNCF Graduated, originated at Sysdig | A Cilium-family project (same eBPF foundation and Kubernetes-identity awareness as Cilium's networking side) | | Best fit | Broad, mature runtime detection rule library across many attack patterns | A team already standardized on Cilium wanting runtime enforcement (not just detection) sharing the same eBPF/identity model | **Naming this here rather than only in Part 11 matters because of the shared architectural DNA**: Tetragon's Kubernetes-identity-aware event correlation works the same way Hubble's does — full pod/namespace/label context attached to every kernel-level event, for the same underlying reason this chapter's identity section explained for networking. A platform team already running Cilium and Hubble gets a head start understanding Tetragon's own event model, since it's the same design applied to process/syscall events instead of network flows. ## A Full Worked Scenario: Migrating `checkout` to Cilium With Zero-Downtime **Bringing several sections together for a realistic migration**: the platform team is moving the whole cluster (including `checkout`, `catalog`, and `inventory`) from a standard CNI plus `kube-proxy` onto Cilium with kube-proxy replacement enabled. 1. **Pre-migration validation** (this chapter's installation section): install Cilium in chaining mode alongside the existing CNI on a non-production cluster first, run `cilium connectivity test`, and confirm every existing `NetworkPolicy` in the `checkout` namespace still enforces correctly under Cilium before touching production. 2. **Sequence the node-by-node migration** (this chapter's migration section) starting with a canary AZ, cordoning and draining nodes one at a time per Part 15's rolling pattern, running `cilium connectivity test` after each node before proceeding to the next. 3. **Confirm identity-based policy correctness** using Hubble mid-migration: `hubble observe --namespace checkout` during the rollout, watching for any unexpected `DROPPED` verdicts that would indicate an existing `NetworkPolicy` didn't translate as expected onto Cilium's enforcement. 4. **Enable kube-proxy replacement** only after every node is confirmed running Cilium as the sole CNI — removing `kube-proxy` before every node has Cilium's replacement active would leave nodes mid-migration with neither mechanism handling Service routing. 5. **Verify Service routing specifically**, since this is the highest-risk single change in the whole migration: `cilium status | grep KubeProxyReplacement` on every node, plus a full `cilium connectivity test` pass, before considering the migration complete. 6. **Leave Hubble running as standing observability** afterward, not just as a migration-verification tool — the same flow visibility that confirmed the migration's correctness becomes the team's ongoing Part 10-style troubleshooting entry point going forward. **The single riskiest step is step 4** — a partial kube-proxy removal, done before every node has confirmed Cilium replacement active, is exactly the kind of mistake that produces a cluster-wide Service-routing outage with no single obvious cause, which is precisely why this sequence gates it behind full node-by-node confirmation rather than a single cluster-wide toggle. ## A Full Worked Scenario: Catching a Data Exfiltration Attempt With DNS Policy and Hubble **A security-flavored scenario tying together DNS-based egress policy, Hubble, and Part 11's supply-chain security coverage**: a compromised dependency in `inventory-service`'s container image attempts to exfiltrate data to an unrecognized external domain. Because `inventory-service` runs under a `toFQDNs` egress policy (this chapter's DNS-policy section) explicitly allowlisting only its known legitimate external dependencies, the exfiltration attempt's DNS lookup for an unrecognized domain never resolves to an allowed destination, and the subsequent connection attempt is dropped at the eBPF layer before a single byte leaves the pod. ```bash hubble observe --pod inventory/inventory-service-x7k2p --verdict DROPPED -f # TIMESTAMP SOURCE DESTINATION VERDICT SUMMARY # 14:12:08 inventory/inventory-svc-x7k2p 93.184.xx.xx:443 DROPPED Policy denied: no matching toFQDNs rule ``` **Explaining why this works, two levels deep**: the symptom the security team actually notices is a Hubble alert (wired to their SIEM via Hubble's exportable flow format) for a `DROPPED` egress connection from a production pod to an unrecognized external IP. The immediate cause is the compromised dependency's outbound connection attempt failing a `toFQDNs` policy match. The underlying condition that made this catchable at all is that the platform team had already applied default-deny egress with an explicit allowlist to every production namespace as a standing security baseline (not something newly added in response to this incident) — without that pre-existing baseline, the exfiltration attempt would have succeeded silently, since a default-allow egress posture has no policy boundary for Hubble to generate a `DROPPED` verdict against in the first place. > [!IMPORTANT] > **Best Practice**: Default-deny egress with an explicit `toFQDNs`/`toEndpoints` allowlist is what turns > Hubble from "visibility into what already happened" into "a real preventive control" — Hubble alone, on a > default-allow cluster, would have observed the exfiltration attempt succeed and merely logged it after the > fact. The security value in this scenario comes specifically from the policy denying the connection, with > Hubble providing the forensic detail about exactly what was attempted and when. ## A Full Worked Scenario: Replacing an Istio Sidecar Mesh With Cilium Service Mesh **A capstone scenario combining this chapter's service-mesh sections with Part 4's sidecar-mesh coverage**: the platform team running Istio's sidecar mesh (Part 4) across `checkout`, `catalog`, and `inventory` decides to migrate to Cilium Service Mesh, driven by rising per-pod sidecar resource overhead as the `catalog` namespace's replica count grew past 200 pods during peak traffic. 1. **Quantify the actual overhead first** (per this chapter's resource-cost comparison): 200 sidecar proxies each consuming a modest but nonzero CPU/memory footprint sums to a real, measurable chunk of the `catalog` namespace's total cluster footprint — the team confirms this via Kubecost/OpenCost (Part 16) before committing to a migration, rather than migrating on a hunch. 2. **Confirm the required L7 feature set is covered**: the team's actual mesh usage is mTLS everywhere plus basic traffic-splitting for canary rollouts — neither requires Istio's more advanced fault-injection or mesh-federation features, so Cilium Service Mesh's narrower L7 feature set (this chapter's comparison table) is confirmed sufficient before migrating, rather than discovered insufficient afterward. 3. **Migrate the CNI first, mesh second**: since the cluster wasn't already running Cilium as its CNI, the team follows this chapter's CNI-migration sequence completely before layering Cilium Service Mesh on top — attempting both changes simultaneously would make it far harder to isolate which layer caused any problem that surfaced during rollout. 4. **Remove sidecar injection namespace-by-namespace**, verifying mTLS and traffic-splitting behavior with Hubble after each namespace, rather than disabling Istio sidecar injection cluster-wide in one step. 5. **Measure the actual outcome**: post-migration, the `catalog` namespace's total resource footprint drops measurably (the exact sidecar-elimination savings this chapter's resource-scaling comparison predicted), confirmed again via Kubecost/OpenCost rather than assumed from the architecture diagram alone. **The generalizable lesson**: a mesh-architecture migration this consequential should be justified by a measured, specific cost (200 sidecars' real resource footprint, confirmed via cost tooling) and a confirmed feature-sufficiency check, not by "sidecar-less is architecturally newer" alone — the same evidence-based discipline this whole series applies to every other consequential infrastructure decision. ## Part 17 CLI Cheat Sheet | Command | Purpose | |---|---| | `cilium install` / `cilium status --wait` | Install Cilium and wait for every node's agent to report healthy | | `cilium connectivity test` | Full pod-to-pod/Service/DNS connectivity self-test — the primary post-install and post-upgrade verification | | `cilium status \| grep KubeProxyReplacement` | Confirm kube-proxy replacement is actually active, not just configured | | `hubble observe --namespace -f` | Live-tail every flow in or out of a namespace, with policy verdicts | | `hubble observe --pod / --verdict DROPPED` | Show only denied traffic for one pod — the fastest path to "which policy is blocking this" | | `cilium-dbg bpf endpoint list` | List every pod (endpoint) a node's Cilium agent currently manages | | `cilium-dbg monitor --type drop` | Live stream of every packet the eBPF datapath is actively dropping, with the reason | | `kubectl get ciliumnetworkpolicies -A` | Audit every `CiliumNetworkPolicy` cluster-wide | | `cilium clustermesh status` | Confirm Cluster Mesh connectivity/health across participating clusters | ## A Cilium Adoption Checklist - [ ] Confirmed node kernel versions meet Cilium's minimum requirements for the specific features being enabled (kube-proxy replacement, WireGuard encryption, L7 policy) - [ ] Ran `cilium connectivity test` clean before considering any install/upgrade/migration complete - [ ] Existing `NetworkPolicy` objects confirmed to still enforce correctly under Cilium (both formats are honored simultaneously, but verify rather than assume) - [ ] `kube-proxy` removal sequenced only after every node confirms Cilium's replacement is active — never removed cluster-wide before every node is migrated - [ ] Default-deny egress with explicit `toFQDNs`/`toEndpoints` allowlists applied to production namespaces handling sensitive data, per this chapter's exfiltration-prevention scenario - [ ] Hubble Relay and UI deployed and confirmed reachable — not just the per-node agent-embedded Hubble server - [ ] L7 policy usage scoped deliberately, not applied broadly "for safety," given its real per-request Envoy-hop cost - [ ] If migrating from a sidecar mesh, the resource-overhead and feature-sufficiency case documented and confirmed via real cost data (Part 16), not assumed from architecture alone - [ ] Cluster Mesh CIDR non-overlap confirmed in advance for any cluster that might eventually need cross-cluster connectivity ## Common Mistakes and Interview Traps | Mistake | Why it's wrong | Correct approach | |---|---|---| | Enabling `kubeProxyReplacement` while `kube-proxy` is still running | Creates two competing Service-routing mechanisms simultaneously | Remove `kube-proxy` in the same coordinated change, sequenced per-node, never as an afterthought | | Treating `CiliumNetworkPolicy` as a full replacement requiring standard `NetworkPolicy` to be deleted | Cilium enforces both formats simultaneously — standard `NetworkPolicy` remains fully valid and portable | Keep L3/L4 rules as standard `NetworkPolicy` where sufficient; use `CiliumNetworkPolicy` specifically for L7/DNS capabilities | | Applying L7 policy broadly "to be thorough" | Every L7-matched flow pays a real Envoy userspace-hop cost the pure eBPF fast path avoids | Scope L7 rules to the specific traffic that genuinely needs method/path-level enforcement | | Assuming Cilium's WireGuard/IPsec encryption is a drop-in replacement for a sidecar mesh's per-pod mTLS certificates | The two operate at different trust boundaries — node-to-node vs. per-workload identity — and may not satisfy the same compliance requirement | Verify the specific requirement that drove the original mTLS adoption before assuming equivalence | | Migrating every node's CNI simultaneously on a live cluster | A subtle misconfiguration affecting every node at once has cluster-wide blast radius | Migrate node-by-node with a connectivity-test gate after each one, per Part 15's rolling-change discipline | | Forgetting to explicitly allow DNS egress alongside a `toFQDNs` rule | `toFQDNs` policy depends on observing the pod's own DNS resolution — blocking DNS breaks the FQDN policy it's meant to support | Always pair a `toFQDNs` egress rule with an explicit allow for DNS (UDP/TCP 53) to CoreDNS | | Choosing a mesh architecture based on "sidecar-less is newer" alone | Architecture age isn't a substitute for confirming the required L7 feature set and measuring actual resource savings | Justify a mesh migration with measured cost data and a feature-sufficiency check, per this chapter's capstone scenario | ## Worked Practice Problems **Problem 1**: A cluster runs both `kube-proxy` and Cilium with `kubeProxyReplacement=true` enabled at the same time, and Services intermittently route to the wrong or stale backend. What's the root cause, and how should the migration have been sequenced to avoid it? *Answer*: Two independent Service-routing mechanisms — `kube-proxy`'s iptables rules and Cilium's eBPF-based replacement — are both actively making routing decisions simultaneously, and neither is aware of the other, producing inconsistent or stale routing depending on which mechanism's rules a given packet happens to hit first. The migration should have removed `kube-proxy` in the same coordinated, node-by-node sequenced change that enabled Cilium's replacement, confirming via `cilium status | grep KubeProxyReplacement` and `kubectl get pods -n kube-system -l k8s-app=kube-proxy` (expecting no results) on each node before moving to the next, never running both simultaneously as an ongoing state. **Problem 2**: A team applies an L7 `CiliumNetworkPolicy` restricting `checkout-service` to only `GET` requests against `catalog-service`, and afterward notices a measurable latency increase on every request between the two services, even ones that were already compliant with the new rule. Why, and is this avoidable? *Answer*: Any traffic matched by an L7 rule is transparently redirected through Cilium's per-node Envoy proxy for HTTP parsing, reintroducing a userspace hop that the pure eBPF fast path (used for L3/L4-only rules) doesn't pay — this cost applies to every request matched by the L7 rule, compliant or not, since the proxy has to parse the request to confirm compliance in the first place. It's not fully avoidable if genuine method/path-level enforcement is required, but it is avoidable in cases where L3/L4 enforcement (just allowing the port) would have been sufficient — the fix is confirming the L7 rule's specificity is actually needed before applying it, not applying it as a default level of caution. **Problem 3**: A `toFQDNs` policy allowlisting `api.stripe.com` starts blocking legitimate traffic for about 90 seconds immediately after Stripe fails over to new infrastructure with a new IP address. What's happening, and does this indicate the policy is misconfigured? *Answer*: `toFQDNs` enforcement is based on IPs Cilium has actually observed being resolved for the allowlisted hostname — a DNS failover to a new IP creates a brief window where the new IP hasn't yet been observed and correlated to the allowed FQDN, during which legitimate traffic to the new IP can be denied. This isn't a misconfiguration; it's an inherent, narrow edge case of DNS-based policy enforcement, and it self-resolves once Cilium observes and correlates the new resolution — it's worth knowing about and monitoring for during a known third-party infrastructure failover, but doesn't indicate the policy itself is wrong. **Problem 4**: A platform team migrating from Istio's sidecar mesh to Cilium Service Mesh wants to justify the migration to leadership. What's the strongest form of justification per this chapter, and what's a justification this chapter would consider insufficient on its own? *Answer*: The strongest justification is a measured, specific cost figure — actual per-pod sidecar resource consumption at the cluster's real replica count, confirmed via cost-visibility tooling (Kubecost/OpenCost), combined with an explicit confirmation that the team's actual L7 feature usage (not Istio's full feature set, just what's actually used) is covered by Cilium Service Mesh's narrower feature set. "Sidecar-less architecture is newer/more elegant" alone, without measuring actual resource savings or confirming feature sufficiency first, is exactly the kind of unsubstantiated justification this chapter's capstone scenario warns against — a mesh migration is consequential enough to require evidence, not architectural preference. ## Summary and What's Next Cilium's entire feature set in this chapter traces back to two ideas: eBPF lets verified programs run directly in the kernel's packet path, replacing `iptables`'s O(n) chain traversal with O(1) hash-map lookups and eliminating per-pod sidecar overhead; and identity-based enforcement, keying policy off stable Kubernetes labels instead of ephemeral IPs, is what makes fast policy updates, L7 rules, DNS-based egress, and Cluster Mesh all work consistently at scale. Hubble turns that same eBPF visibility into structured, already-correlated observability, collapsing much of Part 10's manual diagnostic ladder into a single command with a named policy verdict. Cilium Service Mesh proves the same architectural idea extends beyond networking and security into service-mesh territory, trading some of a sidecar mesh's L7 maturity for a fundamentally flatter resource-cost curve as pod density grows. Part 18 shifts from how traffic moves through a cluster to what happens when a cluster or its workloads are lost entirely: Backup & Disaster Recovery for Workloads. It picks up directly from this chapter's Cluster Mesh coverage (a disaster-recovery plan spanning multiple clusters needs exactly the cross-cluster connectivity model this chapter introduced) and from Part 4's etcd disaster-recovery drill, extending both into a full workload-level backup and recovery strategy — the resilience question every capacity and networking decision in this series has been building toward answering.