Cilium & eBPF Networking Deep Dive

Assumes you're comfortable with the base networking model, CNI, Services, and NetworkPolicy from Part 3, the sidecar-based service mesh architecture and mTLS from Part 4, and the Gateway API resource model from Part 8 (including that chapter's brief mention of Cilium Gateway) — this chapter goes deep on Cilium's own eBPF data plane, kube-proxy replacement, identity-based policy, Hubble, and Cilium's sidecar-less service mesh, rather than re-explaining Kubernetes networking fundamentals or the Gateway API resource model itself.

Table of Contents#

1. Why This Part Exists
2. eBPF Fundamentals — What It Actually Is and Why It Replaces iptables
3. Where eBPF Programs Actually Run: Hooks and Program Types
4. Cilium's Architecture — Agent, Operator, and the eBPF Maps
5. Identity-Based Networking — Cilium's Core Architectural Idea
6. Installing Cilium and Confirming It's Actually Running
7. kube-proxy Replacement — eBPF-Based Service Load Balancing
8. Why the eBPF Datapath Outperforms iptables at Scale
9. CiliumNetworkPolicy — Beyond Standard NetworkPolicy
10. L7-Aware Policy — HTTP-, gRPC-, and Kafka-Level Rules
11. DNS-Based Egress Policy — Securing Calls to the Outside World
12. Hubble — Cilium's Observability Layer
13. Using Hubble to Debug a Real Connectivity Problem
14. Cilium Service Mesh — the Sidecar-less Architecture
15. mTLS Without a Sidecar — How It Actually Works
16. Cilium vs. a Sidecar Mesh — a Direct Comparison
17. Cilium Gateway API Support — Building on Part 8
18. Cluster Mesh — Cilium Across Multiple Clusters
19. Bandwidth Management and Quality of Service
20. Migrating an Existing Cluster From Another CNI to Cilium
21. Troubleshooting Cilium — When the eBPF Layer Itself Is the Problem
22. Security Beyond NetworkPolicy: Tetragon and eBPF-Based Runtime Enforcement
23. A Full Worked Scenario: Migrating checkout to Cilium With Zero-Downtime
24. A Full Worked Scenario: Catching a Data Exfiltration Attempt With DNS Policy and Hubble
25. A Full Worked Scenario: Replacing an Istio Sidecar Mesh With Cilium Service Mesh
26. Part 17 CLI Cheat Sheet
27. A Cilium Adoption Checklist
28. Common Mistakes and Interview Traps
29. Worked Practice Problems
30. Summary and What's Next

Why This Part Exists#

Part 3 taught the CNI contract and the Kubernetes networking model in general terms; this chapter goes one specific implementation deep, because Cilium has moved from "one CNI option among many" to the default or recommended CNI on every major managed Kubernetes offering — GKE Autopilot defaults to it, it's Amazon EKS's recommended CNI for clusters wanting kube-proxy replacement and L7 policy, and it ships as AKS's built-in eBPF dataplane option. Understanding Cilium specifically, not just "a CNI" abstractly, is now a practical requirement for operating a large fraction of real production clusters, not an optional deep cut.

The throughline system continues here: checkout-service (namespace checkout), catalog-service (namespace catalog), and inventory-service (namespace inventory) are the running examples for every policy, and this chapter treats them as a cluster mid-migration from a standard CNI plus kube-proxy to Cilium's eBPF datapath — a realistic scenario for a platform team adopting Cilium on an already-running cluster rather than starting from a green field.

eBPF Fundamentals — What It Actually Is and Why It Replaces iptables#

eBPF (extended Berkeley Packet Filter) lets a sandboxed, verified program run directly inside the Linux kernel, triggered by specific kernel events — network packets arriving, a socket being created, a syscall being made — without needing a kernel module, a context switch to userspace, or a kernel recompile. This single capability is what every Cilium feature in this chapter builds on: instead of asking the kernel to consult a long, sequentially-evaluated list of iptables rules for every packet (iptables's actual mechanism, still the default in most non-Cilium clusters), Cilium loads compiled eBPF programs that make routing, load-balancing, and policy decisions directly in the kernel's own packet-processing path.

The complexity difference is the entire performance story. iptables rules for Service routing and NetworkPolicy grow roughly linearly with the number of Services and endpoints in a cluster, and every packet pays the cost of walking that chain from the top — a cluster with thousands of Services can measurably slow down every single packet, including ones with no logical relationship to most of those rules. eBPF's kernel-resident hash maps look up the exact entry a packet needs directly, in constant time, regardless of how many other Services or endpoints exist elsewhere in the cluster — this is the concrete mechanism behind every "Cilium scales better than iptables-based CNIs at scale" claim in this chapter.

Where eBPF Programs Actually Run: Hooks and Program Types#

eBPF programs attach to specific kernel "hook points," each seeing a packet or event at a different stage of its journey — Cilium composes several of these together, and which hook a given feature uses explains a lot about what that feature can and can't see.

Cilium doesn't pick one hook exclusively — it composes several deliberately, choosing the earliest hook that still has enough information to make a given decision correctly, which is exactly what lets it make some decisions (an early DDoS drop) far cheaper than others (a decision that genuinely needs full L7 payload visibility, which requires cooperation with a userspace proxy — covered later in this chapter's L7 policy section).

Cilium's Architecture — Agent, Operator, and the eBPF Maps#

Cilium runs as a DaemonSet (cilium-agent, one pod per node) plus a small cluster-wide cilium-operator Deployment — no sidecar, no per-pod proxy process for basic CNI/policy functionality, which is the architectural root of the "sidecar-less" claim covered fully in this chapter's service-mesh section.

The absence of a control-plane-wide single point of decision is deliberate — each node's cilium-agent makes its own local packet-forwarding and policy decisions from its own kernel maps, so an API Server or cilium-operator outage degrades to "no new policy/endpoint updates are propagated" rather than "packets stop being forwarded correctly," a resilience property directly comparable to how kube-proxy's own iptables rules keep working on a node even if the control plane is briefly unreachable (Part 10's control-plane failure diagnostics).

Identity-Based Networking — Cilium's Core Architectural Idea#

Standard Kubernetes NetworkPolicy and iptables-based CNIs enforce rules against IP addresses — Cilium's core architectural departure is enforcing policy against a numeric security identity derived from a pod's Kubernetes labels instead, and this single idea is what makes several of this chapter's later features (fast policy updates at scale, cluster mesh, L7 policy) actually work the way they do.

Why this matters in practice, not just architecturally: when checkout-service scales from 3 replicas to 30, or gets rescheduled onto entirely different nodes with entirely different pod IPs, its security identity — derived from its stable labels, not its ephemeral IP — never changes. An iptables-based CNI has to update rules referencing the new IPs every time a pod is rescheduled; Cilium's policy rules reference the identity once and never need to change as pods churn underneath it, which is a direct, measurable win for policy-update latency in a cluster with high pod churn (a CI/CD-heavy namespace redeploying frequently, or an HPA scaling aggressively under load — Part 12).

Installing Cilium and Confirming It's Actually Running#

Cilium installs via its own CLI or a Helm chart — verifying it's actually healthy afterward matters more than the install command itself, since a Cilium agent that fails to load its eBPF programs can leave a node with silently broken networking that looks fine at a glance.

cilium install --version 1.20.1 --set kubeProxyReplacement=true
cilium status --wait

# The single most useful post-install command — a full connectivity
# self-test across the cluster, covering pod-to-pod, pod-to-service,
# and DNS in one pass
cilium connectivity test

kube-proxy Replacement — eBPF-Based Service Load Balancing#

kube-proxy's traditional job — translating a Service's stable ClusterIP into one of its backing pods — is exactly the kind of per-packet lookup eBPF's kernel-resident hash maps are built for, and Cilium's replacement eliminates kube-proxy and its iptables rules entirely, not just supplementing them.

The socket-layer hook placement matters specifically here — by intercepting at connect() time rather than per-packet, Cilium's kube-proxy replacement can perform the ClusterIP-to-backend translation once per connection instead of on every single packet in that connection's lifetime, which is a meaningfully different (and cheaper) cost model than an iptables DNAT rule re-evaluated on every packet.

cilium status | grep KubeProxyReplacement
# KubeProxyReplacement: True   [eth0 (Direct Routing)]
kubectl get pods -n kube-system -l k8s-app=kube-proxy
# No resources found — confirms kube-proxy itself is genuinely absent, not just idle

Why the eBPF Datapath Outperforms iptables at Scale#

The performance argument isn't abstract — it's specifically about how each mechanism's cost scales as a cluster grows, and the crossover point is exactly where most organizations start to feel Kubernetes networking as a real operational cost rather than a solved problem.

Per-CPU eBPF maps are the second half of the scaling story, beyond the O(1)-vs-O(n) lookup cost alone: Cilium structures many of its kernel maps per-CPU specifically to eliminate lock contention between cores processing packets concurrently — a design choice that lets throughput scale close to linearly with core count, rather than flattening out as more cores contend for the same global lock a naive shared-map design would need.

CiliumNetworkPolicy — Beyond Standard NetworkPolicy#

Standard Kubernetes NetworkPolicy (Part 3, Part 11) only expresses L3/L4 rules — pod/namespace selectors and ports. CiliumNetworkPolicy, Cilium's own CRD, is a strict superset: every standard NetworkPolicy is still fully honored (Cilium enforces both formats simultaneously), and CiliumNetworkPolicy additionally expresses L7 rules and DNS-based egress this chapter covers in the next two sections.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: checkout-allow-catalog
  namespace: checkout
spec:
  endpointSelector:
    matchLabels:
      app: checkout-service
  egress:
    - toEndpoints:
        - matchLabels:
            app: catalog-service
            k8s:io.kubernetes.pod.namespace: catalog
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP

The toEndpoints selector above is enforced against the identity model from earlier in this chapter, not against IPs — this rule stays correct through any number of catalog-service pod reschedulings or replica count changes, for exactly the reason the identity-based networking section explained.

L7-Aware Policy — HTTP-, gRPC-, and Kafka-Level Rules#

An L7 rule lets a policy say "this caller may only issue a GET to /catalog/items," not just "this caller may reach port 8080 at all" — a meaningfully stronger boundary than L3/L4 alone, and one standard NetworkPolicy has no vocabulary for expressing.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: catalog-l7-read-only-for-checkout
  namespace: catalog
spec:
  endpointSelector:
    matchLabels:
      app: catalog-service
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: checkout-service
            k8s:io.kubernetes.pod.namespace: checkout
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/catalog/items.*"

L7 visibility requires a userspace proxy, which is the one place Cilium deliberately steps outside pure in-kernel eBPF processing — enforcing an HTTP method/path match needs to actually parse the HTTP request, which eBPF's kernel-verifier constraints make impractical to do safely and fully in-kernel. Cilium transparently redirects only the specific traffic an L7 rule applies to through its own managed Envoy instance (cilium-envoy, mentioned in the architecture section) for that parsing, then hands the already-decided verdict back to the eBPF datapath — every other packet Envoy is not implicated in still takes the pure eBPF fast path.

DNS-Based Egress Policy — Securing Calls to the Outside World#

Standard Kubernetes NetworkPolicy can restrict egress to a CIDR block, but most external dependencies (a third-party API, a SaaS webhook target) don't have a stable IP a CIDR rule can pin to — DNS-based egress policy lets a rule reference the hostname directly, with Cilium resolving and tracking the IPs behind it continuously.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: checkout-allow-stripe-egress
  namespace: checkout
spec:
  endpointSelector:
    matchLabels:
      app: checkout-service
  egress:
    - toFQDNs:
        - matchName: "api.stripe.com"
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP

The second rule block above — explicitly allowing egress to CoreDNS — is not optional boilerplate. toFQDNs policy only works because Cilium can observe and correlate the pod's own DNS lookups; if a default-deny egress policy blocks port 53 to CoreDNS, the FQDN policy has nothing to resolve against and the "allowed" hostname's traffic fails anyway — the exact same DNS-egress trap Part 10's NetworkPolicy troubleshooting chapter covers for standard NetworkPolicy, reappearing here in FQDN-policy form.

Hubble — Cilium's Observability Layer#

Hubble observes every flow eBPF already sees passing through Cilium's datapath and turns it into structured, Kubernetes-aware flow events — pod names, namespaces, labels, Service names, and the actual policy verdict — without packet sampling, a separate packet-capture pipeline, or any sidecar.

hubble observe --namespace checkout --to-namespace catalog -f
# TIMESTAMP  SOURCE                    DESTINATION                VERDICT  SUMMARY
# 10:42:03   checkout/checkout-svc-x   catalog/catalog-svc-y      FORWARDED  TCP Flags: SYN
# 10:42:03   checkout/checkout-svc-x   catalog/catalog-svc-y      FORWARDED  HTTP/1.1 GET /catalog/items -> 200

Because the Hubble agent is embedded directly in cilium-agent — no additional pod, no sidecar injection — it adds essentially zero deployment overhead beyond Cilium itself, which is a real architectural difference from bolting a separate observability agent onto an existing CNI. This is the concrete mechanism behind "network observability built into Cilium" rather than "a network observability tool that also happens to run alongside Cilium."

Using Hubble to Debug a Real Connectivity Problem#

Part 10's NetworkPolicy troubleshooting chapter used a pod-IP-then-ClusterIP-then-DNS diagnostic ladder built from generic kubectl/curl commands — on a Cilium cluster, Hubble replaces most of that ladder with one direct query that names the actual policy verdict, instead of inferring it indirectly.

hubble observe --pod checkout/checkout-service-7d8f9c --to-pod catalog/catalog-service-x2k9p -f
# TIMESTAMP  SOURCE   DESTINATION   VERDICT  SUMMARY
# 10:51:02   ...      ...           DROPPED  Policy denied by
#                                             (CiliumNetworkPolicy) catalog-l7-read-only-for-checkout

Compare this to the Part 10 approach, which required curling the pod IP directly, then the ClusterIP, then checking kubectl get endpoints, inferring the failing layer from which step succeeded — Hubble's output above names the exact policy object responsible in a single command, collapsing several manual diagnostic steps into one structured, already-correlated answer.

Cilium Service Mesh — the Sidecar-less Architecture#

Part 4 covered the sidecar pattern in depth — a proxy container injected into every mesh-participating pod — as the traditional service mesh architecture. Cilium Service Mesh implements the same core mesh capabilities (mTLS, L7 traffic management, observability) without injecting anything into application pods at all, by pushing that functionality into the eBPF datapath every pod already traverses.

The resource-overhead argument is the most concrete, easiest-to-measure difference: a sidecar mesh adds one proxy container's CPU/memory footprint (Part 4's resource-cost coverage) per pod, which scales linearly with replica count — a Deployment with 50 replicas carries 50 sidecar processes. Cilium's mesh functionality runs once per node in the shared cilium-agent/cilium-envoy processes, regardless of how many pods that node hosts, which is a fundamentally different (and, at high pod density, much cheaper) resource-scaling curve.

mTLS Without a Sidecar — How It Actually Works#

Part 4 explained sidecar mTLS as two Envoy proxies negotiating a TLS handshake on the application's behalf — Cilium achieves the same outcome (encrypted, mutually-authenticated pod-to-pod traffic) by having eBPF programs intercept traffic and hand it to per-node processes (WireGuard or IPsec, depending on configuration) that perform the encryption, again without any per-pod proxy.

cilium config view | grep -i encryption
# encryption: wireguard
cilium status | grep Encryption
# Encryption: Wireguard  [NodeEncryption: Disabled]

The practical takeaway for a team migrating off a sidecar mesh (this chapter's closing worked scenario): Cilium's node-to-node encryption modes (WireGuard/IPsec) protect traffic between nodes transparently and without application changes, but map onto a different trust boundary than per-pod mTLS certificates — verify which specific compliance or security requirement drove the original sidecar-mesh mTLS adoption before assuming Cilium's encryption modes satisfy it as a drop-in equivalent.

Cilium vs. a Sidecar Mesh — a Direct Comparison#

Bringing Part 4's sidecar-mesh coverage and this chapter's sidecar-less coverage together into one decision framework, since "which mesh architecture" is a real, consequential choice a platform team makes once and lives with for years.

Cilium Gateway API Support — Building on Part 8#

Part 8 introduced Cilium Gateway as one of several Gateway API implementations, briefly — this section goes one layer deeper into what makes it specifically an eBPF-native implementation rather than "yet another Envoy-based Gateway controller with a different name."

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: checkout-gateway
  namespace: checkout
spec:
  gatewayClassName: cilium
  listeners:
    - name: https
      protocol: HTTPS
      port: 443

Cilium's Gateway API implementation reuses the same eBPF datapath and per-node Envoy instances this chapter has already covered for L7 policy and service mesh — a Gateway/HTTPRoute on a Cilium-managed cluster is implemented by the same cilium-envoy processes handling L7 CiliumNetworkPolicy rules, not a separate, independently-deployed ingress controller. This is the practical advantage of adopting Cilium Gateway specifically on an already-Cilium cluster: one less independent networking component to operate, patch, and reason about compared to running a different vendor's Gateway API implementation (Envoy Gateway, NGINX Gateway Fabric — both covered in Part 8) alongside Cilium as the CNI.

Cluster Mesh — Cilium Across Multiple Clusters#

Cluster Mesh extends Cilium's identity-based networking model (this chapter, earlier) across multiple independent Kubernetes clusters, letting a pod in one cluster reach a Service in another using the exact same identity-based policy model as within a single cluster — directly building on why identity, not IP, is Cilium's foundational idea.

Because identity is a logical construct derived from labels rather than tied to any one cluster's IP address space, a CiliumNetworkPolicy written once continues to apply correctly whether the traffic it governs stays within one cluster or crosses into another — this is precisely the property Part 13's separate-physical-clusters isolation tier and its fleet-management escape hatch would benefit from if the tenants involved needed connectivity, not just consistent configuration, across clusters. Cluster Mesh and Part 13's fleet-management tooling (Cluster API, Rancher Fleet) solve adjacent but distinct problems: fleet tools keep many clusters configured consistently; Cluster Mesh lets them actually talk to each other securely.

Bandwidth Management and Quality of Service#

Beyond policy and observability, Cilium's eBPF datapath also enforces bandwidth limits directly at the kernel level — a capability that has no equivalent in standard Kubernetes NetworkPolicy at all, since NetworkPolicy only ever expresses allow/deny, never rate.

apiVersion: v1
kind: Pod
metadata:
  name: recommendation-model-batch
  namespace: recommendations
  annotations:
    kubernetes.io/egress-bandwidth: "50M"   # Cilium-enforced egress cap

This is directly relevant to the noisy-neighbor concerns Part 13 raised about shared node-level resources that CPU/memory limits alone can't hard-partition — Part 13 explicitly named network throughput as a gap plain ResourceQuota/limits leave open; Cilium's bandwidth-manager feature is one concrete way to close that specific gap, capping a tenant's egress bandwidth the same way a CPU limit caps its compute consumption.

Migrating an Existing Cluster From Another CNI to Cilium#

Migrating a live, already-running cluster's CNI is one of the riskiest single operations covered in this whole series, precisely because networking is the one layer nearly everything else depends on — this section sequences it deliberately rather than treating it as a simple helm upgrade.

Node-by-node, verify-after-each-step is the discipline that makes this safe, and it's the same discipline Part 15 already established for cluster upgrades generally — a CNI migration is, in effect, a rolling node-replacement operation with an unusually high blast radius per mistake, so the same "one node, verify, next node" cadence that Part 15 uses for version upgrades applies here with even less tolerance for skipping the verification step.

Troubleshooting Cilium — When the eBPF Layer Itself Is the Problem#

Part 10's general troubleshooting chapter covers the standard pod/Service/DNS diagnostic ladder — Cilium adds one more layer beneath all of that worth checking specifically: is the eBPF datapath itself healthy on this node.

cilium status                          # per-node agent health summary
cilium-dbg bpf endpoint list           # every endpoint (pod) this node's agent knows about
cilium-dbg monitor --type drop         # live stream of every packet eBPF is actively dropping, and why

Security Beyond NetworkPolicy: Tetragon and eBPF-Based Runtime Enforcement#

Part 11 covered Falco for runtime security — Tetragon is Cilium's own sibling project applying the same eBPF foundation to runtime security observability and enforcement, worth naming here specifically because of how closely its architecture parallels everything else in this chapter.

Naming this here rather than only in Part 11 matters because of the shared architectural DNA: Tetragon's Kubernetes-identity-aware event correlation works the same way Hubble's does — full pod/namespace/label context attached to every kernel-level event, for the same underlying reason this chapter's identity section explained for networking. A platform team already running Cilium and Hubble gets a head start understanding Tetragon's own event model, since it's the same design applied to process/syscall events instead of network flows.

A Full Worked Scenario: Migrating checkout to Cilium With Zero-Downtime#

Bringing several sections together for a realistic migration: the platform team is moving the whole cluster (including checkout, catalog, and inventory) from a standard CNI plus kube-proxy onto Cilium with kube-proxy replacement enabled.

1. Pre-migration validation (this chapter's installation section): install Cilium in chaining mode alongside the existing CNI on a non-production cluster first, run cilium connectivity test, and confirm every existing NetworkPolicy in the checkout namespace still enforces correctly under Cilium before touching production.
2. Sequence the node-by-node migration (this chapter's migration section) starting with a canary AZ, cordoning and draining nodes one at a time per Part 15's rolling pattern, running cilium connectivity test after each node before proceeding to the next.
3. Confirm identity-based policy correctness using Hubble mid-migration: hubble observe --namespace checkout during the rollout, watching for any unexpected DROPPED verdicts that would indicate an existing NetworkPolicy didn't translate as expected onto Cilium's enforcement.
4. Enable kube-proxy replacement only after every node is confirmed running Cilium as the sole CNI — removing kube-proxy before every node has Cilium's replacement active would leave nodes mid-migration with neither mechanism handling Service routing.
5. Verify Service routing specifically, since this is the highest-risk single change in the whole migration: cilium status | grep KubeProxyReplacement on every node, plus a full cilium connectivity test pass, before considering the migration complete.
6. Leave Hubble running as standing observability afterward, not just as a migration-verification tool — the same flow visibility that confirmed the migration's correctness becomes the team's ongoing Part 10-style troubleshooting entry point going forward.

The single riskiest step is step 4 — a partial kube-proxy removal, done before every node has confirmed Cilium replacement active, is exactly the kind of mistake that produces a cluster-wide Service-routing outage with no single obvious cause, which is precisely why this sequence gates it behind full node-by-node confirmation rather than a single cluster-wide toggle.

A Full Worked Scenario: Catching a Data Exfiltration Attempt With DNS Policy and Hubble#

A security-flavored scenario tying together DNS-based egress policy, Hubble, and Part 11's supply-chain security coverage: a compromised dependency in inventory-service's container image attempts to exfiltrate data to an unrecognized external domain.

Because inventory-service runs under a toFQDNs egress policy (this chapter's DNS-policy section) explicitly allowlisting only its known legitimate external dependencies, the exfiltration attempt's DNS lookup for an unrecognized domain never resolves to an allowed destination, and the subsequent connection attempt is dropped at the eBPF layer before a single byte leaves the pod.

hubble observe --pod inventory/inventory-service-x7k2p --verdict DROPPED -f
# TIMESTAMP  SOURCE                        DESTINATION            VERDICT  SUMMARY
# 14:12:08   inventory/inventory-svc-x7k2p  93.184.xx.xx:443       DROPPED  Policy denied: no matching toFQDNs rule

Explaining why this works, two levels deep: the symptom the security team actually notices is a Hubble alert (wired to their SIEM via Hubble's exportable flow format) for a DROPPED egress connection from a production pod to an unrecognized external IP. The immediate cause is the compromised dependency's outbound connection attempt failing a toFQDNs policy match. The underlying condition that made this catchable at all is that the platform team had already applied default-deny egress with an explicit allowlist to every production namespace as a standing security baseline (not something newly added in response to this incident) — without that pre-existing baseline, the exfiltration attempt would have succeeded silently, since a default-allow egress posture has no policy boundary for Hubble to generate a DROPPED verdict against in the first place.

A Full Worked Scenario: Replacing an Istio Sidecar Mesh With Cilium Service Mesh#

A capstone scenario combining this chapter's service-mesh sections with Part 4's sidecar-mesh coverage: the platform team running Istio's sidecar mesh (Part 4) across checkout, catalog, and inventory decides to migrate to Cilium Service Mesh, driven by rising per-pod sidecar resource overhead as the catalog namespace's replica count grew past 200 pods during peak traffic.

1. Quantify the actual overhead first (per this chapter's resource-cost comparison): 200 sidecar proxies each consuming a modest but nonzero CPU/memory footprint sums to a real, measurable chunk of the catalog namespace's total cluster footprint — the team confirms this via Kubecost/OpenCost (Part 16) before committing to a migration, rather than migrating on a hunch.
2. Confirm the required L7 feature set is covered: the team's actual mesh usage is mTLS everywhere plus basic traffic-splitting for canary rollouts — neither requires Istio's more advanced fault-injection or mesh-federation features, so Cilium Service Mesh's narrower L7 feature set (this chapter's comparison table) is confirmed sufficient before migrating, rather than discovered insufficient afterward.
3. Migrate the CNI first, mesh second: since the cluster wasn't already running Cilium as its CNI, the team follows this chapter's CNI-migration sequence completely before layering Cilium Service Mesh on top — attempting both changes simultaneously would make it far harder to isolate which layer caused any problem that surfaced during rollout.
4. Remove sidecar injection namespace-by-namespace, verifying mTLS and traffic-splitting behavior with Hubble after each namespace, rather than disabling Istio sidecar injection cluster-wide in one step.
5. Measure the actual outcome: post-migration, the catalog namespace's total resource footprint drops measurably (the exact sidecar-elimination savings this chapter's resource-scaling comparison predicted), confirmed again via Kubecost/OpenCost rather than assumed from the architecture diagram alone.

The generalizable lesson: a mesh-architecture migration this consequential should be justified by a measured, specific cost (200 sidecars' real resource footprint, confirmed via cost tooling) and a confirmed feature-sufficiency check, not by "sidecar-less is architecturally newer" alone — the same evidence-based discipline this whole series applies to every other consequential infrastructure decision.

Part 17 CLI Cheat Sheet#

A Cilium Adoption Checklist#

• Confirmed node kernel versions meet Cilium's minimum requirements for the specific features being enabled (kube-proxy replacement, WireGuard encryption, L7 policy)
• Ran cilium connectivity test clean before considering any install/upgrade/migration complete
• Existing NetworkPolicy objects confirmed to still enforce correctly under Cilium (both formats are honored simultaneously, but verify rather than assume)
• kube-proxy removal sequenced only after every node confirms Cilium's replacement is active — never removed cluster-wide before every node is migrated
• Default-deny egress with explicit toFQDNs/toEndpoints allowlists applied to production namespaces handling sensitive data, per this chapter's exfiltration-prevention scenario
• Hubble Relay and UI deployed and confirmed reachable — not just the per-node agent-embedded Hubble server
• L7 policy usage scoped deliberately, not applied broadly "for safety," given its real per-request Envoy-hop cost
• If migrating from a sidecar mesh, the resource-overhead and feature-sufficiency case documented and confirmed via real cost data (Part 16), not assumed from architecture alone
• Cluster Mesh CIDR non-overlap confirmed in advance for any cluster that might eventually need cross-cluster connectivity

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A cluster runs both kube-proxy and Cilium with kubeProxyReplacement=true enabled at the same time, and Services intermittently route to the wrong or stale backend. What's the root cause, and how should the migration have been sequenced to avoid it?

Answer: Two independent Service-routing mechanisms — kube-proxy's iptables rules and Cilium's eBPF-based replacement — are both actively making routing decisions simultaneously, and neither is aware of the other, producing inconsistent or stale routing depending on which mechanism's rules a given packet happens to hit first. The migration should have removed kube-proxy in the same coordinated, node-by-node sequenced change that enabled Cilium's replacement, confirming via cilium status | grep KubeProxyReplacement and kubectl get pods -n kube-system -l k8s-app=kube-proxy (expecting no results) on each node before moving to the next, never running both simultaneously as an ongoing state.

Problem 2: A team applies an L7 CiliumNetworkPolicy restricting checkout-service to only GET requests against catalog-service, and afterward notices a measurable latency increase on every request between the two services, even ones that were already compliant with the new rule. Why, and is this avoidable?

Answer: Any traffic matched by an L7 rule is transparently redirected through Cilium's per-node Envoy proxy for HTTP parsing, reintroducing a userspace hop that the pure eBPF fast path (used for L3/L4-only rules) doesn't pay — this cost applies to every request matched by the L7 rule, compliant or not, since the proxy has to parse the request to confirm compliance in the first place. It's not fully avoidable if genuine method/path-level enforcement is required, but it is avoidable in cases where L3/L4 enforcement (just allowing the port) would have been sufficient — the fix is confirming the L7 rule's specificity is actually needed before applying it, not applying it as a default level of caution.

Problem 3: A toFQDNs policy allowlisting api.stripe.com starts blocking legitimate traffic for about 90 seconds immediately after Stripe fails over to new infrastructure with a new IP address. What's happening, and does this indicate the policy is misconfigured?

Answer: toFQDNs enforcement is based on IPs Cilium has actually observed being resolved for the allowlisted hostname — a DNS failover to a new IP creates a brief window where the new IP hasn't yet been observed and correlated to the allowed FQDN, during which legitimate traffic to the new IP can be denied. This isn't a misconfiguration; it's an inherent, narrow edge case of DNS-based policy enforcement, and it self-resolves once Cilium observes and correlates the new resolution — it's worth knowing about and monitoring for during a known third-party infrastructure failover, but doesn't indicate the policy itself is wrong.

Problem 4: A platform team migrating from Istio's sidecar mesh to Cilium Service Mesh wants to justify the migration to leadership. What's the strongest form of justification per this chapter, and what's a justification this chapter would consider insufficient on its own?

Answer: The strongest justification is a measured, specific cost figure — actual per-pod sidecar resource consumption at the cluster's real replica count, confirmed via cost-visibility tooling (Kubecost/OpenCost), combined with an explicit confirmation that the team's actual L7 feature usage (not Istio's full feature set, just what's actually used) is covered by Cilium Service Mesh's narrower feature set. "Sidecar-less architecture is newer/more elegant" alone, without measuring actual resource savings or confirming feature sufficiency first, is exactly the kind of unsubstantiated justification this chapter's capstone scenario warns against — a mesh migration is consequential enough to require evidence, not architectural preference.

Summary and What's Next#

Cilium's entire feature set in this chapter traces back to two ideas: eBPF lets verified programs run directly in the kernel's packet path, replacing iptables's O(n) chain traversal with O(1) hash-map lookups and eliminating per-pod sidecar overhead; and identity-based enforcement, keying policy off stable Kubernetes labels instead of ephemeral IPs, is what makes fast policy updates, L7 rules, DNS-based egress, and Cluster Mesh all work consistently at scale. Hubble turns that same eBPF visibility into structured, already-correlated observability, collapsing much of Part 10's manual diagnostic ladder into a single command with a named policy verdict. Cilium Service Mesh proves the same architectural idea extends beyond networking and security into service-mesh territory, trading some of a sidecar mesh's L7 maturity for a fundamentally flatter resource-cost curve as pod density grows.

Part 18 shifts from how traffic moves through a cluster to what happens when a cluster or its workloads are lost entirely: Backup & Disaster Recovery for Workloads. It picks up directly from this chapter's Cluster Mesh coverage (a disaster-recovery plan spanning multiple clusters needs exactly the cross-cluster connectivity model this chapter introduced) and from Part 4's etcd disaster-recovery drill, extending both into a full workload-level backup and recovery strategy — the resilience question every capacity and networking decision in this series has been building toward answering.