Networking, Routes & OpenShift Service Mesh

Table of Contents#

1. Networking Layers in an OpenShift Cluster
2. OVN-Kubernetes — OpenShift's Default CNI
3. How Pod-to-Pod Traffic Actually Flows
4. NetworkPolicy in Full Depth
5. Controlling Egress: EgressIP and the Egress Firewall
6. Routes — OpenShift's Layer Above Kubernetes Ingress
7. TLS Termination Types: Edge, Passthrough, and Re-Encrypt
8. A Worked Example: Creating and Securing a Route
9. The Ingress Operator and the Default Router
10. Router Sharding
11. Kubernetes Ingress on OpenShift — Interop, Not Replacement
12. Multus — Attaching Additional Networks to a Pod
13. A Worked Example: A NetworkAttachmentDefinition
14. Cluster DNS: CoreDNS and the DNS Operator
15. OpenShift Service Mesh — When Routes and NetworkPolicy Aren't Enough
16. The Sail Operator and Istio's Control Plane on OpenShift
17. Quick Reference: Key Terms From This Chapter
18. Common Mistakes and Interview Traps
19. Worked Practice Problems
20. Summary and What's Next

Networking Layers in an OpenShift Cluster#

Part 2 closed with a deliberate promise: NetworkPolicy's default deny-across-Projects posture is a safe starting point with a well-defined escape hatch, not the whole story. This chapter is that whole story, and it spans four genuinely distinct layers that are easy to conflate if introduced all at once: the cluster network (how any Pod reaches any other Pod, cluster-wide, via the default CNI), NetworkPolicy (which of those reachable paths are actually permitted), ingress (how traffic from outside the cluster reaches a Pod at all, via Routes and/or Kubernetes Ingress), and service mesh (a layer some workloads add on top of all three for traffic shaping, mutual TLS, and observability that Routes and NetworkPolicy alone can't express).

Each layer builds on the one above it in this table, and a real production networking incident is often a confusion about which layer is actually the problem — a request timing out could be a missing NetworkPolicy rule, a misconfigured Route, or a service mesh sidecar's own mTLS policy, and this chapter is organized specifically so each layer's own failure signature is recognizable on its own.

A single failed request can fail at any one of these five points, and this diagram is deliberately the mental checklist worth working through top-to-bottom during an incident: is the Route/Ingress even admitted and routing correctly; if meshed, is the sidecar's own policy (covered later in this chapter) rejecting the request; is a NetworkPolicy denying it; and only once those are ruled out, is the underlying CNI layer itself actually broken — a genuinely rare last possibility, since OVN-Kubernetes failures tend to be all-or-nothing (a node losing connectivity entirely) rather than selectively blocking one specific request pattern.

OVN-Kubernetes — OpenShift's Default CNI#

Every OpenShift cluster's Pod-to-Pod networking runs on OVN-Kubernetes, a CNI plugin built on Open Virtual Network (OVN), itself built on Open vSwitch (OVS) — the same virtual switching technology widely used in OpenStack and other virtualization platforms, repurposed here to give every node a programmable virtual network fabric rather than relying purely on Linux's own routing tables and iptables rules.

OVN-Kubernetes builds an overlay network: every node runs an OVS instance, and pod-to-pod traffic between nodes is encapsulated using the Geneve protocol (a more extensible successor to VXLAN) and tunneled across the underlying physical network, meaning Pods get a flat, cluster-wide IP space regardless of the actual physical network topology underneath. OVN's own centralized control plane (the northbound/southbound databases, running as part of the network Cluster Operator's own managed components) computes the logical flows every node's OVS instance needs, and pushes them down — the same "centralized desired state, continuously reconciled to every node" pattern Part 1 established for the cluster's own components, applied here specifically to network flow rules.

OVN-Kubernetes runs by default in shared gateway mode, where egress and ingress traffic for a node is handled through OVS directly rather than the host's own routing stack — a design choice made specifically to enable hardware offloading (SmartNICs capable of executing OVS flow rules in silicon) on platforms that support it, reducing CPU overhead for network-heavy workloads compared to routing every packet through the host kernel's own network stack.

Shared Gateway Mode vs. Local Gateway Mode#

OVN-Kubernetes's gateway mode is a real, cluster-wide configuration choice made at install time (changing it later is a disruptive, supported-but-nontrivial migration), and it's worth understanding both options rather than assuming the default is automatically the right fit for every environment:

A platform team hitting an unexpected interaction between OVN-Kubernetes and a host-level network security tool is a common trigger for evaluating local gateway mode — worth knowing the option exists by name before assuming the interaction is an unfixable OVN-Kubernetes limitation.

Diagnosing Connectivity with ovnkube-trace#

oc exec -n openshift-ovn-kubernetes ds/ovnkube-node -- \
  ovnkube-trace -src-namespace payments-dev -src pod-a \
                -dst-namespace payments-dev -dst pod-b -tcp -dst-port 5432

ovn-trace source pod to destination pod indicates success
ovn-controller flow output:
  table=8 (ls_in_acl), priority=2000, match=(...), action=allow

Reading ovnkube-trace's output directly names the specific logical flow table and rule that allowed or denied the simulated packet — a materially faster diagnosis path than manually cross-referencing NetworkPolicy objects against OVN's own internal flow tables by hand, especially once several overlapping policies are in play across a namespace.

From the Trenches: A team debugging intermittent cross-node Pod connectivity spent a day suspecting application-level retries and load-balancer health checks before realizing the actual symptom correlated exactly with nodes behind a specific top-of-rack switch that silently dropped packets above a certain MTU. Geneve's encapsulation overhead reduces the effective MTU available to Pod traffic below the physical network's own MTU, and that specific switch's configuration hadn't been updated to account for the overlay's larger frame size on the physical uplinks — a classic overlay-network MTU mismatch, invisible at the Kubernetes API level entirely, only visible once someone thought to check ovnkube node logs for fragmentation-related drops and cross-referenced them against the physical switch topology.

How Pod-to-Pod Traffic Actually Flows#

Tracing one real request end to end grounds the diagram above in an actual packet's journey, and is worth being able to narrate precisely in an interview or an incident:

Every hop in this diagram is a place a NetworkPolicy denial, an MTU mismatch, or a node-level firewall rule can silently drop a packet — ovnkube-trace (a diagnostic tool shipped with OVN-Kubernetes) simulates exactly this flow and reports which specific logical flow rule allowed or denied the packet, the single most direct way to answer "why can't Pod A reach Pod B" without manually reconstructing OVN's flow tables by hand.

Same-node traffic (both Pods scheduled onto the same physical machine) skips the Geneve-encapsulation hop entirely — OVS routes directly between the two Pods' local interfaces without ever touching the physical network, which is both a performance characteristic worth knowing (same-node traffic is meaningfully cheaper than cross-node traffic) and a diagnostic one: a connectivity problem that only reproduces for cross-node traffic, never same-node, points specifically at the physical network or the Geneve overlay itself, rather than at OVN's logical flow rules, which apply identically in both cases.

NetworkPolicy in Full Depth#

Part 2 introduced the default allow-from-same-namespace policy a fresh Project receives; this section covers the full NetworkPolicy object model those defaults are built from. A NetworkPolicy is fundamentally a allow-list mechanism scoped to a namespace: once any NetworkPolicy selects a given Pod, that Pod's traffic (in whichever direction — ingress, egress, or both — the policy covers) is denied by default except for what the policy's rules explicitly permit; a Pod matched by no NetworkPolicy at all remains fully open in that direction, which is exactly the gap the Project template's default policy exists to close from the moment a namespace is created.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-allow-db-egress
  namespace: payments-dev
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: ["Egress"]
  egress:
    - to:
        - podSelector:
            matchLabels: { app: postgres }
      ports:
        - protocol: TCP
          port: 5432
    - to:                         # DNS must be explicitly allowed too
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53

That second egress rule is a common, easy-to-miss requirement: once a Pod is selected by any NetworkPolicy covering egress, DNS resolution itself is subject to the same default-deny unless explicitly allowed — a Pod that can otherwise reach its database perfectly well over a raw IP will still fail if it resolves that database's address by hostname first and the policy never explicitly permitted port 53 traffic to the cluster's DNS Pods.

A Default-Deny-Everything Baseline#

The Project-template default from Part 2 only covers ingress; a namespace with a genuinely strict posture typically starts from an explicit deny-everything baseline for both directions, then layers specific allow rules on top:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments-prod
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

An empty podSelector: {} matches every Pod in the namespace, and an empty ingress/egress rule list (implicit here, since neither is specified) denies everything in both directions covered by policyTypes — the strictest possible starting point, from which every subsequent NetworkPolicy in the namespace adds back exactly the specific paths that namespace's workloads actually need, including the DNS-egress rule from the trenches story below.

Allowing a Specific External CIDR#

ipBlock is the selector type worth knowing for the "allow traffic to/from a specific external range, not another Pod or namespace" case — a partner's API range, or a specific external monitoring collector:

  ingress:
    - from:
        - ipBlock:
            cidr: 203.0.113.0/24
            except: ["203.0.113.128/25"]

except carves a sub-range back out of an otherwise-allowed block — useful for "allow this partner's whole range except their known-decommissioned segment" without needing two separate, harder-to-reconcile policies.

From the Trenches: A team added an egress-restricting NetworkPolicy to lock down a namespace after a security review, tested it against every internal service dependency, and shipped it — only to find every Pod in the namespace immediately unable to resolve any hostname at all, including ones needed for basic cluster operation (pulling images by tag from an internal registry hostname). The fix wasn't reverting the policy; it was adding exactly the DNS-egress rule shown above, which the team had genuinely not realized was a separate, explicit requirement — a strong argument for testing any new default-deny NetworkPolicy in a non-production namespace first, specifically watching for silent DNS failures, which don't always announce themselves as clearly as a direct connection refusal would.

Controlling Egress: EgressIP and the Egress Firewall#

NetworkPolicy governs traffic within the cluster; two further OVN-Kubernetes-native mechanisms govern traffic leaving the cluster toward external systems — a common enterprise requirement (a partner API allow-listing specific source IPs, a compliance policy restricting which external destinations a Project may reach) that NetworkPolicy alone can't express, since NetworkPolicy has no concept of "the cluster's own outbound source IP."

EgressIP assigns a specific, consistent external-facing IP address to a Project's outbound traffic — so a partner's firewall can allow-list one stable IP rather than an unpredictable, potentially-changing node IP, even as the actual Pods generating that traffic get rescheduled across different nodes over time.

Egress Firewall (an EgressFirewall custom resource) restricts which external destinations a Project's Pods may reach at all, functioning like a namespace-scoped outbound allow/deny list independent of NetworkPolicy's ingress/egress-between-Pods model:

apiVersion: k8s.ovn.org/v1
kind: EgressFirewall
metadata:
  name: default
  namespace: payments-dev
spec:
  egress:
    - type: Allow
      to:
        cidrSelector: 203.0.113.0/24    # the approved partner API range
    - type: Deny
      to:
        cidrSelector: 0.0.0.0/0          # deny everything else

From the Trenches: A team's EgressFirewall was written and tested against a single Pod replica, allowing only the partner's documented CIDR range — and passed every test. Weeks later, a routine node-pool rebalance rescheduled that Deployment's Pods onto different nodes, and the partner integration started failing intermittently. The EgressFirewall itself was correct and unaffected by the reschedule; the actual root cause was that the team had never configured EgressIP alongside it, so the Pods' outbound traffic was presenting whichever node's own external IP happened to host them at any given moment — the partner's own firewall, allow-listing a specific IP the team had shared informally rather than through EgressIP, started rejecting traffic the moment the Pods landed on a node with a different external IP. Configuring EgressIP to pin a single, stable, allow-listable IP to the Project — independent of which node its Pods actually land on — was the fix, and the underlying lesson is that EgressFirewall and EgressIP solve genuinely different problems and are frequently both needed together, not interchangeably.

Routes — OpenShift's Layer Above Kubernetes Ingress#

Everything above this point governs traffic already inside the cluster network. Routes are OpenShift's answer to the question every cluster eventually has to solve: how does traffic from outside the cluster reach a Service at all? A Route is conceptually similar to a Kubernetes Ingress object — both bind an external hostname to an internal Service — but Routes predate Ingress in OpenShift's own history and carry a richer, more opinionated feature set built directly into the object itself, rather than delegated to whichever ingress controller a cluster happens to run.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: web
  namespace: payments-dev
spec:
  host: payments.apps.prod-east.example.com
  to:
    kind: Service
    name: web
  port:
    targetPort: 8080
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

Note spec.host: if left unset, OpenShift generates one automatically from the cluster's own wildcard domain (<route-name>-<namespace>.apps.<cluster-domain>) — the *.apps.<cluster-domain> wildcard DNS record every IPI installation provisions automatically is exactly what makes this "just works with no DNS request" behavior possible, a genuinely different default experience from a vanilla-Kubernetes Ingress, which has no equivalent automatic hostname provisioning at all.

TLS Termination Types: Edge, Passthrough, and Re-Encrypt#

A Route's tls.termination field is one of the more consequential decisions a team makes when exposing a service, and the three options trade off differently on where encryption actually happens:

Edge termination's insecureEdgeTerminationPolicy: Redirect (used in the example above) is worth calling out specifically: it automatically redirects any plain-HTTP request to HTTPS, rather than either serving insecure HTTP silently or rejecting it outright — the sensible default for nearly every public-facing Route, and a one-line fix for the common "site works over HTTPS but plain HTTP requests fail confusingly" complaint.

From the Trenches: A team migrating a legacy application to OpenShift chose passthrough termination by habit, assuming it was "more secure" without a specific mTLS requirement driving the choice. The consequence: the router could no longer perform host-based routing decisions using the request's actual hostname (since it never decrypts the traffic at all), and several unrelated routing and metrics features the team expected "for free" from the router simply didn't work for that specific Route. Re-encrypt termination — chosen once the team named their actual requirement (end-to-end encryption, without needing client-certificate inspection) — restored full router-level routing and metrics visibility while keeping the Pod-to-router hop encrypted, the correct fit once the actual requirement was made explicit instead of defaulting to whichever option sounded most secure in the abstract.

A Worked Example: Creating and Securing a Route#

The fastest path from a Service to a secured, externally-reachable Route:

oc expose service web --hostname=payments.apps.prod-east.example.com

oc patch route web -p '{"spec":{"tls":{"termination":"edge","insecureEdgeTerminationPolicy":"Redirect"}}}'

oc get route web -o jsonpath='{.spec.host}{"\n"}{.status.ingress[0].conditions}'

status.ingress[].conditions is worth reading directly rather than assuming the Route is live the moment it's created: a Route can be accepted by the API but not yet admitted by every router that should be serving it (relevant once router sharding, covered next, is in play), and the condition array names exactly which router has and hasn't admitted it yet.

Route-Specific Annotations Worth Knowing#

Beyond spec.tls, a handful of Route annotations cover common production requirements a raw Kubernetes Ingress typically needs a controller-specific annotation (with no portability guarantee) to express at all:

metadata:
  annotations:
    haproxy.router.openshift.io/timeout: "30s"
    haproxy.router.openshift.io/balance: "leastconn"

leastconn in particular is worth knowing as a deliberate alternative to the default round-robin balancing: for backends with meaningfully uneven per-request processing time (a mix of fast read endpoints and slow write endpoints behind the same Service), round-robin can leave some backend Pods overloaded while others sit idle, whereas least-connections actively accounts for each backend's current in-flight request count.

source-based balancing is the third option worth naming explicitly: it hashes the client's source IP to consistently route the same client to the same backend Pod, a lightweight session-affinity mechanism that doesn't require the cookie insertion disable_cookies turns off — useful when session affinity is wanted but cookie-based affinity specifically isn't, such as for a non-HTTP protocol passthrough Route where HAProxy has no cookie to insert in the first place.

Weighted Routing Across Multiple Backends#

A Route can split traffic across more than one backend Service by weight — the mechanism most commonly used for a manual canary rollout before a full GitOps-driven progressive delivery pipeline (Part 4 covers this) is in place:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: web
spec:
  host: payments.apps.prod-east.example.com
  to:
    kind: Service
    name: web-stable
    weight: 90
  alternateBackends:
    - kind: Service
      name: web-canary
      weight: 10

This sends roughly 10% of traffic to web-canary and 90% to web-stable, adjustable by editing the weights directly — a simple, immediately-available mechanism worth knowing before reaching for a heavier tool, though it lacks the automated analysis and rollback a dedicated progressive-delivery controller provides. Part 4's coverage of OpenShift GitOps and progressive delivery revisits this exact mechanism as the low-level primitive a higher-level automated canary controller ultimately drives on a team's behalf, rather than something to abandon once that automation is in place.

The Ingress Operator and the Default Router#

Every Route ultimately gets served by a router — an HAProxy-based load balancer, managed by the ingress Cluster Operator (Part 1's roster), that watches every Route object cluster-wide and continuously regenerates its HAProxy configuration to match. The default IngressController object (oc get ingresscontroller default -n openshift-ingress-operator) is what the Ingress Operator manages, and it's directly configurable for the properties that matter operationally: replica count, exposure strategy (a cloud load balancer, a NodePort, or host networking), and TLS security profile.

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
  domain: apps.prod-east.example.com
  replicas: 3
  endpointPublishingStrategy:
    type: LoadBalancerService

Bumping replicas beyond the default is a common, real Day-2 tuning decision for any cluster serving meaningful external traffic — the router is a genuine potential bottleneck and single point of contention if under-scaled, and unlike most Kubernetes workloads, it's not something a HorizontalPodAutoscaler manages automatically by default; capacity planning for it is a deliberate, manual sizing decision tied to expected request volume.

Endpoint Publishing Strategies#

endpointPublishingStrategy.type decides how the router itself becomes reachable, and the right choice depends entirely on the underlying infrastructure and network model:

Bare-metal and disconnected environments (Part 1's install-method coverage) commonly land on NodePortService or HostNetwork, since a cloud provider's own load-balancer API — what LoadBalancerService depends on — simply doesn't exist in those environments.

Router Sharding#

A single default router serving every Route in the cluster is the common case, but a real, supported requirement — isolating one Project's ingress traffic from another's at the router level, or dedicating a router to a specific set of high-traffic public-facing Routes — is solved by router sharding: running multiple IngressController objects, each scoped to a subset of Routes via a namespaceSelector or routeSelector.

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: internal-only
  namespace: openshift-ingress-operator
spec:
  domain: internal.prod-east.example.com
  namespaceSelector:
    matchLabels:
      network-tier: internal
  endpointPublishingStrategy:
    type: HostNetwork

A subtlety worth internalizing: the default IngressController continues serving every Route not explicitly excluded from it, even after additional sharded routers are introduced — a common misconfiguration is assuming a new sharded router automatically "removes" its matched Routes from the default router's own scope, when in fact the default router needs its own namespaceSelector/routeSelector exclusion configured explicitly if truly exclusive routing between shards is the goal; each shard also needs its own DNS record pointed at its own router, since nothing wires that up automatically once more than one router exists.

Kubernetes Ingress on OpenShift — Interop, Not Replacement#

A raw Kubernetes Ingress object still works on OpenShift — the route-controller-manager component (part of the ingress Cluster Operator) watches Ingress objects and automatically generates a corresponding Route for each one, translating Ingress's TLS Secret references into the equivalent Route TLS configuration. This is precisely what makes OpenShift's Kubernetes conformance claim from Part 1 hold at the ingress layer too: a Helm chart authored purely against upstream Ingress deploys and works on OpenShift without modification, gaining the underlying Route mechanism transparently.

The practical guidance: author a raw Ingress when portability to a non-OpenShift cluster genuinely matters (a Helm chart meant to ship to customers running vanilla Kubernetes too), and author a Route directly when OpenShift-specific features (passthrough/re-encrypt TLS, fine-grained router tuning) are needed — both coexist on the same cluster without conflict, and nothing about choosing one forecloses using the other elsewhere in the same cluster.

Watching the Automatic Conversion Happen#

oc apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: payments-dev
spec:
  rules:
    - host: web.apps.prod-east.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: web, port: { number: 8080 } }
EOF

oc get routes -n payments-dev

The oc get routes output shows a Route named after the Ingress object, owned by it (visible in metadata.ownerReferences) — deleting the Ingress deletes the generated Route automatically, since the route-controller-manager treats the Route as a fully owned, derived resource rather than an independent object a team should hand-edit directly. Editing the generated Route directly is a common early mistake: any manual change is overwritten the next time the controller reconciles it against the source Ingress, the same "edit the source of truth, not the derived object" lesson Part 1's Machine Config Operator and Part 2's default ClusterRole reconciliation both taught in their own layers.

Multus — Attaching Additional Networks to a Pod#

Every Pod's primary network interface (eth0) is attached to the cluster's default OVN-Kubernetes network — sufficient for the overwhelming majority of workloads. Multus is a meta-CNI plugin that lets a Pod attach one or more additional network interfaces (net1, net2, ...) beyond that default, without replacing it — a genuine, if specialized, requirement for network functions virtualization (NFV) workloads, high-throughput data-plane applications needing direct access to a specific physical NIC, or any workload needing to participate in more than one distinct network segment simultaneously.

Multus itself doesn't implement any network — it's a wrapper that invokes whichever real CNI plugin (macvlan, ipvlan, an SR-IOV device plugin) a NetworkAttachmentDefinition names, purely responsible for attaching the additional interface(s) a Pod's annotation requests, then handing control back to the default network for everything else.

A Worked Example: A NetworkAttachmentDefinition#

Defining an additional macvlan-based network, then attaching a Pod to it:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: high-speed-net
  namespace: telco-workloads
spec:
  config: '{
    "cniVersion": "0.4.0",
    "type": "macvlan",
    "master": "eth1",
    "mode": "bridge",
    "ipam": { "type": "whereabouts", "range": "192.168.100.0/24" }
  }'

apiVersion: v1
kind: Pod
metadata:
  name: nfv-workload
  namespace: telco-workloads
  annotations:
    k8s.v1.cni.cncf.io/networks: high-speed-net
spec:
  containers:
    - name: nfv-workload
      image: quay.io/myorg/nfv-workload:v1

The Pod's annotation is the only wiring needed — Multus reads it, invokes the named NetworkAttachmentDefinition's configured CNI plugin, and the Pod ends up with eth0 on the default cluster network plus net1 on the dedicated high-speed-net segment, both fully functional simultaneously. This is squarely a specialized-workload feature — telco/NFV, certain ML data-pipeline workloads with dedicated storage-network requirements — and not something an ordinary application team should reach for without a specific, concrete networking requirement Routes/Services can't already satisfy.

Choosing an Underlying CNI Plugin for a Multus Network#

NetworkAttachmentDefinition.spec.config names the actual CNI plugin Multus should invoke, and the choice matters:

SR-IOV in particular requires the underlying node hardware and a dedicated SriovNetworkNodePolicy (managed by the SR-IOV Network Operator, installed via OLM from Part 1) to actually expose virtual functions as schedulable node resources — a meaningfully deeper hardware dependency than macvlan/ipvlan, and worth confirming node hardware support before committing a design to it.

Multi-Cluster Networking: Submariner#

Everything in this chapter assumes traffic within a single cluster. Organizations running multiple clusters (Part 1's ROSA/ARO/OKD family, or simply several self-managed clusters across regions) that need Pods in one cluster to reach Services in another directly — rather than only through public ingress — reach for Submariner, a separate CNCF project OpenShift supports installing via OLM, which establishes secure tunnels between clusters and extends Service discovery across the cluster boundary. It's mentioned here by name specifically because it's the natural next question once a team internalizes this chapter's single-cluster networking model and then encounters a genuine multi-cluster connectivity requirement — out of scope for this series' depth, but worth recognizing rather than assuming multi-cluster Pod-to-Pod connectivity requires routing everything through public Routes.

Cluster DNS: CoreDNS and the DNS Operator#

Every Service's cluster-internal DNS name (web.payments-dev.svc.cluster.local) is resolved by CoreDNS, managed by the dns Cluster Operator (Part 1's roster) — the same CoreDNS project used across virtually every Kubernetes distribution, configured here through a DNS custom resource rather than a hand-edited Corefile directly:

apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  servers:
    - name: internal-corp
      zones: ["corp.internal"]
      forwardPlugin:
        upstreams: ["10.10.0.53"]

This pattern — forwarding a specific internal zone to a corporate DNS server while every other query resolves normally through the cluster's own CoreDNS — is the standard way to make an existing internal DNS namespace resolvable from inside the cluster, without forwarding every query externally and losing the performance and isolation benefits of in-cluster DNS resolution for the cluster's own Service names.

Troubleshooting DNS Resolution Directly#

When a Pod reports a hostname it can't resolve, checking resolution directly from inside the cluster network — rather than guessing from the application's own error message — is the fastest way to isolate whether the problem is DNS itself or something further downstream:

oc run dns-debug --image=registry.redhat.io/rhel9/support-tools:latest --restart=Never -it -- \
  dig web.payments-dev.svc.cluster.local

oc get pods -n openshift-dns -o wide
oc logs -n openshift-dns -l dns.operator.openshift.io/daemonset-dns --tail=50

A resolution failure that reproduces from this debug Pod but not from a Pod in a namespace with an egress-restricting NetworkPolicy is a strong signal pointing straight back at this chapter's own DNS-egress gotcha, rather than an actual CoreDNS problem — worth checking before escalating to the DNS Operator's own component logs.

OpenShift Service Mesh — When Routes and NetworkPolicy Aren't Enough#

Routes solve external ingress; NetworkPolicy solves coarse-grained allow/deny between Pods. Neither solves a different, more specific class of requirement that emerges once an application decomposes into enough services talking to each other: mutual TLS between every service automatically (not just at the cluster edge), fine-grained traffic shaping (canary releases by percentage, retries and circuit breaking per service-to-service call), and deep observability into service-to-service call patterns without instrumenting every application individually. OpenShift Service Mesh, built on Istio, exists specifically for that gap.

Every application container in the mesh gets a transparent Envoy proxy sidecar injected alongside it; the application itself talks to localhost exactly as if calling the destination service directly, while the sidecar transparently intercepts that traffic, wraps it in mutual TLS to the destination's own sidecar, and reports detailed telemetry (latency, error rate, retry counts) back to the mesh's observability stack — all without a single line of application code aware the mesh exists.

The honest trade-off, matching the same opinionation-vs-overhead framing this series applies throughout: a service mesh is a real, ongoing operational commitment (another control plane to run, upgrade, and understand) that pays off specifically once an application's actual service-to-service complexity (dozens of services, a real need for mTLS everywhere, canary rollouts by traffic percentage) outgrows what Routes and NetworkPolicy can express — adopting it preemptively, before that complexity is real, adds the same kind of unjustified overhead this series has warned against at every other layer.

The Sail Operator and Istio's Control Plane on OpenShift#

OpenShift Service Mesh 3.x is installed and managed through the Sail Operator, a from-scratch rewrite of how OpenShift packages Istio that replaced the older ServiceMeshControlPlane-based installation model with a much thinner wrapper around upstream Istio's own Istio custom resource — a deliberate convergence with the community Istio project rather than a Red-Hat-specific control plane API, making skills and configuration substantially more portable to a non-OpenShift Istio deployment than the earlier model was.

apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
spec:
  version: v1.26.0
  namespace: istio-system

apiVersion: v1
kind: Namespace
metadata:
  name: payments-dev
  labels:
    istio-injection: enabled

Labeling a namespace istio-injection: enabled is the actual, minimal step that brings its workloads into the mesh — the sidecar-injection webhook then automatically adds the Envoy proxy container to every Pod created in that namespace afterward, with zero change required to the applications' own Deployment manifests.

Traffic Management: VirtualService and DestinationRule#

Once services are in the mesh, Istio's own traffic-management CRDs express exactly the kind of fine-grained routing this chapter's earlier Route-weighting example only approximated:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: web
  namespace: payments-dev
spec:
  hosts: ["web"]
  http:
    - route:
        - destination: { host: web, subset: stable }
          weight: 90
        - destination: { host: web, subset: canary }
          weight: 10
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: web
  namespace: payments-dev
spec:
  host: web
  subsets:
    - name: stable
      labels: { version: stable }
    - name: canary
      labels: { version: canary }

The meaningful upgrade over a Route's own weighted backends: a VirtualService can route on request headers, retry automatically on specific error classes, and inject deliberate fault delays for chaos-testing purposes — none of which a Route's simpler weight-based model expresses at all, since a Route operates purely at the ingress edge while VirtualService/DestinationRule govern every hop inside the mesh, not just the initial entry point.

Observability: Kiali and Distributed Tracing#

The telemetry every Envoy sidecar reports feeds two tools worth knowing by name: Kiali renders the mesh's actual service-to-service call graph visually, derived from real observed traffic rather than a diagram someone drew and hoped stayed current, making it the fastest way to answer "what does this service actually call, right now" for a system too complex to reason about from source code alone; and a distributed tracing backend (commonly Jaeger, or Red Hat's OpenTelemetry-based Tempo stack) reconstructs one request's full path across every service hop it touched, with per-hop latency — the tool that turns "the request was slow somewhere in that chain of six services" into a specific answer naming which one.

Istio 1.26's ambient mode (referenced as a tech-preview feature in current OpenShift Service Mesh releases) is worth knowing by name as an emerging alternative: it removes the per-Pod sidecar entirely in favor of a shared per-node proxy, trading some of the sidecar model's per-Pod isolation for meaningfully lower resource overhead — a space actively evolving, and worth checking current release notes against before committing to one mode for a new mesh adoption.

NetworkPolicy and the Mesh — Layered, Not Replaced#

A common point of confusion once a namespace joins the mesh: NetworkPolicy does not stop being enforced. OVN-Kubernetes still evaluates every NetworkPolicy targeting a meshed Pod exactly as before — the mesh's mTLS and traffic-management layer sits on top of, not instead of, the CNI-level enforcement this chapter covered earlier. A request that a NetworkPolicy would deny is still denied, regardless of Istio's own AuthorizationPolicy rules (Istio's mesh-native equivalent of an allow/deny policy, enforced at the sidecar rather than the CNI level) permitting it — the two layers are independent, and both must permit a request for it to succeed, the same "every layer in the chain must agree" model this chapter's opening sequence diagram laid out.

A team migrating a namespace into the mesh and expecting Istio's AuthorizationPolicy to become the only access-control layer going forward is a common early mistake — any pre-existing NetworkPolicy in that namespace keeps applying in full, and the two need to be reconciled deliberately rather than assumed redundant with each other.

Quick Reference: Key Terms From This Chapter#

Common Mistakes and Interview Traps#

Worked Practice Problems#

1. A team adds a default-deny NetworkPolicy to a namespace, tests every known internal service dependency, and ships it. The next day, Pods in that namespace can't pull images from an internal registry hostname, though direct-IP connections still work. What's the most likely cause, and what's the fix?#

The most likely cause is exactly this chapter's DNS-egress gotcha: once a Pod is selected by an egress-covering NetworkPolicy, DNS resolution itself is subject to the same default-deny unless an explicit rule permits it, and the team's dependency testing evidently didn't specifically probe hostname-based resolution the way an image pull (which resolves the registry's hostname before connecting) does. The fix is adding an explicit egress rule allowing UDP/TCP port 53 traffic to the cluster's DNS Pods (typically selected via a namespaceSelector targeting openshift-dns), alongside the existing application-specific egress rules, rather than reverting the policy entirely.

2. A public-facing application needs end-to-end encryption for compliance reasons, but the team also wants the router to perform host-based routing and collect its own request metrics — a passthrough Route breaks both. What's the right TLS termination type, and why?#

Re-encrypt termination is the right fit: it terminates the client's TLS session at the router (satisfying the compliance requirement that traffic is encrypted from the client all the way to the router), then establishes a second, separate TLS session from the router to the Pod (satisfying "end-to-end encryption" in the sense of never traversing the network in plaintext), while still letting the router see the decrypted request in between — which is exactly what host-based routing and router-level metrics collection require. Passthrough would satisfy the encryption requirement but sacrifice the routing/metrics requirement entirely, since the router never decrypts passthrough traffic at all; re-encrypt is the specific option designed for exactly this combination of requirements.

3. A platform team wants to isolate a set of untrusted, multi-tenant-facing Routes onto their own router, separate from the cluster's internal Routes, without accidentally leaving those Routes also served by the default router. What steps does this actually require?#

Beyond creating the new sharded IngressController with a namespaceSelector/routeSelector matching the untrusted Routes, the team must also explicitly configure the default IngressController to exclude that same selector — otherwise, per this chapter's sharding section, the default router continues serving those Routes in addition to the new sharded one, defeating the isolation goal entirely. The team also needs a separate DNS record pointing specifically at the new sharded router's own exposure endpoint, since DNS routing between multiple routers isn't wired up automatically the moment a second IngressController is created.

4. An application team wants universal mutual TLS between a dozen internal microservices and asks whether NetworkPolicy can provide it. What's the honest answer, and what's the actual right tool?#

NetworkPolicy cannot provide this — it's an allow/deny mechanism operating on IP/port-level reachability between Pod and namespace selectors, with no concept of encryption or certificate-based identity at all; a NetworkPolicy allowing traffic between two Pods says nothing about whether that traffic is encrypted or authenticated in any cryptographic sense. OpenShift Service Mesh is the right tool for exactly this requirement: enrolling the relevant namespaces in the mesh (via the istio-injection: enabled label) gives every enrolled service automatic, transparent mutual TLS to every other enrolled service through their Envoy sidecars, with no application code changes — the honest trade-off to name alongside recommending it is the added operational cost of running and understanding a service mesh control plane, which is worth being upfront about rather than presenting the adoption as free.

5. A cluster's default router is showing elevated latency during peak traffic hours, but every backend Pod's own metrics look healthy. What's the first capacity-related hypothesis to check, and why might it be easy to overlook?#

The first hypothesis worth checking is that the router itself — not any backend Pod — is under-provisioned for current traffic volume, since this chapter's own coverage of the Ingress Operator names it explicitly as a component that is not autoscaled by default and requires deliberate, manual replica sizing. It's easy to overlook precisely because most other OpenShift workloads either autoscale automatically or are sized once and rarely revisited, so a platform team's usual capacity-monitoring habits may simply never have been pointed at the router's own replica count and per-instance resource usage — checking oc get ingresscontroller default -n openshift-ingress-operator -o yaml for the current replica count against the cluster's actual peak external request volume is the concrete next step, before investigating any backend-side explanation further.

6. Two services in the same mesh-enrolled namespace can't reach each other. Istio's AuthorizationPolicy for that namespace looks correct and permissive. What's the next layer to check, and why might a team miss it?#

The next layer to check is NetworkPolicy at the CNI level — since Istio's AuthorizationPolicy and Kubernetes' NetworkPolicy are enforced completely independently (per this chapter's layering section), a correct and permissive AuthorizationPolicy says nothing about whether an existing NetworkPolicy in that namespace is silently denying the same traffic. A team is likely to miss this specifically because joining the mesh is often mentally framed as "now Istio handles access control here," leading to NetworkPolicy being overlooked entirely once the mesh is in place — oc get networkpolicy -n <namespace> -o yaml and tracing the specific denied path with ovnkube-trace from earlier in this chapter is the concrete next diagnostic step, treating the mesh's own policy layer as necessary but not sufficient to explain the failure.

Summary and What's Next#

This chapter moved through four distinct networking layers in the order traffic actually experiences them: OVN-Kubernetes as the CNI making Pod-to-Pod connectivity physically possible cluster-wide via a Geneve-encapsulated overlay; NetworkPolicy as the allow-list layer governing which of those physically-possible paths are actually permitted, including the easy-to-miss requirement that DNS itself needs an explicit egress rule; Routes (and their EgressIP/Egress Firewall counterparts for outbound traffic) as OpenShift's opinionated, richer-than-Ingress answer to external traffic reaching a Service, with three TLS termination types trading off differently between security posture and router-level functionality; and OpenShift Service Mesh as the optional, higher-cost layer for the specific subset of applications whose service-to-service complexity has genuinely outgrown what the first three layers can express. Multus and cluster DNS configuration rounded out the specialized and supporting mechanisms most clusters touch rarely, but need to recognize by name when a real requirement surfaces.

The five-hop request trace this chapter opened with — Route/Ingress, mesh sidecar, NetworkPolicy, CNI, destination Pod — is worth keeping as a standing mental model for any future networking incident on this platform: naming which specific layer actually failed, rather than describing the symptom generically as "networking is broken," is what turns an open-ended investigation into a targeted one, and every tool this chapter introduced (ovnkube-trace, oc adm policy scc-subject-review's RBAC-layer cousin from Part 2, Route status conditions, Kiali's call graph) exists to answer exactly one layer of that question directly rather than requiring it to be inferred.

Part 4 moves from how traffic reaches a running workload to how that workload's container image gets built and delivered in the first place: BuildConfigs and Source-to-Image as OpenShift's built-in build mechanism, ImageStreams and the integrated internal registry, and OpenShift Pipelines (Tekton) paired with OpenShift GitOps (the ArgoCD Operator) as the modern, Kubernetes-native way most teams now assemble their actual delivery pipeline on top of those primitives.

Cross-reference: this catalog's Kubernetes Deep Dive series covers CNI, Service, and Ingress from a vanilla-Kubernetes vantage point in its own Networking (CNI) & Storage (CSI) chapter and its Gateway API chapters — worth revisiting in parallel with this chapter for the parts of the networking stack (Services, DNS, CNI fundamentals) that are genuinely unmodified between the two platforms, rather than re-deriving them here.

Sources consulted for this chapter: Red Hat's OpenShift Container Platform Networking documentation (OVN-Kubernetes, NetworkPolicy, EgressIP, Egress Firewall, Ingress sharding, Multus/multiple networks, DNS Operator), and Red Hat's OpenShift Service Mesh 3.x and Sail Operator documentation and release notes, current as of OpenShift 4.19/4.20 and OpenShift Service Mesh 3.1.

Confirm these version references against the cluster's own oc get clusterversion and oc get csv -n openshift-operators output before relying on any specific version number in this chapter for a real deployment decision — both OpenShift and OpenShift Service Mesh ship new minor versions on an ongoing cadence, per Part 1's release-cadence coverage.