Part 13 of 1932 min read · 9 diagramsAI-assisted

Multi-Tenancy & Cluster Sharing at Scale

Assumes you're comfortable with ResourceQuota/LimitRange from Part 2, RBAC and NetworkPolicy hardening from Part 11, and admission control from Part 1 — this chapter is about combining those primitives into a real tenant-isolation boundary, not re-introducing any of them individually.

Table of Contents#

  1. Why This Part Exists
  2. The Isolation Spectrum: Soft to Hard Multi-Tenancy
  3. Namespace-Based Soft Multi-Tenancy — the Baseline
  4. Where Namespace Isolation Breaks Down
  5. Hierarchical Namespaces — the Idea, and a Retirement Worth Knowing
  6. Tenant-Aware Policy Engines: Capsule's Tenant Abstraction
  7. A Worked Capsule Example: Onboarding a New Tenant
  8. Virtual Clusters — vCluster Architecture Deep Dive
  9. vCluster Isolation Tiers
  10. Runtime Isolation for Untrusted Tenants: gVisor and Kata Containers
  11. Choosing a Multi-Tenancy Model — Decision Framework
  12. Noisy-Neighbor Mitigation Beyond ResourceQuota
  13. Multi-Tenant Observability — Isolating Metrics and Logs Per Tenant
  14. Multi-Tenant GitOps — Scoping Who Can Deploy Where
  15. A Worked Numeric Example: Sizing Tenant Quotas
  16. Cost Allocation and Chargeback in a Shared Cluster
  17. Admission Policy as a Tenant Boundary
  18. Choosing How Often to Re-Verify
  19. Testing Tenant Isolation — How to Actually Verify It Holds
  20. A Full Worked Scenario: Onboarding Three Teams Safely
  21. When Multi-Tenancy Isn't Enough — Fleet Management as the Escape Hatch
  22. Part 13 CLI Cheat Sheet
  23. A Tenant Onboarding Checklist
  24. Common Mistakes and Interview Traps
  25. Worked Practice Problems
  26. Summary and What's Next

Why This Part Exists#

Every mechanism this chapter needs already exists somewhere earlier in this series — ResourceQuota and LimitRange from Part 2, RBAC and NetworkPolicy from Part 11, admission control from Part 1 — but no part so far has addressed the actual platform-engineering question a growing organization eventually has to answer: "how many teams can safely share one cluster, and where does that stop being true?" This is squarely a platform-engineering concern — per the industry research behind this series' chapter list, platform engineering is forecast to reach roughly 80% enterprise adoption, and multi-tenancy design is one of its most concrete, recurring decisions.

The throughline system gains new neighbors in this chapter: alongside the existing checkout team, imagine a catalog team and a newly onboarded recommendations team, all sharing the same cluster the checkout namespace already lives in. Every worked example in this chapter treats these three teams as cooperative but independent — exactly the trust profile most internal platforms are actually designing for.

The Isolation Spectrum: Soft to Hard Multi-Tenancy#

Every multi-tenancy approach sits somewhere on one spectrum: how much of the underlying cluster does a tenant actually get to see and potentially affect, versus how much operational overhead and cost does providing that isolation require.

Diagram
ModelWhat a tenant can seeTypical use
Plain namespaces + RBAC + quotasTheir own namespace's objects only, but shares one API server, one etcd, one set of CRDs cluster-wideTrusted internal teams, same organization
Namespaces + a tenant-policy engine (Capsule)Same as above, plus enforced guardrails (network, quota, RBAC templates) applied consistently per tenantMany internal teams, self-service namespace provisioning
Virtual clusters (vCluster)Their own API server and (virtual) control plane, even their own CRDs — but shares the underlying nodesSemi-trusted tenants, or teams needing cluster-admin-like self-service without touching the real cluster-admin plane
Separate physical clustersNothing shared at all except perhaps a network boundaryRegulatory/compliance-mandated hard separation, or genuinely untrusted external tenants

Important

There is no universally "correct" point on this spectrum — the right answer is always a function of how much you trust the tenants involved and what a worst-case failure in isolation would actually cost. A platform serving mutually trusted internal teams gains little from the overhead of separate clusters; a platform selling isolated environments to external, mutually distrusting customers has a very different risk calculus. Revisit the choice periodically, too — a tenant's trust profile and workload sensitivity can shift well after the original onboarding decision was made.

Namespace-Based Soft Multi-Tenancy — the Baseline#

Every stronger model in this chapter builds on top of this one — a namespace per tenant, RBAC scoping each tenant's ServiceAccounts and human users to their own namespace, a ResourceQuota capping their consumption, and a default-deny NetworkPolicy (Part 11) as the network boundary.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: recommendations-quota
  namespace: recommendations
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"
    persistentvolumeclaims: "10"
    services.loadbalancers: "0"   # a common tenant restriction: no direct cloud LB provisioning
Diagram

This baseline is genuinely sufficient for a large fraction of real internal platforms — most organizations aren't isolating mutually hostile parties, they're isolating cooperative teams who occasionally make mistakes, and RBAC plus quotas plus NetworkPolicy stops the overwhelming majority of accidental cross-tenant impact at a fraction of the operational cost of anything stronger.

Where Namespace Isolation Breaks Down#

The honest limitation of the namespace baseline is what it doesn't separate — and every stronger tier in this chapter exists specifically to close one of these gaps.

Shared resourceRisk if not addressed
API serverA tenant's controller/operator with a runaway watch loop, or a burst of API calls, can degrade API server latency for every other tenant sharing it
etcdOne tenant creating an enormous number of objects (a CRD instance explosion from a misbehaving operator) consumes shared etcd storage and I/O capacity
Cluster-scoped resources (CRDs, StorageClasses, admission webhooks)A tenant with permission to install a CRD or webhook can affect every other tenant's objects cluster-wide — namespacing doesn't apply to cluster-scoped kinds at all
The node's kernelEvery pod on shared nodes runs containers sharing the same host kernel — a container-escape vulnerability in the runtime affects every tenant co-located on that node, not just the compromised one
Node-level resource contentionCPU/memory limits (Part 2) prevent a container exceeding its own allocation, but shared node-level resources like disk I/O bandwidth and network throughput are much harder to hard-partition per pod

Admission webhooks deserve the same scrutiny for an identical reason: a ValidatingWebhookConfiguration or MutatingWebhookConfiguration is cluster-scoped, and its namespaceSelector/rules determine which objects across the entire cluster it intercepts — a tenant's own policy engine or operator, installed with a webhook scoped too broadly, can start validating or mutating every other tenant's objects, not just its own. This is the same failure shape as the CRD risk above, applied to a different cluster-scoped kind.

Warning

A tenant granted permission to create CustomResourceDefinition objects — even scoped "only in their own namespace" via RBAC, since CRDs are themselves cluster-scoped regardless of who creates them — can register a CRD name colliding with, or shadowing, one another tenant or the platform team depends on. This is a genuinely easy mistake to make when writing "self-service" RBAC for platform teams: verify a tenant's Role never grants create/update/delete on customresourcedefinitions.apiextensions.k8s.io unless that's a deliberate, reviewed capability.

Hierarchical Namespaces — the Idea, and a Retirement Worth Knowing#

Hierarchical namespaces let a parent namespace's policies (RBAC, ResourceQuota, NetworkPolicy) propagate automatically to child namespaces, so a platform team can define a tenant's guardrails once at the parent level instead of copy-pasting the same RBAC/quota YAML into every namespace a growing team spins up.

Diagram

This capability lived in the hierarchical-namespaces (HNC) project from the Kubernetes Working Group for Multi-Tenancy — worth naming precisely because that repository is now retired, per its own GitHub archive status, and is not the recommendation to reach for on a new platform build today. The underlying idea — inherited, hierarchical policy propagation — remains valuable and has been carried forward primarily by tenant-policy engines like Capsule (next section), which implement a comparable propagation model as one feature of a broader, more actively maintained tool rather than as a standalone, now-unmaintained controller.

Note

This is a deliberate accuracy callout, not a minor detail: any tutorial, course, or older blog post still recommending HNC as the current path to hierarchical namespace policy is describing a project that is no longer maintained. Confirm a tool's current maintenance status before adopting it for new platform work — this exact gap between "still frequently recommended in older material" and "actually maintained today" is a recurring trap across the whole Kubernetes ecosystem, not unique to HNC.

Tenant-Aware Policy Engines: Capsule's Tenant Abstraction#

Capsule (a CNCF Sandbox project) introduces a Tenant custom resource that groups one or more namespaces under a single owner and a single set of enforced policies — the platform team defines a Tenant once, and Capsule's admission webhooks enforce it consistently across every namespace that tenant creates.

Diagram

Note

Capsule is a CNCF Sandbox project (an earlier maturity stage than KEDA's Graduated status from Part 12) — actively developed and used in production by real platform teams, but worth knowing precisely where it sits on the CNCF maturity ladder before treating "CNCF project" alone as a maturity signal. Sandbox-stage projects are a reasonable adoption choice for a specific, well-understood problem like this one, but merit a closer look at release cadence and community size than a Graduated project would need.

The self-service angle is the practical win here — without Capsule, a platform team is either the bottleneck for every new namespace request (they must apply the RBAC/quota/NetworkPolicy boilerplate by hand each time) or trusts every team with enough cluster-wide permission to create namespaces freely, which reintroduces the cluster-scoped-resource risk from the previous section. A Tenant object lets the tenant owner create new namespaces within their own tenant boundary self-service, with Capsule's webhooks guaranteeing every one of them inherits the same guardrails automatically.

A Worked Capsule Example: Onboarding a New Tenant#

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: recommendations-team
spec:
  owners:
    - name: recommendations-lead@company.com
      kind: User
  namespaceOptions:
    quota: 5   # max namespaces this tenant can self-service create
  resourceQuotas:
    scope: Tenant   # aggregated across every namespace the tenant owns, not per-namespace
    items:
      - hard:
          requests.cpu: "30"
          requests.memory: 60Gi
          pods: "80"
  networkPolicy:
    ingress:
      - from:
          - namespaceSelector:
              matchLabels:
                capsule.clastix.io/tenant: recommendations-team   # only same-tenant traffic by default
# The tenant owner, with no cluster-admin access at all, can now self-service:
kubectl create namespace recommendations-experiments \
  --as recommendations-lead@company.com
# Capsule's webhook validates this against the Tenant's namespaceOptions.quota
# and automatically applies the Tenant's RBAC/quota/NetworkPolicy templates

The scope: Tenant ResourceQuota aggregation is worth calling out specifically — a per-namespace ResourceQuota (the Part 2 mechanism) caps each namespace independently, which a tenant with namespaceOptions.quota: 5 could otherwise use to multiply their effective total allocation by simply creating more namespaces. Tenant-scoped aggregation closes that loophole by summing usage across every namespace the tenant owns against one shared ceiling.

Virtual Clusters — vCluster Architecture Deep Dive#

A vCluster runs an entire, real, additional Kubernetes control plane — its own API server, its own etcd (or a lightweight equivalent), its own controller manager — inside a single namespace of the underlying "host" cluster, giving a tenant something that looks and behaves like their own dedicated cluster while still sharing the host cluster's actual compute nodes.

Diagram

The "sync" step is the architectural core of how this works: a tenant creates a Pod against the virtual API server exactly as if it were a real, independent cluster; vCluster's synchronization controller translates that object down into a real Pod on the host cluster (in the vCluster's backing namespace), where it actually runs on shared physical nodes. The tenant never sees the host cluster's API server, other tenants' namespaces, or any cluster-scoped objects outside their own virtual control plane — from inside a vCluster, even kubectl get namespaces only shows namespaces the tenant themselves created, solving the cluster-scoped-resource visibility gap that plain namespace isolation cannot.

vCluster Isolation Tiers#

vCluster's isolation strength is not one fixed level — it's configurable per deployment, trading cost against isolation strength the same way the isolation-spectrum diagram earlier in this chapter suggested.

TierNode sharingBest for
Shared NodesTenants' pods run on the same physical nodes as other tenants, same kernelTrusted tenants — internal dev/CI/CD environments, cost-sensitive internal platforms
Dedicated NodesA labeled, reserved subset of the host cluster's nodes, via node selectors/taintsTenants needing predictable, uncontended compute without provisioning entirely separate infrastructure
Private NodesNodes enrolled specifically for one tenant, with network/storage/compute isolated at the infrastructure levelSemi-trusted or regulatory-sensitive tenants where "shares a kernel with someone else" is unacceptable

Each tier only changes the underlying compute-sharing story — the virtual control plane's own isolation (separate API server, separate etcd, invisible host-cluster internals) is identical across all three. This means a platform team can start a tenant on Shared Nodes for cost efficiency and move them to Dedicated or Private Nodes later purely as an infrastructure change, without the tenant's own workload manifests, RBAC, or CRDs needing to change at all — the virtual cluster they interact with looks the same regardless of which physical isolation tier backs it.

Runtime Isolation for Untrusted Tenants: gVisor and Kata Containers#

Every model so far still shares the host kernel for at least some tier of tenant — gVisor and Kata Containers address the "shared kernel" risk specifically, by changing what actually executes a container's syscalls.

Diagram
RuntimeIsolation mechanismOverhead
Standard (runc)None beyond namespaces/cgroups — shares the host kernel directlyLowest
gVisor (runsc)Intercepts and re-implements syscalls in a userspace "sentry" process, presenting a much narrower attack surface to the real host kernelModerate — some syscall-heavy workloads see a real performance cost
Kata ContainersEach pod runs inside its own lightweight virtual machine with a genuinely separate kernelHighest — closest to true hardware-level isolation, at real memory/startup-time cost

A third option worth naming alongside these two: AWS Fargate for EKS (Part 7) and similar serverless-container offerings use a comparable microVM-per-pod model (built on the same Firecracker technology Kata Containers can also use as a backend) as their default isolation posture, not an opt-in RuntimeClass — a relevant data point when comparing self-managed Kata adoption against simply running sensitive tenant workloads on a managed serverless-container tier instead.

These are selected per-workload via a RuntimeClass object, not cluster-wide — a platform can run most pods on standard runc for performance and reserve gVisor or Kata specifically for the namespaces/tenants where "this tenant's code might be actively malicious, not just occasionally buggy" is a genuine possibility, such as a platform running arbitrary customer-submitted code.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
---
apiVersion: v1
kind: Pod
spec:
  runtimeClassName: gvisor   # opts this specific pod into the stronger runtime

Choosing a Multi-Tenancy Model — Decision Framework#

Choose...When
Plain namespaces + RBAC + quotas + NetworkPolicyTrusted internal teams, cooperative, no regulatory isolation mandate
Namespaces + Capsule (or a similar tenant-policy engine)Same trust level, but self-service namespace provisioning at a scale where hand-applying guardrails per team doesn't scale
vCluster, Shared NodesA tenant genuinely needs cluster-admin-like control (their own CRDs, their own controllers) without touching the real cluster's control plane
vCluster, Dedicated/Private NodesSame as above, plus a requirement for predictable or fully isolated compute
gVisor/Kata RuntimeClass, layered on any of the aboveThe specific risk is kernel-level container escape — a platform running genuinely untrusted or externally-submitted code
Fully separate physical clustersA hard regulatory/compliance boundary, or tenants who must never share even a virtual control plane's underlying host

Noisy-Neighbor Mitigation Beyond ResourceQuota#

ResourceQuota caps a tenant's total declared consumption, but says nothing about where their pods land or how they're prioritized against other tenants' pods under real contention — three more primitives, all covered individually in earlier parts, combine to close that gap.

# 1. A PriorityClass per tenant tier — determines who gets evicted first
# under real node pressure (Part 2's preemption mechanics)
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-standard
value: 100
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-premium
value: 1000
---
# 2. Taints reserving a node pool for one tenant tier only
# kubectl taint nodes -l tenant-tier=premium dedicated=premium-tenants:NoSchedule
PrimitiveWhat it adds beyond ResourceQuotaCovered in
PriorityClass per tenant tierUnder real node memory/CPU pressure, lower-priority tenants' pods are evicted first, protecting higher-tier tenants' availabilityPart 2
Taints/tolerations for tenant-dedicated node poolsPhysically separates a tenant's pods onto specific nodes, reducing (though not eliminating, on Shared Nodes) contention with other tenantsPart 2
Pod anti-affinity spreading a tenant's own replicasPrevents one tenant's own workload from concentrating on a single node and becoming a single point of resource contention for whoever else lands therePart 2
Topology spread constraints across tenant node poolsEnsures a tenant's replicas spread evenly, rather than clusteringPart 2

Combining these with the ResourceQuota baseline turns "no single tenant can request too much in total" into "no single tenant can crowd out another tenant's pods when the cluster is genuinely under pressure" — a meaningfully stronger guarantee, since quotas alone say nothing about eviction order or physical placement once contention actually happens.

Multi-Tenant Observability — Isolating Metrics and Logs Per Tenant#

A shared Prometheus/Loki stack scraping every tenant's namespace by default creates two problems this chapter's isolation model needs to answer: can Tenant A query Tenant B's metrics/logs, and can Tenant B's high-cardinality metrics degrade query performance for everyone else sharing that same backend?

Diagram
LayerHow tenant isolation is enforced
IngestionA label (tenant or the namespace itself) is attached to every metric series/log line as it's collected — this must happen at ingestion, not query time, since query-time filtering alone can't stop a tenant from ever seeing another's raw data if the backend doesn't enforce it structurally
Storage/query (Mimir, Cortex, Thanos, Loki)These systems support native multi-tenancy via an X-Scope-OrgID header — each tenant's queries are scoped to their own tenant ID at the storage-query layer itself, not just via a Grafana dashboard filter that a determined user could bypass
DashboardsGrafana's own per-tenant data source/folder permissions, layered on top of the backend's own tenant scoping — defense in depth, not the primary control
# A query scoped to exactly one tenant via Loki's native multi-tenancy header —
# the backend rejects this if the caller's auth doesn't match the claimed tenant
curl -H "X-Scope-OrgID: recommendations" \
  http://loki.observability.svc.cluster.local:3100/loki/api/v1/query \
  --data-urlencode 'query={namespace="recommendations"}'

Caution

Relying on a Grafana dashboard filter (e.g. a templated namespace variable) as the only tenant isolation for observability data is not real isolation — anyone with raw query access to the underlying data source (Prometheus's own API, a shared Loki endpoint) can bypass a dashboard-level filter entirely. Genuine tenant isolation for observability data must be enforced at the storage/query backend itself (Mimir/Cortex/Loki's native multi-tenancy), the same principle as never relying on a UI-level control for something RBAC or a NetworkPolicy should actually enforce.

Multi-Tenant GitOps — Scoping Who Can Deploy Where#

A shared cluster running GitOps (the automation-cicd-gitops tutorial series covers GitOps mechanics in full — this section is specifically the multi-tenancy angle) needs the deployment pipeline itself scoped per tenant, or GitOps becomes a way to bypass every RBAC boundary this chapter has built.

# ArgoCD AppProject — scopes what a tenant's own Application objects
# are allowed to deploy, and where
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: recommendations-team
  namespace: argocd
spec:
  sourceRepos:
    - "https://github.com/company/recommendations-*"
  destinations:
    - namespace: "recommendations*"
      server: "https://kubernetes.default.svc"
  clusterResourceWhitelist: []   # deliberately empty — no cluster-scoped resources allowed
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota   # tenants can't self-service raise their own quota via GitOps

The clusterResourceWhitelist: [] and quota-blacklist lines matter for exactly the same reason RBAC should never grant CRD lifecycle permissions to a tenant — without them, a tenant's Git repository becomes an alternate, unmonitored path to the same over-privileged actions this chapter has spent several sections closing off through RBAC and admission policy. GitOps doesn't bypass Kubernetes RBAC on its own (ArgoCD still applies objects through the API server as whatever identity it's configured to run as), but a single shared, overly-permissive ArgoCD ServiceAccount used across every tenant's AppProject re-creates exactly the over-broad-grant risk this whole chapter exists to avoid — scoping AppProject per tenant is what keeps GitOps consistent with the rest of the tenant boundary rather than quietly undermining it.

A Worked Numeric Example: Sizing Tenant Quotas#

Abstract guidance ("set a reasonable quota") is less useful than working through the actual math a platform team would do — here's a realistic sizing exercise for the recommendations team's ResourceQuota introduced earlier in this chapter.

The team runs 3 services, each currently deployed at 4 replicas with 500m CPU / 1Gi memory requests per pod, plus headroom for Part 12's HPA to scale up to double that under load:

Baseline: 3 services × 4 replicas × 500m CPU = 6 CPU, × 1Gi memory = 12Gi memory HPA headroom (up to 2x under load): 12 CPU, 24Gi memory Platform buffer (batch jobs, one-off debug pods, ~20%): +2.4 CPU, +4.8Gi memory
spec:
  hard:
    requests.cpu: "15"      # rounded up from 14.4
    requests.memory: 30Gi   # rounded up from 28.8Gi
    limits.cpu: "20"        # limits set higher than requests per Part 2's Burstable QoS pattern
    limits.memory: 32Gi

Sizing a quota from a real HPA ceiling, not a guess, is the actual discipline here — a quota set below what a tenant's own HPA maxReplicas could legitimately request under real load doesn't stop overspend, it just relocates the failure: instead of a clean cost conversation, the tenant's pods start failing scheduling (Part 10's Pending-pod diagnostic tree) the moment real traffic actually needs to scale, which reads to the tenant as a platform outage rather than the capacity-planning gap it actually is.

Cost Allocation and Chargeback in a Shared Cluster#

A shared cluster's bill is one number from the cloud provider — attributing it back to individual tenants requires tagging and aggregation that Kubernetes doesn't provide natively.

kubectl get resourcequota -A -o json | \
  jq -r '.items[] | "\(.metadata.namespace): \(.status.used["requests.cpu"]) / \(.status.hard["requests.cpu"]) CPU"'
ApproachHow it attributes cost
Manual: aggregate ResourceQuota usage per namespace/TenantCheap to start, but only proxies actual cost via requested (not necessarily used) resources
A cost-visibility tool (Kubecost or similar)Combines actual node cost with real per-pod resource usage for a genuine per-tenant/per-namespace dollar figure, including shared control-plane overhead allocation
Capsule Tenant-scoped quota reportingAggregates naturally per tenant across every namespace they own, matching how the tenant boundary is actually defined

Tip

Resource requests, not actual usage, is what most chargeback models bill against — this creates a direct incentive for a tenant to right-size their requests down (Part 12's VPA is directly relevant here) rather than defensively over-requesting "to be safe," since over-requesting under a chargeback model has a real, visible cost consequence that a purely technical ResourceQuota ceiling alone doesn't create.

Admission Policy as a Tenant Boundary#

Part 11 covered Kyverno/OPA Gatekeeper for security policy — the same admission-control mechanism is equally central to multi-tenancy, enforcing rules that RBAC alone can't express.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-storageclass-per-tenant
spec:
  validationFailureAction: Enforce
  rules:
    - name: only-standard-storageclass-for-non-platform-tenants
      match:
        any:
          - resources:
              kinds: ["PersistentVolumeClaim"]
              namespaces: ["catalog", "recommendations", "checkout"]
      validate:
        message: "Tenant namespaces may only use the 'standard' StorageClass"
        pattern:
          spec:
            storageClassName: "standard"

RBAC answers "who can create what kind of object" — it has no vocabulary for "what values are allowed inside that object once created," which is exactly the gap a policy engine fills. RBAC alone cannot stop a tenant with legitimate create permission on PersistentVolumeClaim objects from requesting an expensive, high-performance StorageClass never intended for their tier — only an admission policy inspecting the object's actual field values can enforce that kind of tenant-tier boundary.

Choosing How Often to Re-Verify#

A one-time isolation test at onboarding proves the boundary was correct on day one — it says nothing about whether a subsequent, unrelated platform change quietly reopened a gap months later.

Trigger for re-running the isolation test suiteWhy
Any RBAC change (new ClusterRole, edited aggregation rule)The single most common source of an accidentally widened grant
Any NetworkPolicy change in a shared/platform namespaceA relaxed rule for one debugging session that never gets reverted is a genuinely common real-world gap
A new admission webhook or CRD installed cluster-wideBoth are cluster-scoped by nature — a new one can unintentionally intersect with an existing tenant boundary
On a fixed cadence regardless of known changesCatches drift from changes nobody remembered to flag as isolation-relevant in the first place

Treating this as a scheduled, low-effort recurring check (the small script from the previous section, run on a cadence or wired into the same pipeline pattern Part 15 covers for cluster upgrades) costs far less than discovering an isolation gap the hard way, during an actual cross-tenant incident.

Testing Tenant Isolation — How to Actually Verify It Holds#

Every control in this chapter can be configured correctly on paper and still fail in practice — the only way to know tenant isolation actually holds is to actively try to violate it, the same "verify, don't just configure" discipline Part 11 applied to security controls generally.

# 1. Cross-namespace RBAC check: can tenant A's ServiceAccount read tenant B's Secrets?
kubectl auth can-i get secrets \
  --as=system:serviceaccount:recommendations:default -n checkout

# 2. Cross-namespace network check: does NetworkPolicy actually block traffic
# from a pod outside the expected tenant boundary?
kubectl run isolation-test --rm -it --image=busybox -n catalog \
  -- wget -T 3 -O- http://checkout-svc.checkout.svc.cluster.local

# 3. Quota-bypass check: can the tenant exceed their aggregate quota by
# creating an additional namespace, if Tenant-scoped aggregation isn't enforced?
kubectl create namespace recommendations-shadow --as recommendations-lead@company.com
TestWhat a failure would mean
kubectl auth can-i cross-tenant, using --as to impersonate the other tenant's ServiceAccountAn RBAC grant is broader than intended — the single most common isolation gap, and the cheapest one to check
A NetworkPolicy cross-namespace connectivity attempt that should be blockedThe default-deny baseline has a gap — often a missing policyTypes entry, or a policy that was applied but never actually took effect (Part 10's NetworkPolicy troubleshooting applies directly here)
Attempting to exceed tenant-aggregate quota via a second self-service namespaceConfirms whether Tenant-scoped aggregation (Capsule) is actually wired up, versus quotas silently reverting to per-namespace only
Attempting to install a CustomResourceDefinition or cluster-scoped object as a tenant identityConfirms the RBAC gap flagged earlier in this chapter (tenants should never have CRD lifecycle permissions) isn't present

Tip

Turn this into a small, repeatable test suite (a script or a CI job run against a staging cluster after every RBAC/NetworkPolicy/Capsule policy change) rather than a one-time manual check — tenant isolation is exactly the kind of guarantee that silently regresses when an unrelated change (a new ClusterRole, a relaxed NetworkPolicy for an unrelated debugging session that never got reverted) accidentally reopens a gap that was correctly closed months earlier.

A Full Worked Scenario: Onboarding Three Teams Safely#

Bringing every tier in this chapter together for a platform team onboarding checkout, catalog, and a brand-new recommendations team onto one shared cluster:

Diagram

Working through the actual decision for recommendations: it's a standard internal service, doesn't need its own CRDs, and the team is cooperative with no regulatory sensitivity — the correct-sized answer is the cheapest one on the isolation spectrum that still solves the real problem: a Capsule Tenant wrapping one or more namespaces, with the standard ResourceQuota/RBAC/NetworkPolicy guardrails from earlier in this chapter. Reaching for a vCluster or a separate physical cluster here would be over-engineering relative to the actual risk being managed — a useful, generally applicable lesson: isolation strength should be sized to the tenant's actual trust level and workload needs, not applied uniformly at the strongest tier "to be safe" everywhere.

When Multi-Tenancy Isn't Enough — Fleet Management as the Escape Hatch#

Every tier this chapter covers still shares something — at minimum, the cloud account and billing boundary. When even a Private-Nodes vCluster or a fully separate physical cluster is the right call per the decision framework above, the new problem becomes operating potentially dozens of separate clusters consistently — which is exactly what fleet-management tooling exists to solve.

Diagram
ToolRole
Cluster API (CAPI)Declaratively provisions and lifecycle-manages entire clusters (not just objects within one) via Kubernetes-native CRDs — a Cluster/MachineDeployment object per tenant cluster, reconciled the same way a Deployment reconciles pods
Rancher FleetGitOps specifically aimed at pushing consistent configuration (policies, workloads, upgrades) across a fleet of many clusters from one control point
A cloud provider's own fleet console (e.g. GKE Fleet, EKS with multiple clusters under one Organization)Centralized visibility and some policy consistency, without necessarily the full declarative lifecycle management CAPI provides
Cluster upgrade orchestration across a fleet (Part 15)Rolling out a version upgrade consistently across dozens of separate tenant clusters is itself a scheduling problem — a fleet tool sequences it deliberately rather than leaving each cluster's owner to upgrade on their own timeline

The tradeoff this escape hatch makes explicit: separate clusters solve the "shares nothing" isolation requirement completely, but only if the fleet-management layer actually keeps every cluster consistently configured — without one, "give every tenant their own cluster" quietly becomes "give every tenant their own snowflake cluster that drifts further from every other one over time," which is a worse operational outcome than the shared-cluster noisy-neighbor problems this chapter otherwise solves. Choosing separate clusters as an isolation tier is really choosing separate clusters plus a real fleet-management investment — treating it as a lighter-weight option than vCluster because "it's just normal clusters" is a common and costly underestimate.

Part 13 CLI Cheat Sheet#

CommandPurpose
kubectl get resourcequota -A -o json | jq ...Aggregate per-namespace consumption for a cost/chargeback estimate
kubectl get tenants (Capsule CRD)List every tenant a Capsule-managed cluster currently has onboarded
kubectl create namespace <ns> --as <tenant-owner>Test that a tenant's self-service namespace creation is correctly scoped by Capsule's webhook
kubectl get runtimeclassConfirm which RuntimeClasses (standard, gVisor, Kata) are available for tenant-tier assignment
vcluster create <name> -n <host-namespace>Provision a new virtual cluster for a tenant needing control-plane isolation
vcluster connect <name>Retrieve a kubeconfig scoped to one tenant's virtual cluster
kubectl get networkpolicy -AAudit that every tenant namespace actually has its expected default-deny + allow rules in place
argocd appproject get <tenant>Confirm a tenant's GitOps AppProject scoping matches what RBAC/Capsule already enforce

A Tenant Onboarding Checklist#

Every control this chapter covers, as a single checklist a platform team can run through for each new tenant — the kind of artifact worth keeping next to Part 10's troubleshooting runbook.

  • Namespace(s) created, scoped under a Tenant object if using Capsule (or an equivalent grouping)
  • RBAC: tenant's human users and ServiceAccounts scoped to their own namespace(s) only, no wildcard apiGroups/resources (Part 11)
  • ResourceQuota sized from real projected usage plus HPA headroom, aggregated at the tenant level if the tenant can create more than one namespace
  • Default-deny NetworkPolicy applied, with explicit allow rules for every real dependency including DNS and monitoring scrape traffic (Part 10, Part 11)
  • PriorityClass assigned matching the tenant's tier, if running a tiered platform
  • Admission policy confirms the tenant cannot create cluster-scoped objects (CRDs, ClusterRoles, webhooks) or select a disallowed StorageClass/RuntimeClass
  • Observability isolation confirmed — tenant's Grafana/dashboard access scoped to their own tenant ID at the backend level, not just a dashboard filter
  • GitOps AppProject (or equivalent) scoped to the tenant's own repos and destination namespaces
  • Isolation test suite (previous section) run once against the new tenant before considering onboarding complete

Common Mistakes and Interview Traps#

MistakeWhy it's wrongCorrect approach
Treating a per-namespace ResourceQuota as a hard cap on a team's total cluster consumptionA team with multiple namespaces multiplies their effective allocationAggregate at the tenant level (Capsule's scope: Tenant) if a team can create more than one namespace
Granting broad create RBAC on CustomResourceDefinition to any tenant "to enable self-service"CRDs are cluster-scoped — one tenant's CRD can collide with or shadow another's, or the platform's ownScope CRD lifecycle management to the platform team only; give tenants Custom Resources, not Custom Resource Definitions
Recommending Hierarchical Namespace Controller (HNC) for new platform workThe upstream project is retired/unmaintainedUse a currently-maintained tenant-policy engine (Capsule or similar) for hierarchical-style policy propagation
Applying gVisor/Kata cluster-wide by defaultReal, measurable performance overhead for workloads that never needed the stronger isolation in the first placeScope RuntimeClass selection per-tenant/per-namespace to where kernel-level isolation is genuinely warranted
Assuming a vCluster fully isolates computeThe virtual control plane is isolated; the underlying nodes are still shared unless Dedicated/Private Nodes are explicitly configuredConfirm which isolation tier is actually deployed before making an isolation claim to a tenant or auditor
Billing tenants purely on actual usage, ignoring requested-but-unused capacityRewards over-requesting with no visible cost, undermining the incentive VPA/right-sizing depends onChargeback models should reflect requests, since that's what actually reserves capacity other tenants can't use
Letting a tenant's own operator install a cluster-scoped admission webhookThe webhook's rules/namespaceSelector can intercept every other tenant's objects, not just the installing tenant's ownRestrict ValidatingWebhookConfiguration/MutatingWebhookConfiguration creation to the platform team, same as CRDs
Relying on a Grafana dashboard filter as the only observability tenant boundaryBypassable by anyone with direct query access to the underlying data sourceEnforce tenant scoping at the storage/query backend itself (Mimir/Cortex/Loki native multi-tenancy)

Worked Practice Problems#

Problem 1: A platform team gives every internal team a namespace, RBAC scoped to that namespace, and a ResourceQuota. Six months later, one team's misbehaving custom controller has degraded API server latency for the entire cluster. What isolation gap does this reveal, and which tier in this chapter's spectrum would have prevented it?

Answer: Namespace-based RBAC and quotas cap what a tenant can do inside their own namespace, but they don't isolate shared control-plane resources like API server request capacity — a controller with an inefficient or excessive watch/list pattern degrades API server performance for every tenant sharing that same API server, regardless of namespace boundaries. A vCluster would have given that team their own virtual API server, entirely isolating this specific failure mode from other tenants' experience of the shared cluster (the underlying nodes would still be shared, but the control-plane contention would not be).

Problem 2: A team's platform documentation still recommends installing the Hierarchical Namespace Controller (HNC) for propagating quota and RBAC to child namespaces. What should a reviewer flag, and what should the documentation point to instead?

Answer: HNC is a retired project (per its own GitHub repository status) and shouldn't be recommended as the path for new platform work, regardless of how well-established it appears in older material. The underlying capability it provided — inherited policy propagation across related namespaces — is available today via a currently-maintained tenant-policy engine like Capsule, which implements a comparable model (a Tenant grouping namespaces with propagated RBAC/quota/NetworkPolicy) as an actively maintained alternative.

Problem 3: A tenant with legitimate create permission on PersistentVolumeClaim objects in their namespace repeatedly requests an expensive, high-IOPS StorageClass reserved for a different tier of customer. RBAC review confirms their Role is correctly scoped to only PersistentVolumeClaim resources — no over-broad grant is found. What's the actual gap, and how is it closed?

Answer: RBAC controls which kinds of objects a subject can act on, not what values those objects may contain — a Role correctly scoped to PersistentVolumeClaim still has no vocabulary for restricting which storageClassName a tenant may request within that resource kind. The gap is closed with an admission policy (Kyverno/OPA Gatekeeper, Part 11) validating the PVC's actual field values against the tenant's allowed StorageClasses, layered on top of an otherwise-correct RBAC configuration rather than as a replacement for it.

Problem 4: A platform team decides that, since Capsule already scopes tenants well, they'll skip a fleet-management tool and give their three most sensitive tenants entirely separate physical clusters provisioned by hand. A year later, the three clusters are on different Kubernetes versions and have drifted NetworkPolicy configurations. What was the actual decision the team made without realizing it, and what should they have planned for upfront?

Answer: Choosing separate physical clusters as an isolation tier is really choosing separate clusters plus an ongoing fleet-consistency commitment — without a fleet-management layer (Cluster API, Rancher Fleet, or equivalent GitOps-across-clusters tooling) actively pushing consistent configuration and upgrade schedules to all of them, each cluster inevitably drifts independently over time, since nothing is reconciling them back to a shared baseline. The team should have planned for a fleet-management investment at the same time they chose the separate-clusters isolation tier, not treated it as a simpler, lower-effort option than a shared-cluster approach with vCluster or Capsule.

Summary and What's Next#

Every tier in this chapter is additive, not exclusive — a mature platform typically runs plain namespace isolation as the universal floor for every tenant, layers Capsule-style policy enforcement on top once self-service scale demands it, and reserves vCluster or separate clusters for the specific tenants whose trust level or regulatory requirements genuinely need it, rather than picking exactly one model cluster-wide.

Multi-tenancy is a spectrum, not a single decision — plain namespaces with RBAC, quotas, and NetworkPolicy solve the majority of real internal-platform cases cheaply; Capsule-style tenant-policy engines add self-service at scale; vCluster adds genuine control-plane isolation for tenants that need it; and gVisor/Kata add kernel-level isolation for the specific risk of an actively malicious, not just occasionally buggy, workload. The one consistent lesson across every tier is sizing isolation strength to actual tenant trust and workload needs, rather than defaulting to the strongest (and most expensive) option everywhere.

Part 14 turns to a workload type this chapter's isolation models increasingly need to account for: AI/ML workloads. GPU scheduling, device plugins, and the platform patterns for serving models at scale are, per this series' industry research, the fastest-growing driver of new Kubernetes adoption in 2026 — and several of this chapter's multi-tenancy patterns (chargeback, RuntimeClass-based isolation, dedicated node pools) apply directly to a cluster sharing expensive GPU capacity across teams. GPU capacity in particular sharpens every tradeoff this chapter walked through: a GPU node sits idle at real dollar cost the moment it isn't scheduled, which makes the noisy-neighbor mitigation and cost-chargeback sections above far higher-stakes for a shared GPU pool than for the general-purpose CPU/memory tenancy this chapter mostly illustrated with.