Assumes you're comfortable with ResourceQuota/LimitRange from Part 2, RBAC and NetworkPolicy hardening from Part 11, and admission control from Part 1 — this chapter is about combining those primitives into a real tenant-isolation boundary, not re-introducing any of them individually.
Table of Contents#
- Why This Part Exists
- The Isolation Spectrum: Soft to Hard Multi-Tenancy
- Namespace-Based Soft Multi-Tenancy — the Baseline
- Where Namespace Isolation Breaks Down
- Hierarchical Namespaces — the Idea, and a Retirement Worth Knowing
- Tenant-Aware Policy Engines: Capsule's Tenant Abstraction
- A Worked Capsule Example: Onboarding a New Tenant
- Virtual Clusters — vCluster Architecture Deep Dive
- vCluster Isolation Tiers
- Runtime Isolation for Untrusted Tenants: gVisor and Kata Containers
- Choosing a Multi-Tenancy Model — Decision Framework
- Noisy-Neighbor Mitigation Beyond ResourceQuota
- Multi-Tenant Observability — Isolating Metrics and Logs Per Tenant
- Multi-Tenant GitOps — Scoping Who Can Deploy Where
- A Worked Numeric Example: Sizing Tenant Quotas
- Cost Allocation and Chargeback in a Shared Cluster
- Admission Policy as a Tenant Boundary
- Choosing How Often to Re-Verify
- Testing Tenant Isolation — How to Actually Verify It Holds
- A Full Worked Scenario: Onboarding Three Teams Safely
- When Multi-Tenancy Isn't Enough — Fleet Management as the Escape Hatch
- Part 13 CLI Cheat Sheet
- A Tenant Onboarding Checklist
- Common Mistakes and Interview Traps
- Worked Practice Problems
- Summary and What's Next
Why This Part Exists#
Every mechanism this chapter needs already exists somewhere earlier in this series — ResourceQuota and LimitRange from Part 2, RBAC and NetworkPolicy from Part 11, admission control from Part 1 — but no part so far has addressed the actual platform-engineering question a growing organization eventually has to answer: "how many teams can safely share one cluster, and where does that stop being true?" This is squarely a platform-engineering concern — per the industry research behind this series' chapter list, platform engineering is forecast to reach roughly 80% enterprise adoption, and multi-tenancy design is one of its most concrete, recurring decisions.
The throughline system gains new neighbors in this chapter: alongside the existing checkout team,
imagine a catalog team and a newly onboarded recommendations team, all sharing the same cluster the
checkout namespace already lives in. Every worked example in this chapter treats these three teams as
cooperative but independent — exactly the trust profile most internal platforms are actually designing for.
The Isolation Spectrum: Soft to Hard Multi-Tenancy#
Every multi-tenancy approach sits somewhere on one spectrum: how much of the underlying cluster does a tenant actually get to see and potentially affect, versus how much operational overhead and cost does providing that isolation require.
| Model | What a tenant can see | Typical use |
|---|---|---|
| Plain namespaces + RBAC + quotas | Their own namespace's objects only, but shares one API server, one etcd, one set of CRDs cluster-wide | Trusted internal teams, same organization |
| Namespaces + a tenant-policy engine (Capsule) | Same as above, plus enforced guardrails (network, quota, RBAC templates) applied consistently per tenant | Many internal teams, self-service namespace provisioning |
| Virtual clusters (vCluster) | Their own API server and (virtual) control plane, even their own CRDs — but shares the underlying nodes | Semi-trusted tenants, or teams needing cluster-admin-like self-service without touching the real cluster-admin plane |
| Separate physical clusters | Nothing shared at all except perhaps a network boundary | Regulatory/compliance-mandated hard separation, or genuinely untrusted external tenants |
Important
There is no universally "correct" point on this spectrum — the right answer is always a function of how much you trust the tenants involved and what a worst-case failure in isolation would actually cost. A platform serving mutually trusted internal teams gains little from the overhead of separate clusters; a platform selling isolated environments to external, mutually distrusting customers has a very different risk calculus. Revisit the choice periodically, too — a tenant's trust profile and workload sensitivity can shift well after the original onboarding decision was made.
Namespace-Based Soft Multi-Tenancy — the Baseline#
Every stronger model in this chapter builds on top of this one — a namespace per tenant, RBAC scoping each tenant's ServiceAccounts and human users to their own namespace, a ResourceQuota capping their consumption, and a default-deny NetworkPolicy (Part 11) as the network boundary.
apiVersion: v1
kind: ResourceQuota
metadata:
name: recommendations-quota
namespace: recommendations
spec:
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.cpu: "40"
limits.memory: 80Gi
pods: "50"
persistentvolumeclaims: "10"
services.loadbalancers: "0" # a common tenant restriction: no direct cloud LB provisioningThis baseline is genuinely sufficient for a large fraction of real internal platforms — most organizations aren't isolating mutually hostile parties, they're isolating cooperative teams who occasionally make mistakes, and RBAC plus quotas plus NetworkPolicy stops the overwhelming majority of accidental cross-tenant impact at a fraction of the operational cost of anything stronger.
Where Namespace Isolation Breaks Down#
The honest limitation of the namespace baseline is what it doesn't separate — and every stronger tier in this chapter exists specifically to close one of these gaps.
| Shared resource | Risk if not addressed |
|---|---|
| API server | A tenant's controller/operator with a runaway watch loop, or a burst of API calls, can degrade API server latency for every other tenant sharing it |
| etcd | One tenant creating an enormous number of objects (a CRD instance explosion from a misbehaving operator) consumes shared etcd storage and I/O capacity |
| Cluster-scoped resources (CRDs, StorageClasses, admission webhooks) | A tenant with permission to install a CRD or webhook can affect every other tenant's objects cluster-wide — namespacing doesn't apply to cluster-scoped kinds at all |
| The node's kernel | Every pod on shared nodes runs containers sharing the same host kernel — a container-escape vulnerability in the runtime affects every tenant co-located on that node, not just the compromised one |
| Node-level resource contention | CPU/memory limits (Part 2) prevent a container exceeding its own allocation, but shared node-level resources like disk I/O bandwidth and network throughput are much harder to hard-partition per pod |
Admission webhooks deserve the same scrutiny for an identical reason: a ValidatingWebhookConfiguration
or MutatingWebhookConfiguration is cluster-scoped, and its namespaceSelector/rules determine which
objects across the entire cluster it intercepts — a tenant's own policy engine or operator, installed with
a webhook scoped too broadly, can start validating or mutating every other tenant's objects, not just its
own. This is the same failure shape as the CRD risk above, applied to a different cluster-scoped kind.
Warning
A tenant granted permission to create CustomResourceDefinition objects — even scoped "only in their own
namespace" via RBAC, since CRDs are themselves cluster-scoped regardless of who creates them — can
register a CRD name colliding with, or shadowing, one another tenant or the platform team depends on. This
is a genuinely easy mistake to make when writing "self-service" RBAC for platform teams: verify a tenant's
Role never grants create/update/delete on customresourcedefinitions.apiextensions.k8s.io unless
that's a deliberate, reviewed capability.
Hierarchical Namespaces — the Idea, and a Retirement Worth Knowing#
Hierarchical namespaces let a parent namespace's policies (RBAC, ResourceQuota, NetworkPolicy) propagate automatically to child namespaces, so a platform team can define a tenant's guardrails once at the parent level instead of copy-pasting the same RBAC/quota YAML into every namespace a growing team spins up.
This capability lived in the hierarchical-namespaces (HNC) project from the Kubernetes Working Group for
Multi-Tenancy — worth naming precisely because that repository is now retired, per its own GitHub
archive status, and is not the recommendation to reach for on a new platform build today. The underlying
idea — inherited, hierarchical policy propagation — remains valuable and has been carried forward
primarily by tenant-policy engines like Capsule (next section), which implement a comparable propagation
model as one feature of a broader, more actively maintained tool rather than as a standalone,
now-unmaintained controller.
Note
This is a deliberate accuracy callout, not a minor detail: any tutorial, course, or older blog post still recommending HNC as the current path to hierarchical namespace policy is describing a project that is no longer maintained. Confirm a tool's current maintenance status before adopting it for new platform work — this exact gap between "still frequently recommended in older material" and "actually maintained today" is a recurring trap across the whole Kubernetes ecosystem, not unique to HNC.
Tenant-Aware Policy Engines: Capsule's Tenant Abstraction#
Capsule (a CNCF Sandbox project) introduces a Tenant custom resource that groups one or more namespaces
under a single owner and a single set of enforced policies — the platform team defines a Tenant once, and
Capsule's admission webhooks enforce it consistently across every namespace that tenant creates.
Note
Capsule is a CNCF Sandbox project (an earlier maturity stage than KEDA's Graduated status from Part 12) — actively developed and used in production by real platform teams, but worth knowing precisely where it sits on the CNCF maturity ladder before treating "CNCF project" alone as a maturity signal. Sandbox-stage projects are a reasonable adoption choice for a specific, well-understood problem like this one, but merit a closer look at release cadence and community size than a Graduated project would need.
The self-service angle is the practical win here — without Capsule, a platform team is either the
bottleneck for every new namespace request (they must apply the RBAC/quota/NetworkPolicy boilerplate by
hand each time) or trusts every team with enough cluster-wide permission to create namespaces freely, which
reintroduces the cluster-scoped-resource risk from the previous section. A Tenant object lets the tenant
owner create new namespaces within their own tenant boundary self-service, with Capsule's webhooks
guaranteeing every one of them inherits the same guardrails automatically.
A Worked Capsule Example: Onboarding a New Tenant#
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: recommendations-team
spec:
owners:
- name: recommendations-lead@company.com
kind: User
namespaceOptions:
quota: 5 # max namespaces this tenant can self-service create
resourceQuotas:
scope: Tenant # aggregated across every namespace the tenant owns, not per-namespace
items:
- hard:
requests.cpu: "30"
requests.memory: 60Gi
pods: "80"
networkPolicy:
ingress:
- from:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: recommendations-team # only same-tenant traffic by default# The tenant owner, with no cluster-admin access at all, can now self-service:
kubectl create namespace recommendations-experiments \
--as recommendations-lead@company.com
# Capsule's webhook validates this against the Tenant's namespaceOptions.quota
# and automatically applies the Tenant's RBAC/quota/NetworkPolicy templatesThe scope: Tenant ResourceQuota aggregation is worth calling out specifically — a per-namespace
ResourceQuota (the Part 2 mechanism) caps each namespace independently, which a tenant with
namespaceOptions.quota: 5 could otherwise use to multiply their effective total allocation by simply
creating more namespaces. Tenant-scoped aggregation closes that loophole by summing usage across every
namespace the tenant owns against one shared ceiling.
Virtual Clusters — vCluster Architecture Deep Dive#
A vCluster runs an entire, real, additional Kubernetes control plane — its own API server, its own etcd (or a lightweight equivalent), its own controller manager — inside a single namespace of the underlying "host" cluster, giving a tenant something that looks and behaves like their own dedicated cluster while still sharing the host cluster's actual compute nodes.
The "sync" step is the architectural core of how this works: a tenant creates a Pod against the virtual
API server exactly as if it were a real, independent cluster; vCluster's synchronization controller
translates that object down into a real Pod on the host cluster (in the vCluster's backing namespace),
where it actually runs on shared physical nodes. The tenant never sees the host cluster's API server,
other tenants' namespaces, or any cluster-scoped objects outside their own virtual control plane — from
inside a vCluster, even kubectl get namespaces only shows namespaces the tenant themselves created,
solving the cluster-scoped-resource visibility gap that plain namespace isolation cannot.
vCluster Isolation Tiers#
vCluster's isolation strength is not one fixed level — it's configurable per deployment, trading cost against isolation strength the same way the isolation-spectrum diagram earlier in this chapter suggested.
| Tier | Node sharing | Best for |
|---|---|---|
| Shared Nodes | Tenants' pods run on the same physical nodes as other tenants, same kernel | Trusted tenants — internal dev/CI/CD environments, cost-sensitive internal platforms |
| Dedicated Nodes | A labeled, reserved subset of the host cluster's nodes, via node selectors/taints | Tenants needing predictable, uncontended compute without provisioning entirely separate infrastructure |
| Private Nodes | Nodes enrolled specifically for one tenant, with network/storage/compute isolated at the infrastructure level | Semi-trusted or regulatory-sensitive tenants where "shares a kernel with someone else" is unacceptable |
Each tier only changes the underlying compute-sharing story — the virtual control plane's own isolation (separate API server, separate etcd, invisible host-cluster internals) is identical across all three. This means a platform team can start a tenant on Shared Nodes for cost efficiency and move them to Dedicated or Private Nodes later purely as an infrastructure change, without the tenant's own workload manifests, RBAC, or CRDs needing to change at all — the virtual cluster they interact with looks the same regardless of which physical isolation tier backs it.
Runtime Isolation for Untrusted Tenants: gVisor and Kata Containers#
Every model so far still shares the host kernel for at least some tier of tenant — gVisor and Kata Containers address the "shared kernel" risk specifically, by changing what actually executes a container's syscalls.
| Runtime | Isolation mechanism | Overhead |
|---|---|---|
Standard (runc) | None beyond namespaces/cgroups — shares the host kernel directly | Lowest |
gVisor (runsc) | Intercepts and re-implements syscalls in a userspace "sentry" process, presenting a much narrower attack surface to the real host kernel | Moderate — some syscall-heavy workloads see a real performance cost |
| Kata Containers | Each pod runs inside its own lightweight virtual machine with a genuinely separate kernel | Highest — closest to true hardware-level isolation, at real memory/startup-time cost |
A third option worth naming alongside these two: AWS Fargate for EKS (Part 7) and similar
serverless-container offerings use a comparable microVM-per-pod model (built on the same Firecracker
technology Kata Containers can also use as a backend) as their default isolation posture, not an opt-in
RuntimeClass — a relevant data point when comparing self-managed Kata adoption against simply running
sensitive tenant workloads on a managed serverless-container tier instead.
These are selected per-workload via a RuntimeClass object, not cluster-wide — a platform can run most
pods on standard runc for performance and reserve gVisor or Kata specifically for the namespaces/tenants
where "this tenant's code might be actively malicious, not just occasionally buggy" is a genuine
possibility, such as a platform running arbitrary customer-submitted code.
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
---
apiVersion: v1
kind: Pod
spec:
runtimeClassName: gvisor # opts this specific pod into the stronger runtimeChoosing a Multi-Tenancy Model — Decision Framework#
| Choose... | When |
|---|---|
| Plain namespaces + RBAC + quotas + NetworkPolicy | Trusted internal teams, cooperative, no regulatory isolation mandate |
| Namespaces + Capsule (or a similar tenant-policy engine) | Same trust level, but self-service namespace provisioning at a scale where hand-applying guardrails per team doesn't scale |
| vCluster, Shared Nodes | A tenant genuinely needs cluster-admin-like control (their own CRDs, their own controllers) without touching the real cluster's control plane |
| vCluster, Dedicated/Private Nodes | Same as above, plus a requirement for predictable or fully isolated compute |
| gVisor/Kata RuntimeClass, layered on any of the above | The specific risk is kernel-level container escape — a platform running genuinely untrusted or externally-submitted code |
| Fully separate physical clusters | A hard regulatory/compliance boundary, or tenants who must never share even a virtual control plane's underlying host |
Noisy-Neighbor Mitigation Beyond ResourceQuota#
ResourceQuota caps a tenant's total declared consumption, but says nothing about where their pods land or how they're prioritized against other tenants' pods under real contention — three more primitives, all covered individually in earlier parts, combine to close that gap.
# 1. A PriorityClass per tenant tier — determines who gets evicted first
# under real node pressure (Part 2's preemption mechanics)
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: tenant-standard
value: 100
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: tenant-premium
value: 1000
---
# 2. Taints reserving a node pool for one tenant tier only
# kubectl taint nodes -l tenant-tier=premium dedicated=premium-tenants:NoSchedule| Primitive | What it adds beyond ResourceQuota | Covered in |
|---|---|---|
PriorityClass per tenant tier | Under real node memory/CPU pressure, lower-priority tenants' pods are evicted first, protecting higher-tier tenants' availability | Part 2 |
| Taints/tolerations for tenant-dedicated node pools | Physically separates a tenant's pods onto specific nodes, reducing (though not eliminating, on Shared Nodes) contention with other tenants | Part 2 |
| Pod anti-affinity spreading a tenant's own replicas | Prevents one tenant's own workload from concentrating on a single node and becoming a single point of resource contention for whoever else lands there | Part 2 |
| Topology spread constraints across tenant node pools | Ensures a tenant's replicas spread evenly, rather than clustering | Part 2 |
Combining these with the ResourceQuota baseline turns "no single tenant can request too much in total" into "no single tenant can crowd out another tenant's pods when the cluster is genuinely under pressure" — a meaningfully stronger guarantee, since quotas alone say nothing about eviction order or physical placement once contention actually happens.
Multi-Tenant Observability — Isolating Metrics and Logs Per Tenant#
A shared Prometheus/Loki stack scraping every tenant's namespace by default creates two problems this chapter's isolation model needs to answer: can Tenant A query Tenant B's metrics/logs, and can Tenant B's high-cardinality metrics degrade query performance for everyone else sharing that same backend?
| Layer | How tenant isolation is enforced |
|---|---|
| Ingestion | A label (tenant or the namespace itself) is attached to every metric series/log line as it's collected — this must happen at ingestion, not query time, since query-time filtering alone can't stop a tenant from ever seeing another's raw data if the backend doesn't enforce it structurally |
| Storage/query (Mimir, Cortex, Thanos, Loki) | These systems support native multi-tenancy via an X-Scope-OrgID header — each tenant's queries are scoped to their own tenant ID at the storage-query layer itself, not just via a Grafana dashboard filter that a determined user could bypass |
| Dashboards | Grafana's own per-tenant data source/folder permissions, layered on top of the backend's own tenant scoping — defense in depth, not the primary control |
# A query scoped to exactly one tenant via Loki's native multi-tenancy header —
# the backend rejects this if the caller's auth doesn't match the claimed tenant
curl -H "X-Scope-OrgID: recommendations" \
http://loki.observability.svc.cluster.local:3100/loki/api/v1/query \
--data-urlencode 'query={namespace="recommendations"}'Caution
Relying on a Grafana dashboard filter (e.g. a templated namespace variable) as the only tenant
isolation for observability data is not real isolation — anyone with raw query access to the underlying
data source (Prometheus's own API, a shared Loki endpoint) can bypass a dashboard-level filter entirely.
Genuine tenant isolation for observability data must be enforced at the storage/query backend itself
(Mimir/Cortex/Loki's native multi-tenancy), the same principle as never relying on a UI-level control for
something RBAC or a NetworkPolicy should actually enforce.
Multi-Tenant GitOps — Scoping Who Can Deploy Where#
A shared cluster running GitOps (the automation-cicd-gitops tutorial series covers GitOps mechanics in
full — this section is specifically the multi-tenancy angle) needs the deployment pipeline itself scoped
per tenant, or GitOps becomes a way to bypass every RBAC boundary this chapter has built.
# ArgoCD AppProject — scopes what a tenant's own Application objects
# are allowed to deploy, and where
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: recommendations-team
namespace: argocd
spec:
sourceRepos:
- "https://github.com/company/recommendations-*"
destinations:
- namespace: "recommendations*"
server: "https://kubernetes.default.svc"
clusterResourceWhitelist: [] # deliberately empty — no cluster-scoped resources allowed
namespaceResourceBlacklist:
- group: ""
kind: ResourceQuota # tenants can't self-service raise their own quota via GitOpsThe clusterResourceWhitelist: [] and quota-blacklist lines matter for exactly the same reason RBAC
should never grant CRD lifecycle permissions to a tenant — without them, a tenant's Git repository
becomes an alternate, unmonitored path to the same over-privileged actions this chapter has spent several
sections closing off through RBAC and admission policy. GitOps doesn't bypass Kubernetes RBAC on its own
(ArgoCD still applies objects through the API server as whatever identity it's configured to run as), but a
single shared, overly-permissive ArgoCD ServiceAccount used across every tenant's AppProject re-creates
exactly the over-broad-grant risk this whole chapter exists to avoid — scoping AppProject per tenant is
what keeps GitOps consistent with the rest of the tenant boundary rather than quietly undermining it.
A Worked Numeric Example: Sizing Tenant Quotas#
Abstract guidance ("set a reasonable quota") is less useful than working through the actual math a
platform team would do — here's a realistic sizing exercise for the recommendations team's ResourceQuota
introduced earlier in this chapter.
The team runs 3 services, each currently deployed at 4 replicas with 500m CPU / 1Gi memory requests per pod, plus headroom for Part 12's HPA to scale up to double that under load:
Baseline: 3 services × 4 replicas × 500m CPU = 6 CPU, × 1Gi memory = 12Gi memory
HPA headroom (up to 2x under load): 12 CPU, 24Gi memory
Platform buffer (batch jobs, one-off debug pods, ~20%): +2.4 CPU, +4.8Gi memory
spec:
hard:
requests.cpu: "15" # rounded up from 14.4
requests.memory: 30Gi # rounded up from 28.8Gi
limits.cpu: "20" # limits set higher than requests per Part 2's Burstable QoS pattern
limits.memory: 32GiSizing a quota from a real HPA ceiling, not a guess, is the actual discipline here — a quota set below
what a tenant's own HPA maxReplicas could legitimately request under real load doesn't stop overspend, it
just relocates the failure: instead of a clean cost conversation, the tenant's pods start failing
scheduling (Part 10's Pending-pod diagnostic tree) the moment real traffic actually needs to scale, which
reads to the tenant as a platform outage rather than the capacity-planning gap it actually is.
Cost Allocation and Chargeback in a Shared Cluster#
A shared cluster's bill is one number from the cloud provider — attributing it back to individual tenants requires tagging and aggregation that Kubernetes doesn't provide natively.
kubectl get resourcequota -A -o json | \
jq -r '.items[] | "\(.metadata.namespace): \(.status.used["requests.cpu"]) / \(.status.hard["requests.cpu"]) CPU"'| Approach | How it attributes cost |
|---|---|
| Manual: aggregate ResourceQuota usage per namespace/Tenant | Cheap to start, but only proxies actual cost via requested (not necessarily used) resources |
| A cost-visibility tool (Kubecost or similar) | Combines actual node cost with real per-pod resource usage for a genuine per-tenant/per-namespace dollar figure, including shared control-plane overhead allocation |
Capsule Tenant-scoped quota reporting | Aggregates naturally per tenant across every namespace they own, matching how the tenant boundary is actually defined |
Tip
Resource requests, not actual usage, is what most chargeback models bill against — this creates a direct incentive for a tenant to right-size their requests down (Part 12's VPA is directly relevant here) rather than defensively over-requesting "to be safe," since over-requesting under a chargeback model has a real, visible cost consequence that a purely technical ResourceQuota ceiling alone doesn't create.
Admission Policy as a Tenant Boundary#
Part 11 covered Kyverno/OPA Gatekeeper for security policy — the same admission-control mechanism is equally central to multi-tenancy, enforcing rules that RBAC alone can't express.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-storageclass-per-tenant
spec:
validationFailureAction: Enforce
rules:
- name: only-standard-storageclass-for-non-platform-tenants
match:
any:
- resources:
kinds: ["PersistentVolumeClaim"]
namespaces: ["catalog", "recommendations", "checkout"]
validate:
message: "Tenant namespaces may only use the 'standard' StorageClass"
pattern:
spec:
storageClassName: "standard"RBAC answers "who can create what kind of object" — it has no vocabulary for "what values are allowed
inside that object once created," which is exactly the gap a policy engine fills. RBAC alone cannot stop
a tenant with legitimate create permission on PersistentVolumeClaim objects from requesting an
expensive, high-performance StorageClass never intended for their tier — only an admission policy
inspecting the object's actual field values can enforce that kind of tenant-tier boundary.
Choosing How Often to Re-Verify#
A one-time isolation test at onboarding proves the boundary was correct on day one — it says nothing about whether a subsequent, unrelated platform change quietly reopened a gap months later.
| Trigger for re-running the isolation test suite | Why |
|---|---|
| Any RBAC change (new ClusterRole, edited aggregation rule) | The single most common source of an accidentally widened grant |
| Any NetworkPolicy change in a shared/platform namespace | A relaxed rule for one debugging session that never gets reverted is a genuinely common real-world gap |
| A new admission webhook or CRD installed cluster-wide | Both are cluster-scoped by nature — a new one can unintentionally intersect with an existing tenant boundary |
| On a fixed cadence regardless of known changes | Catches drift from changes nobody remembered to flag as isolation-relevant in the first place |
Treating this as a scheduled, low-effort recurring check (the small script from the previous section, run on a cadence or wired into the same pipeline pattern Part 15 covers for cluster upgrades) costs far less than discovering an isolation gap the hard way, during an actual cross-tenant incident.
Testing Tenant Isolation — How to Actually Verify It Holds#
Every control in this chapter can be configured correctly on paper and still fail in practice — the only way to know tenant isolation actually holds is to actively try to violate it, the same "verify, don't just configure" discipline Part 11 applied to security controls generally.
# 1. Cross-namespace RBAC check: can tenant A's ServiceAccount read tenant B's Secrets?
kubectl auth can-i get secrets \
--as=system:serviceaccount:recommendations:default -n checkout
# 2. Cross-namespace network check: does NetworkPolicy actually block traffic
# from a pod outside the expected tenant boundary?
kubectl run isolation-test --rm -it --image=busybox -n catalog \
-- wget -T 3 -O- http://checkout-svc.checkout.svc.cluster.local
# 3. Quota-bypass check: can the tenant exceed their aggregate quota by
# creating an additional namespace, if Tenant-scoped aggregation isn't enforced?
kubectl create namespace recommendations-shadow --as recommendations-lead@company.com| Test | What a failure would mean |
|---|---|
kubectl auth can-i cross-tenant, using --as to impersonate the other tenant's ServiceAccount | An RBAC grant is broader than intended — the single most common isolation gap, and the cheapest one to check |
A NetworkPolicy cross-namespace connectivity attempt that should be blocked | The default-deny baseline has a gap — often a missing policyTypes entry, or a policy that was applied but never actually took effect (Part 10's NetworkPolicy troubleshooting applies directly here) |
| Attempting to exceed tenant-aggregate quota via a second self-service namespace | Confirms whether Tenant-scoped aggregation (Capsule) is actually wired up, versus quotas silently reverting to per-namespace only |
Attempting to install a CustomResourceDefinition or cluster-scoped object as a tenant identity | Confirms the RBAC gap flagged earlier in this chapter (tenants should never have CRD lifecycle permissions) isn't present |
Tip
Turn this into a small, repeatable test suite (a script or a CI job run against a staging cluster after every RBAC/NetworkPolicy/Capsule policy change) rather than a one-time manual check — tenant isolation is exactly the kind of guarantee that silently regresses when an unrelated change (a new ClusterRole, a relaxed NetworkPolicy for an unrelated debugging session that never got reverted) accidentally reopens a gap that was correctly closed months earlier.
A Full Worked Scenario: Onboarding Three Teams Safely#
Bringing every tier in this chapter together for a platform team onboarding checkout, catalog, and a
brand-new recommendations team onto one shared cluster:
Working through the actual decision for recommendations: it's a standard internal service, doesn't
need its own CRDs, and the team is cooperative with no regulatory sensitivity — the correct-sized answer is
the cheapest one on the isolation spectrum that still solves the real problem: a Capsule Tenant wrapping
one or more namespaces, with the standard ResourceQuota/RBAC/NetworkPolicy guardrails from earlier in this
chapter. Reaching for a vCluster or a separate physical cluster here would be over-engineering relative to
the actual risk being managed — a useful, generally applicable lesson: isolation strength should be sized
to the tenant's actual trust level and workload needs, not applied uniformly at the strongest tier "to be
safe" everywhere.
When Multi-Tenancy Isn't Enough — Fleet Management as the Escape Hatch#
Every tier this chapter covers still shares something — at minimum, the cloud account and billing boundary. When even a Private-Nodes vCluster or a fully separate physical cluster is the right call per the decision framework above, the new problem becomes operating potentially dozens of separate clusters consistently — which is exactly what fleet-management tooling exists to solve.
| Tool | Role |
|---|---|
| Cluster API (CAPI) | Declaratively provisions and lifecycle-manages entire clusters (not just objects within one) via Kubernetes-native CRDs — a Cluster/MachineDeployment object per tenant cluster, reconciled the same way a Deployment reconciles pods |
| Rancher Fleet | GitOps specifically aimed at pushing consistent configuration (policies, workloads, upgrades) across a fleet of many clusters from one control point |
| A cloud provider's own fleet console (e.g. GKE Fleet, EKS with multiple clusters under one Organization) | Centralized visibility and some policy consistency, without necessarily the full declarative lifecycle management CAPI provides |
| Cluster upgrade orchestration across a fleet (Part 15) | Rolling out a version upgrade consistently across dozens of separate tenant clusters is itself a scheduling problem — a fleet tool sequences it deliberately rather than leaving each cluster's owner to upgrade on their own timeline |
The tradeoff this escape hatch makes explicit: separate clusters solve the "shares nothing" isolation requirement completely, but only if the fleet-management layer actually keeps every cluster consistently configured — without one, "give every tenant their own cluster" quietly becomes "give every tenant their own snowflake cluster that drifts further from every other one over time," which is a worse operational outcome than the shared-cluster noisy-neighbor problems this chapter otherwise solves. Choosing separate clusters as an isolation tier is really choosing separate clusters plus a real fleet-management investment — treating it as a lighter-weight option than vCluster because "it's just normal clusters" is a common and costly underestimate.
Part 13 CLI Cheat Sheet#
| Command | Purpose |
|---|---|
kubectl get resourcequota -A -o json | jq ... | Aggregate per-namespace consumption for a cost/chargeback estimate |
kubectl get tenants (Capsule CRD) | List every tenant a Capsule-managed cluster currently has onboarded |
kubectl create namespace <ns> --as <tenant-owner> | Test that a tenant's self-service namespace creation is correctly scoped by Capsule's webhook |
kubectl get runtimeclass | Confirm which RuntimeClasses (standard, gVisor, Kata) are available for tenant-tier assignment |
vcluster create <name> -n <host-namespace> | Provision a new virtual cluster for a tenant needing control-plane isolation |
vcluster connect <name> | Retrieve a kubeconfig scoped to one tenant's virtual cluster |
kubectl get networkpolicy -A | Audit that every tenant namespace actually has its expected default-deny + allow rules in place |
argocd appproject get <tenant> | Confirm a tenant's GitOps AppProject scoping matches what RBAC/Capsule already enforce |
A Tenant Onboarding Checklist#
Every control this chapter covers, as a single checklist a platform team can run through for each new tenant — the kind of artifact worth keeping next to Part 10's troubleshooting runbook.
- Namespace(s) created, scoped under a
Tenantobject if using Capsule (or an equivalent grouping) - RBAC: tenant's human users and ServiceAccounts scoped to their own namespace(s) only, no wildcard
apiGroups/resources(Part 11) -
ResourceQuotasized from real projected usage plus HPA headroom, aggregated at the tenant level if the tenant can create more than one namespace - Default-deny
NetworkPolicyapplied, with explicit allow rules for every real dependency including DNS and monitoring scrape traffic (Part 10, Part 11) -
PriorityClassassigned matching the tenant's tier, if running a tiered platform - Admission policy confirms the tenant cannot create cluster-scoped objects (CRDs, ClusterRoles,
webhooks) or select a disallowed
StorageClass/RuntimeClass - Observability isolation confirmed — tenant's Grafana/dashboard access scoped to their own tenant ID at the backend level, not just a dashboard filter
- GitOps
AppProject(or equivalent) scoped to the tenant's own repos and destination namespaces - Isolation test suite (previous section) run once against the new tenant before considering onboarding complete
Common Mistakes and Interview Traps#
| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Treating a per-namespace ResourceQuota as a hard cap on a team's total cluster consumption | A team with multiple namespaces multiplies their effective allocation | Aggregate at the tenant level (Capsule's scope: Tenant) if a team can create more than one namespace |
Granting broad create RBAC on CustomResourceDefinition to any tenant "to enable self-service" | CRDs are cluster-scoped — one tenant's CRD can collide with or shadow another's, or the platform's own | Scope CRD lifecycle management to the platform team only; give tenants Custom Resources, not Custom Resource Definitions |
| Recommending Hierarchical Namespace Controller (HNC) for new platform work | The upstream project is retired/unmaintained | Use a currently-maintained tenant-policy engine (Capsule or similar) for hierarchical-style policy propagation |
| Applying gVisor/Kata cluster-wide by default | Real, measurable performance overhead for workloads that never needed the stronger isolation in the first place | Scope RuntimeClass selection per-tenant/per-namespace to where kernel-level isolation is genuinely warranted |
| Assuming a vCluster fully isolates compute | The virtual control plane is isolated; the underlying nodes are still shared unless Dedicated/Private Nodes are explicitly configured | Confirm which isolation tier is actually deployed before making an isolation claim to a tenant or auditor |
| Billing tenants purely on actual usage, ignoring requested-but-unused capacity | Rewards over-requesting with no visible cost, undermining the incentive VPA/right-sizing depends on | Chargeback models should reflect requests, since that's what actually reserves capacity other tenants can't use |
| Letting a tenant's own operator install a cluster-scoped admission webhook | The webhook's rules/namespaceSelector can intercept every other tenant's objects, not just the installing tenant's own | Restrict ValidatingWebhookConfiguration/MutatingWebhookConfiguration creation to the platform team, same as CRDs |
| Relying on a Grafana dashboard filter as the only observability tenant boundary | Bypassable by anyone with direct query access to the underlying data source | Enforce tenant scoping at the storage/query backend itself (Mimir/Cortex/Loki native multi-tenancy) |
Worked Practice Problems#
Problem 1: A platform team gives every internal team a namespace, RBAC scoped to that namespace, and a ResourceQuota. Six months later, one team's misbehaving custom controller has degraded API server latency for the entire cluster. What isolation gap does this reveal, and which tier in this chapter's spectrum would have prevented it?
Answer: Namespace-based RBAC and quotas cap what a tenant can do inside their own namespace, but they don't isolate shared control-plane resources like API server request capacity — a controller with an inefficient or excessive watch/list pattern degrades API server performance for every tenant sharing that same API server, regardless of namespace boundaries. A vCluster would have given that team their own virtual API server, entirely isolating this specific failure mode from other tenants' experience of the shared cluster (the underlying nodes would still be shared, but the control-plane contention would not be).
Problem 2: A team's platform documentation still recommends installing the Hierarchical Namespace Controller (HNC) for propagating quota and RBAC to child namespaces. What should a reviewer flag, and what should the documentation point to instead?
Answer: HNC is a retired project (per its own GitHub repository status) and shouldn't be recommended as
the path for new platform work, regardless of how well-established it appears in older material. The
underlying capability it provided — inherited policy propagation across related namespaces — is available
today via a currently-maintained tenant-policy engine like Capsule, which implements a comparable model
(a Tenant grouping namespaces with propagated RBAC/quota/NetworkPolicy) as an actively maintained
alternative.
Problem 3: A tenant with legitimate create permission on PersistentVolumeClaim objects in their
namespace repeatedly requests an expensive, high-IOPS StorageClass reserved for a different tier of
customer. RBAC review confirms their Role is correctly scoped to only PersistentVolumeClaim resources —
no over-broad grant is found. What's the actual gap, and how is it closed?
Answer: RBAC controls which kinds of objects a subject can act on, not what values those objects may
contain — a Role correctly scoped to PersistentVolumeClaim still has no vocabulary for restricting which
storageClassName a tenant may request within that resource kind. The gap is closed with an admission
policy (Kyverno/OPA Gatekeeper, Part 11) validating the PVC's actual field values against the tenant's
allowed StorageClasses, layered on top of an otherwise-correct RBAC configuration rather than as a
replacement for it.
Problem 4: A platform team decides that, since Capsule already scopes tenants well, they'll skip a fleet-management tool and give their three most sensitive tenants entirely separate physical clusters provisioned by hand. A year later, the three clusters are on different Kubernetes versions and have drifted NetworkPolicy configurations. What was the actual decision the team made without realizing it, and what should they have planned for upfront?
Answer: Choosing separate physical clusters as an isolation tier is really choosing separate clusters plus an ongoing fleet-consistency commitment — without a fleet-management layer (Cluster API, Rancher Fleet, or equivalent GitOps-across-clusters tooling) actively pushing consistent configuration and upgrade schedules to all of them, each cluster inevitably drifts independently over time, since nothing is reconciling them back to a shared baseline. The team should have planned for a fleet-management investment at the same time they chose the separate-clusters isolation tier, not treated it as a simpler, lower-effort option than a shared-cluster approach with vCluster or Capsule.
Summary and What's Next#
Every tier in this chapter is additive, not exclusive — a mature platform typically runs plain namespace isolation as the universal floor for every tenant, layers Capsule-style policy enforcement on top once self-service scale demands it, and reserves vCluster or separate clusters for the specific tenants whose trust level or regulatory requirements genuinely need it, rather than picking exactly one model cluster-wide.
Multi-tenancy is a spectrum, not a single decision — plain namespaces with RBAC, quotas, and NetworkPolicy solve the majority of real internal-platform cases cheaply; Capsule-style tenant-policy engines add self-service at scale; vCluster adds genuine control-plane isolation for tenants that need it; and gVisor/Kata add kernel-level isolation for the specific risk of an actively malicious, not just occasionally buggy, workload. The one consistent lesson across every tier is sizing isolation strength to actual tenant trust and workload needs, rather than defaulting to the strongest (and most expensive) option everywhere.
Part 14 turns to a workload type this chapter's isolation models increasingly need to account for: AI/ML workloads. GPU scheduling, device plugins, and the platform patterns for serving models at scale are, per this series' industry research, the fastest-growing driver of new Kubernetes adoption in 2026 — and several of this chapter's multi-tenancy patterns (chargeback, RuntimeClass-based isolation, dedicated node pools) apply directly to a cluster sharing expensive GPU capacity across teams. GPU capacity in particular sharpens every tradeoff this chapter walked through: a GPU node sits idle at real dollar cost the moment it isn't scheduled, which makes the noisy-neighbor mitigation and cost-chargeback sections above far higher-stakes for a shared GPU pool than for the general-purpose CPU/memory tenancy this chapter mostly illustrated with.