Projects, Security Context Constraints & Multi-Tenancy

Table of Contents#

1. What Multi-Tenancy Actually Requires
2. Projects vs. Namespaces — What a Project Actually Adds
3. Project Templates and What Gets Created by Default
4. A Worked Example: Creating and Inspecting a New Project
5. RBAC in OpenShift — Roles, ClusterRoles, and the Default Set
6. Self-Provisioning and Restricting It
7. Security Context Constraints — The Core Model
8. The Default SCC Roster
9. How SCC Admission Actually Works: Priority and "Most Restrictive That Fits"
10. Namespace UID Ranges — Why Containers Rarely Run as a Fixed UID
11. SCCs vs. Pod Security Standards — Two Overlapping Models
12. A Worked Example: Diagnosing and Fixing an SCC Denial
13. Granting a Workload a Less-Restrictive SCC, Safely
14. ResourceQuotas and LimitRanges in Depth
15. Multi-Tenancy Patterns: Namespace-per-Team vs. Cluster-per-Team
16. Default Network Isolation Between Projects
17. Chargeback: Attributing Shared Cluster Cost to Projects
18. Quick Reference: Key Terms From This Chapter
19. Common Mistakes and Interview Traps
20. Worked Practice Problems
21. Summary and What's Next

What Multi-Tenancy Actually Requires#

Part 1 closed with a deliberate hand-off: everything from here on lives above the RHCOS/CVO/MCO line, in the territory a developer or a namespace-level administrator actually touches. This chapter starts at the layer that makes "many teams safely sharing one cluster" possible at all, because it's the layer where OpenShift's opinionated defaults are most visible and most consequential.

Real multi-tenancy — multiple teams, each trusted with their own workloads but not with each other's, sharing one cluster — needs at least four things working together: an isolation boundary for objects and RBAC (a namespace), a way to bound what a workload can consume (quotas and limits), a way to bound what a workload can do at the kernel level regardless of who deployed it (a pod-security model), and default network isolation so one team's pods can't simply talk to another team's pods just because they happen to share a cluster. Vanilla Kubernetes ships the primitives for all four — Namespace, ResourceQuota, Pod Security Admission, NetworkPolicy — but ships none of them wired together by default; a bare kubectl create namespace produces an empty namespace with no quota, no security floor, and (depending on the CNI) no network isolation at all. OpenShift's Project is the object that bundles all four together as a single, opinionated default the moment a namespace is created.

Every row's right-hand column exists specifically because OpenShift ships an opinionated default for it; the rest of this chapter walks through each pillar in the order a platform team actually encounters them — starting from the namespace boundary itself.

From the Trenches: A platform team migrating from a shared vanilla Kubernetes cluster to OpenShift initially treated the migration as "same objects, different UI," and was surprised when their existing container images — which ran fine on their old cluster, all as UID 0 — failed to schedule at all after the move. The old cluster had no Pod Security Admission configured, so nothing had ever stopped those images from running as root; OpenShift's default Security Context Constraint rejected every one of them immediately, on the very first deploy attempt. This wasn't a bug in the migration — it was the multi-tenancy floor this chapter describes doing exactly what it's designed to do, surfacing a security gap the old cluster had silently tolerated for years.

This is also why "migrate to OpenShift" and "adopt OpenShift's multi-tenancy model" are not two separate, sequenceable projects — the moment workloads land on the cluster, every mechanism in this chapter is already active by default, whether or not the migrating team has budgeted time to understand it first. Reading this chapter before a migration, not during an incident caused by one, is the cheaper way to encounter these defaults.

Projects vs. Namespaces — What a Project Actually Adds#

A Project is a Namespace — oc get namespace <name> and oc get project <name> return the same underlying object, and any tool that only understands vanilla Kubernetes namespaces (a Helm chart, a raw manifest, a third-party Operator) works against a Project without modification. What OpenShift adds sits entirely in what happens automatically when a Project is created, and in a small set of additional annotations and RBAC bindings layered onto that namespace.

That last row is worth sitting with: allowing arbitrary users to create Kubernetes Namespace objects directly on vanilla Kubernetes requires a cluster-scoped RBAC grant that also happens to imply broad namespace-level power (namespace deletion, in particular, is a genuinely dangerous permission to hand out cluster-wide). OpenShift's ProjectRequest API is a narrower, purpose-built self-service surface — a user can request a new project for themselves without ever being granted the broader Namespace verbs a vanilla-Kubernetes equivalent would require.

Project Templates and What Gets Created by Default#

The concrete objects a new Project receives are defined by a project template — a cluster-wide Template object (default name project-request) that a cluster administrator can customize to inject organization-specific defaults. A representative default project template's contents:

apiVersion: template.openshift.io/v1
kind: Template
metadata:
  name: project-request
objects:
  - apiVersion: project.openshift.io/v1
    kind: Project
    metadata:
      annotations:
        openshift.io/description: "${PROJECT_DESCRIPTION}"
        openshift.io/display-name: "${PROJECT_DISPLAYNAME}"
      name: "${PROJECT_NAME}"
  - apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: default-quota
      namespace: "${PROJECT_NAME}"
    spec:
      hard:
        requests.cpu: "4"
        requests.memory: 8Gi
        limits.cpu: "8"
        limits.memory: 16Gi
        pods: "50"
  - apiVersion: v1
    kind: LimitRange
    metadata:
      name: default-limits
      namespace: "${PROJECT_NAME}"
    spec:
      limits:
        - type: Container
          defaultRequest: { cpu: 100m, memory: 128Mi }
          default: { cpu: 500m, memory: 512Mi }
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: admin
      namespace: "${PROJECT_NAME}"
    roleRef:
      kind: ClusterRole
      name: admin
    subjects:
      - kind: User
        name: "${PROJECT_ADMIN_USER}"
parameters:
  - name: PROJECT_NAME
  - name: PROJECT_DISPLAYNAME
  - name: PROJECT_DESCRIPTION
  - name: PROJECT_ADMIN_USER

A cluster administrator installs a customized version of this template with oc create -f template.yaml -n openshift-config and points the cluster's Project configuration resource at it, so every future oc new-project picks up the organization's own quota defaults, required labels, or additional NetworkPolicies — without every requesting user needing to know any of that exists.

Customizing the Template Cluster-Wide#

The full workflow for putting a customized template into effect starts from the cluster's actual default, rather than writing one from scratch:

# Export the cluster's current default as a starting point
oc adm create-bootstrap-project-template -o yaml > template.yaml

# Edit template.yaml — add organization-specific ResourceQuota values,
# required labels, additional NetworkPolicies, etc.

oc create -f template.yaml -n openshift-config

# Point the cluster at it
oc edit project.config.openshift.io/cluster

apiVersion: config.openshift.io/v1
kind: Project
metadata:
  name: cluster
spec:
  projectRequestTemplate:
    name: project-request

Every oc new-project from that point forward uses the customized template — a one-time platform-team change that applies retroactively to every future Project, with zero action required from any individual requester.

Project Lifecycle: Deletion Isn't Always Instant#

Deleting a Project (oc delete project payments-dev) doesn't remove it immediately — it moves to a Terminating phase while the namespace controller works through every object inside it, and a Project can stay Terminating indefinitely if one of its objects has a finalizer that never completes (a PersistentVolumeClaim whose underlying storage backend is unreachable, or a custom resource whose owning Operator has itself been uninstalled and can no longer process the finalizer). oc get project payments-dev -o yaml showing a non-empty status.conditions naming a specific stuck resource is the fast path to diagnosing this, rather than assuming deletion is simply slow and waiting indefinitely.

A Worked Example: Creating and Inspecting a New Project#

oc new-project payments-dev \
  --display-name="Payments Team - Dev" \
  --description="Development environment for the Payments squad"

# Confirm what actually landed
oc get resourcequota,limitrange -n payments-dev
oc get rolebindings -n payments-dev
oc get project payments-dev -o yaml | grep -A2 "sa.scc"

That last command surfaces two annotations worth understanding on sight, since they're the mechanism the UID-range and SCC sections later in this chapter depend on:

metadata:
  annotations:
    openshift.io/sa.scc.uid-range: 1000600000/10000
    openshift.io/sa.scc.mcs: s0:c26,c10

Every Project gets its own dedicated slice of the UID space and its own SELinux Multi-Category Security (MCS) label the moment it's created — a concrete, automatic isolation guarantee this chapter returns to in depth once SCCs are introduced, and one no bare kubectl create namespace provides on its own.

RBAC in OpenShift — Roles, ClusterRoles, and the Default Set#

RBAC itself is unmodified upstream Kubernetes, and the four core objects are worth naming precisely before the OpenShift-specific additions on top of them: a Role defines a set of permitted verbs (get, list, create, update, delete, and so on) against specific resource types, scoped to one namespace; a ClusterRole defines the same kind of permission set but can be bound either cluster-wide or within a single namespace, making it the reusable building block for anything meant to apply consistently across many Projects; a RoleBinding grants a Role or ClusterRole to a specific user, group, or Service Account within one namespace; and a ClusterRoleBinding grants a ClusterRole cluster-wide, across every namespace at once — the distinction between binding a ClusterRole via a namespace-scoped RoleBinding versus a cluster-wide ClusterRoleBinding is exactly how the same reusable role definition (admin, edit, view) ends up granting drastically different actual scope depending on which binding type is used. Role, ClusterRole, RoleBinding, and ClusterRoleBinding behave identically to any other conformant cluster. What OpenShift adds is a small, curated set of default ClusterRoles covering the common cases most organizations would otherwise define themselves from scratch:

A subtlety worth being precise about, since it's a common interview trap: admin (Project-scoped) and cluster-admin are not the same role at a different scope — admin deliberately excludes the ability to modify the Project's own ResourceQuota or grant SCCs, specifically so a Project's own trusted user still can't unilaterally escape the multi-tenancy boundaries a platform team set for them. Widening a Project's quota or SCC access is a decision that has to come from outside that Project, by design.

Default Groups Worth Knowing#

Alongside the default roles, OpenShift maintains a small set of automatically-managed groups that RBAC bindings commonly target instead of individual users:

The self-provisioner binding to system:authenticated:oauth mentioned above is a concrete example of why these group names matter precisely: a security review reading that binding needs to know it covers every interactively-logged-in user, not Service Accounts, without having to reverse-engineer the distinction from OpenShift's OAuth implementation directly.

Creating a Custom ClusterRole for a Narrow Permission#

When none of the six default roles fits — a support engineer who should be able to view Pod logs and exec into containers for debugging, but nothing else — the correct pattern is a purpose-built ClusterRole, never editing a default one (which the reconciliation behavior above would silently undo anyway):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-debugger
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]

oc adm policy add-role-to-user pod-debugger jane.doe -n payments-dev --role-namespace=default

Bound via a RoleBinding rather than a ClusterRoleBinding, this grants exactly the narrow permission to one user in one Project — a ClusterRole is reusable across as many RoleBindings as needed, but a RoleBinding is what actually scopes it to a specific namespace, the same reusable-definition-vs-scoped-grant separation Kubernetes RBAC uses everywhere else.

Aggregated ClusterRoles — Extending admin/edit/view Without Editing Them#

A related but distinct need from the custom role above: an Operator that introduces its own custom resource (a BackupSchedule CRD, say) often wants Project admin/edit users to automatically be able to manage it, without a platform team having to hand-edit the default admin/edit ClusterRoles for every new Operator installed — which the reconciliation behavior earlier in this chapter would silently revert anyway. Aggregated ClusterRoles solve exactly this: the default admin ClusterRole is defined with an aggregationRule matching a specific label, and any new ClusterRole carrying that label is automatically merged into admin's effective permission set:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: backup-operator-admin-permissions
  labels:
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
  - apiGroups: ["backup.example.com"]
    resources: ["backupschedules"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

This ClusterRole never needs its own RoleBinding at all — the label alone is what causes the default admin ClusterRole's aggregation controller to fold its rules into admin automatically, meaning every existing admin-bound user across every Project gains the new permission the moment this object is created, with zero further action and zero risk of the change being reverted by the default-role reconciliation this chapter covered earlier (since admin itself was never directly edited — only extended through the mechanism it was built to be extended through).

Another operational detail worth knowing: OpenShift automatically reconciles its default ClusterRoles on every control-plane restart and cluster upgrade, restoring any permission a cluster administrator may have accidentally (or deliberately, without realizing the consequence) removed from a default role. This is a deliberate safety net — it means a well-intentioned but mistaken edit to edit or view doesn't create a silent, permanent security drift across upgrades — but it also means a genuinely intentional customization to a default role won't survive the next reconciliation; the correct pattern for custom permissions is always a new, custom ClusterRole, never an edit to one of the six above.

Self-Provisioning and Restricting It#

The self-provisioner ClusterRoleBinding being bound to system:authenticated:oauth by default is a genuine, deliberate trade-off: it's what makes oc new-project a true self-service action for any logged-in developer, matching the "a new team can safely self-service a new environment" promise from Part 1 — but it also means, out of the box, any authenticated user can create an unbounded number of Projects, each consuming cluster resources and API-object overhead, unless a platform team decides to restrict it.

# Remove self-service project creation cluster-wide
oc patch clusterrolebinding self-provisioners \
  -p '{"subjects": null}'

# Or, more surgically, restrict it to a specific group instead of all authenticated users
oc adm groups new platform-approved-creators
oc adm policy add-cluster-role-to-group self-provisioner platform-approved-creators

Restricting self-provisioning is a common early Day-2 decision for a platform team adopting OpenShift at scale, and it's worth making deliberately rather than leaving the aggressive default in place simply because nobody thought to revisit it.

From the Trenches: An organization left self-provisioning open by default for over a year, and a routine cost review found several hundred abandoned Projects — created for one-off demos, proof-of-concepts, and long-departed contractors' experiments — each still holding its default ResourceQuota's worth of reserved (if unused) capacity, and each still a live RBAC surface nobody was actively reviewing. Nothing about this was a security breach; it was simply the predictable, compounding cost of a genuinely convenient default with no offsetting cleanup process. The fix wasn't disabling self-provisioning — the team still wanted fast self-service — it was adding a mandatory openshift.io/requester label (populated automatically by a customized project-request template) plus a scheduled job flagging any Project with no workload activity for 90 days, giving the convenience of self-service an equally automatic accountability mechanism instead of relying on nobody ever needing cleanup.

Security Context Constraints — The Core Model#

Everything above governs objects — what a user can create, read, or modify. Security Context Constraints (SCCs) govern something different and more fundamental: what a Pod itself is allowed to do at the kernel/runtime level, regardless of which user or Project it's running in. An SCC is OpenShift's own admission-control mechanism, predating and functionally overlapping with (but not identical to) Kubernetes' own Pod Security Admission — it constrains things like whether a container may run as root, whether it may request additional Linux capabilities, whether it may use host networking or host paths, and what SELinux context it's assigned.

Every Pod, once admitted, carries an openshift.io/scc annotation recording exactly which SCC it was matched against — oc get pod <name> -o yaml | grep scc is the fastest way to answer "why is this Pod allowed to do X" or, more commonly, "why was this Pod rejected."

Reading an SCC Object Directly#

An SCC's own YAML is worth reading once in full, since every field maps directly onto a specific admission check from the sequence diagram above:

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: restricted-v2
priority: null
allowPrivilegedContainer: false
allowPrivilegeEscalation: false
allowHostNetwork: false
allowHostPorts: false
allowHostPID: false
allowHostIPC: false
readOnlyRootFilesystem: false
requiredDropCapabilities: ["ALL"]
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs
fsGroup:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users: []
groups:
  - system:authenticated

The type field on runAsUser/seLinuxContext/fsGroup names the actual enforcement strategy: MustRunAsRange requires a value from the namespace's allocated range (and supplies it automatically if the Pod spec leaves it unset — exactly the mechanism behind "leaving runAsUser unset" being the correct fix earlier in this chapter); MustRunAs similarly requires and can supply a specific value; RunAsAny places no constraint at all, letting the Pod's own spec decide freely. A custom SCC swapping supplementalGroups.type from RunAsAny to MustRunAs with a specific range, for instance, is exactly how a platform team would tighten an existing SCC's behavior for a specific compliance requirement without inventing an entirely new enforcement mechanism from scratch — every SCC is built from this same small set of strategy types.

The Default SCC Roster#

A default OpenShift installation ships roughly ten SCCs, ordered here from least to most permissive:

The naming convention (-v2 suffixes) reflects the same Pod Security Standards alignment work referenced in Part 1's OLM v1 note: OpenShift 4.11 introduced versioned SCCs specifically to map more precisely onto Kubernetes' own upstream privileged/baseline/restricted Pod Security Standard levels, so a workload's actual admitted permissions can be reasoned about in both vocabularies at once.

Reading this roster top to bottom also maps cleanly onto a typical organization's own separation of duties: application developers' Service Accounts should almost always land on restricted-v2 or, occasionally, nonroot-v2; a platform team's own infrastructure Operators are the only legitimate grantees of hostnetwork-v2, hostaccess, or privileged; and anyuid sits deliberately in between as the narrow, reviewable escalation for the specific "legacy image needs root, nothing else" case this chapter keeps returning to — a useful mental shortcut for a security review scanning a cluster's SCC grants for anything that looks out of place at a glance.

From the Trenches: A team granted their entire application namespace the anyuid SCC as a blanket fix after one legacy image (which happened to run as root) failed to schedule, rather than diagnosing that specific image's actual requirement. Every other workload in that namespace — including several that ran perfectly well under restricted-v2 — silently inherited the broader anyuid grant too, widening the namespace's real attack surface far beyond what the one legacy image actually needed. The fix, once a security review caught it, was scoping the anyuid grant to only the one Service Account the legacy image's Deployment actually used, and leaving every other workload's Service Account on the namespace default — the SCC-per-workload discipline the next section covers.

How SCC Admission Actually Works: Priority and "Most Restrictive That Fits"#

A subtlety that trips up even experienced Kubernetes engineers new to OpenShift: SCC selection is not something a Pod spec requests directly. A Pod's securityContext expresses preferences (a specific runAsUser, specific capabilities), and the SCC admission plugin evaluates every SCC the requesting Service Account is authorized to use, in priority order (an explicit priority field, ties broken by restrictiveness), applying the first one whose constraints the Pod's requested spec satisfies — not necessarily the Pod's own stated preference, and not necessarily the most permissive SCC available to that Service Account.

This explains a specific, common confusion: a Service Account granted both restricted-v2 and anyuid will still have its Pods admitted under restricted-v2 if the Pod's spec doesn't request anything restricted-v2 disallows — anyuid being available to a Service Account doesn't mean every Pod that Service Account creates actually uses it. The practical rule worth internalizing: grant the least-permissive SCC that actually satisfies a workload's real requirement, and trust the admission plugin to keep using it — the mechanism is specifically designed to avoid granting more privilege than a given Pod actually exercises, but only if the underlying SCC grant itself is scoped narrowly in the first place, exactly the gap the trenches story above fell into.

Namespace UID Ranges — Why Containers Rarely Run as a Fixed UID#

The UID-range annotation surfaced earlier (openshift.io/sa.scc.uid-range: 1000600000/10000) is the mechanism behind one of the most common points of confusion for engineers arriving from a Docker or vanilla-Kubernetes background: under the default restricted-v2 SCC, a container does not run as whatever UID its Dockerfile's USER instruction specifies — it runs as an arbitrary UID from that Project's own pre-allocated range, different for every Project, and the same image can end up running as a different UID in every namespace it's deployed to.

This is a deliberate additional isolation guarantee: even if two Projects on the same cluster somehow both ran a process that escaped its container's namespace isolation, they'd still be running as different host-level UIDs, giving defense-in-depth against exactly that failure mode. The practical consequence for image builders: never hard-code a specific numeric UID as a security assumption, and instead make the image work correctly for an arbitrary, unknown UID — the same guidance this catalog's Docker Container Fundamentals series gives for --user, generalized to "arbitrary, not just non-root."

# Correct pattern: don't assume a fixed UID, make arbitrary UIDs work
RUN chgrp -R 0 /app/data && chmod -R g=u /app/data
# GID 0 (root group) is guaranteed regardless of the actual UID OpenShift assigns,
# so granting the group the same permissions as the (unknown) owning user
# lets an arbitrary-UID process still read/write its own data directory.
USER 1000670000

A Pod that explicitly declares runAsUser: 0 (or a Dockerfile that hard-codes USER root) is exactly the kind of request restricted-v2 denies outright — the fix is almost never "grant a more permissive SCC," it's fixing the image to genuinely not need root, unless there's a real, specific reason (a legacy binary requiring a privileged bind, for instance) that a narrowly-scoped anyuid grant is the honest answer to.

SCCs vs. Pod Security Standards — Two Overlapping Models#

Kubernetes' own upstream Pod Security Admission enforces the Pod Security Standards (privileged, baseline, restricted) via namespace labels — a mechanism that exists independently of OpenShift and that OpenShift's newer SCC versions were deliberately aligned with, but the two mechanisms are not interchangeable, and this is a real, documented gap worth knowing precisely:

The specific, real gotcha worth memorizing: OpenShift's restricted-v2 SCC requires runAsUser to be either unset or explicitly within the namespace's allocated UID range — a Pod spec that hard-codes a specific non-root UID (satisfying the upstream restricted Pod Security Standard perfectly well on a vanilla cluster) can still fail SCC admission on OpenShift for that exact reason, which is precisely why the "arbitrary UID, not just non-root UID" framing in the previous section matters in practice, not just in theory.

Setting Pod Security Admission Labels Alongside SCCs#

Because Pod Security Admission is a separate, upstream mechanism, it's still enforced on OpenShift independently of whatever SCC a Pod is ultimately admitted under — a defense-in-depth pairing worth configuring explicitly for any namespace with a genuinely strict security posture, rather than relying on SCCs alone:

apiVersion: v1
kind: Namespace
metadata:
  name: payments-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

With both mechanisms active, a Pod must satisfy the namespace's labeled Pod Security Standard and get admitted under an available SCC — two independent checks, from two independently-maintained projects (upstream Kubernetes and OpenShift), that happen to overlap heavily in intent but not in exact mechanics, which is precisely why the runAsUser gotcha above is worth memorizing rather than assuming one check subsumes the other.

A Worked Example: Diagnosing and Fixing an SCC Denial#

A Pod stuck never scheduling, with an event like this, is an SCC denial:

Warning  FailedCreate  replicaset-controller
Error creating: pods "web-7d9f6-" is forbidden: unable to validate against
any security context constraint:
[provider "restricted-v2": .spec.containers[0].securityContext.runAsUser:
Invalid value: 0: must be in the ranges: [1000670000, 1000679999]]

The message directly names the offending field and the actual allowed range — the fastest diagnosis path is reading it literally rather than guessing. The corrected Pod spec, letting the SCC assign the UID rather than forcing one:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  template:
    spec:
      containers:
        - name: web
          image: quay.io/myorg/web:v3
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            # No runAsUser set — let restricted-v2 assign one
            # from this namespace's allocated range.

Leaving runAsUser unset is the correct fix in the overwhelming majority of cases — it's a request pattern restricted-v2 was specifically designed to admit, and it keeps the workload on the least-permissive SCC available.

Predicting an SCC Denial Before It Happens#

oc adm policy scc-subject-review answers "which SCC, if any, would admit this Pod" as a dry run — genuinely useful as a CI pipeline gate that catches an SCC denial before a manifest ever reaches the cluster, rather than discovering it only when a Deployment fails to schedule:

oc get deployment web -o yaml | oc adm policy scc-subject-review -f -

RESOURCE          ALLOWED BY
Deployment/web     restricted-v2

An empty ALLOWED BY column means no SCC currently available to that manifest's Service Account would admit it — exactly the check worth running before deploying to a namespace whose SCC grants a team doesn't fully control, catching the same class of denial the worked example above walked through after the fact, before it ever reaches a real cluster. -u/-g/-z flags let the same check be run against a specific user, group, or Service Account other than the one the manifest itself specifies, useful for answering "could a different team's Service Account deploy this same manifest" without actually granting them access to try.

Granting a Workload a Less-Restrictive SCC, Safely#

For the legitimate minority of cases — a legacy image genuinely requiring root, or a specific capability restricted-v2 denies — the correct grant targets a Service Account, never a whole namespace or a whole user group, so exactly one workload's Pods are affected:

oc create serviceaccount legacy-app-sa -n payments-dev

oc adm policy add-scc-to-user anyuid \
  -z legacy-app-sa -n payments-dev

apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: payments-dev
spec:
  template:
    spec:
      serviceAccountName: legacy-app-sa   # only this Deployment's Pods get anyuid
      containers:
        - name: legacy-app
          image: quay.io/myorg/legacy-app:v1

Every other workload in payments-dev continues to use its Service Account's own default (restricted-v2), completely unaffected by this grant — the direct, practical fix for the "blanket namespace-wide anyuid grant" trenches story earlier in this chapter.

Choosing the Right SCC — A Decision Tree#

Auditing SCC and RBAC Grants Across a Cluster#

Periodically auditing who actually holds an elevated SCC or RBAC grant — not just checking at the moment a grant is made — is what catches the "granted six months ago for a since-decommissioned workload, never revoked" class of drift:

# Every SCC and its priority, at a glance
oc get scc -o custom-columns=NAME:.metadata.name,PRIORITY:.priority,RUNASUSER:.runAsUser.type

# Which Service Accounts are bound to a specific, more-permissive SCC
oc get clusterrolebindings,rolebindings --all-namespaces \
  -o json | jq -r '.items[] | select(.roleRef.name=="system:openshift:scc:anyuid")'

# Can this specific Service Account do a specific action, right now
oc adm policy who-can use scc anyuid
oc adm policy can-i create pods --as=system:serviceaccount:payments-dev:legacy-app-sa

oc adm policy who-can/can-i answer the RBAC question precisely rather than requiring a manual trace through every RoleBinding in the cluster — the same "ask the system directly instead of reconstructing the answer by hand" discipline this catalog applies to debugging generally.

ResourceQuotas and LimitRanges in Depth#

Multi-tenancy isn't just about what a Pod is allowed to do — it's also about how much of the cluster's shared capacity one team can consume, since a single Project's runaway resource usage can degrade every other Project sharing the same nodes. ResourceQuota caps a Project's aggregate consumption; LimitRange fills in sane defaults so an individual Pod that specifies no resources at all doesn't silently consume unboundedly.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: payments-dev-quota
  namespace: payments-dev
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "50"
    persistentvolumeclaims: "10"
    services.loadbalancers: "0"

That last line — capping services.loadbalancers at zero — is a common, deliberate pattern for non-production Projects specifically: it prevents a developer from accidentally provisioning a real cloud load balancer (a genuine, billable cloud resource) from a dev namespace that should only ever use internal Routes, closing a cost-control gap a CPU/memory quota alone wouldn't catch.

A more complete LimitRange also bounds the maximum a single container may request, not just the default — closing the gap where one deliberately (or accidentally) oversized container spec could still consume the Project's entire quota by itself:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: payments-dev
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }
      default: { cpu: 500m, memory: 512Mi }
      min: { cpu: 50m, memory: 64Mi }
      max: { cpu: "2", memory: 4Gi }

min matters as much as max in practice: without it, a developer can request a container with an unrealistically tiny CPU/memory footprint that then gets OOM-killed or CPU-throttled constantly in production, a failure mode that shows up as "the application is flaky" rather than as an obviously-wrong resource spec, unless the LimitRange itself rejects the request at creation time instead of letting it schedule and fail later.

Auditing Quota Usage Across the Cluster#

A single Project's quota is easy to check in isolation (oc describe resourcequota -n payments-dev); the more useful platform-team question is which Projects, across the whole cluster, are actually close to their ceiling right now, before a team's next deploy fails unexpectedly:

for ns in $(oc get projects -o jsonpath='{.items[*].metadata.name}'); do
  oc get resourcequota -n "$ns" -o json 2>/dev/null | \
    jq -r --arg ns "$ns" \
    '.items[] | "\($ns): cpu \(.status.used["requests.cpu"] // "0")/\(.status.hard["requests.cpu"] // "-") mem \(.status.used["requests.memory"] // "0")/\(.status.hard["requests.memory"] // "-")"'
done

A representative result, scanned quickly:

payments-dev: cpu 3800m/4 mem 7.6Gi/8Gi
fraud-detection-dev: cpu 1200m/4 mem 2.1Gi/8Gi
platform-shared: cpu 3950m/4 mem 15.8Gi/16Gi

platform-shared sitting at roughly 99% of both its CPU and memory quota is the signal worth acting on proactively — reaching out to that team before their next scale-up or deploy fails with an opaque exceeded quota error, rather than waiting for that failure to become an incident report. Part 5's monitoring chapter covers turning exactly this kind of query into a standing Prometheus alert rather than a manually-run script, once the built-in monitoring stack is in scope.

From the Trenches: A team's ResourceQuota had gone unreviewed for over a year while the team itself tripled in size and workload count; the quota was never hit because nobody had increased it to match, so — instead of a clean rejection — every new Deployment attempt simply failed with exceeded quota at the exact moment the team most needed to ship a fix under time pressure during an incident. The underlying lesson: a ResourceQuota that's correct on day one silently becomes wrong as a team grows, and treating it as a set-once value rather than something reviewed alongside the team's own headcount and workload growth is how a control meant to protect shared infrastructure ends up blocking exactly the team it was scoped for.

A Project whose ResourceQuota is exhausted doesn't degrade gracefully by default — new Pod creation is rejected outright with a clear exceeded quota admission error, which is a materially better failure mode than silent resource starvation, but still worth alerting on proactively (Part 5 covers the built-in monitoring stack this kind of alert belongs in) rather than discovering it only when a developer's deploy unexpectedly fails.

Multi-Tenancy Patterns: Namespace-per-Team vs. Cluster-per-Team#

Everything in this chapter assumes namespace-level (Project-level) multi-tenancy — many teams sharing one cluster, isolated by Project, RBAC, SCC, quota, and network policy. That's not the only model, and the choice between it and running a separate cluster per team is a real, consequential trade-off worth naming explicitly rather than defaulting into.

A namespace-per-team baseline sits in the lower-cost, moderate-isolation quadrant by default; layering the Pod Security Admission pairing from earlier in this chapter on top pushes it meaningfully further up the isolation axis for a modest additional configuration cost, before an organization needs to consider the much larger cost jump to dedicated clusters at all — worth walking through in that order with a security team asking for "more isolation," rather than jumping straight to the most expensive option as the only way to satisfy the request.

Plotting hosted control planes (Part 1's HyperShift model) on this same chart is deliberate: it recovers a meaningful share of the isolation gain a fully dedicated cluster provides, at a cost premium well short of provisioning entirely separate infrastructure per team — a genuinely useful middle option once a team's isolation requirement has outgrown a plain shared namespace but hasn't yet justified the full cost of dedicated per-team clusters.

Everything this chapter documents — Projects, SCCs, RBAC, quotas, default network isolation — exists specifically to make the left column a defensible, auditable choice rather than a risky shortcut; an organization that finds itself repeatedly working around these defaults to achieve acceptable isolation is often a signal the actual requirement has crossed into the right column, and that's worth surfacing to whoever owns that infrastructure cost trade-off rather than layering ever-more-custom policy onto a shared cluster that structurally wasn't the right fit for that specific tenant.

A Middle Ground: Fleet Management Across Many Clusters#

Organizations that do land on cluster-per-team (or cluster-per-environment, or cluster-per-region) rarely want to operate each cluster as a fully independent island — Red Hat Advanced Cluster Management (RHACM) is the product name worth recognizing for this specific gap: a hub cluster that pushes consistent Projects, RBAC, SCC, and quota policy out to every spoke cluster in a fleet, giving back much of the "one consistent policy, centrally managed" benefit of a shared cluster without actually sharing the underlying infrastructure. RHACM itself is a separate product from anything this series covers in depth, but it's worth knowing by name as the answer to "how do we keep dozens of separate clusters from drifting into dozens of inconsistent multi-tenancy configurations" once an organization has genuinely outgrown the single-shared-cluster model this chapter otherwise assumes.

Default Network Isolation Between Projects#

The last piece of the multi-tenancy floor is network-level: by default, a fresh Project's pods can talk freely to each other, but are not reachable from pods in any other Project, via a set of NetworkPolicy objects the project template (or, on some CNI configurations, the network plugin's own default project isolation mode) installs automatically.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-same-namespace
  namespace: payments-dev
spec:
  podSelector: {}
  ingress:
    - from:
        - podSelector: {}

With no other NetworkPolicy present, this single rule is the entire ingress policy for the namespace: pods within payments-dev can reach each other, and nothing else can reach into payments-dev at all — a default-deny-across-namespaces, allow-within-namespace posture, out of the box, with zero action required from the team that owns the Project.

A Taste of Deliberately Opening a Cross-Project Path#

When two teams' services genuinely need to talk — an internal payments-api in payments-dev needs to call a fraud-scoring service in fraud-detection-dev — the fix is an additional, narrowly-scoped NetworkPolicy in the target namespace, layered alongside (not replacing) the default deny:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-payments-api-to-fraud-scoring
  namespace: fraud-detection-dev
spec:
  podSelector:
    matchLabels: { app: fraud-scoring }
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: payments-dev }
          podSelector:
            matchLabels: { app: payments-api }
      ports:
        - port: 8443
          protocol: TCP

This opens exactly one path — from Pods labeled app: payments-api in payments-dev, to Pods labeled app: fraud-scoring in fraud-detection-dev, on exactly one port — leaving the default deny intact for every other cross-namespace combination. Part 3 of this series covers NetworkPolicy and OpenShift's broader networking stack (OVN-Kubernetes, Routes, Multus) in full depth; this section exists here specifically to show that the default from the previous section is a safe starting point with a well-defined escape hatch, not a permanent restriction with no supported way to relax it where a real, specific requirement exists.

Chargeback: Attributing Shared Cluster Cost to Projects#

A shared cluster's economics raise a question every one of the previous sections quietly sidesteps: if payments-dev and fraud-detection-dev share the same node pool, how does a platform team fairly attribute the cluster's actual infrastructure bill back to each team, rather than treating cluster cost as one undifferentiated line item nobody can act on? This is the same "chargeback vs. showback" problem this catalog's FinOps & Cost Optimization roadmap content covers in general terms, applied specifically to the Project-level isolation this chapter builds.

The practical mechanism starts from what this chapter already establishes: since every Project carries its own ResourceQuota (actual requested/limit values) and every Pod's real usage is visible per-namespace through the cluster's built-in monitoring stack (Part 5 covers Prometheus/Grafana in depth), a cost-allocation tool needs only two real inputs — each Project's actual CPU/memory consumption over a billing period, and the cluster's total infrastructure cost for that period — to compute a fair per-Project share, typically via an Operator like Kube-bench's cost-focused siblings or a dedicated cost-management tool (OpenShift's own Cost Management service, when subscribed, ingests exactly this Prometheus-sourced per-namespace usage data automatically).

Whichever model a platform team picks, the Project boundary this chapter builds is what makes the attribution possible at all — without a Project-per-team (or Project-per-environment) convention consistently applied, there's no clean unit to attribute cost to in the first place, which is one more reason the multi-tenancy discipline this chapter covers pays for itself well beyond the security and isolation framing it's usually discussed under.

A consistent labeling convention is the one prerequisite every attribution model above shares: a cost-center or team label applied to every Project at creation time (the project template from earlier in this chapter is exactly where to enforce this, as a required parameter rather than an optional convention) is what lets a cost-management tool group Prometheus usage data by team automatically, rather than a platform engineer manually reconciling namespace names against a separate spreadsheet of team ownership every billing cycle.

Quick Reference: Key Terms From This Chapter#

Common Mistakes and Interview Traps#

Worked Practice Problems#

1. A developer's Pod fails to schedule with an error naming restricted-v2 and a specific UID range. Their Dockerfile has USER 1001. Why does this fail, and what's the honest fix?#

The restricted-v2 SCC requires runAsUser to be either unset or within the Project's own allocated UID range (e.g., 1000670000-1000679999) — a hard-coded USER 1001 from the Dockerfile falls outside that range on essentially every real cluster, so admission rejects it regardless of 1001 being a legitimate non-root UID in the abstract. The honest fix is removing the hard-coded UID assumption entirely: rely on the SCC to assign an arbitrary UID from the namespace's range, and make the image's file permissions work for an unknown UID via GID 0 group ownership, rather than trying to negotiate a specific UID into the allowed range (which would also break the moment the same image is deployed to a different Project with a different allocated range).

2. A platform team wants developers to be able to create new Projects quickly, but also wants an audit trail of who approved each new environment for compliance reasons. Is restricting self-provisioner the right call?#

Restricting self-provisioner outright would solve the audit-trail requirement but directly conflicts with the "quickly" requirement, since every new Project would then require a platform-team member to manually create it. The better fit, given both constraints, is keeping self-provisioning available but routing it through an automated, auditable path — a GitOps-driven Project-request workflow (Part 4 covers GitOps in depth) where a merged pull request against a Projects repository is what actually triggers oc new-project, giving both fast self-service (from the developer's perspective, it's just a PR) and a durable, reviewable audit trail (the PR history itself) — rather than trading one requirement off against the other by picking either extreme.

3. A security team asks why an application team's namespace has the anyuid SCC granted at the namespace level (via a RoleBinding to the system:serviceaccounts:app-team group) rather than to a specific Service Account. What's the risk, and how should it be remediated?#

The risk is exactly the trenches story earlier in this chapter: every Service Account in that namespace — including any created later, by anyone with namespace-level create access — automatically inherits the anyuid grant, meaning a single legitimate need (one legacy image requiring root) has silently widened every current and future workload's available privilege in that namespace. Remediation means auditing which specific Deployment(s) actually require anyuid, creating a dedicated Service Account for each, re-scoping the grant to those specific Service Accounts via oc adm policy add-scc-to-user, removing the namespace/group-level binding, and confirming every other workload in the namespace still schedules correctly under its default restricted-v2 SCC afterward.

4. A platform team installs an Operator that introduces a new custom resource, and wants every Project's admin-bound user to be able to manage it without editing the default admin ClusterRole directly. What's the correct mechanism, and why would editing admin directly be a mistake?#

The correct mechanism is an aggregated ClusterRole: a new, purpose-built ClusterRole carrying the rbac.authorization.k8s.io/aggregate-to-admin: "true" label and rules scoped to the new custom resource, which the aggregation controller automatically folds into admin's effective permissions across every Project cluster-wide, with no RoleBinding of its own required. Editing admin directly would be a mistake for the same reason described earlier in this chapter: OpenShift reconciles its default ClusterRoles on every control-plane restart and upgrade, so a direct edit to admin would silently revert at the next reconciliation — the aggregation label is specifically the supported extension point precisely because it doesn't touch the default role's own definition at all.

5. A monitoring workload's DaemonSet needs to bind to a host network port on every node to scrape metrics. A developer requests the privileged SCC for its Service Account, reasoning "it needs host access, and privileged covers that." Is this the right grant?#

No — privileged is a far broader grant than the actual requirement, and this chapter's SCC roster names a narrower fit directly: hostnetwork-v2 grants exactly the host-network-namespace access this workload needs, without also granting the unrestricted host filesystem, capability, and PID-namespace access privileged carries. The correct approach is the same least-permissive-that-satisfies-the-requirement discipline as every other SCC decision in this chapter — walk the decision tree from the actual concrete requirement (host network binding, nothing else) rather than reaching for the broadest available grant because it's guaranteed to work; privileged should be reserved for genuine infrastructure components (CNI plugins, CSI drivers) whose actual requirements span multiple host-level dimensions at once, not used as a default "make the error go away" grant for a narrower need.

Summary and What's Next#

This chapter covered the layer that makes OpenShift's multi-tenancy promise real rather than aspirational: Projects bundle a namespace with default quotas, limits, RBAC, and network isolation in one self-service action; RBAC's default role set (cluster-admin, admin, edit, view, basic-user, self-provisioner) covers the common cases most organizations would otherwise hand-roll, with custom ClusterRoles as the correct escape hatch rather than editing a reconciled default; and Security Context Constraints enforce a default-restrictive floor on what a Pod is allowed to do at the kernel level, independent of who deployed it or which Project it lives in — admitting each Pod under the most restrictive SCC that its actual requested spec satisfies, never the most permissive one merely available to it. The namespace UID-range mechanism and the SCC-vs-Pod-Security-Standards gap are the two most consequential, least-obvious details underneath all of it, and both trace back to the same design goal: real isolation enforced by default, not isolation that depends on every team remembering to configure it themselves correctly.

Every mechanism this chapter covered shares one further property worth carrying forward explicitly: each is auditable from the command line — oc adm policy who-can/can-i for RBAC, oc get scc and the per-Pod openshift.io/scc annotation for security posture, and a per-namespace quota scan for consumption — meaning "is our multi-tenancy actually configured the way we think it is" is always a question the platform itself can answer directly, rather than one that depends on tribal knowledge or out-of-date documentation. That auditability is what makes chargeback, compliance review, and onboarding a new platform engineer all tractable at once, on the same underlying data — a new platform engineer can genuinely learn this cluster's actual security and cost posture by running the commands this chapter walked through, rather than by hunting through institutional knowledge that may or may not still be accurate.

Keep the Project boundary this chapter builds in mind as the unit everything downstream attaches to: RBAC bindings, SCC grants, quotas, network policies, and — as the chargeback section showed — cost attribution all key off the same Project, which is exactly why getting this chapter's defaults right once, cluster-wide, pays off across every one of those concerns simultaneously rather than needing to be solved separately for each.

Part 3 moves from who can do what to how traffic actually moves — OVN-Kubernetes as the default CNI, Routes as OpenShift's layer built on top of (and interoperable with) standard Kubernetes Ingress, NetworkPolicy in the depth this chapter's closing section only introduced, Multus for workloads needing more than one network interface, and OpenShift Service Mesh for the subset of workloads whose traffic-management needs outgrow what Routes and NetworkPolicy alone can express.

Sources consulted for this chapter: Red Hat's OpenShift Container Platform Authentication and Authorization documentation (RBAC, Security Context Constraints, Pod Security Standards alignment), the OpenShift Projects and Namespaces documentation, Red Hat's own guidance on OpenShift UID range allocation and multi-tenancy patterns, and the oc adm policy command reference for SCC review and RBAC auditing.