# Red Hat OpenShift: Enterprise Kubernetes Platform — Part 1: Architecture & What OpenShift Adds Over Vanilla Kubernetes

> **Series:** Red Hat OpenShift: Enterprise Kubernetes Platform (1 of 5)
> **Part 1:** This file — Architecture & What OpenShift Adds Over Vanilla Kubernetes
> **Part 2:** `02-projects-sccs-and-multi-tenancy.md` — Projects, Security Context Constraints & Multi-Tenancy
> **Part 3:** `03-networking-routes-and-service-mesh.md` — Networking, Routes & OpenShift Service Mesh
> **Part 4:** `04-buildconfigs-s2i-imagestreams-and-pipelines.md` — BuildConfigs, S2I, ImageStreams & OpenShift Pipelines/GitOps
> **Part 5:** `05-day-2-operations-cluster-operators-and-observability.md` — Day-2 Operations: Cluster Operators, Upgrades & Observability
> **Questions:** `questions.md`

## Table of Contents

1. [What Problem OpenShift Actually Solves](#what-problem-openshift-actually-solves)
2. [The Big Picture: OpenShift's Layered Architecture](#the-big-picture-openshifts-layered-architecture)
3. [RHCOS — The Immutable Operating System Every Node Runs](#rhcos--the-immutable-operating-system-every-node-runs)
4. [Ignition and the First-Boot Provisioning Model](#ignition-and-the-first-boot-provisioning-model)
5. [CRI-O — The Container Runtime Purpose-Built for Kubernetes](#cri-o--the-container-runtime-purpose-built-for-kubernetes)
6. [The Bootstrap Process: From Nothing to a Running Control Plane](#the-bootstrap-process-from-nothing-to-a-running-control-plane)
7. [The Cluster Version Operator and "Operators All the Way Down"](#the-cluster-version-operator-and-operators-all-the-way-down)
8. [The Machine Config Operator and MachineConfigPools](#the-machine-config-operator-and-machineconfigpools)
9. [Cluster Operators — The Full Roster and What Each Owns](#cluster-operators--the-full-roster-and-what-each-owns)
10. [The Operator Lifecycle Manager (OLM) and OperatorHub](#the-operator-lifecycle-manager-olm-and-operatorhub)
11. [Installer-Provisioned vs. User-Provisioned Infrastructure](#installer-provisioned-vs-user-provisioned-infrastructure)
12. [A Minimal `install-config.yaml`, Walked Through](#a-minimal-install-configyaml-walked-through)
13. [Disconnected and Mirrored-Registry Installations](#disconnected-and-mirrored-registry-installations)
14. [Control Plane Topology and High Availability](#control-plane-topology-and-high-availability)
15. [Verifying Cluster Health After a Fresh Install](#verifying-cluster-health-after-a-fresh-install)
16. [Telemetry, Insights, and the Support Lifecycle](#telemetry-insights-and-the-support-lifecycle)
17. [The `oc` CLI and Web Console](#the-oc-cli-and-web-console)
18. [OpenShift vs. Vanilla Kubernetes — A Full Feature Comparison](#openshift-vs-vanilla-kubernetes--a-full-feature-comparison)
19. [Multi-Architecture and Hybrid Cloud Reach](#multi-architecture-and-hybrid-cloud-reach)
20. [Distributions in the OpenShift Family](#distributions-in-the-openshift-family)
21. [Quick Reference: Key Terms From This Chapter](#quick-reference-key-terms-from-this-chapter)
22. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
23. [Worked Practice Problems](#worked-practice-problems)
24. [Summary and What's Next](#summary-and-whats-next)

## What Problem OpenShift Actually Solves

The Kubernetes Deep Dive series in this catalog already covers the API server, etcd, the scheduler, and the controller manager in depth. Everything in that series is still true here — OpenShift ships upstream Kubernetes underneath, unmodified at the API level. So the first question worth answering honestly is: if a team already knows how to run `kubeadm` or a managed offering like EKS, what does adopting OpenShift actually buy them, and what does it cost?

The honest answer is that vanilla Kubernetes deliberately ships a minimal, unopinionated core. The project's own governance model explicitly pushes container runtime choice, ingress implementation, registry, CI/CD, developer self-service tooling, cluster-wide security policy, and the operating system itself out of scope — every team that runs raw Kubernetes has to make dozens of these decisions themselves, then build and maintain the glue that wires them together. That glue is exactly where a huge fraction of platform-engineering effort at a mid-size company actually goes: which CNI, which ingress controller, how developers get a container built and pushed without a laptop `docker build`, how the OS underneath the kubelet gets patched across a fleet, how a security team enforces "no privileged pods" without every namespace owner writing their own admission policy.

OpenShift's core bet is that a curated, batteries-included distribution of Kubernetes — with the operating system, container runtime, networking, an internal registry, build tooling, a security model, and Day-2 lifecycle operations pre-integrated and jointly supported by one vendor — is worth the opinionation it imposes, for an organization whose priority is running Kubernetes reliably at scale rather than assembling a bespoke platform from parts. That trade only pays off when the organization actually needs enterprise support, a hardened default security posture across many teams, or predictable Day-2 operations more than it needs to hand-pick every component; a five-person startup running one cluster with one team on it may find the opinionation is pure overhead relative to a managed EKS/GKE cluster with a hand-picked ingress controller.

> **From the Trenches:** A financial-services platform team migrated from a self-managed `kubeadm` cluster to OpenShift specifically because an external auditor flagged that "any developer with `kubectl` access can deploy a privileged pod" as a finding in a compliance review. On vanilla Kubernetes, closing that gap meant standing up and maintaining Pod Security Admission policies, an OPA/Gatekeeper or Kyverno policy engine, and a registry-scanning pipeline — all bespoke, all owned by the platform team going forward. On OpenShift, Security Context Constraints (Part 2 of this series) closed the same gap as a cluster default on day one, and the audit finding was resolved in the same sprint. The trade-off the team accepted in return: every existing workload that assumed it could run as root needed to be fixed before it would schedule at all, which took three additional sprints of remediation the team hadn't budgeted for.

### Where the Platform Team's Responsibility Ends

Everything this chapter covers — RHCOS, CRI-O, the CVO, the MCO, Cluster Operators, OLM — is squarely a platform team's responsibility, not an application team's. This division of labor is itself one of OpenShift's structural bets: because so much of the "keep the underlying platform healthy" work is automated by operators rather than left as bespoke tooling, a platform team of a given size can support meaningfully more clusters, or more application teams on a shared cluster, than the same team could support hand-maintaining the equivalent components on vanilla Kubernetes. An application team, in turn, interacts almost entirely with the layer Part 2 onward covers — Projects, Routes, BuildConfigs — and rarely needs to reason about RHCOS or the CVO directly at all, in the same way a web developer rarely needs to reason about the Linux kernel scheduler underneath their application server. Keeping this boundary explicit is worth doing early in any organization adopting OpenShift, since a common early-adoption friction point is an application team assuming they need platform-level knowledge (or a platform team assuming application teams should self-serve platform-level changes) that the architecture was specifically designed to abstract away from them.

This chapter covers what's structurally different underneath OpenShift before any of the later chapters' developer- or operator-facing features make sense — the operating system, the runtime, the bootstrap sequence, and the operator-driven model that every later chapter assumes.

### A Brief History: Why OpenShift Looks the Way It Does

OpenShift's current shape is the direct result of one specific architectural decision made in 2018, and understanding it explains why so much of this platform is built around Operators rather than around a more conventional configuration-management tool. OpenShift 3, released in 2015, was already Kubernetes-based, but its Day-2 operations — installing the platform, applying updates, managing the underlying hosts — still leaned heavily on Ansible playbooks run by a human or a CI job, a model that worked but left "keep the cluster's own components consistent and up to date" as a largely manual, error-prone responsibility.

Red Hat's 2018 acquisition of CoreOS brought two things into the platform: Container Linux's "the OS is a single, atomically-updated unit" philosophy (which became RHCOS), and the **Operator pattern** itself, which CoreOS had originated as a way to encode human operational knowledge — how to safely upgrade a database, how to fail over a stateful service — into software that runs continuously inside the cluster instead of living in a runbook a human executes by hand. OpenShift 4, released in 2019, rebuilt the platform around that idea completely: instead of Ansible playbooks configuring hosts and components from the outside, the Cluster Version Operator and the roster of Cluster Operators from later in this chapter manage the entire platform from the inside, continuously, the same way a `Deployment` controller continuously reconciles a workload's replica count. Every architectural choice this chapter documents — RHCOS's immutability, the CVO's single tested release payload, OLM extending the same pattern to third-party software — traces back to that one 2018-2019 shift from "a human/pipeline configures the cluster" to "the cluster configures itself, continuously, and reports its own health."

### OpenShift and CNCF Conformance

It's worth being precise about a claim made earlier in this chapter: OpenShift is a **Certified Kubernetes** distribution under the Cloud Native Computing Foundation's own conformance program, meaning it passes the same conformance test suite every other certified distribution (EKS, GKE, AKS, `kubeadm` itself) must pass, and carries no API-level deviations that would break a workload's portability. This is a meaningful, checkable claim, not marketing language — the practical consequence is that a Helm chart, a raw Kubernetes manifest, or an Operator built against upstream Kubernetes APIs deploys onto OpenShift without modification for anything that doesn't specifically depend on a vanilla-cluster default OpenShift replaces (an ingress-nginx `Ingress` resource still works; it just won't get a Route's additional features without also creating one, covered in Part 3).

### When OpenShift's Opinionation Pays Off — A Decision Framework

Every later chapter in this series assumes the platform's opinionated defaults are worth adopting; it's worth stating explicitly, once, when that assumption actually holds:

| Situation | OpenShift's opinionation | Vanilla Kubernetes / a managed offering |
|---|---|---|
| Many teams, one shared cluster, compliance-driven security requirements | Default-restrictive SCCs and Project-scoped quotas close common audit findings on day one | Each requirement typically needs its own bespoke admission-policy and quota tooling, owned and maintained by the platform team |
| A platform team supporting dozens of clusters across a hybrid fleet | One CVO-managed release payload per cluster, one supported upgrade graph across the whole fleet | Each cluster's component versions (CNI, ingress, registry) can drift independently, multiplying the version-compatibility matrix the team must track |
| A single small team, one cluster, minimal compliance overhead | The security/registry/build defaults are often unused overhead relative to a hand-picked, lighter stack | A managed offering (EKS/GKE/AKS) with a hand-picked ingress controller and cert-manager is frequently the lower-overhead choice |
| Regulated industry requiring vendor-backed support commitments (SLA on CVE response, defined lifecycle) | Red Hat's subscription and lifecycle policy (covered later in this chapter) gives one contractual point of accountability across the whole stack | Support is either self-owned or split across several independent vendors/projects, each with its own commitment (or lack of one) |
| Team explicitly wants full control over every component's version and configuration | The tested release payload is a real constraint on swapping any default component | Every component is independently replaceable, at the cost of independently validating every combination |

The pattern worth internalizing: OpenShift's opinionation is a genuine trade, not a strictly-better option — it pays off precisely when the organization's actual pain is coordination and compliance overhead across many teams or clusters, and it's overhead precisely when that pain doesn't yet exist.

## The Big Picture: OpenShift's Layered Architecture

Every OpenShift cluster is Kubernetes plus four structural additions layered around it: an immutable, container-optimized operating system on every node; a fleet of first-party Operators that own the cluster's own configuration instead of a human running `apt upgrade`; a curated set of platform services (networking, registry, builds, routing) integrated by default instead of chosen à la carte; and a security model (Security Context Constraints, Projects) that's default-restrictive instead of default-open.

```mermaid
flowchart TB
    subgraph K8S["Unmodified Upstream Kubernetes API"]
        API["kube-apiserver"]
        ETCD["etcd"]
        SCHED["kube-scheduler"]
        CM["kube-controller-manager"]
    end

    subgraph OS_LAYER["Operating System Layer"]
        RHCOS["RHCOS on every node"]
        CRIO["CRI-O container runtime"]
        IGN["Ignition first-boot config"]
    end

    subgraph OPERATOR_LAYER["Operator-Driven Cluster Management"]
        CVO["Cluster Version Operator"]
        MCO["Machine Config Operator"]
        CLUSTER_OPS["~25 Cluster Operators (network, ingress, registry, monitoring, etc.)"]
        OLM["Operator Lifecycle Manager"]
    end

    subgraph PLATFORM_LAYER["Curated Platform Services"]
        ROUTES["Router / Routes"]
        REGISTRY["Internal image registry"]
        BUILDS["BuildConfigs / Source-to-Image"]
        SCC["Security Context Constraints"]
        PROJECTS["Projects (namespace + defaults)"]
    end

    OS_LAYER --> K8S
    CVO --> CLUSTER_OPS
    CLUSTER_OPS --> PLATFORM_LAYER
    MCO --> OS_LAYER
    OLM -.manages 3rd-party Operators.-> CLUSTER_OPS
    K8S --> PLATFORM_LAYER

    classDef upstream fill:#1d4ed8,color:#fff,stroke:#1e40af
    classDef os fill:#7c3aed,color:#fff,stroke:#5b21b6
    classDef operator fill:#b45309,color:#fff,stroke:#92400e
    classDef platform fill:#047857,color:#fff,stroke:#065f46
    class API,ETCD,SCHED,CM upstream
    class RHCOS,CRIO,IGN os
    class CVO,MCO,CLUSTER_OPS,OLM operator
    class ROUTES,REGISTRY,BUILDS,SCC,PROJECTS platform
```

Reading this diagram bottom-up mirrors how a cluster actually comes to life: RHCOS boots via Ignition, CRI-O starts, the Kubernetes control plane comes up on top of that OS layer, then the Cluster Version Operator takes over and installs every other Cluster Operator, which in turn stand up the platform services a developer or administrator actually interacts with day to day. Nothing in the `K8S` box is OpenShift-specific — a `kubectl get pods` behaves identically to any other conformant Kubernetes cluster, because OpenShift is a Cloud Native Computing Foundation certified Kubernetes distribution, not a fork.

## RHCOS — The Immutable Operating System Every Node Runs

Every control plane node, and by default every worker node, in a self-managed OpenShift cluster runs **Red Hat Enterprise Linux CoreOS (RHCOS)** — a purpose-built, immutable variant of RHEL designed to be managed entirely by the cluster rather than by an administrator logging in and running `yum update`. "Immutable" here specifically means the root filesystem itself is not meant to be edited directly on a running node; instead, the whole operating system image is updated atomically as a unit, using **OSTree** as the underlying content-addressed versioning mechanism, conceptually closer to how a container image is a versioned, replaceable artifact than to a traditional package-managed Linux install where thousands of independent package updates can drift a fleet of machines apart from each other over months.

This matters operationally in a way that's easy to underestimate: on a traditional fleet of hand-patched Linux hosts, "what OS version and packages are actually running on node 47" is a question that requires SSH-ing in and checking, and the honest answer is often "we're not entirely sure, it depends on which patch cycles that node happened to catch." On RHCOS, every node's OS state is a single OSTree commit hash the Machine Config Operator can report on, diff, and roll every node in a pool to in lockstep — the same "declared, reproducible, auditable" property this site's Docker Container Fundamentals series argues for at the image layer, applied one layer further down, to the node's own operating system.

RHCOS bundles the kubelet (Kubernetes' per-node agent) and CRI-O (the container runtime, covered below) as part of the base image rather than as separately-installed packages, and runs SELinux in enforcing mode by default — a mandatory access control layer that constrains what a compromised process can do even if it escapes its container's namespace isolation, layered underneath everything this series' Part 2 covers about Security Context Constraints.

| RHCOS property | Traditional package-managed RHEL/Ubuntu node |
|---|---|
| OS update unit | Whole-image OSTree commit, applied atomically | Individually-versioned packages, applied incrementally |
| Who applies updates | The Machine Config Operator, cluster-driven | An administrator or a configuration-management tool, human-driven |
| Rollback | Boot the previous OSTree deployment | Reverse each package update individually, if possible at all |
| Drift risk across a fleet | Structurally low — every node in a pool is the same commit | Structurally high — nodes patched at different times diverge |
| Direct SSH package installs | Actively discouraged; changes don't survive the next MCO rollout | The normal operating model |

> **From the Trenches:** A team new to OpenShift tried to `ssh` onto a worker node and run `rpm -Uvh` to install a diagnostic tool their monitoring vendor required, following habits from their previous RHEL fleet. The install appeared to succeed, but the very next Machine Config Operator rollout — triggered by an unrelated `MachineConfig` change the platform team pushed that same week — silently reverted it, because the node's actual desired state lives in the `MachineConfig` object, not in whatever happens to be installed on the running filesystem. The fix was packaging the diagnostic tool as a layered RHCOS image extension (via `rpm-ostree` overlay through a `MachineConfig`) so it survives every future rollout, instead of a one-off manual install that the platform's own reconciliation loop would keep undoing.

### Inspecting a Node's RHCOS State Without SSH

Because direct SSH access is discouraged and, on many managed or locked-down clusters, not even available, the supported way to inspect a node's actual operating-system state is `oc debug node`, which schedules a privileged debug pod on the target node and `chroot`s into its filesystem:

```bash
oc debug node/worker-0.example.com
# Once inside the debug pod's shell:
chroot /host
rpm-ostree status
```

A healthy node reports a single `rpm-ostree status` deployment marked `booted`, with a version string matching the OpenShift release the rest of the cluster is running. Two competing deployments listed (one booted, one merely staged) is the normal, transient state of a node mid-way through an MCO-driven update — it resolves itself once the node finishes rebooting into the staged deployment; it is only a problem if a node is stuck in that state well past the rest of the pool's update window, which is the first thing worth checking when a `MachineConfigPool` reports as `Updating` for longer than its siblings.

### Layering Additional Software onto RHCOS

The earlier trenches story about a manually-`rpm`-installed diagnostic tool being silently reverted has a real, supported answer: RHCOS supports **extensions** — a small, curated set of additional RPM packages (kernel modules, `usbguard`, sandboxed-containers support, and similar) that can be declared directly in a `MachineConfig` and layered onto the base image through the same `rpm-ostree` mechanism the OS itself uses, rather than a manual, unmanaged install:

```yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: worker-usbguard
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  extensions:
    - usbguard
```

Because this is a `MachineConfig` like any other, it goes through the same render/roll-out/reboot sequence as any other node-level change, and — critically — survives node replacement identically to the sysctl example earlier, since the extension is now part of what "correctly configured" means for that pool rather than a one-off manual step. Extensions are deliberately a short, curated list, not a general-purpose package-installation mechanism — arbitrary RPM installs still aren't a supported pattern, and software with more complex runtime needs belongs in a container, not layered onto the host OS at all.

## Ignition and the First-Boot Provisioning Model

RHCOS nodes don't run a traditional `cloud-init`-style provisioning script on every boot; instead, they use **Ignition**, a provisioning tool that runs exactly once, during the very first boot of a machine, before the node's own init system starts. An Ignition config is a declarative JSON document describing users, filesystems, systemd units, and files that should exist on the node the moment it comes up — conceptually similar to how a Kubernetes manifest declares desired state for a workload, but applied to the raw machine before Kubernetes itself is even running on it.

This is the mechanism that makes the bootstrap process below possible at all: the installer generates Ignition configs for the bootstrap machine, the control plane machines, and the worker machines, and every one of those machines fetches and applies its own config on first boot, with no manual "log in and configure this box" step anywhere in the sequence. After first boot, ongoing configuration changes are handled by the Machine Config Operator rewriting and rolling out new `MachineConfig` objects — Ignition itself is a one-shot bootstrapping mechanism, not an ongoing configuration-management tool a cluster administrator interacts with directly after day one.

### What's Actually Inside an Ignition Config

A trimmed real Ignition config — the artifact `openshift-install` renders from the Kubernetes-level `MachineConfig` objects before any machine boots — makes the "declarative, one-shot" claim concrete:

```json
{
  "ignition": { "version": "3.2.0" },
  "passwd": {
    "users": [
      { "name": "core", "sshAuthorizedKeys": ["ssh-ed25519 AAAA..."] }
    ]
  },
  "storage": {
    "files": [
      {
        "path": "/etc/kubernetes/kubelet.conf",
        "mode": 420,
        "contents": { "source": "data:text/plain;base64,..." }
      }
    ]
  },
  "systemd": {
    "units": [
      { "name": "kubelet.service", "enabled": true }
    ]
  }
}
```

Note what is deliberately absent: there's no script that runs a sequence of imperative commands, no conditional logic, and no retry loop — every field describes a piece of desired end state (a user should exist with this key, this file should exist with this content, this systemd unit should be enabled), and Ignition's own job on first boot is entirely to make that state true once, the same declarative philosophy as a Kubernetes manifest, just applied one layer below Kubernetes itself.

## CRI-O — The Container Runtime Purpose-Built for Kubernetes

Vanilla Kubernetes clusters commonly run `containerd` as their Container Runtime Interface (CRI) implementation — the same runtime underneath Docker Engine, covered in this catalog's Docker Container Fundamentals series. OpenShift instead ships **CRI-O** by default, a container runtime built from the ground up to implement exactly the CRI specification Kubernetes needs, and nothing more.

The distinction matters less at the API level (both are CRI-compliant, and `kubectl` behaves identically against either) and more in scope and attack surface: `containerd` grew out of the Docker Engine's own internals and historically carried API surface unrelated to the Kubernetes CRI contract, some of which has been a source of CVEs unrelated to anything a Kubernetes cluster actually uses. CRI-O was designed to implement only what `kubelet` needs — pull an OCI image, start and stop an OCI-compliant container, report status — deliberately minimizing the runtime's own surface for exactly the kind of "we shipped a feature nobody uses but everybody has to patch" risk that a security-conscious enterprise distribution wants to avoid.

| Property | CRI-O | containerd |
|---|---|---|
| Design goal | Implement exactly the Kubernetes CRI, nothing more | General-purpose runtime, CRI is one consumer among others (also used directly by Docker Engine) |
| Governance | Kubernetes SIG-Node sub-project | CNCF graduated project |
| OpenShift default | Yes, on every RHCOS node | N/A (not used by OpenShift) |
| Typical vanilla-Kubernetes default | Less common as the sole default | Very common (kubeadm, EKS, GKE, AKS default in many setups) |
| Image/runtime spec | OCI, same as containerd | OCI |

Both runtimes ultimately shell out to `runc` (or a compatible OCI runtime) to actually create the namespaces and cgroups a container needs — the choice between them is a supply-chain and attack-surface decision, not a functional one a developer running `oc apply` will ever notice.

### Talking to CRI-O Directly with `crictl`

`kubectl`/`oc` operate at the Kubernetes API level and never talk to the container runtime directly — but when a pod is stuck in a state the API server's own view can't explain (a pod wedged in `ContainerCreating` with no useful events, for instance), `crictl`, the CRI-compatible debugging CLI, talks straight to CRI-O on the node itself and is the tool this catalog's own Kubernetes Deep Dive series already recommends for exactly this class of problem:

```bash
oc debug node/worker-0.example.com -- chroot /host crictl ps -a
oc debug node/worker-0.example.com -- chroot /host crictl logs <container-id>
oc debug node/worker-0.example.com -- chroot /host crictl inspect <container-id>
```

`crictl ps -a` shows every container CRI-O currently knows about on that node, including ones the kubelet has already given up reporting to the API server — a pod that disappeared from `oc get pods` but left an orphaned container consuming resources on the node is a real, if uncommon, failure mode this command surfaces directly, where the Kubernetes API level shows nothing wrong at all.

## The Bootstrap Process: From Nothing to a Running Control Plane

An OpenShift cluster does not come up with three control-plane nodes electing a leader from a cold start the way a hand-rolled `kubeadm` HA cluster might. Instead, the installer stands up a **temporary bootstrap machine** that acts as a scaffold: it hosts a temporary single-node control plane and etcd instance just long enough for the real control plane nodes to join, form their own etcd cluster, and take over — at which point the bootstrap machine is destroyed.

```mermaid
sequenceDiagram
    participant Installer as openshift-install
    participant Bootstrap as Bootstrap machine
    participant CP as Control plane nodes
    participant CVO as Cluster Version Operator

    Installer->>Bootstrap: Boot with Ignition config (temp API server + etcd)
    Bootstrap->>Bootstrap: Start temporary Kubernetes control plane
    Installer->>CP: Boot control plane nodes with their own Ignition configs
    CP->>Bootstrap: Fetch remaining assets, join temporary control plane
    CP->>CP: Form the real etcd cluster across control plane nodes
    CP->>CVO: Temporary control plane installs the Cluster Version Operator
    CVO->>CP: CVO installs every other Cluster Operator in dependency order
    CP->>CP: Production control plane takes over from the temporary one
    Installer->>Bootstrap: Tear down bootstrap machine
    Installer->>Installer: Wait for "install-complete", cluster is self-sufficient
```

The practical consequence of this sequence for anyone debugging a failed installation is that `openshift-install wait-for bootstrap-complete --log-level debug` and the bootstrap machine's own logs are the correct first place to look when a cluster never comes up — by the time `oc` even has a cluster to talk to, the bootstrap machine has already done most of the interesting work, and a failure at that stage (a DNS record that doesn't resolve, a load balancer that isn't forwarding the right ports, a firewall blocking node-to-node etcd traffic) never surfaces as a Kubernetes-level symptom at all, because Kubernetes itself never successfully came up.

> **From the Trenches:** A team's first bare-metal OpenShift install hung indefinitely at "waiting for bootstrap to complete." The actual root cause was that the load balancer in front of the API server's port 6443 was configured to health-check the control plane nodes before they existed, and — because the health check kept failing — the load balancer never added the bootstrap machine's temporary API server as a backend either, even though the DNS record itself resolved correctly. `openshift-install`'s own bootstrap logs showed nothing wrong on the bootstrap machine itself; the actual failure only became visible from the load balancer's own connection logs, which is exactly why this installation model requires validating the load balancer and DNS layer *before* running the installer, not debugging them reactively after a stuck bootstrap.

## The Cluster Version Operator and "Operators All the Way Down"

Once the production control plane is up, day-to-day management of the cluster's own components is handed to the **Cluster Version Operator (CVO)** — the operator that manages every other operator. The CVO reads a `ClusterVersion` custom resource describing the desired OpenShift version, and reconciles the cluster toward a release payload: a signed, versioned manifest bundling the exact set of Cluster Operators, their versions, and their intended configuration for that OpenShift release.

This is the architectural idea that most distinguishes OpenShift's Day-2 model from a hand-assembled Kubernetes cluster: instead of a platform team independently choosing, installing, and upgrading a CNI plugin, an ingress controller, a metrics stack, and a registry — each on its own release cadence, each a separate point of potential version-skew breakage — the CVO treats the entire set of cluster components as **one versioned, tested unit**. An OpenShift upgrade from one minor version to the next is a single `oc adm upgrade` operation that rolls the whole release payload forward together, because Red Hat has already validated that exact combination of component versions works together — the same guarantee a Linux distribution's package repository gives you for a set of interdependent system packages, applied to a Kubernetes platform's own components.

```mermaid
stateDiagram-v2
    [*] --> Reconciling: ClusterVersion updated
    Reconciling --> Verifying: New release image pulled and signature verified
    Verifying --> ApplyingManifests: Signature valid
    Verifying --> Failed: Signature invalid or unreachable
    ApplyingManifests --> UpdatingOperators: Manifests applied in dependency order
    UpdatingOperators --> Available: All Cluster Operators report Available and not Degraded
    UpdatingOperators --> Degraded: One or more operators fail to reconcile
    Degraded --> UpdatingOperators: Operator retries reconciliation
    Available --> [*]
    Failed --> [*]
```

Every Cluster Operator reports its own `Available`, `Progressing`, and `Degraded` conditions, and the CVO aggregates them into the cluster's overall version state — `oc get clusteroperators` is the single command that answers "is anything in this cluster currently broken," which is a materially different debugging experience from a hand-assembled cluster where a broken CNI plugin, a broken ingress controller, and a broken metrics pipeline would each need their own separate `kubectl get pods -n <namespace>` investigation with no unified status view.

### The Upgrade Graph: Cincinnati and Conditional Updates

Not every OpenShift version can safely upgrade directly to every later version — a known regression discovered after a release shipped, affecting only clusters with a specific configuration, is a real and recurring reason a specific upgrade edge needs to be blocked or flagged rather than offered unconditionally. OpenShift tracks this through the **Cincinnati** update graph (served by the OpenShift Update Service, OSUS): a graph of releases where each edge represents a validated, recommended upgrade path, and `oc adm upgrade` only offers edges the graph currently marks as safe for that specific cluster's configuration.

Some edges carry a **conditional update** risk — recommended for most clusters, but explicitly flagged as `SupportedButNotRecommended` for clusters matching a specific known-affected condition (a particular platform, a specific feature gate enabled, a specific node count range). `oc adm upgrade` surfaces these conditions directly, including a link to the specific known issue, rather than silently offering an upgrade path Red Hat's own telemetry has flagged as risky for that cluster's actual configuration — a meaningfully more cautious default than "any newer version is presumed fine to upgrade to."

## The Machine Config Operator and MachineConfigPools

Where the Cluster Version Operator manages the Kubernetes-level components, the **Machine Config Operator (MCO)** manages everything on the node itself, between the kernel and the kubelet: `systemd` units, `kubelet`/`crio` configuration, kernel arguments, `NetworkManager` settings, and RHCOS's own OSTree-based updates. It is the mechanism that makes RHCOS's "cluster-managed operating system" promise from earlier in this chapter real rather than aspirational.

Nodes are grouped into **MachineConfigPools** — by default, one `master` pool and one `worker` pool, though custom pools (e.g., a pool for GPU nodes needing different kernel arguments) are a common Day-2 pattern. A `MachineConfig` object is associated with a pool via a label selector, and the MCO's `RenderController` merges every `MachineConfig` targeting a pool into one rendered configuration, which the `UpdateController` then rolls out to that pool's nodes one at a time — cordoning, draining, applying the update, rebooting if the change requires it (like a kernel argument), and uncordoning the node before moving to the next, so a fleet-wide OS-level change never takes an entire pool of nodes offline simultaneously.

| Concept | What it actually is |
|---|---|
| `MachineConfig` | A declared piece of desired node-level state (a file, a systemd unit, a kernel argument) |
| `MachineConfigPool` | A named group of nodes (by label) that share a rendered configuration |
| `RenderController` | Merges every `MachineConfig` targeting a pool into one rendered config |
| `UpdateController` | Rolls the rendered config out node-by-node: cordon, drain, apply, reboot if needed, uncordon |
| `MachineConfigDaemon` | The per-node agent that actually applies the rendered config on that node |

> **From the Trenches:** A platform team needed a custom `sysctl` value (`vm.max_map_count`) raised for a set of nodes running an Elasticsearch workload, without changing it cluster-wide. Applying it via a `DaemonSet` `initContainer` (the vanilla-Kubernetes approach) worked, but drifted silently whenever a node was replaced through cluster autoscaling, since the new node never re-ran that specific `DaemonSet` pod deterministically before the workload scheduled onto it. Moving the same setting into a `MachineConfig` targeting a custom `MachineConfigPool` labeled onto exactly those nodes made the setting part of the node's own declared, MCO-enforced state — surviving node replacement, reboot, and re-provisioning identically, because it's now baked into what "this node is correctly configured" means at the OS layer, not something a separately-scheduled pod has to re-apply every time.

### A Worked MachineConfig Example

The Elasticsearch `sysctl` scenario above, expressed as an actual object, shows how much of a `MachineConfig` is really just Ignition's own file/unit format wrapped in a Kubernetes-style envelope:

```yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-elasticsearch-sysctl
  labels:
    machineconfiguration.openshift.io/role: worker-elasticsearch
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/sysctl.d/99-elasticsearch.conf
          mode: 0644
          overwrite: true
          contents:
            source: data:,vm.max_map_count%3D262144
```

And the matching `MachineConfigPool` that targets it at exactly the labeled nodes, leaving the standard `worker` pool untouched:

```yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker-elasticsearch
spec:
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values: [worker, worker-elasticsearch]
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker-elasticsearch: ""
  paused: false
```

Applying both, then labeling the target nodes with `node-role.kubernetes.io/worker-elasticsearch=""`, is enough for the MCO to render the combined configuration, roll it out one node at a time to only that pool, and keep enforcing it against any future replacement node carrying the same label — with zero ongoing action from the team that requested the change.

## Cluster Operators — The Full Roster and What Each Owns

A default OpenShift installation runs roughly two dozen Cluster Operators, each owning one platform concern and each independently reporting its own health to the CVO. Recognizing what each one owns is the fastest way to triage "something in the cluster is broken" down to the right log to read.

| Cluster Operator | Owns |
|---|---|
| `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` | The core Kubernetes control plane components themselves, wrapped as operators |
| `etcd` | etcd cluster membership, scaling, backup/restore coordination |
| `network` | Cluster networking (OVN-Kubernetes by default) — see Part 3 |
| `ingress` | The default HAProxy-based router that implements Routes — see Part 3 |
| `dns` | Cluster-internal DNS (CoreDNS) configuration |
| `image-registry` | The internal integrated image registry — see Part 4 |
| `machine-config` | Everything covered in the previous section |
| `machine-api` | Provisioning and scaling of `Machine`/`MachineSet` objects — see Part 5 |
| `authentication` | The OAuth server backing `oc login` and console SSO |
| `console` | The web console itself |
| `monitoring` | The built-in Prometheus/Alertmanager/Grafana stack — see Part 5 |
| `cluster-logging` (add-on) | The optional built-in logging stack — see Part 5 |
| `operator-lifecycle-manager` | OLM itself, covered next |
| `storage` | Default `StorageClass` and CSI driver lifecycle |
| `marketplace` | OperatorHub's default catalog sources |

`oc get clusteroperators` lists every one of these with its `AVAILABLE`, `PROGRESSING`, and `DEGRADED` columns in a single view — the single most useful first command when a cluster is behaving unexpectedly, before diving into any individual component's own pods and logs.

### A Note on `openshift-*` Namespaces

Every Cluster Operator's own pods run in a dedicated, reserved `openshift-<component>` namespace (`openshift-ingress`, `openshift-image-registry`, `openshift-monitoring`, and so on) — the platform-level equivalent of upstream Kubernetes' own `kube-system`. These namespaces are not meant to host application workloads, and by default carry tighter RBAC than a regular developer-created Project (Part 2 covers exactly what a Project adds over a bare namespace, and why the distinction matters for exactly this kind of default). Recognizing this convention on sight is a fast way to distinguish "this is a platform component, check its Cluster Operator first" from "this is an application workload, check its own Deployment/Pod events first" when triaging any issue from a namespace name alone.

### Reading a Degraded ClusterOperator's Conditions

A truncated real-world `oc get clusteroperators` during an incident might look like this:

```text
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication   4.17.5    True        False         False      45d
etcd             4.17.5    True        False         False      45d
ingress          4.17.5    True        True          True       12m
kube-apiserver   4.17.5    True        False         False      45d
network          4.17.5    True        False         False      45d
```

`ingress` being `AVAILABLE=True` but also `DEGRADED=True` and `PROGRESSING=True` at once is a common, specific pattern: the router is still serving traffic (available), but is actively trying to reconcile toward a new desired state and failing part of that reconciliation (degraded) — not a total outage, but a real problem worth investigating before it escalates. `oc describe clusteroperator ingress` surfaces the operator's own condition `Message` field directly:

```text
Conditions:
  Type            Status  Message
  Degraded        True    The "default" ingresscontroller reports
                           Degraded=True: DegradedConditions: 1 of 2
                           requirements not met: canary route checks
                           are failing on 1 of 3 router pods
  Progressing     True    Not all ingress controllers are available.
  Available       True    The "default" ingresscontroller reports
                           Available=True
```

This message almost always names the specific sub-resource failing (here, a canary route check on one specific router replica) rather than requiring a blind search through every pod in the `openshift-ingress-operator` namespace — reading this field first, before diving into pod-level logs, is consistently the fastest path from "something is degraded" to "here is the actual failing component."

## The Operator Lifecycle Manager (OLM) and OperatorHub

Everything above describes how OpenShift manages its *own* first-party components. **Operator Lifecycle Manager (OLM)** is the mechanism that extends the same "operator manages a piece of software's whole lifecycle" pattern to third-party and customer-installed software — a database operator, a service mesh, a certificate manager, a logging aggregator — installed through the same declarative model rather than a bespoke Helm chart or manual YAML apply per vendor.

OLM is itself composed of two cooperating operators. The **Catalog Operator** watches `CatalogSource` objects (a registry of available Operators, packaged as `ClusterServiceVersion` (CSV) manifests plus their CRDs) and resolves a user's `Subscription` to a specific CSV, handling dependency resolution between Operators. The **OLM Operator** then takes a resolved CSV and actually deploys what it describes — typically a `Deployment` running the operator's own controller — and grants it exactly the RBAC permissions the CSV declares it needs, nothing broader.

```mermaid
stateDiagram-v2
    [*] --> Available: CSV published to a CatalogSource
    Available --> Subscribed: A Subscription targets this package/channel
    Subscribed --> InstallPlanPending: Catalog Operator resolves dependencies
    InstallPlanPending --> Installing: InstallPlan approved (manual or automatic)
    Installing --> Succeeded: OLM Operator deploys CSV's resources, RBAC applied
    Installing --> Failed: Resource creation or RBAC grant fails
    Succeeded --> Replacing: Newer CSV published in the subscribed channel
    Replacing --> Succeeded: New CSV installs, old CSV's resources cleaned up
    Failed --> [*]
```

For a cluster administrator, the practical surface of all this is **OperatorHub** — a catalog inside the web console (or `oc get packagemanifests` from the CLI) where installing a supported Operator is a "pick a channel, click subscribe" operation, with OLM handling CRD installation, RBAC, and future upgrades within that channel automatically. This is a genuinely different day-to-day experience from vanilla Kubernetes, where installing something like a certificate manager or a service mesh means the team owns tracking its Helm chart or manifest repository, its CRD versioning, and its upgrade compatibility matrix entirely by hand.

> **From the Trenches:** A team manually upgraded a third-party Operator's CSV by editing its `Deployment` image tag directly, bypassing OLM's `Subscription` mechanism, to get a bug fix faster than waiting for the vendor's next catalog channel update. The next day, OLM's Catalog Operator reconciled the `Subscription` back to the channel's last-known CSV, silently reverting the manual fix — because OLM treats the `Subscription` object, not the running `Deployment`'s current state, as the source of truth for what should be installed, exactly the same reconciliation-loop lesson as the MCO story above, just at the Operator layer instead of the node layer. The correct fix was a `CatalogSource` pointing at a private, pinned index image containing the specific CSV version needed, not a manual edit that OLM's own control loop would keep undoing.

Red Hat is also incrementally rolling out **OLM v1**, a next-generation redesign that flattens the CSV/Subscription/InstallPlan model into a simpler `ClusterExtension` API with more predictable upgrade semantics — worth being aware of as a name if reading current release notes, though the classic OLM model described above remains the one running in most production clusters as of this writing.

### A Worked Subscription Example

Installing a third-party Operator through OLM, rather than clicking through OperatorHub's console, is a `Subscription` object a platform team can commit to Git alongside everything else on the cluster:

```yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cert-manager-operator
  namespace: openshift-operators
spec:
  channel: stable-v1
  name: openshift-cert-manager-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual
```

`installPlanApproval: Manual` is the deliberate choice for anything running in production: OLM still computes the `InstallPlan` for a new version the moment one appears in the subscribed channel, but waits for an administrator to explicitly approve it (`oc patch installplan ... --type merge --patch '{"spec":{"approved":true}}'`) rather than rolling it out unattended the instant the vendor publishes a new CSV — trading a small amount of manual approval friction for the ability to review a third-party Operator's changelog before it touches a production cluster, the same judgment call this catalog's CI/CD content makes about automated deployments generally.

## Installer-Provisioned vs. User-Provisioned Infrastructure

OpenShift ships one installer binary, `openshift-install`, but it supports two fundamentally different deployment models depending on who owns the underlying infrastructure.

**Installer-Provisioned Infrastructure (IPI)** has the installer itself create every piece of infrastructure a cluster needs — VMs, load balancers, DNS records, security groups — directly against a cloud provider's API (AWS, Azure, GCP, vSphere, bare metal via a provisioning service), driven entirely by an `install-config.yaml` file. **User-Provisioned Infrastructure (UPI)** instead expects the operator to have already stood up the load balancers, DNS records, and machines themselves, following Red Hat's documented requirements, with the installer only handling the Ignition-driven bootstrap and cluster formation on top of infrastructure that already exists.

| Decision factor | Choose IPI when | Choose UPI when |
|---|---|---|
| Cloud provider support | Running on a fully-supported IPI platform (AWS, Azure, GCP, vSphere, bare metal) | Running on infrastructure IPI doesn't support, or infrastructure with constraints IPI's automation can't express |
| Existing infrastructure standards | The team is fine letting the installer own naming, tagging, and networking conventions | Infrastructure must integrate with an existing IPAM, DNS, or naming scheme the org already mandates |
| Air-gapped / disconnected environments | Rarely — IPI assumes it can reach the target cloud's API directly | Very common — UPI's staged process fits a disconnected mirror registry workflow more naturally |
| Operational ownership | The platform team is comfortable with the installer managing infrastructure lifecycle going forward | Infrastructure teams (networking, IPAM) need to retain direct ownership of the underlying resources |
| Day-2 infrastructure changes (e.g., scaling control plane) | Largely automated via `Machine`/`MachineSet` objects | Requires manually provisioning and joining machines matching the installer's expectations |

Most new deployments on a supported cloud default to IPI specifically because it also wires up the **Machine API** (covered in Part 5) automatically, giving the cluster the ability to scale its own worker nodes up and down — a capability that requires substantially more manual scaffolding to replicate under UPI.

### Two More Installation Methods: Assisted and Agent-Based

IPI and UPI are the two foundational models, but two additional installer front-ends solve specific pain points neither one addresses well on its own. The **Assisted Installer** is a hosted, Red-Hat-operated service (`console.redhat.com`) that walks an administrator through a guided, validated bare-metal or virtualized installation — it continuously validates the target hardware and network configuration against known requirements *before* the install runs, catching a large class of installation-day failures earlier than IPI/UPI's own error reporting would, at the cost of requiring the installing machine to reach Red Hat's hosted service during the process.

The **Agent-based Installer** takes the same guided-validation experience the Assisted Installer offers and packages it into a single bootable ISO with no dependency on reaching Red Hat's hosted service at all — solving exactly the gap a disconnected bare-metal deployment has: UPI's flexibility without UPI's much larger manual burden, and Assisted Installer's validation without its connectivity requirement. It also removes IPI bare-metal's need for a separate provisioning host, since one of the cluster's own nodes temporarily runs the bootstrap role in place, then rejoins the cluster as a normal node once bootstrapping finishes.

| Installer | Connectivity to Red Hat required | Validates hardware/network before install | Best fit |
|---|---|---|---|
| IPI | Yes (to the target cloud API) | Partial | Supported public clouds, standard deployments |
| UPI | No | No | Unsupported platforms, existing infra ownership, disconnected with more manual effort accepted |
| Assisted Installer | Yes (hosted service) | Yes, extensively | Bare metal/vSphere teams wanting guided validation and are connected |
| Agent-based Installer | No | Yes, extensively | Disconnected bare metal/vSphere wanting the same validation without connectivity |

```mermaid
flowchart TD
    Q1{"Is the target a fully-supported\ncloud platform (AWS, Azure, GCP, vSphere)?"}
    Q2{"Can the installing machine\nreach Red Hat's hosted services?"}
    Q3{"Does existing infra\n(DNS/IPAM/naming) need to be preserved?"}

    Q1 -->|Yes| IPI["Use IPI"]
    Q1 -->|No, bare metal/vSphere| Q2
    Q2 -->|Yes| Assisted["Use the Assisted Installer"]
    Q2 -->|No, disconnected| Agent["Use the Agent-based Installer"]
    Q1 -->|No, unsupported platform| Q3
    Q3 -->|Yes| UPI["Use UPI"]
    Q3 -->|No| IPI

    classDef choice fill:#047857,color:#fff,stroke:#065f46
    class IPI,Assisted,Agent,UPI choice
```

## A Minimal `install-config.yaml`, Walked Through

Every IPI installation starts from one file, generated interactively by `openshift-install create install-config` and then hand-edited before the actual install runs. A minimal AWS example makes the moving pieces concrete:

```yaml
apiVersion: v1
baseDomain: example.com
metadata:
  name: prod-east
controlPlane:
  name: master
  replicas: 3
  platform:
    aws:
      type: m6i.xlarge
compute:
  - name: worker
    replicas: 3
    platform:
      aws:
        type: m6i.large
networking:
  networkType: OVNKubernetes
  clusterNetwork:
    - cidr: 10.128.0.0/14
      hostPrefix: 23
  serviceNetwork:
    - 172.30.0.0/16
platform:
  aws:
    region: us-east-1
pullSecret: '<redacted — from console.redhat.com>'
sshKey: '<redacted — public key for debug node access>'
```

A few fields are worth understanding rather than copy-pasting: `metadata.name` combined with `baseDomain` forms the cluster's actual DNS domain (`prod-east.example.com` here), and every generated DNS record — the API server's, the default wildcard router route — hangs off that combination, so getting it wrong means re-running the install from scratch rather than patching it after the fact. `controlPlane.replicas: 3` and the standard/compact/SNO trade-off from later in this chapter are decided right here, before a single machine boots. `networking.networkType: OVNKubernetes` selects the default CNI this series' Part 3 covers in depth; changing it after installation is a supported but nontrivial migration, not a config-file edit. The `pullSecret` is the credential that authorizes pulling Red Hat's own release images and any subscribed content from `registry.redhat.io` — without it, the installer fails before a single machine is even provisioned.

Once edited, `openshift-install create cluster --dir=./prod-east --log-level=info` consumes this file, generates the Ignition configs referenced in the bootstrap sequence above, and drives the entire IPI provisioning flow against the target cloud's API.

### What the Installer Actually Produces on Disk

Running the installer in stages (`create manifests`, then `create ignition-configs`, rather than the all-in-one `create cluster`) — a common pattern when a team needs to hand-edit a generated manifest before it's baked into an Ignition config, for instance to tweak a control-plane machine's scheduling taints — leaves a working directory worth understanding:

```text
prod-east/
├── auth/
│   ├── kubeconfig              # admin kubeconfig for oc/kubectl once the cluster is up
│   └── kubeadmin-password      # one-time initial admin credential
├── manifests/                  # every Kubernetes-level object the installer will apply,
│                                # editable before ignition-configs are generated
├── openshift/                  # OpenShift-specific manifests (Cluster Operators' own config)
├── bootstrap.ign                # Ignition config for the temporary bootstrap machine
├── master.ign                   # Ignition config for control plane machines
└── worker.ign                    # Ignition config for worker machines
```

`manifests/` and `openshift/` are the last point at which a Kubernetes-level object (a `MachineConfig` limiting a control-plane node's own workload scheduling, for instance) can be hand-edited before it's baked into the `.ign` files the actual machines boot from — once `create ignition-configs` runs, further changes belong in a post-install `MachineConfig` applied through the normal MCO flow covered earlier in this chapter, not a re-edit of these files, since the bootstrap sequence has already consumed them by the time real machines exist to apply a change to.

## Disconnected and Mirrored-Registry Installations

A meaningful fraction of real enterprise OpenShift deployments — government, defense, and heavily regulated financial environments especially — run in **disconnected** (air-gapped) environments with no direct route to the public internet, and therefore no direct route to Red Hat's own release image registries or `registry.redhat.io`. These environments install and upgrade OpenShift from a **mirrored registry**: a private registry inside the disconnected network, populated ahead of time (from a connected machine, or a physical media transfer) with exactly the release images, Operator catalog images, and any application images the cluster will need.

The `oc-mirror` plugin (or its v2 successor) is the tool that automates this: given a manifest describing the OpenShift release and the specific Operator catalog channels needed, it pulls every referenced image from the public registries on a connected machine, packages them, and produces the exact `ImageContentSourcePolicy`/`ImageDigestMirrorSet` objects the disconnected cluster needs to transparently redirect every pull to the mirror instead of the public registry — meaning workload manifests referencing `registry.redhat.io/...` images don't need to be rewritten by hand; the cluster's own pull-through mirroring config handles the redirect.

> **From the Trenches:** A defense-sector team's disconnected cluster upgrade failed midway with image pull errors for a specific Operator dependency the platform team hadn't realized was part of the target release's dependency graph. The mirror had been populated using an `oc-mirror` manifest built for the *previous* release's Operator catalog, and the newer release pulled in an updated dependency the old mirror manifest never captured. The fix — and the resulting standing practice — was regenerating the mirror manifest from the target release's actual catalog before every upgrade, rather than reusing a manifest that happened to work for a prior version, since OLM's dependency resolution (covered earlier in this chapter) can pull in a different transitive dependency set release to release.

## Control Plane Topology and High Availability

A standard OpenShift cluster runs **three control plane nodes**, each running its own etcd member, kube-apiserver, kube-controller-manager, and kube-scheduler instance — the same odd-numbered-quorum requirement covered in this catalog's Kubernetes Deep Dive series applies identically here, since etcd's Raft consensus needs a strict majority to remain available, and three nodes tolerate exactly one node failure without losing quorum.

OpenShift also supports two topologies beyond the standard three-node HA control plane, each trading availability guarantees for a smaller resource footprint:

| Topology | Control plane nodes | Fits when |
|---|---|---|
| Standard HA | 3 dedicated control plane nodes + separate worker nodes | Production clusters where control plane resource contention with workloads is unacceptable |
| Compact (three-node) | 3 nodes acting as both control plane and worker | Smaller production or edge deployments where dedicating 3 nodes purely to control-plane duty isn't justified |
| Single Node OpenShift (SNO) | 1 node, no HA at all | Edge/far-edge deployments (telco, retail, disconnected sites) where the workload's own redundancy model tolerates a single point of cluster failure, and the physical footprint for 3+ nodes doesn't exist |
| Hosted control plane (HyperShift) | 0 dedicated nodes in the workload cluster — control plane runs as pods on a separate management cluster | Fleet operators running many clusters who want per-cluster control-plane cost and blast radius reduced, at the cost of depending on the management cluster's own availability |

A fourth, architecturally distinct option is worth naming even though it inverts the whole "3 dedicated nodes" framing above: **hosted control planes**, built on the open-source **HyperShift** project, run a cluster's control plane components as ordinary pods inside a separate, shared **management cluster**, isolated per hosted cluster by namespace, rather than on dedicated machines inside the cluster being served. ROSA with Hosted Control Planes is the commercial productization of this model on AWS. The trade-off is a genuine inversion of the standard-HA story: a hosted cluster has zero dedicated control-plane infrastructure of its own (lower cost, faster provisioning, since spinning up a new hosted cluster is "create a namespace of pods on infrastructure that already exists" rather than "provision three new machines"), but its availability now depends on the shared management cluster's own health — a real consideration for an organization evaluating this model at fleet scale, not a strictly-better replacement for the classic topologies above.

Choosing compact, SNO, or hosted control planes is a deliberate trade of cluster-level HA for footprint and cost — it does not change anything about how a developer interacts with the cluster day to day, but it materially changes the blast radius of a single node's hardware failure, and that trade-off needs to be made explicitly by whoever owns the deployment's availability requirements, not defaulted into silently by whoever happened to provision the fewest nodes to save cost.

Regardless of topology, etcd's own operational health deserves its own attention separate from the workloads running above it — this catalog's Kubernetes Deep Dive series covers etcd's Raft consensus, compaction, and defragmentation in full depth, and every word of it applies unchanged here, since OpenShift's `etcd` Cluster Operator automates the same backup and defragmentation operations a self-managed cluster would otherwise need a human or cron job to run.

### Control Plane Node Sizing

Unlike a workload node, where sizing is a straightforward function of the pods scheduled onto it, control plane node sizing has to account for a failure scenario that never applies to worker capacity planning: when one of three control plane nodes is down (a planned reboot for an MCO rollout, or an unplanned hardware failure), the remaining two must absorb the full API server, etcd, and controller-manager load without becoming resource-starved themselves, since a resource-starved API server during exactly the window when the cluster has the least redundancy is the worst possible time for it to also become slow or unresponsive.

| Cluster size | Minimum per control plane node | Why |
|---|---|---|
| Small (fewer than ~500 pods, ~25 nodes) | 4 vCPU / 16 GB RAM | Baseline production HA requirement even at small scale — etcd and the API server have a real fixed cost regardless of workload count |
| Medium (hundreds of nodes, thousands of pods) | 8 vCPU / 32 GB RAM or more | API server request volume and etcd's working set both scale with total object count, not just node count |
| Rule of thumb across all sizes | Keep steady-state utilization under ~60% of capacity | Leaves headroom for the two-of-three failover scenario above without the surviving nodes themselves becoming the bottleneck |

Under-sizing the control plane is a mistake that often stays invisible during normal operation and only surfaces during an incident — exactly the failure mode a genuinely defensive capacity plan accounts for deliberately, per the same "size for the failure scenario, not just the steady state" discipline this catalog's Reliability & SRE content applies to workload capacity planning generally.

## Verifying Cluster Health After a Fresh Install

Once `openshift-install create cluster` reports `install-complete`, a short, repeatable verification sequence confirms the cluster is actually healthy rather than merely finished — the same "verify before you're done" discipline this catalog applies everywhere else, not just to code changes.

```bash
export KUBECONFIG=./prod-east/auth/kubeconfig

# Every cluster operator should report Available=True, Progressing=False, Degraded=False
oc get clusteroperators

# Every node should be Ready, with no unexpected taints
oc get nodes

# The overall cluster version and update history
oc get clusterversion

# Confirm the default router and internal registry actually came up
oc get pods -n openshift-ingress
oc get pods -n openshift-image-registry
```

A cluster that passes all four checks is genuinely ready for workloads; a cluster where `oc get clusteroperators` shows anything other than a clean `True/False/False` pattern, or where `oc get nodes` lists a node still in `NotReady`, is not — and chasing down which specific operator or node is unhealthy right after install, while the bootstrap and Ignition logs are still fresh and available, is dramatically cheaper than debugging the same gap weeks later once a real workload has already been deployed on top of an incompletely-healthy cluster.

## Telemetry, Insights, and the Support Lifecycle

The "vendor support spanning the entire stack" row in the feature-comparison table later in this chapter is worth making concrete, since it's the practical payoff of everything covered so far, not just a marketing line.

### The Insights Operator

Every connected OpenShift cluster runs the **Insights Operator**, which periodically uploads a defined set of cluster health and configuration data (never workload data or secrets) to Red Hat, and in return receives back proactive recommendations surfaced directly in the web console — a known misconfiguration, an approaching certificate expiry, a resource nearing a hard platform limit, flagged before it becomes an outage rather than diagnosed after one. This is also the data source behind the conditional-update risk flags on the Cincinnati upgrade graph covered earlier: Red Hat's own telemetry across the whole install base is what allows a specific upgrade edge to be flagged as risky for clusters matching a specific configuration, before an individual cluster administrator would have any way to know that themselves. A cluster running fully disconnected, per the earlier section on air-gapped installs, doesn't get this proactive feedback loop at all — one of the genuine, if often overlooked, costs of a disconnected deployment beyond the mirroring overhead itself.

### Extended Update Support and the Release Cadence

OpenShift ships a new minor version roughly every few months, considerably faster than most organizations want to chase on every single release for a production fleet. **Extended Update Support (EUS)** exists specifically to make that sustainable: even-numbered minor releases (4.14, 4.16, 4.18, and so on) receive an extended support window — 24 months as a baseline, extendable to 36 months with an additional subscription term — specifically so a platform team can standardize on upgrading once per EUS cycle rather than chasing every minor release, while still receiving CVE backports and critical bug fixes throughout that whole window.

| Release type | Support window | Typical fit |
|---|---|---|
| Standard (odd-numbered) minor release | Shorter, standard support window | Teams that want to stay on the latest features and are comfortable upgrading every release |
| EUS (even-numbered) minor release | 24 months baseline, extendable to 36 | Production fleets standardizing on a slower, predictable upgrade cadence |

A team planning a fleet-wide upgrade strategy should pick EUS versions as landing points deliberately, the same way a team on a long-term-support Linux distribution plans around LTS release boundaries rather than chasing every point release — this is a direct, practical consequence of the CVO's "one tested release payload" model from earlier in this chapter, since it's what makes committing to stay on a specific version for two-plus years a supportable choice rather than a security liability.

## The `oc` CLI and Web Console

Every example so far has used `oc`, and it's worth being precise about what it actually is: a superset of `kubectl` that talks to exactly the same API server, understands every native Kubernetes resource identically, and adds a handful of OpenShift-specific ergonomics on top. Nothing about switching from `kubectl` to `oc` changes how a raw Kubernetes manifest behaves — `oc apply -f deployment.yaml` and `kubectl apply -f deployment.yaml` do exactly the same thing against an OpenShift cluster, because they're issuing the same API request.

What `oc` adds is mostly around faster, more opinionated workflows for OpenShift-specific resources and common developer tasks:

```bash
oc whoami                       # who the current login token authenticates as
oc new-project my-team-dev      # create a Project (Part 2) with sane default RBAC/quotas
oc new-app quay.io/myorg/api:v3 # generate an ImageStream, DeploymentConfig/Deployment,
                                 # and Service from an image in one command
oc get all                      # a broader "everything relevant" view than plain kubectl
oc explain route.spec           # inline API documentation for OpenShift-native resources
oc status                       # a human-readable summary of a project's deployed topology
```

`oc new-app` in particular is worth calling out: it inspects the target image (or a source Git repository, tying into the BuildConfig/S2I mechanism Part 4 covers) and generates a reasonable set of starting objects in one command, a genuinely faster "get something running to iterate on" path than hand-writing a Deployment/Service pair from scratch — though, like any scaffolding tool, the objects it generates are a starting point for a real manifest a team commits to Git, not a substitute for reviewing and owning that manifest afterward.

The **web console** mirrors this same relationship: a Developer perspective aimed at application deployment and topology visualization, and an Administrator perspective exposing cluster-level configuration (the same `MachineConfigPool`, `ClusterOperator`, and `Subscription` objects covered earlier in this chapter, presented as forms and status views instead of YAML) — genuinely useful for a team getting oriented, but every action either perspective takes ultimately reduces to the same Kubernetes API calls `oc`/`kubectl` would issue directly, which is why nothing in this series treats the console as a separate mechanism from the CLI-driven examples used throughout.

> **From the Trenches:** A new platform hire, used to a previous shop's plain-`kubectl` workflow, assumed `oc new-app` was a special OpenShift-only deployment mechanism distinct from writing a Deployment manifest by hand, and treated the objects it generated as something to leave alone rather than review — until a routine security review flagged that the generated `Deployment` had no resource limits set at all, which is exactly what `oc new-app`'s reasonable-but-generic defaults produce when given no further input. The fix wasn't avoiding `oc new-app` — it's a genuinely useful scaffold — but treating its output the same as any other manifest a team commits to Git: reviewed, tuned, and owned, not a black box because it happened to come from a convenience command instead of a text editor.

## OpenShift vs. Vanilla Kubernetes — A Full Feature Comparison

Bringing every thread in this chapter together, the practical, feature-by-feature difference a team evaluating both options actually needs to weigh:

| Concern | Vanilla Kubernetes (e.g., `kubeadm`, a bare EKS/GKE cluster) | OpenShift |
|---|---|---|
| Operating system | Team's choice — commonly Ubuntu or Amazon Linux, self-managed patching | RHCOS, immutable, MCO-managed |
| Container runtime | Team's choice, commonly `containerd` | CRI-O by default |
| Ingress | Team installs and maintains an ingress controller (nginx, Traefik, etc.) | Router/Routes built in by default — see Part 3 |
| Internal registry | Team stands up and operates one (Harbor, ECR, GCR) | Integrated registry built in — see Part 4 |
| Build tooling | Team wires up CI to build images externally | BuildConfigs/Source-to-Image built in — see Part 4 |
| Pod-level security default | Open by default; team must add Pod Security Admission/OPA/Kyverno | Security Context Constraints enforced by default — see Part 2 |
| Multi-tenant namespace defaults | Namespaces are bare; team adds quotas/network policies themselves | Projects auto-apply quotas, limits, and network isolation defaults — see Part 2 |
| Third-party software lifecycle | Team tracks Helm charts/manifests and upgrade compatibility itself | OLM + OperatorHub, curated and version-tested |
| Cluster upgrades | Team upgrades each component (CNI, ingress, metrics stack) independently | CVO upgrades the whole tested release payload as one unit |
| Built-in monitoring/logging | Team selects and operates its own stack | Prometheus/Alertmanager/Grafana and optional logging stack built in — see Part 5 |
| Vendor support | Depends entirely on the managed offering (or none, if self-managed) | Red Hat-supported, single point of accountability across the whole stack |
| Flexibility to swap any component | High — every piece is independently replaceable | Lower — swapping a default component (e.g., the CNI) is a supported-but-nontrivial deviation from the tested combination |
| Underlying Kubernetes API | Standard | Standard, unmodified, CNCF-conformant |
| Default storage provisioning | Team selects and wires up a CSI driver and `StorageClass` | The `storage` Cluster Operator ships a sensible default `StorageClass` per supported platform out of the box |
| Air-gapped/disconnected support | Possible, but every component's mirroring is the team's own responsibility | First-class `oc-mirror` tooling and a documented disconnected install path across the whole stack |
| Node OS patch/CVE response | Team's own patching cadence and tooling per node | MCO-driven, fleet-wide, coordinated with the same release payload as everything else |
| Telemetry-driven upgrade safety | Not present unless the team builds it | Cincinnati conditional updates, informed by Red Hat's own install-base telemetry |
| Long-term support planning | Depends entirely on the managed offering's own policy, if any | EUS releases give a documented 24-36 month support window to plan a fleet upgrade cadence around |

The pattern across nearly every row is the same: OpenShift trades flexibility for a jointly-tested, jointly-supported, opinionated default. Neither direction is universally correct — it's the same "does the opinionation solve a real, currently-felt pain, or add overhead against a problem that doesn't exist yet" judgment call this catalog's Docker Container Fundamentals series makes about adopting Kubernetes in the first place, one layer up the stack.

## Multi-Architecture and Hybrid Cloud Reach

One more structural property worth naming before the family-of-distributions comparison below: the same OpenShift release is supported across x86_64, 64-bit ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) architectures, and across every major public cloud plus on-premises virtualization and bare metal — a materially broader reach than most individual pieces of a hand-assembled Kubernetes stack can claim on their own, since a hand-picked CNI plugin, ingress controller, or storage driver each carry their own independent architecture and platform support matrix that a platform team would otherwise have to reconcile manually.

This matters concretely for two common enterprise scenarios. A financial or telecom organization running IBM Z mainframes alongside x86 cloud infrastructure can run the same OpenShift version, the same Operators, and the same application manifests across both architectures, rather than maintaining a structurally different container platform per architecture. A multi-cloud strategy (this catalog's Multi-Cloud Architecture content covers the broader trade-offs of that strategy itself) can standardize on one platform's operational model — the same CVO upgrade cadence, the same OLM catalog, the same Security Context Constraints — across AWS, Azure, GCP, and on-premises clusters simultaneously, which is a meaningfully different starting point than reconciling each cloud's own managed Kubernetes offering's independent feature set and upgrade cadence.

| Consideration | Vanilla Kubernetes across clouds/architectures | OpenShift across clouds/architectures |
|---|---|---|
| Operational model consistency | Each managed offering (EKS/GKE/AKS) has its own upgrade cadence, defaults, and quirks | One consistent operational model (CVO, MCO, OLM) regardless of underlying cloud or architecture |
| Multi-architecture workload support | Depends on each component's own architecture support matrix | Officially supported across x86_64, ARM, Power, and Z as one product line |
| Skills transfer between environments | Partial — cloud-specific tooling and defaults differ meaningfully | High — the same `oc`, the same Projects/Routes/BuildConfigs model, everywhere |

This reach also underlies **OpenShift Virtualization**, an optional Operator (built on KubeVirt) that runs traditional virtual machines as first-class objects alongside containers on the same cluster — a genuinely separate capability from anything covered in this series, worth knowing exists as a name for organizations mid-migration from a VM-only estate, but out of scope for a series focused on OpenShift as a container platform.

## Distributions in the OpenShift Family

"OpenShift" itself names a family of related offerings sharing the same core architecture but differing in who operates the control plane and where it runs:

| Distribution | Who manages the control plane | Typical fit |
|---|---|---|
| OpenShift Container Platform (OCP) | The customer, self-managed, on their own infrastructure or a supported cloud | Full control over placement, networking, and upgrade timing; the customer owns Day-2 operations |
| Red Hat OpenShift Service on AWS (ROSA) | Jointly managed by Red Hat and AWS as a managed service | Teams wanting OpenShift's model without operating the control plane themselves, already committed to AWS |
| Azure Red Hat OpenShift (ARO) | Jointly managed by Red Hat and Microsoft | Same trade-off as ROSA, for teams committed to Azure |
| OKD | The upstream, community-maintained, unsupported distribution OCP is built from | Evaluation, learning, and non-production use where Red Hat's commercial support isn't needed |
| MicroShift | Self-managed, single small binary | Edge and resource-constrained device deployments needing OpenShift's API surface without the full control-plane footprint |

Everything this series covers applies to OCP directly, and applies to ROSA/ARO from an application-developer and cluster-administrator perspective identically — the difference between them is almost entirely in who is paged when the control plane itself has a problem, not in how a workload is deployed or how Projects, Routes, or BuildConfigs behave.

```mermaid
quadrantChart
    title Control-plane ownership vs. operational footprint
    x-axis Self-managed control plane --> Vendor-managed control plane
    y-axis Small footprint --> Full enterprise footprint
    quadrant-1 Managed, full-featured
    quadrant-2 Self-managed, full-featured
    quadrant-3 Self-managed, minimal
    quadrant-4 Managed, minimal
    OCP: [0.2, 0.85]
    OKD: [0.15, 0.55]
    ROSA: [0.8, 0.85]
    ARO: [0.8, 0.8]
    MicroShift: [0.15, 0.15]
```

Reading this diagram alongside the table above, the two axes map directly onto the two questions worth asking when picking a distribution: who is operationally responsible if the control plane itself misbehaves, and how much of the full enterprise feature set (OperatorHub's full catalog, the built-in logging stack, multi-cluster management add-ons) the deployment actually needs versus a minimal single-purpose footprint.

## Quick Reference: Key Terms From This Chapter

A fast lookup for the vocabulary this chapter introduced, before moving on to Part 2's developer- and administrator-facing concepts:

| Term | What it is |
|---|---|
| RHCOS | The immutable, OSTree-based operating system every RHCOS-based node runs |
| Ignition | The one-shot, first-boot provisioning tool that applies a node's initial declared state |
| CRI-O | The Kubernetes-only container runtime OpenShift ships by default, in place of `containerd` |
| Bootstrap machine | A temporary control plane used only to stand up the real one, then destroyed |
| Cluster Version Operator (CVO) | The operator that manages every other Cluster Operator as one tested release payload |
| Machine Config Operator (MCO) | The operator managing everything between the kernel and the kubelet on every node |
| `MachineConfig` / `MachineConfigPool` | Declared node-level state, and the named group of nodes it targets |
| Cluster Operator | Any of the ~25 operators each owning one platform concern (networking, ingress, registry, etc.) |
| Operator Lifecycle Manager (OLM) | The Catalog Operator + OLM Operator pair managing third-party Operator installs and upgrades |
| CSV (ClusterServiceVersion) | The manifest describing one version of an Operator and what it needs to run |
| IPI / UPI | Installer-provisioned vs. user-provisioned infrastructure, the two foundational install models |
| Assisted / Agent-based Installer | Guided, validation-heavy installers for bare metal, connected and disconnected respectively |
| Hosted control planes (HyperShift) | Running a cluster's control plane as pods on a separate management cluster instead of dedicated nodes |
| Cincinnati | The upgrade-graph service behind `oc adm upgrade`'s recommended and conditional update paths |
| Insights Operator | The telemetry agent providing proactive, cluster-specific recommendations and upgrade-risk data |
| EUS (Extended Update Support) | The longer support window on even-numbered minor releases, for fleet-wide upgrade planning |
| `oc` | The OpenShift CLI, a superset of `kubectl` against the same conformant API |

## Common Mistakes and Interview Traps

| Mistake or claim | Why it is wrong | Better answer |
|---|---|---|
| "OpenShift is a fork of Kubernetes with a different API." | The Kubernetes API surface is unmodified and CNCF-conformant; OpenShift adds operators and platform services around it, not a divergent core API. | Describe OpenShift as a curated distribution, not a fork — `kubectl` works against it exactly as it would against any conformant cluster. |
| "RHCOS nodes can be patched like any RHEL server via SSH and a package manager." | Direct package installs don't survive the Machine Config Operator's next reconciliation of the node's declared state. | Any persistent OS-level change goes through a `MachineConfig`, not a manual SSH session. |
| "CRI-O and containerd are functionally interchangeable, so the choice doesn't matter." | Both are CRI-compliant, but they differ meaningfully in scope and attack surface, which is exactly why OpenShift defaults to the narrower one. | Frame the choice as a supply-chain/attack-surface decision, not a functional one. |
| "Editing a Deployment's image tag is a valid way to upgrade an OLM-managed Operator faster." | OLM's Catalog Operator reconciles the Subscription back to the channel's CSV, silently reverting a manual edit. | Point the Subscription at a catalog with the desired CSV version, or wait for the channel to publish it. |
| "IPI is always the better choice because it's more automated." | UPI exists specifically for cases IPI's automation can't serve — disconnected environments, existing IPAM/DNS ownership requirements, unsupported platforms. | Choose based on the decision factors (existing infra standards, connectivity, ownership), not "more automated is always better." |
| "A compact three-node cluster has the same availability guarantees as a standard HA cluster." | Compact clusters run workloads on the same nodes as the control plane, so a workload-induced resource exhaustion can affect control-plane stability in a way a dedicated HA topology avoids. | Name the actual trade-off (footprint/cost vs. blast-radius isolation), not "they're equivalent." |
| "ROSA and ARO are just OpenShift installed on someone else's cloud account." | The control plane itself is jointly managed by Red Hat and the cloud provider, a materially different operational model than self-managed OCP even running on the same cloud. | Distinguish "who is paged for the control plane" as the actual differentiator, not just "whose data center." |
| "An Ignition config can be edited after a node has already booted to change its configuration." | Ignition runs exactly once, on first boot, before the node's own init system starts — it has no mechanism to re-apply later. | Ongoing configuration changes go through the Machine Config Operator rewriting `MachineConfig` objects, not a re-run of Ignition. |
| "RHCOS extensions are a general-purpose way to install any RPM package the cluster needs." | Extensions are a small, curated, Red-Hat-supported list; arbitrary package installation still isn't a supported pattern on an immutable OS. | Software with more complex runtime needs than a short curated extension list belongs in a container, not layered onto the host OS. |
| "Any newer OpenShift version is safe to upgrade to once it's generally available." | The Cincinnati upgrade graph can mark a specific edge as conditionally risky for a cluster's specific configuration, even after general availability. | Always run `oc adm upgrade` and read any conditional-update warnings before choosing a target version, rather than assuming newer is always safe. |

## Worked Practice Problems

### 1. A cluster administrator wants to add a custom `sysctl` setting to only the nodes running a specific stateful workload, and wants it to survive both reboots and node replacement via autoscaling. What's the correct mechanism, and why would a `DaemonSet initContainer` be the wrong tool here?

The correct mechanism is a `MachineConfig` targeting a custom `MachineConfigPool` whose node selector matches only the nodes running that workload — this makes the setting part of the node's own MCO-reconciled desired state, so it's re-applied identically whenever a node in that pool is replaced or rebooted. A `DaemonSet initContainer` only runs when its pod is scheduled, and a freshly-provisioned replacement node has no guarantee that pod runs and completes before the actual workload schedules onto it — creating a race where the sysctl might not be set yet when the stateful workload starts, exactly the kind of silent drift the Machine Config Operator exists to prevent.

### 2. `oc get clusteroperators` shows the `monitoring` operator as `Degraded` while every other operator reports healthy. A teammate suggests restarting the whole cluster to fix it. Is that the right first step?

No — restarting the whole cluster is a broad, high-blast-radius action for what the CVO's own reporting has already scoped down to one specific operator. Since Cluster Operators report their health independently, the correct first step is investigating the `monitoring` operator's own pods, events, and logs in its namespace (typically checking `oc describe clusteroperator monitoring` for the specific condition message first), the same targeted diagnosis this chapter's cluster-operator roster table is meant to enable — a full cluster restart risks introducing new problems across every other healthy component while not being guaranteed to fix an issue that's likely specific to that operator's own configuration or a resource it depends on.

### 3. A team evaluating OpenShift for a five-person startup running one small cluster asks whether it's worth adopting over a managed EKS cluster with a hand-picked ingress controller and Helm-installed cert-manager. What's the honest framing of that decision, based on this chapter?

The honest framing is that OpenShift's core value — jointly-tested Day-2 upgrades across the whole platform, a default-restrictive security posture enforced cluster-wide, and vendor support spanning the entire stack — pays off most clearly for organizations running many teams/clusters where the coordination and audit overhead of hand-assembling and maintaining that many independent components becomes the actual bottleneck. A five-person team running one cluster is exactly the profile this chapter's opening section flags as a case where the opinionation may be pure overhead relative to picking a lightweight ingress controller and cert-manager directly — the decision should hinge on whether the team's actual pain point (audit/compliance requirements, multi-team coordination, vendor-support requirements) matches what OpenShift specifically solves, not on OpenShift being categorically "more enterprise" and therefore better by default.

### 4. A cluster administrator runs `oc adm upgrade` and sees the desired target version listed under a "Supported but not recommended" heading with a linked known-issue. A teammate argues the cluster should upgrade anyway since the version is generally available and the alternative is falling further behind on patches. How should this be resolved?

"Generally available" and "safe for this specific cluster" are different claims, and the Cincinnati upgrade graph's conditional-update flag exists precisely to distinguish them — the linked known issue should be read first to determine whether it actually applies to this cluster's configuration (a specific platform, a specific feature gate, a specific node count range) before deciding. If the known issue doesn't apply to this cluster's actual configuration, proceeding is reasonable and the "falling behind on patches" concern is valid; if it does apply, the correct response is waiting for a subsequent patch release that resolves the flagged issue (which the upgrade graph will surface once available) rather than proceeding into a documented, cluster-relevant risk purely to avoid the discomfort of staying on the current version slightly longer.

## Summary and What's Next

OpenShift's architecture is best understood as four additions wrapped around an unmodified Kubernetes core: RHCOS and Ignition give every node a declared, atomically-updated operating system instead of a hand-patched one; CRI-O narrows the container runtime's own attack surface to exactly what Kubernetes needs; the Cluster Version Operator and Machine Config Operator extend the "declared desired state, continuously reconciled" model from workloads all the way down to the cluster's own components and the node's own operating system; and the Operator Lifecycle Manager extends that same pattern to third-party software, replacing ad hoc Helm-chart tracking with a curated, dependency-aware catalog. None of this changes how `kubectl`/`oc` talks to the API server — it changes who owns keeping the layers underneath that API server correct, patched, and consistent across every node in the fleet.

The installation model — IPI, UPI, Assisted, and Agent-based, plus the hosted-control-plane inversion of the whole "dedicated control-plane nodes" assumption — is a genuine set of trade-offs a team makes deliberately based on their actual infrastructure ownership and connectivity constraints, not a single "best" choice; the same is true of choosing a standard, compact, or single-node control-plane topology, and of choosing which OpenShift family member (self-managed OCP, ROSA, ARO, OKD, MicroShift) fits a given team's appetite for owning the control plane themselves. Layered on top of all of it, telemetry-driven safety nets — the Insights Operator's proactive recommendations and the Cincinnati upgrade graph's conditional-update warnings — and a deliberate Extended Update Support cadence are what make committing to run this platform at multi-year, multi-cluster scale a supportable decision rather than a leap of faith.

Part 2 moves up from the platform's own architecture to what a developer or namespace administrator actually experiences day to day: Projects as OpenShift's namespace abstraction, Security Context Constraints as the default-restrictive security model this chapter's audit story referenced, and the multi-tenancy defaults that make "a new team can safely self-service a new environment without a platform engineer manually configuring quotas and network isolation for them" a built-in property rather than something every organization re-invents for itself. Keep this chapter's platform/application boundary in mind going in: everything from here through the rest of the series lives above the line this chapter drew, and rarely needs to reach back down into RHCOS, the CVO, or the MCO to make sense.

Sources consulted for this chapter: Red Hat's OpenShift Container Platform Architecture documentation (RHCOS, CRI-O, installation overview), the OpenShift Operator Lifecycle Manager documentation, the `openshift/installer` project's IPI/UPI and Agent-based installer documentation, the `openshift/machine-config-operator` project's MachineConfig/MachineConfigPool design documentation, Red Hat's Hosted Control Planes/HyperShift documentation, and Red Hat's OpenShift Container Platform Life Cycle and Extended Update Support policy pages.