# Red Hat OpenShift: Enterprise Kubernetes Platform — Part 5: Day-2 Operations: Cluster Operators, Upgrades & Observability

> **Series:** Red Hat OpenShift: Enterprise Kubernetes Platform (5 of 5)
> **Part 1:** `01-architecture-and-openshift-vs-kubernetes.md` — Architecture & What OpenShift Adds Over Vanilla Kubernetes
> **Part 2:** `02-projects-sccs-and-multi-tenancy.md` — Projects, Security Context Constraints & Multi-Tenancy
> **Part 3:** `03-networking-routes-and-service-mesh.md` — Networking, Routes & OpenShift Service Mesh
> **Part 4:** `04-buildconfigs-s2i-imagestreams-and-pipelines.md` — BuildConfigs, S2I, ImageStreams & OpenShift Pipelines/GitOps
> **Part 5:** This file — Day-2 Operations: Cluster Operators, Upgrades & Observability
> **Questions:** `questions.md`

## Table of Contents

1. [What "Day 2" Means, and Why It Deserves Its Own Chapter](#what-day-2-means-and-why-it-deserves-its-own-chapter)
2. [Performing a Cluster Upgrade in Practice](#performing-a-cluster-upgrade-in-practice)
3. [Handling a Stuck or Degraded Upgrade](#handling-a-stuck-or-degraded-upgrade)
4. [The Machine API — Machines and MachineSets](#the-machine-api--machines-and-machinesets)
5. [Cluster Autoscaling: ClusterAutoscaler and MachineAutoscaler](#cluster-autoscaling-clusterautoscaler-and-machineautoscaler)
6. [MachineHealthChecks — Automatic Node Remediation](#machinehealthchecks--automatic-node-remediation)
7. [Node Maintenance: Cordon, Drain, and Safe Reboot](#node-maintenance-cordon-drain-and-safe-reboot)
8. [etcd Backup and Disaster Recovery](#etcd-backup-and-disaster-recovery)
9. [Restoring From an etcd Backup](#restoring-from-an-etcd-backup)
10. [Beyond etcd: Application Backup With OADP](#beyond-etcd-application-backup-with-oadp)
11. [The Built-In Monitoring Stack](#the-built-in-monitoring-stack)
12. [User Workload Monitoring](#user-workload-monitoring)
13. [A Worked Example: A Custom Alert for a User Workload](#a-worked-example-a-custom-alert-for-a-user-workload)
14. [The Built-In Logging Stack: Loki and the Cluster Logging Operator](#the-built-in-logging-stack-loki-and-the-cluster-logging-operator)
15. [OLM Day-2 Patterns: Channels and Approval Strategies Revisited](#olm-day-2-patterns-channels-and-approval-strategies-revisited)
16. [Putting It Together: A Day-2 Operations Runbook](#putting-it-together-a-day-2-operations-runbook)
17. [Quick Reference: Key Terms From This Chapter](#quick-reference-key-terms-from-this-chapter)
18. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
19. [Worked Practice Problems](#worked-practice-problems)
20. [Summary and Series Wrap-Up](#summary-and-series-wrap-up)

## What "Day 2" Means, and Why It Deserves Its Own Chapter

Every prior chapter in this series covered a piece of standing up and using the platform: architecture (Part 1), multi-tenancy (Part 2), networking (Part 3), delivery (Part 4). "Day 2" is the industry term for everything that happens *after* a cluster is running production workloads — upgrades, capacity changes, disaster recovery, and the ongoing observability practice that tells a team whether the platform is actually healthy — and it deserves its own chapter because a cluster's Day 2 operational quality is what determines whether it survives years of real production use, not just an initial rollout.

This chapter revisits several objects earlier chapters already introduced — the Cluster Version Operator (Part 1), the Machine Config Operator (Part 1), OLM (Part 1) — and goes one level deeper into the actual Day-2 mechanics a platform team runs against them repeatedly over a cluster's lifetime, rather than the one-time architectural understanding those earlier chapters focused on.

| Day-2 concern | What it actually protects against |
|---|---|
| Upgrades | Falling behind on security patches and losing EUS-window support (Part 1) |
| Capacity management (Machine API, autoscaling) | Under-provisioned capacity during real demand, or paying for permanently over-provisioned idle nodes |
| Node/hardware health (`MachineHealthCheck`) | A slow-to-notice hardware failure quietly degrading one workload's reliability |
| Disaster recovery (etcd, OADP) | Total, unrecoverable loss of cluster configuration or application data |
| Observability (monitoring, logging) | Not knowing the platform is unhealthy until a customer notices first |

Each row in this table is a distinct failure mode this chapter's mechanisms exist specifically to close — worth keeping explicit, since Day-2 operations work is easy to under-invest in precisely because its payoff is an incident that never happens, rather than a visible feature shipped.

> **From the Trenches:** An organization's platform team spent its first year almost entirely on Part 1 through Part 4 concerns — getting workloads onto the cluster, securing multi-tenancy, wiring up delivery pipelines — and treated Day-2 operations as something to "get to eventually." The eventual reckoning came during the cluster's first real upgrade a year in: no one had ever actually tested an etcd restore, the `ClusterAutoscaler` had been configured once and never revisited despite the workload footprint tripling, and nobody could say with confidence which OLM Operators were even on a supported upgrade path. None of this was a crisis in isolation, but all of it surfaced simultaneously during the highest-stakes operation the cluster had faced yet — the fix wasn't heroics during that upgrade, it was retroactively building exactly the standing runbook this chapter closes with, then treating it as a genuinely first-class, continuously-maintained practice going forward rather than a one-time cleanup exercise.

## Performing a Cluster Upgrade in Practice

Part 1 covered the CVO's architecture and the Cincinnati upgrade graph's conditional-update warnings; this section covers actually running one. The full sequence, in order:

```bash
oc adm upgrade

oc adm upgrade channel stable-4.19

oc get clusterversion version -o jsonpath='{.status.availableUpdates}'

oc adm upgrade --to=4.19.5

watch oc get clusteroperators

oc get clusterversion version -o jsonpath='{.status.history}'
```

Before triggering the upgrade itself, two preconditions from this chapter's own disaster-recovery section deserve to be non-negotiable, not optional: a **recent etcd backup**, and every installed OLM Operator (Part 1) already updated to a version compatible with the target OpenShift release, since an incompatible third-party Operator can itself become the reason a cluster looks unhealthy immediately post-upgrade, muddying the diagnosis of what's actually wrong.

```mermaid
flowchart TD
    A["Pre-flight: etcd backup taken,\nOLM Operators compatibility-checked"] --> B["oc adm upgrade --to=X"]
    B --> C["Control plane updates first:\nAPI server, etcd, controllers"]
    C --> D["Machine Config Operator rolls\nnew RHCOS/config to master pool"]
    D --> E["MCO rolls new config\nto worker pool(s)"]
    E --> F["Every Cluster Operator\nreconciles to the new release"]
    F --> G["oc get clusterversion reports\nthe new version, Available=True"]

    classDef step fill:#047857,color:#fff,stroke:#065f46
    class A,B,C,D,E,F,G step
```

The control plane always updates before worker nodes — a deliberate ordering, since the API server and its supporting components need to be running the new version before it's safe for anything else in the cluster to start assuming that version's behavior. Worker node updates then roll one node at a time within each `MachineConfigPool` (Part 1), cordoning, draining, and uncordoning exactly as any other MCO-driven change does, meaning an upgrade's actual worker-facing disruption is bounded to whatever workloads' own `PodDisruptionBudget`s permit during that rolling drain — a `PodDisruptionBudget` that's too strict (allowing zero disruption) can stall an upgrade indefinitely waiting for a node to safely drain, which is exactly the next section's most common failure mode.

### Pausing a MachineConfigPool During a Sensitive Window

A `MachineConfigPool` can be explicitly paused, deferring any pending worker-node rollout (including one triggered by an in-progress upgrade) until a team is ready for it — useful for a scenario where the control plane has already updated, but rolling worker changes during a specific business-critical window (a retail Black Friday peak, a financial close period) is unacceptable:

```bash
oc patch mcp worker --type merge -p '{"spec":{"paused":true}}'

# ... business-critical window passes ...

oc patch mcp worker --type merge -p '{"spec":{"paused":false}}'
oc get mcp worker -w
```

Pausing is a deliberate, temporary override — the pool's rollout resumes exactly where it left off once unpaused, but a pool left paused indefinitely means that pool's nodes silently drift further behind the rest of the cluster's own declared state with every passing day, the same "the reconciler is what actually keeps the cluster correct" property this series has emphasized throughout; pausing should always have a planned un-pause, not be treated as a permanent configuration.

## Handling a Stuck or Degraded Upgrade

```bash
oc get clusterversion version -o jsonpath='{.status.conditions}'

oc get clusteroperators | grep -v "True.*False.*False"

oc get nodes -o wide

oc get pdb --all-namespaces

oc get mcp
```

| Symptom | Likely cause | Fix |
|---|---|---|
| A specific `ClusterOperator` shows `Degraded=True` | That operator's own reconciliation is failing against the new release | `oc describe clusteroperator <name>` for the specific condition message, per Part 1's diagnosis pattern |
| A worker node stuck `NotReady` or not draining | An overly strict `PodDisruptionBudget` blocking eviction | Temporarily relax the PDB, or confirm the workload actually tolerates a brief disruption |
| The upgrade appears to hang with no operator reporting a problem | A slow, still-legitimately-in-progress MCO rollout on a large node count | Confirm via `oc get mcp` that pools are genuinely still `Updating`, not stalled, before assuming a failure |
| A third-party Operator's Pods crash-loop post-upgrade | The Operator wasn't updated to a release-compatible version first | Update the Operator's `Subscription` channel and re-verify before resuming |
| The upgrade appears to have reversed itself, back to the old version | A misread of `oc get clusterversion` mid-rollout — the field showing the *target*, not current, version was checked | Read `status.history` explicitly for the actual completed-vs-in-progress entries, not just the top-level version field |
| A node reboots repeatedly during the MCO rollout without ever completing | A kernel-argument or `MachineConfig` change conflicting with that node's specific hardware | Isolate to a custom `MachineConfigPool` (Part 1) and investigate that node's own console/serial logs directly |

> **From the Trenches:** A cluster upgrade stalled for hours with every `ClusterOperator` reporting healthy and no obvious error anywhere in the CVO's own status. The actual cause was a single, forgotten `PodDisruptionBudget` on a stateful workload set to `maxUnavailable: 0` — the MCO's own drain step was correctly refusing to violate it, silently blocking that one node (and, transitively, the rest of that `MachineConfigPool`'s rollout) indefinitely rather than proceeding unsafely. `oc get pdb --all-namespaces` surfaced the offending PDB in seconds once someone thought to check it — the upgrade wasn't broken at all; it was correctly waiting on a constraint nobody had remembered to account for before starting.

## The Machine API — Machines and MachineSets

The **Machine API** — a set of Cluster Operators and CRDs, distinct from Kubernetes' own `Node` object — is what gives OpenShift (on a supported IPI platform) the ability to provision, scale, and replace its own worker nodes declaratively, the same "declared desired state, continuously reconciled" model this series has applied at every other layer, now extended to the infrastructure the cluster itself runs on.

```yaml
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: prod-east-worker-us-east-1a
  namespace: openshift-machine-api
  labels:
    machine.openshift.io/cluster-api-cluster: prod-east
spec:
  replicas: 3
  selector:
    matchLabels: { machine.openshift.io/cluster-api-machineset: prod-east-worker-us-east-1a }
  template:
    spec:
      providerSpec:
        value:
          instanceType: m6i.large
          placement: { availabilityZone: us-east-1a }
```

| Object | Role |
|---|---|
| `MachineSet` | Declares a desired count of identically-configured `Machine`s in one zone/instance-type combination |
| `Machine` | One actual provisioned instance — the Machine API's equivalent of a `Node`, but describing the underlying infrastructure, not just the kubelet-registered object |
| `MachineConfigPool` (Part 1) | A separate concept — governs OS-level configuration, not provisioning/scaling |
| `MachineSet.status.replicas` vs `.readyReplicas` | The declared count versus how many have actually finished joining as healthy Nodes — worth checking both during a scale-out, not just the first |

```bash
oc scale machineset prod-east-worker-us-east-1a --replicas=5 -n openshift-machine-api
oc get machines -n openshift-machine-api
oc get machineset prod-east-worker-us-east-1a -n openshift-machine-api \
  -o jsonpath='{.status.replicas}/{.status.readyReplicas}{"\n"}'
```

Scaling a `MachineSet` up or down is the direct, manual mechanism for capacity changes — provisioning a new cloud instance, waiting for it to join the cluster as a `Node`, or (scaling down) cordoning and draining a `Machine` before terminating its underlying instance, all handled by the Machine API's own controllers without a human needing to touch the cloud provider's own console or API directly.

### Controlling Which Machine Gets Removed on Scale-Down

Scaling a `MachineSet` down doesn't let a team choose which specific `Machine` is removed by default — the controller picks based on its own internal heuristics (newest first, by default). A `machine.openshift.io/delete-machine: "true"` annotation on a specific `Machine` overrides that default, marking it as the preferred deletion target the next time that `MachineSet` scales down:

```bash
oc annotate machine prod-east-worker-us-east-1a-x7k2p \
  machine.openshift.io/delete-machine="true" -n openshift-machine-api

oc scale machineset prod-east-worker-us-east-1a --replicas=2 -n openshift-machine-api

oc get machines -n openshift-machine-api -w
oc get machine prod-east-worker-us-east-1a-x7k2p -n openshift-machine-api
```

This is the mechanism worth knowing for a real, common scenario: a specific node has developed a known-flaky pattern (not yet unhealthy enough for `MachineHealthCheck` to intervene, covered later in this chapter, but clearly worth replacing proactively) — annotating it for deletion and then scaling down and back up replaces exactly that machine, rather than leaving the outcome to the controller's own default heuristic and hoping it happens to pick the right one.

### Provider-Specific Differences Worth Knowing

The Machine API's object model (`Machine`, `MachineSet`) is identical across every supported platform, but `providerSpec` — the platform-specific configuration block — differs meaningfully by cloud:

| Platform | `providerSpec` covers |
|---|---|
| AWS | Instance type, AMI, availability zone, IAM instance profile, security groups |
| Azure | VM size, image reference, availability zone, network security group |
| GCP | Machine type, image, zone, service account |
| vSphere/bare metal | Template, resource pool, network, disk size — no "instance type" concept at all |
| IBM Cloud / IBM Z / IBM Power | Profile, image, zone/LPAR-specific fields reflecting that architecture's own provisioning model |

A `MachineSet` manifest authored for AWS is not portable as-is to Azure or GCP — only the surrounding object model (`replicas`, `selector`, the overall shape) is shared; the `providerSpec` block itself needs to be authored per-platform, which is worth setting expectations around explicitly for any team scripting `MachineSet` creation across a genuinely multi-cloud OpenShift footprint (Part 1's multi-architecture/hybrid-cloud coverage).

## Cluster Autoscaling: ClusterAutoscaler and MachineAutoscaler

Manually scaling `MachineSet`s works, but doesn't respond to actual demand automatically — **two** cooperating objects close that gap, and it's a common interview trap to describe only one of them as "the autoscaler."

```yaml
apiVersion: autoscaling.openshift.io/v1
kind: ClusterAutoscaler
metadata:
  name: default
spec:
  resourceLimits:
    maxNodesTotal: 24
  scaleDown:
    enabled: true
    delayAfterAdd: 10m
    delayAfterDelete: 10s
    delayAfterFailure: 3m
---
apiVersion: autoscaling.openshift.io/v1beta1
kind: MachineAutoscaler
metadata:
  name: prod-east-worker-us-east-1a
  namespace: openshift-machine-api
spec:
  minReplicas: 3
  maxReplicas: 10
  scaleTargetRef:
    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet
    name: prod-east-worker-us-east-1a
```

| Object | Role | Without it |
|---|---|---|
| `ClusterAutoscaler` | The single, cluster-wide policy object — overall node limits, scale-down timing/aggressiveness | No autoscaling happens at all, regardless of any `MachineAutoscaler` present |
| `MachineAutoscaler` | Per-`MachineSet` min/max bounds — which specific `MachineSet`(s) are actually allowed to scale, and how far | The `ClusterAutoscaler` has nothing to scale — it never adjusts a `MachineSet` it has no `MachineAutoscaler` bound to |
| `scaleDown.delayAfterAdd` | How long to wait after a scale-up before considering scaling back down | Without a sensible delay, a brief demand spike can trigger a wasteful add-then-immediately-remove cycle |

Exactly one `ClusterAutoscaler` object can exist cluster-wide (named `default`), while any number of `MachineAutoscaler`s can target different `MachineSet`s — a common real design is a `MachineAutoscaler` per availability zone, letting the autoscaler spread new capacity across zones for resilience rather than piling every new node into a single zone's `MachineSet`.

```bash
oc get clusterautoscaler default -o yaml
oc get machineautoscaler --all-namespaces
oc logs -n openshift-machine-api deployment/cluster-autoscaler-default
oc get events -n openshift-machine-api --field-selector reason=FailedScaleUp
```

The `cluster-autoscaler-default` Deployment's own logs are the direct way to see the autoscaler's actual scaling decisions and reasoning in real time — including why it declined to scale up (a Pod's specific node-affinity or taint requirement no available `MachineSet` can satisfy is a common, otherwise-invisible reason a Pod stays `Pending` despite autoscaling being fully enabled and configured correctly).

## MachineHealthChecks — Automatic Node Remediation

A **`MachineHealthCheck`** closes a different gap: what happens when a `Machine`'s underlying infrastructure genuinely fails (a hung kubelet, an unresponsive instance) rather than simply needing more or fewer nodes.

```yaml
apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
  name: prod-east-worker-health
  namespace: openshift-machine-api
spec:
  selector:
    matchLabels: { machine.openshift.io/cluster-api-machineset: prod-east-worker-us-east-1a }
  unhealthyConditions:
    - type: Ready
      status: "Unknown"
      timeout: 300s
    - type: Ready
      status: "False"
      timeout: 300s
  maxUnhealthy: "40%"
  nodeStartupTimeout: 20m
```

Once a `Machine`'s `Node` reports `Ready=Unknown` or `Ready=False` for longer than the configured timeout, the `MachineHealthCheck` controller deletes the unhealthy `Machine`, and its owning `MachineSet` automatically provisions a replacement — fully automatic node replacement, with no human paged for what would otherwise be a routine hardware/instance failure. `maxUnhealthy: "40%"` is a deliberate circuit breaker: if more than 40% of the targeted `Machine`s are simultaneously unhealthy, the controller stops remediating entirely, on the reasoning that a failure affecting that many machines at once is more likely a systemic problem (a zone outage, a networking issue) than N independent hardware failures, and mass-replacing machines in that scenario would likely make things worse, not better.

```mermaid
stateDiagram-v2
    [*] --> Healthy: Node reports Ready=True
    Healthy --> Suspect: Ready becomes False/Unknown
    Suspect --> Healthy: Recovers before timeout
    Suspect --> Remediating: Timeout exceeded, below maxUnhealthy threshold
    Suspect --> CircuitBroken: Timeout exceeded, above maxUnhealthy threshold
    Remediating --> Healthy: Machine deleted, MachineSet\nprovisions a healthy replacement
    CircuitBroken --> [*]: No remediation — likely a\nsystemic issue, needs human investigation
```

> **From the Trenches:** A team's `MachineHealthCheck` correctly remediated a single flaky node several times over a few weeks — each replacement resolved the immediate symptom, so nobody investigated further. The pattern turned out to be a specific availability zone's underlying host hardware experiencing a slow degradation the cloud provider hadn't yet flagged — `MachineHealthCheck` was doing exactly its job, but treating each individual remediation as fully resolving the problem masked a trend that only became visible once someone correlated *which* zone every remediated machine had come from over that multi-week window. The lesson: `MachineHealthCheck` remediation events are themselves worth tracking as a metric/alert (a rising remediation rate, especially concentrated in one zone), not just treated as invisible, successful self-healing with nothing further to look at.

## Node Maintenance: Cordon, Drain, and Safe Reboot

Every automated mechanism above (MCO rollouts, upgrades, `MachineHealthCheck` remediation) ultimately performs the same manual sequence a platform team also needs to know how to run by hand for planned maintenance:

```bash
oc adm cordon worker-3.prod-east.example.com

oc adm drain worker-3.prod-east.example.com \
  --ignore-daemonsets --delete-emptydir-data --force

# ... perform maintenance, reboot, etc. ...

oc adm uncordon worker-3.prod-east.example.com

oc get node worker-3.prod-east.example.com
```

| Step | What it does |
|---|---|
| `cordon` | Marks the node unschedulable — no *new* Pods land on it, but existing Pods keep running |
| `drain` | Evicts every evictable Pod from the node, respecting `PodDisruptionBudget`s |
| `uncordon` | Marks the node schedulable again once maintenance is complete |
| `--force` | Also evicts Pods not managed by a controller (bare Pods) — otherwise `drain` refuses to touch them |
| `--delete-emptydir-data` | Explicitly acknowledges that any `emptyDir` volume's contents on this node will be lost — required for `drain` to proceed past a Pod using one |

`--ignore-daemonsets` is necessary because a `DaemonSet`'s whole purpose is running exactly one Pod per node — draining can't meaningfully "evict" it without contradicting that design, so `drain` skips DaemonSet-managed Pods rather than erroring out entirely. A `drain` that hangs indefinitely is, per this chapter's upgrade-troubleshooting section, very often the same `PodDisruptionBudget` culprit — checking `oc get pdb` is worth doing immediately rather than waiting to see if the drain eventually completes on its own.

### Checking Drain Safety Before Running It

`oc adm drain --dry-run=client` previews exactly what a real drain would evict, without actually touching anything — a fast, safe way to confirm the operation is going to do what's expected before committing to it on a production node:

```bash
oc adm drain worker-3.prod-east.example.com --dry-run=client --ignore-daemonsets

oc get pods --all-namespaces --field-selector spec.nodeName=worker-3.prod-east.example.com

oc describe node worker-3.prod-east.example.com | grep -A5 "Non-terminated Pods"

oc get pdb --all-namespaces -o json | \
  jq -r '.items[] | select(.status.disruptionsAllowed==0) | .metadata.namespace + "/" + .metadata.name'
```

That last command is worth running proactively, not just reactively during a stuck drain — it lists every `PodDisruptionBudget` cluster-wide currently allowing zero disruptions, which is precisely the set that could block a future node drain or upgrade before one is even attempted, giving a platform team advance warning rather than discovering the constraint mid-maintenance-window.

## etcd Backup and Disaster Recovery

etcd is the single source of truth for the entire cluster's state (Part 1) — losing it without a backup means losing the cluster's own configuration entirely, not just application data. `cluster-backup.sh`, run directly on a control-plane node, is the supported mechanism:

```bash
oc debug node/master-0.prod-east.example.com
chroot /host
/usr/local/bin/cluster-backup.sh /home/core/assets/backup
ls -la /home/core/assets/backup
```

This produces two artifacts: a snapshot of etcd's actual data, and a separate archive of the static pod manifests and certificates the control plane needs to bootstrap itself back up — both are required together for a real restore, since the certificates alone can't reconstruct cluster state, and the etcd snapshot alone can't restart a control plane with no static pod definitions to boot from.

```text
/home/core/assets/backup/
├── snapshot_2026-08-25_020000.db                    # the actual etcd data snapshot
└── static_kuberesources_2026-08-25_020000.tar.gz    # static pod manifests + certificates
```

### Automating Backups on a Schedule

Running `cluster-backup.sh` manually before every upgrade is necessary but not sufficient — a hardware failure or a bad manual change can happen on any ordinary Tuesday, not just upgrade day. A `CronJob` running as a privileged, node-scoped Pod is the standard way to automate the same script on a recurring schedule:

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: openshift-etcd
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          nodeSelector: { node-role.kubernetes.io/master: "" }
          tolerations:
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
          containers:
            - name: backup
              image: registry.redhat.io/openshift4/ose-cli:latest
              command: ["/usr/local/bin/cluster-backup.sh", "/etc/kubernetes/cluster-backup"]
          restartPolicy: OnFailure
```

This `CronJob` necessarily runs with node-level access (`hostNetwork`, targeting a control-plane node directly) to invoke the same script this section ran manually — a legitimate, narrowly-scoped exception to the least-privilege SCC discipline Part 2 established, precisely because etcd backup is inherently a node-level, control-plane operation with no meaningful way to perform it from an ordinary namespaced Pod. Pairing this `CronJob` with an off-cluster copy step (syncing the backup artifact to S3 or an equivalent external store) closes the "don't store the backup only on the cluster it protects" gap this chapter's practices table flagged.

### Verifying a Backup Is Actually Restorable

A backup nobody has ever test-restored is, per this catalog's Reliability & SRE content's own guidance on untested recovery procedures, a documented assumption rather than a verified capability. Periodically test-restoring a backup onto a genuinely separate, disposable cluster (never the production cluster the backup came from) is the only way to confirm the backup and restore procedure both actually work end to end, rather than trusting a script exit code alone.

| Practice | Why it matters |
|---|---|
| Run before every upgrade | The single most important, cheapest insurance against an upgrade going wrong |
| Automate on a schedule (not just pre-upgrade) | A hardware failure or a bad manual change can happen any day, not just upgrade day |
| Store the backup off-cluster | A backup stored only on the cluster it protects doesn't survive the disaster scenarios that actually matter (a full control-plane loss) |
| Match z-stream versions on restore | Restoring a backup from a different patch version than the cluster currently runs is explicitly unsupported |
| Retain more than one recent backup | A single most-recent backup taken just before an undetected problem began is itself already corrupted — retaining a short history gives a genuine fallback |

## Restoring From an etcd Backup

```bash
# Copy the backup to the recovery control-plane node's /home/core first, then:
sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup

oc get clusteroperators
oc get nodes
oc get etcd cluster -o jsonpath='{.status.conditions}'
```

The restore script itself stops the other control-plane nodes' static pods, restores the recovery node from the snapshot, and restarts the control plane from that single node — the other control-plane nodes rejoin etcd's cluster afterward, similar in spirit to the bootstrap sequence Part 1 covered, just recovering into a known-good prior state instead of building a fresh cluster. A restore is a genuinely disruptive, last-resort operation — every change made to the cluster after the backup was taken is lost, which is exactly why the backup cadence from the previous section matters as much as the restore mechanism itself; a six-month-old backup restored during an incident recovers the cluster, but discards six months of legitimate configuration changes along with whatever caused the incident.

### When a Restore Is (and Isn't) the Right Call

| Scenario | Restore the right call? |
|---|---|
| A majority of etcd members lost simultaneously, quorum unrecoverable | Yes — this is exactly the scenario the restore procedure exists for |
| A single bad `MachineConfig` change broke the worker pool | Usually no — reverting the specific Git-tracked change (Part 4's GitOps model) is faster and loses nothing |
| A namespace's application data was accidentally deleted | No — this is OADP's job, not etcd restore, since etcd never held the application data itself |
| Genuine uncertainty about what changed or when | A strong signal to reach for the etcd restore, since a targeted fix isn't identifiable |
| A single control-plane node's hardware failed, quorum intact | No — the Machine API/MachineSet replaces the failed control-plane machine, and the surviving etcd members recover it without a restore |

Treating restore as the default response to any cluster-level problem, rather than the last resort it's designed to be, routinely discards more legitimate work than the problem itself would have cost to fix directly — the decision deserves the same explicit, deliberate weighing this series has applied to every other high-blast-radius operation.

## Beyond etcd: Application Backup With OADP

etcd backup protects the cluster's own configuration — it does **not** back up application data living in `PersistentVolume`s, which is a separate, real gap many teams don't realize until they need a restore that etcd backup alone can't provide. The **OpenShift API for Data Protection (OADP)** Operator, built on the open-source Velero project, closes this gap:

```yaml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-payments-backup
  namespace: openshift-adp
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces: ["payments-prod"]
    snapshotVolumes: true
    ttl: 720h0m0s
```

OADP backs up both Kubernetes object manifests and, via cloud-provider volume snapshots, the actual persistent data those objects reference — giving a genuine application-level restore capability (recover one namespace's full state, including data) that etcd backup and restore, being cluster-wide and configuration-only, was never designed to provide. A mature disaster-recovery posture runs both mechanisms, aimed at the two genuinely different failure classes each one covers.

### Restoring a Namespace From an OADP Backup

```yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: payments-prod-restore
  namespace: openshift-adp
spec:
  backupName: daily-payments-backup-20260825020000
  includedNamespaces: ["payments-prod"]
  restorePVs: true
  existingResourcePolicy: update
```

```bash
oc get restore payments-prod-restore -n openshift-adp -o jsonpath='{.status.phase}'
velero backup describe daily-payments-backup-20260825020000
oc get restore payments-prod-restore -n openshift-adp -o jsonpath='{.status.warnings}'
```

A `Restore` object references one specific completed `Backup` by name — `velero backup describe` (or the equivalent `oc get backup`) is the way to list available backups and confirm exactly which point-in-time snapshot a given restore will recover to, the same "know precisely what you're restoring to before running it" discipline the etcd restore section applied to cluster-level recovery, now applied to namespace-level, data-inclusive recovery.

## The Built-In Monitoring Stack

Every OpenShift cluster ships a fully pre-integrated **Prometheus, Alertmanager, and Grafana** stack (via the `monitoring` Cluster Operator from Part 1's roster) covering the platform's own components by default — no separate installation or wiring required, a direct payoff of Part 1's "curated, jointly-tested platform" framing.

```mermaid
flowchart LR
    Targets["Cluster components\n+ platform Operators"] --> Prom["Platform Prometheus\n(openshift-monitoring)"]
    Prom --> Alertmgr["Alertmanager"]
    Prom --> Grafana["Grafana / Console dashboards"]
    Alertmgr --> Notify["PagerDuty / Slack / email"]

    classDef comp fill:#1d4ed8,color:#fff,stroke:#1e40af
    class Targets,Prom,Alertmgr,Grafana,Notify comp
```

```bash
oc get pods -n openshift-monitoring
oc get prometheusrule -n openshift-monitoring
oc get alertmanager main -n openshift-monitoring -o yaml
oc get routes -n openshift-monitoring
```

This default stack monitors the platform itself (API server latency, etcd health, node conditions, Cluster Operator status) — it deliberately does **not** scrape arbitrary user-workload namespaces out of the box, which is exactly the gap the next section closes.

## User Workload Monitoring

Enabling **user workload monitoring** extends the same Prometheus/Alertmanager model to application teams' own namespaces, without requiring cluster-admin privileges for day-to-day use once enabled:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
```

This deploys a **separate** Prometheus instance and Thanos Ruler into `openshift-user-workload-monitoring` — deliberately isolated from the platform's own monitoring Prometheus, so a runaway or misconfigured application-level `PrometheusRule` can't degrade the platform team's own visibility into cluster health, the same isolation principle Part 2 applied to multi-tenancy generally, now applied to the monitoring stack itself.

### Querying Metrics Directly

Both Prometheus instances are reachable through the web console's Observe → Metrics tab, or directly via the CLI for scripting/automation needs:

```bash
oc exec -n openshift-monitoring prometheus-k8s-0 -c prometheus -- \
  curl -s 'http://localhost:9090/api/v1/query?query=up{job="apiserver"}'

oc get route prometheus-k8s -n openshift-monitoring -o jsonpath='{.spec.host}'
oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}'

TOKEN=$(oc create token prometheus-k8s -n openshift-monitoring)
curl -s -H "Authorization: Bearer $TOKEN" \
  --data-urlencode 'query=sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace)' \
  https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query
```

The `thanos-querier` route/service is worth knowing specifically: it federates queries across *both* the platform Prometheus and user-workload Prometheus instances transparently, meaning a single PromQL query issued against Thanos Querier can correlate a platform-level metric (node CPU pressure) against a user-workload metric (an application's own request rate) in one query, without needing to know or care which of the two separate Prometheus instances actually collected each series.

| Component | Platform monitoring | User workload monitoring |
|---|---|---|
| Prometheus instance | `openshift-monitoring` | `openshift-user-workload-monitoring`, separate |
| Who defines alert rules | Platform team, cluster-scoped | Any namespace owner, via a namespaced `PrometheusRule` |
| Required privilege | Cluster-admin | Ordinary namespace `edit`/`admin` role (Part 2) |
| Scrapes | Platform components only | Any namespace opting in via a `ServiceMonitor` |
| Retention | Platform-configured, cluster-wide default | Independently configurable, since it's a separate Prometheus instance |
| Alertmanager routing | The platform's own Alertmanager, or a dedicated user-workload one | Either — a namespace can route to the shared Alertmanager or bring its own `AlertmanagerConfig` |

## A Worked Example: A Custom Alert for a User Workload

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web
  namespace: payments-prod
  labels: { k8s-app: web }
spec:
  selector:
    matchLabels: { app: web }
  endpoints:
    - port: metrics
      interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: web-error-rate
  namespace: payments-prod
spec:
  groups:
    - name: web
      rules:
        - alert: WebHighErrorRate
          expr: |
            sum(rate(http_requests_total{job="web",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="web"}[5m])) > 0.05
          for: 10m
          labels: { severity: warning }
          annotations:
            summary: "web error rate above 5% for 10 minutes"
```

The `ServiceMonitor` tells user-workload Prometheus what to scrape (any `Service` matching the label selector, on the named port); the `PrometheusRule` defines the actual alerting logic — both objects live entirely within the application team's own namespace, editable with the `edit`/`admin` RBAC role from Part 2, with no platform-team involvement required for a team to instrument and alert on their own service once user workload monitoring is enabled cluster-wide.

### Routing the Alert Somewhere a Human Will See It

A firing `PrometheusRule` alert is only useful once it actually reaches a human — a namespaced `AlertmanagerConfig` object routes a team's own alerts to their own notification channel, without touching the platform's shared Alertmanager configuration at all:

```yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: web-alerts
  namespace: payments-prod
spec:
  route:
    receiver: payments-slack
    groupBy: ["alertname"]
  receivers:
    - name: payments-slack
      slackConfigs:
        - apiURL: { name: slack-webhook-secret, key: url }
          channel: "#payments-alerts"
```

This namespaced object is what makes user workload monitoring genuinely self-service end to end — a team can define what to scrape (`ServiceMonitor`), what counts as a problem (`PrometheusRule`), and where to send the resulting alert (`AlertmanagerConfig`), entirely within their own namespace's RBAC boundary, with the platform team's only remaining role being having enabled `enableUserWorkload: true` once, cluster-wide.

### Grafana Dashboards for Custom Metrics

The console's built-in Grafana instance can also visualize any metric either Prometheus instance collects, including a team's own custom application metrics — a `ConfigMap` labeled `console.openshift.io/dashboard: "true"` in `openshift-config-managed` is the mechanism for adding a custom dashboard directly into the console's own Dashboards view, alongside the built-in platform dashboards, rather than requiring a separately-hosted Grafana instance a team has to stand up and secure themselves.

## The Built-In Logging Stack: Loki and the Cluster Logging Operator

Metrics (Prometheus) answer "how much/how often"; **logs** answer "what exactly happened, in this specific request." The **Cluster Logging Operator**, paired with a **LokiStack** (via the separate Loki Operator), is OpenShift's current-generation built-in logging stack — a materially lighter-weight architecture than the older Elasticsearch-based stack it replaced, and one worth knowing the current object model for rather than out-of-date `ClusterLogging`-only documentation.

```yaml
apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.small
  storage:
    secret: { name: loki-s3-secret, type: s3 }
  storageClassName: gp3-csi
  tenants: { mode: openshift-logging }
---
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  serviceAccount: { name: collector }
  outputs:
    - name: default-lokistack
      type: lokiStack
      lokiStack: { target: { name: logging-loki, namespace: openshift-logging } }
  pipelines:
    - name: app-logs
      inputRefs: ["application"]
      outputRefs: ["default-lokistack"]
```

`ClusterLogForwarder` is the object worth focusing on: it defines named **pipelines**, each routing a specific log **input** (`application`, `infrastructure`, or `audit` — three built-in log classes every cluster produces) to one or more **outputs**, which can be the in-cluster LokiStack, or an external system (Splunk, an external Elasticsearch, Kafka) — meaning a single forwarder definition can simultaneously send application logs to Loki for day-to-day querying while also forwarding audit logs to a separate, compliance-mandated external SIEM, without running two separate collection agents to achieve it.

| Log type | What it contains |
|---|---|
| `application` | Every container's own stdout/stderr — what most day-to-day debugging actually needs |
| `infrastructure` | Platform component logs — kubelet, CRI-O, the Cluster Operators themselves |
| `audit` | Every API server request, including who made it — the compliance-relevant log class |
| Custom `inputs` (via label/namespace selectors) | A team-defined subset of `application` logs, useful for routing one namespace's logs differently than the rest |
| Vector | The lightweight log-collection agent running as a DaemonSet, replacing the older Fluentd-based collector |

### Querying Logs With LogQL

Loki's query language, **LogQL**, is deliberately close to PromQL in shape — a label-selector stream expression, optionally piped through filters and aggregations, rather than a full-text search index:

```logql
{ kubernetes_namespace_name="payments-prod", kubernetes_container_name="web" }
  |= "error"
  | json
  | line_format "{{.message}}"

sum(rate({kubernetes_namespace_name="payments-prod"} |= "error" [5m])) by (kubernetes_pod_name)

{ log_type="audit" } | json | user_username="jane.doe"
```

The first query streams raw log lines matching the namespace/container labels, filtered to lines containing "error," then parses each line as JSON and reformats the output; the second aggregates the *rate* of matching log lines per Pod over a 5-minute window — the same rate-based reasoning PromQL applies to metrics, applied here directly to log volume, letting a team alert on "this Pod's error-log rate just spiked" using the same mental model as a metrics-based alert, through the web console's Observe → Logs tab or the `logcli` CLI tool directly.

## OLM Day-2 Patterns: Channels and Approval Strategies Revisited

Part 1 introduced OLM's `Subscription`/`InstallPlan` model; the Day-2 discipline worth adding here is treating **channel** and **`installPlanApproval`** as ongoing operational levers, not one-time install choices.

```yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cert-manager-operator
  namespace: openshift-operators
spec:
  channel: stable-v1
  installPlanApproval: Manual
```

| Channel naming pattern | What it signals |
|---|---|
| `stable` | Generally the safest, most conservative channel for production |
| `stable-v1`, `stable-v2` | A major-version-pinned stable channel, avoiding an unexpected major-version jump |
| `candidate`/`alpha`/`beta` | Pre-release channels — appropriate for testing an upcoming version, not production |
| `fast` | Newer than `stable` but still supported — a middle ground some Operators offer |
| Switching channels mid-lifecycle | Not always a simple downgrade-safe operation — check the specific Operator's own documented upgrade/downgrade support before switching |

A platform team's actual Day-2 discipline is periodically reviewing every installed Operator's pending `InstallPlan`s (`oc get installplan --all-namespaces`), approving them deliberately after checking the target CSV's release notes — exactly the same "review before approving, using `installPlanApproval: Manual`" pattern Part 1 introduced, now framed as a recurring operational habit rather than a one-time install-time decision, alongside the OpenShift cluster's own upgrade cadence from earlier in this chapter, since an incompatible Operator version is one of the most common real causes of a cluster upgrade going wrong.

### The Actual Approval Workflow

```bash
oc get installplan -n openshift-operators

oc get installplan install-abc12 -n openshift-operators -o jsonpath='{.spec.clusterServiceVersionNames}'

oc get subscription cert-manager-operator -n openshift-operators -o jsonpath='{.status.installedCSV}'

# After reviewing the target CSV's release notes:
oc patch installplan install-abc12 -n openshift-operators \
  --type merge -p '{"spec":{"approved":true}}'

oc get csv -n openshift-operators
oc get csv -n openshift-operators -o jsonpath='{.items[*].status.phase}'
```

The `InstallPlan` sits in a pending, unapproved state indefinitely until this patch is applied — nothing times out or auto-approves it, which is exactly the point of `installPlanApproval: Manual`: the Operator's new version simply waits for a human decision, with no risk of an unreviewed change silently applying itself after some delay.

## Putting It Together: A Day-2 Operations Runbook

Every mechanism this chapter covered composes into a standing operational rhythm a platform team actually runs, on some regular cadence, for the life of the cluster:

```text
Daily/automated:
  - etcd backup (cluster-backup.sh, scheduled)
  - OADP application backup (Velero Schedule)
  - Monitor ClusterOperator health, Alertmanager routing
  - Review firing alerts and LogQL error-rate queries for anomalies

Before any upgrade:
  - Fresh etcd backup, confirmed restorable
  - Every OLM Operator's Subscription channel/version compatibility checked
  - Review Cincinnati conditional-update warnings (Part 1)
  - Confirm no PodDisruptionBudget cluster-wide currently allows zero disruptions

Ongoing:
  - Review pending InstallPlans, approve deliberately
  - Review MachineAutoscaler bounds against actual usage trends
  - Confirm MachineHealthCheck coverage on every MachineSet
  - Track MachineHealthCheck remediation rate/zone concentration as its own metric
  - Periodic oc adm prune images (Part 4) and log-retention review
  - Test-restore an etcd backup and an OADP backup onto a disposable cluster
  - Review self-provisioning and quota defaults against actual Project growth (Part 2)
  - Confirm every user-workload-monitoring ServiceMonitor still points at a live Service
  - Rotate the Sealed Secrets key (Part 4) on its own documented schedule
```

Nothing in this runbook is exotic — it's the accumulated discipline every mechanism in this chapter individually justified, assembled into the actual recurring practice that keeps a cluster healthy for years rather than merely for its first successful rollout.

## Quick Reference: Key Terms From This Chapter

| Term | What it is |
|---|---|
| `oc adm upgrade` | The command that reads and drives the Cincinnati-recommended upgrade path |
| `MachineConfigPool` pausing | Deferring a pending MCO rollout, including one triggered by an upgrade, until explicitly unpaused |
| `Machine` / `MachineSet` | The Machine API's provisioned-instance and desired-count objects |
| `providerSpec` | The platform-specific (AWS/Azure/GCP/vSphere/etc.) portion of a `MachineSet`, never portable as-is across clouds |
| `machine.openshift.io/delete-machine` | The annotation marking a specific Machine as the preferred scale-down target |
| `ClusterAutoscaler` / `MachineAutoscaler` | The cluster-wide policy object and the per-MachineSet bounds object, both required together |
| `MachineHealthCheck` | Automatic detection and replacement of an unhealthy Machine |
| `cordon` / `drain` / `uncordon` | The manual node-maintenance sequence every automated mechanism also performs |
| `cluster-backup.sh` / `cluster-restore.sh` | The etcd snapshot backup and restore scripts |
| OADP / Velero | The application-level (including PersistentVolume data) backup mechanism, complementary to etcd backup |
| User workload monitoring | The separate, isolated Prometheus/Thanos Ruler instance for application-defined metrics and alerts |
| `ServiceMonitor` / `PrometheusRule` | The namespaced objects defining what to scrape and what to alert on |
| `ClusterLogForwarder` | Routes named log inputs (application/infrastructure/audit) to one or more outputs |
| LokiStack | The log-storage backend for OpenShift's current logging stack |
| LogQL | Loki's PromQL-like query language for logs |
| `cluster-backup.sh` output pair | The etcd snapshot plus the static-pod-manifest/certificate archive, both required for a real restore |
| `installPlanApproval: Manual` | The OLM setting requiring explicit review before an Operator upgrade applies |
| Cincinnati conditional updates | The upgrade-graph warnings surfaced by `oc adm upgrade`, revisited operationally in this chapter |

## Common Mistakes and Interview Traps

| Mistake or claim | Why it is wrong | Better answer |
|---|---|---|
| "A cluster upgrade updates worker nodes first, then the control plane." | The control plane always updates first — worker nodes assuming new behavior before the API server supports it would be unsafe. | Name the actual order: control plane, then MCO-driven rolling worker updates, then every Cluster Operator reconciling. |
| "A `ClusterAutoscaler` object alone is enough to enable autoscaling." | Without at least one `MachineAutoscaler` targeting a specific `MachineSet`, the `ClusterAutoscaler` has nothing it's permitted to scale. | Both objects are required together — name the specific role each one plays. |
| "etcd backup covers full disaster recovery, including application data." | etcd backup covers cluster configuration only — `PersistentVolume` data needs a separate mechanism. | Name OADP/Velero as the complementary tool for application-level, including-data backup. |
| "A stuck cluster upgrade always means something is broken." | A `MachineConfigPool` genuinely still `Updating` on a large node count, or a strict `PodDisruptionBudget` correctly blocking an unsafe drain, can both look identical to a real failure. | Check `oc get mcp` and `oc get pdb` before assuming failure over a slow-but-healthy rollout. |
| "User workload monitoring shares the same Prometheus instance as the platform's own monitoring." | It deploys a deliberately separate Prometheus/Thanos Ruler instance specifically to isolate the two. | Name the isolation as intentional — protecting platform visibility from a misbehaving application-level rule. |
| "`oc adm drain` fails outright on any node running a DaemonSet-managed Pod." | `--ignore-daemonsets` is the standard, expected flag for exactly this case — it's not an error condition. | Always include `--ignore-daemonsets` for a routine node drain; its absence, not its presence, is the unusual case. |
| "Restoring from an etcd backup recovers the cluster with zero data loss." | Every change made after the backup was taken is lost — a restore recovers to the backup's point in time, not to the moment of failure. | Frame backup cadence itself as the actual mitigation for this gap, not the restore mechanism. |
| "Approving every OLM `InstallPlan` automatically is the safer default for staying current." | Automatic approval applies a new CSV version the instant it's published, with no chance to review its release notes or compatibility against the cluster's own version first. | Use `installPlanApproval: Manual` for anything production-relevant, and review before approving. |
| "A `MachineSet`'s `providerSpec` is portable across cloud providers, since the rest of the object model is identical." | `providerSpec` is genuinely platform-specific (instance type, AMI, VM size, image); only the surrounding `replicas`/`selector` shape is shared. | Author `providerSpec` per platform explicitly — never assume a manifest written for one cloud works unmodified on another. |
| "Pausing a `MachineConfigPool` is a safe way to permanently opt a pool out of future changes." | A pool left paused indefinitely silently drifts further behind the cluster's declared state with every change that accumulates unapplied. | Treat pausing as a deliberate, temporary measure with a planned un-pause, not a permanent configuration. |
| "An untested etcd backup is equivalent to a verified disaster-recovery capability." | A backup script's successful exit code says nothing about whether the resulting snapshot can actually be restored end to end. | Periodically test-restore onto a disposable cluster to confirm the whole procedure genuinely works. |

## Worked Practice Problems

### 1. A cluster upgrade has been "in progress" for over an hour with no `ClusterOperator` reporting `Degraded`. What are the first two things to check before assuming something has failed?

First, check `oc get mcp` to confirm whether the relevant `MachineConfigPool`(s) are genuinely still `Updating` — a large worker node count rolling one node at a time can legitimately take a long time, and this alone can fully explain the apparent stall with nothing actually broken. Second, check `oc get pdb --all-namespaces` for an overly strict `PodDisruptionBudget` blocking the MCO's drain step on a specific node — exactly the failure mode this chapter's own trenches story describes, which produces no `ClusterOperator`-level degradation at all since nothing about the operators themselves is unhealthy, only the node-drain step waiting on a constraint it's correctly refusing to violate.

### 2. A team enables the cluster autoscaler by creating a `ClusterAutoscaler` object, but reports that the cluster never scales up even when Pods are stuck `Pending` due to insufficient resources. What's missing?

The team is very likely missing a `MachineAutoscaler` targeting the specific `MachineSet`(s) they want scaled — a `ClusterAutoscaler` alone defines cluster-wide policy (overall node limits, scale-down timing) but has no `MachineSet` it's actually authorized to adjust without a `MachineAutoscaler` bound to it. The fix is creating at least one `MachineAutoscaler` with `scaleTargetRef` pointing at the relevant `MachineSet`(s) and sensible `minReplicas`/`maxReplicas` bounds — only then does the `ClusterAutoscaler` have anything it's permitted to scale in response to unschedulable Pods.

### 3. A platform team's disaster-recovery plan only includes scheduled etcd backups. A production incident destroys a `PersistentVolume` backing a critical stateful application. What does the etcd backup restore, and what does it not?

Restoring from the etcd backup recovers the cluster's own configuration state — every Kubernetes object definition, including the `PersistentVolumeClaim` object referencing the now-destroyed volume — but it does **not** recover the actual data that lived on that volume, since etcd never stored the data itself, only the cluster's metadata about it. The team's disaster-recovery plan has a real, concrete gap: OADP (or an equivalent Velero-based backup covering volume snapshots) is the missing piece specifically for this failure class, and this incident is exactly the scenario that gap predicts — the fix going forward is adding scheduled OADP backups for any namespace with genuinely critical persistent data, not merely relying on etcd backup to cover disaster recovery as a whole.

### 4. A security team asks why audit logs need to go to an external SIEM while application logs stay in the in-cluster LokiStack, and whether this requires running two separate logging agents. What's the actual answer?

It requires no second agent — a single `ClusterLogForwarder` definition can route different log **inputs** (`application`, `infrastructure`, `audit`) to different **outputs** within the same object, meaning the existing Vector-based collection agent forwards `application` logs to the in-cluster LokiStack for day-to-day debugging while simultaneously forwarding `audit` logs to an external SIEM output for compliance retention, both from the same underlying collected log stream. The distinction the security team is really asking about is a routing/pipeline configuration choice within one forwarder definition, not a requirement for separate collection infrastructure per destination.

### 5. A platform team needs to replace one specific worker node whose disk has developed early SMART warnings, without waiting for it to become unhealthy enough to trigger `MachineHealthCheck` remediation on its own. What's the precise, controlled way to do this?

The precise way is annotating the specific `Machine` backing that node with `machine.openshift.io/delete-machine="true"`, then scaling its `MachineSet` down by one and back up by one (or simply down and up, letting the annotation steer which one is removed) — this proactively removes exactly the flagged machine rather than leaving the choice to the `MachineSet` controller's default newest-first heuristic, which has no awareness of the SMART warning at all and might just as easily remove a perfectly healthy, newer machine instead. This is meaningfully more controlled than waiting for `MachineHealthCheck` to eventually intervene, since that mechanism only acts once the node's `Ready` condition itself degrades, which a disk nearing failure may not yet be doing.

### 6. A team wants a single alert that fires when an application's real end-user error rate is high, correlated against whether the underlying nodes are also under memory pressure, without standing up a separate observability tool. Is this achievable with the built-in stack, and how?

Yes — this is exactly what Thanos Querier's federation across the platform and user-workload Prometheus instances is for: a single PromQL expression can reference both a user-workload metric (the application's own `http_requests_total` error-rate series, scraped via the team's own `ServiceMonitor`) and a platform-level metric (node memory pressure, scraped by the platform's own Prometheus) in one query, since Thanos Querier presents both as one federated queryable source. The alert itself would still be authored as a namespaced `PrometheusRule` in the application's own namespace, referencing both series in its `expr` — no separate observability tool or manual metric-forwarding pipeline is required, since the federation is a built-in property of the stack this chapter described, not something the team needs to wire up themselves.

## Summary and Series Wrap-Up

This chapter closed the operational loop every earlier chapter's architecture made possible: the CVO's tested release payload (Part 1) becomes a real, repeatable upgrade procedure with known failure modes and known fixes; the Machine API extends the same declarative model to the cluster's own infrastructure, with the `ClusterAutoscaler`/`MachineAutoscaler` pair and `MachineHealthCheck` turning capacity management and hardware-failure remediation from manual, paged-at-3am tasks into automated, policy-driven ones; etcd backup and OADP together cover the two genuinely distinct disaster-recovery failure classes — cluster configuration and application data — that a mature Day-2 practice needs both of, not just one; and the built-in monitoring and logging stacks give a platform team (and, via user workload monitoring, every application team) the observability into all of the above without needing to stand up or integrate any of it themselves.

Across all five parts, this series followed one consistent throughline, first named explicitly in Part 1 and revisited at every layer since: OpenShift's core bet is that a curated, jointly-tested, operator-driven platform is worth its opinionation specifically when an organization's actual pain is coordinating and securing many teams or many clusters at once — not a universal improvement over vanilla Kubernetes for every team at every scale. Part 1's architecture, Part 2's default-restrictive multi-tenancy, Part 3's integrated networking and optional service mesh, Part 4's built-in delivery tooling, and this chapter's Day-2 automation are five expressions of the identical trade, each worth adopting exactly as far as an organization's real, current requirements justify it, and no further — the same honest, requirement-driven judgment call this series has asked the reader to make explicitly at every layer, rather than treating any of it as automatically "more enterprise" and therefore unconditionally better.

Cross-reference: this catalog's Kubernetes Deep Dive series covers etcd's Raft consensus, compaction, and defragmentation mechanics, and cluster upgrade/rollout concepts from a vanilla-Kubernetes vantage point — both chapters' depth applies unchanged to the OpenShift-managed etcd and CVO-driven upgrades this chapter covered operationally. Its own Observability domain content (metrics, logging, tracing fundamentals) is likewise worth revisiting for the platform-agnostic half of what this chapter's monitoring and logging sections build on.

Sources consulted for this chapter: Red Hat's OpenShift Container Platform Updating Clusters documentation, the Backup and Restore documentation (etcd disaster recovery, OADP), the Machine Management documentation (Machine API, autoscaling, MachineHealthCheck), the Monitoring Stack for Red Hat OpenShift documentation (user workload monitoring), and Red Hat's OpenShift Logging (LokiStack, ClusterLogForwarder) documentation and release notes.