Day-2 Operations: Cluster Operators, Upgrades & Observability

Table of Contents#

1. What "Day 2" Means, and Why It Deserves Its Own Chapter
2. Performing a Cluster Upgrade in Practice
3. Handling a Stuck or Degraded Upgrade
4. The Machine API — Machines and MachineSets
5. Cluster Autoscaling: ClusterAutoscaler and MachineAutoscaler
6. MachineHealthChecks — Automatic Node Remediation
7. Node Maintenance: Cordon, Drain, and Safe Reboot
8. etcd Backup and Disaster Recovery
9. Restoring From an etcd Backup
10. Beyond etcd: Application Backup With OADP
11. The Built-In Monitoring Stack
12. User Workload Monitoring
13. A Worked Example: A Custom Alert for a User Workload
14. The Built-In Logging Stack: Loki and the Cluster Logging Operator
15. OLM Day-2 Patterns: Channels and Approval Strategies Revisited
16. Putting It Together: A Day-2 Operations Runbook
17. Quick Reference: Key Terms From This Chapter
18. Common Mistakes and Interview Traps
19. Worked Practice Problems
20. Summary and Series Wrap-Up

What "Day 2" Means, and Why It Deserves Its Own Chapter#

Every prior chapter in this series covered a piece of standing up and using the platform: architecture (Part 1), multi-tenancy (Part 2), networking (Part 3), delivery (Part 4). "Day 2" is the industry term for everything that happens after a cluster is running production workloads — upgrades, capacity changes, disaster recovery, and the ongoing observability practice that tells a team whether the platform is actually healthy — and it deserves its own chapter because a cluster's Day 2 operational quality is what determines whether it survives years of real production use, not just an initial rollout.

This chapter revisits several objects earlier chapters already introduced — the Cluster Version Operator (Part 1), the Machine Config Operator (Part 1), OLM (Part 1) — and goes one level deeper into the actual Day-2 mechanics a platform team runs against them repeatedly over a cluster's lifetime, rather than the one-time architectural understanding those earlier chapters focused on.

Each row in this table is a distinct failure mode this chapter's mechanisms exist specifically to close — worth keeping explicit, since Day-2 operations work is easy to under-invest in precisely because its payoff is an incident that never happens, rather than a visible feature shipped.

From the Trenches: An organization's platform team spent its first year almost entirely on Part 1 through Part 4 concerns — getting workloads onto the cluster, securing multi-tenancy, wiring up delivery pipelines — and treated Day-2 operations as something to "get to eventually." The eventual reckoning came during the cluster's first real upgrade a year in: no one had ever actually tested an etcd restore, the ClusterAutoscaler had been configured once and never revisited despite the workload footprint tripling, and nobody could say with confidence which OLM Operators were even on a supported upgrade path. None of this was a crisis in isolation, but all of it surfaced simultaneously during the highest-stakes operation the cluster had faced yet — the fix wasn't heroics during that upgrade, it was retroactively building exactly the standing runbook this chapter closes with, then treating it as a genuinely first-class, continuously-maintained practice going forward rather than a one-time cleanup exercise.

Performing a Cluster Upgrade in Practice#

Part 1 covered the CVO's architecture and the Cincinnati upgrade graph's conditional-update warnings; this section covers actually running one. The full sequence, in order:

oc adm upgrade

oc adm upgrade channel stable-4.19

oc get clusterversion version -o jsonpath='{.status.availableUpdates}'

oc adm upgrade --to=4.19.5

watch oc get clusteroperators

oc get clusterversion version -o jsonpath='{.status.history}'

Before triggering the upgrade itself, two preconditions from this chapter's own disaster-recovery section deserve to be non-negotiable, not optional: a recent etcd backup, and every installed OLM Operator (Part 1) already updated to a version compatible with the target OpenShift release, since an incompatible third-party Operator can itself become the reason a cluster looks unhealthy immediately post-upgrade, muddying the diagnosis of what's actually wrong.

The control plane always updates before worker nodes — a deliberate ordering, since the API server and its supporting components need to be running the new version before it's safe for anything else in the cluster to start assuming that version's behavior. Worker node updates then roll one node at a time within each MachineConfigPool (Part 1), cordoning, draining, and uncordoning exactly as any other MCO-driven change does, meaning an upgrade's actual worker-facing disruption is bounded to whatever workloads' own PodDisruptionBudgets permit during that rolling drain — a PodDisruptionBudget that's too strict (allowing zero disruption) can stall an upgrade indefinitely waiting for a node to safely drain, which is exactly the next section's most common failure mode.

Pausing a MachineConfigPool During a Sensitive Window#

A MachineConfigPool can be explicitly paused, deferring any pending worker-node rollout (including one triggered by an in-progress upgrade) until a team is ready for it — useful for a scenario where the control plane has already updated, but rolling worker changes during a specific business-critical window (a retail Black Friday peak, a financial close period) is unacceptable:

oc patch mcp worker --type merge -p '{"spec":{"paused":true}}'

# ... business-critical window passes ...

oc patch mcp worker --type merge -p '{"spec":{"paused":false}}'
oc get mcp worker -w

Pausing is a deliberate, temporary override — the pool's rollout resumes exactly where it left off once unpaused, but a pool left paused indefinitely means that pool's nodes silently drift further behind the rest of the cluster's own declared state with every passing day, the same "the reconciler is what actually keeps the cluster correct" property this series has emphasized throughout; pausing should always have a planned un-pause, not be treated as a permanent configuration.

Handling a Stuck or Degraded Upgrade#

oc get clusterversion version -o jsonpath='{.status.conditions}'

oc get clusteroperators | grep -v "True.*False.*False"

oc get nodes -o wide

oc get pdb --all-namespaces

oc get mcp

From the Trenches: A cluster upgrade stalled for hours with every ClusterOperator reporting healthy and no obvious error anywhere in the CVO's own status. The actual cause was a single, forgotten PodDisruptionBudget on a stateful workload set to maxUnavailable: 0 — the MCO's own drain step was correctly refusing to violate it, silently blocking that one node (and, transitively, the rest of that MachineConfigPool's rollout) indefinitely rather than proceeding unsafely. oc get pdb --all-namespaces surfaced the offending PDB in seconds once someone thought to check it — the upgrade wasn't broken at all; it was correctly waiting on a constraint nobody had remembered to account for before starting.

The Machine API — Machines and MachineSets#

The Machine API — a set of Cluster Operators and CRDs, distinct from Kubernetes' own Node object — is what gives OpenShift (on a supported IPI platform) the ability to provision, scale, and replace its own worker nodes declaratively, the same "declared desired state, continuously reconciled" model this series has applied at every other layer, now extended to the infrastructure the cluster itself runs on.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: prod-east-worker-us-east-1a
  namespace: openshift-machine-api
  labels:
    machine.openshift.io/cluster-api-cluster: prod-east
spec:
  replicas: 3
  selector:
    matchLabels: { machine.openshift.io/cluster-api-machineset: prod-east-worker-us-east-1a }
  template:
    spec:
      providerSpec:
        value:
          instanceType: m6i.large
          placement: { availabilityZone: us-east-1a }

oc scale machineset prod-east-worker-us-east-1a --replicas=5 -n openshift-machine-api
oc get machines -n openshift-machine-api
oc get machineset prod-east-worker-us-east-1a -n openshift-machine-api \
  -o jsonpath='{.status.replicas}/{.status.readyReplicas}{"\n"}'

Scaling a MachineSet up or down is the direct, manual mechanism for capacity changes — provisioning a new cloud instance, waiting for it to join the cluster as a Node, or (scaling down) cordoning and draining a Machine before terminating its underlying instance, all handled by the Machine API's own controllers without a human needing to touch the cloud provider's own console or API directly.

Controlling Which Machine Gets Removed on Scale-Down#

Scaling a MachineSet down doesn't let a team choose which specific Machine is removed by default — the controller picks based on its own internal heuristics (newest first, by default). A machine.openshift.io/delete-machine: "true" annotation on a specific Machine overrides that default, marking it as the preferred deletion target the next time that MachineSet scales down:

oc annotate machine prod-east-worker-us-east-1a-x7k2p \
  machine.openshift.io/delete-machine="true" -n openshift-machine-api

oc scale machineset prod-east-worker-us-east-1a --replicas=2 -n openshift-machine-api

oc get machines -n openshift-machine-api -w
oc get machine prod-east-worker-us-east-1a-x7k2p -n openshift-machine-api

This is the mechanism worth knowing for a real, common scenario: a specific node has developed a known-flaky pattern (not yet unhealthy enough for MachineHealthCheck to intervene, covered later in this chapter, but clearly worth replacing proactively) — annotating it for deletion and then scaling down and back up replaces exactly that machine, rather than leaving the outcome to the controller's own default heuristic and hoping it happens to pick the right one.

Provider-Specific Differences Worth Knowing#

The Machine API's object model (Machine, MachineSet) is identical across every supported platform, but providerSpec — the platform-specific configuration block — differs meaningfully by cloud:

A MachineSet manifest authored for AWS is not portable as-is to Azure or GCP — only the surrounding object model (replicas, selector, the overall shape) is shared; the providerSpec block itself needs to be authored per-platform, which is worth setting expectations around explicitly for any team scripting MachineSet creation across a genuinely multi-cloud OpenShift footprint (Part 1's multi-architecture/hybrid-cloud coverage).

Cluster Autoscaling: ClusterAutoscaler and MachineAutoscaler#

Manually scaling MachineSets works, but doesn't respond to actual demand automatically — two cooperating objects close that gap, and it's a common interview trap to describe only one of them as "the autoscaler."

apiVersion: autoscaling.openshift.io/v1
kind: ClusterAutoscaler
metadata:
  name: default
spec:
  resourceLimits:
    maxNodesTotal: 24
  scaleDown:
    enabled: true
    delayAfterAdd: 10m
    delayAfterDelete: 10s
    delayAfterFailure: 3m
---
apiVersion: autoscaling.openshift.io/v1beta1
kind: MachineAutoscaler
metadata:
  name: prod-east-worker-us-east-1a
  namespace: openshift-machine-api
spec:
  minReplicas: 3
  maxReplicas: 10
  scaleTargetRef:
    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet
    name: prod-east-worker-us-east-1a

Exactly one ClusterAutoscaler object can exist cluster-wide (named default), while any number of MachineAutoscalers can target different MachineSets — a common real design is a MachineAutoscaler per availability zone, letting the autoscaler spread new capacity across zones for resilience rather than piling every new node into a single zone's MachineSet.

oc get clusterautoscaler default -o yaml
oc get machineautoscaler --all-namespaces
oc logs -n openshift-machine-api deployment/cluster-autoscaler-default
oc get events -n openshift-machine-api --field-selector reason=FailedScaleUp

The cluster-autoscaler-default Deployment's own logs are the direct way to see the autoscaler's actual scaling decisions and reasoning in real time — including why it declined to scale up (a Pod's specific node-affinity or taint requirement no available MachineSet can satisfy is a common, otherwise-invisible reason a Pod stays Pending despite autoscaling being fully enabled and configured correctly).

MachineHealthChecks — Automatic Node Remediation#

A MachineHealthCheck closes a different gap: what happens when a Machine's underlying infrastructure genuinely fails (a hung kubelet, an unresponsive instance) rather than simply needing more or fewer nodes.

apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
  name: prod-east-worker-health
  namespace: openshift-machine-api
spec:
  selector:
    matchLabels: { machine.openshift.io/cluster-api-machineset: prod-east-worker-us-east-1a }
  unhealthyConditions:
    - type: Ready
      status: "Unknown"
      timeout: 300s
    - type: Ready
      status: "False"
      timeout: 300s
  maxUnhealthy: "40%"
  nodeStartupTimeout: 20m

Once a Machine's Node reports Ready=Unknown or Ready=False for longer than the configured timeout, the MachineHealthCheck controller deletes the unhealthy Machine, and its owning MachineSet automatically provisions a replacement — fully automatic node replacement, with no human paged for what would otherwise be a routine hardware/instance failure. maxUnhealthy: "40%" is a deliberate circuit breaker: if more than 40% of the targeted Machines are simultaneously unhealthy, the controller stops remediating entirely, on the reasoning that a failure affecting that many machines at once is more likely a systemic problem (a zone outage, a networking issue) than N independent hardware failures, and mass-replacing machines in that scenario would likely make things worse, not better.

From the Trenches: A team's MachineHealthCheck correctly remediated a single flaky node several times over a few weeks — each replacement resolved the immediate symptom, so nobody investigated further. The pattern turned out to be a specific availability zone's underlying host hardware experiencing a slow degradation the cloud provider hadn't yet flagged — MachineHealthCheck was doing exactly its job, but treating each individual remediation as fully resolving the problem masked a trend that only became visible once someone correlated which zone every remediated machine had come from over that multi-week window. The lesson: MachineHealthCheck remediation events are themselves worth tracking as a metric/alert (a rising remediation rate, especially concentrated in one zone), not just treated as invisible, successful self-healing with nothing further to look at.

Node Maintenance: Cordon, Drain, and Safe Reboot#

Every automated mechanism above (MCO rollouts, upgrades, MachineHealthCheck remediation) ultimately performs the same manual sequence a platform team also needs to know how to run by hand for planned maintenance:

oc adm cordon worker-3.prod-east.example.com

oc adm drain worker-3.prod-east.example.com \
  --ignore-daemonsets --delete-emptydir-data --force

# ... perform maintenance, reboot, etc. ...

oc adm uncordon worker-3.prod-east.example.com

oc get node worker-3.prod-east.example.com

--ignore-daemonsets is necessary because a DaemonSet's whole purpose is running exactly one Pod per node — draining can't meaningfully "evict" it without contradicting that design, so drain skips DaemonSet-managed Pods rather than erroring out entirely. A drain that hangs indefinitely is, per this chapter's upgrade-troubleshooting section, very often the same PodDisruptionBudget culprit — checking oc get pdb is worth doing immediately rather than waiting to see if the drain eventually completes on its own.

Checking Drain Safety Before Running It#

oc adm drain --dry-run=client previews exactly what a real drain would evict, without actually touching anything — a fast, safe way to confirm the operation is going to do what's expected before committing to it on a production node:

oc adm drain worker-3.prod-east.example.com --dry-run=client --ignore-daemonsets

oc get pods --all-namespaces --field-selector spec.nodeName=worker-3.prod-east.example.com

oc describe node worker-3.prod-east.example.com | grep -A5 "Non-terminated Pods"

oc get pdb --all-namespaces -o json | \
  jq -r '.items[] | select(.status.disruptionsAllowed==0) | .metadata.namespace + "/" + .metadata.name'

That last command is worth running proactively, not just reactively during a stuck drain — it lists every PodDisruptionBudget cluster-wide currently allowing zero disruptions, which is precisely the set that could block a future node drain or upgrade before one is even attempted, giving a platform team advance warning rather than discovering the constraint mid-maintenance-window.

etcd Backup and Disaster Recovery#

etcd is the single source of truth for the entire cluster's state (Part 1) — losing it without a backup means losing the cluster's own configuration entirely, not just application data. cluster-backup.sh, run directly on a control-plane node, is the supported mechanism:

oc debug node/master-0.prod-east.example.com
chroot /host
/usr/local/bin/cluster-backup.sh /home/core/assets/backup
ls -la /home/core/assets/backup

This produces two artifacts: a snapshot of etcd's actual data, and a separate archive of the static pod manifests and certificates the control plane needs to bootstrap itself back up — both are required together for a real restore, since the certificates alone can't reconstruct cluster state, and the etcd snapshot alone can't restart a control plane with no static pod definitions to boot from.

/home/core/assets/backup/
├── snapshot_2026-08-25_020000.db                    # the actual etcd data snapshot
└── static_kuberesources_2026-08-25_020000.tar.gz    # static pod manifests + certificates

Automating Backups on a Schedule#

Running cluster-backup.sh manually before every upgrade is necessary but not sufficient — a hardware failure or a bad manual change can happen on any ordinary Tuesday, not just upgrade day. A CronJob running as a privileged, node-scoped Pod is the standard way to automate the same script on a recurring schedule:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: openshift-etcd
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          nodeSelector: { node-role.kubernetes.io/master: "" }
          tolerations:
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
          containers:
            - name: backup
              image: registry.redhat.io/openshift4/ose-cli:latest
              command: ["/usr/local/bin/cluster-backup.sh", "/etc/kubernetes/cluster-backup"]
          restartPolicy: OnFailure

This CronJob necessarily runs with node-level access (hostNetwork, targeting a control-plane node directly) to invoke the same script this section ran manually — a legitimate, narrowly-scoped exception to the least-privilege SCC discipline Part 2 established, precisely because etcd backup is inherently a node-level, control-plane operation with no meaningful way to perform it from an ordinary namespaced Pod. Pairing this CronJob with an off-cluster copy step (syncing the backup artifact to S3 or an equivalent external store) closes the "don't store the backup only on the cluster it protects" gap this chapter's practices table flagged.

Verifying a Backup Is Actually Restorable#

A backup nobody has ever test-restored is, per this catalog's Reliability & SRE content's own guidance on untested recovery procedures, a documented assumption rather than a verified capability. Periodically test-restoring a backup onto a genuinely separate, disposable cluster (never the production cluster the backup came from) is the only way to confirm the backup and restore procedure both actually work end to end, rather than trusting a script exit code alone.

Restoring From an etcd Backup#

# Copy the backup to the recovery control-plane node's /home/core first, then:
sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup

oc get clusteroperators
oc get nodes
oc get etcd cluster -o jsonpath='{.status.conditions}'

The restore script itself stops the other control-plane nodes' static pods, restores the recovery node from the snapshot, and restarts the control plane from that single node — the other control-plane nodes rejoin etcd's cluster afterward, similar in spirit to the bootstrap sequence Part 1 covered, just recovering into a known-good prior state instead of building a fresh cluster. A restore is a genuinely disruptive, last-resort operation — every change made to the cluster after the backup was taken is lost, which is exactly why the backup cadence from the previous section matters as much as the restore mechanism itself; a six-month-old backup restored during an incident recovers the cluster, but discards six months of legitimate configuration changes along with whatever caused the incident.

When a Restore Is (and Isn't) the Right Call#

Treating restore as the default response to any cluster-level problem, rather than the last resort it's designed to be, routinely discards more legitimate work than the problem itself would have cost to fix directly — the decision deserves the same explicit, deliberate weighing this series has applied to every other high-blast-radius operation.

Beyond etcd: Application Backup With OADP#

etcd backup protects the cluster's own configuration — it does not back up application data living in PersistentVolumes, which is a separate, real gap many teams don't realize until they need a restore that etcd backup alone can't provide. The OpenShift API for Data Protection (OADP) Operator, built on the open-source Velero project, closes this gap:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-payments-backup
  namespace: openshift-adp
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces: ["payments-prod"]
    snapshotVolumes: true
    ttl: 720h0m0s

OADP backs up both Kubernetes object manifests and, via cloud-provider volume snapshots, the actual persistent data those objects reference — giving a genuine application-level restore capability (recover one namespace's full state, including data) that etcd backup and restore, being cluster-wide and configuration-only, was never designed to provide. A mature disaster-recovery posture runs both mechanisms, aimed at the two genuinely different failure classes each one covers.

Restoring a Namespace From an OADP Backup#

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: payments-prod-restore
  namespace: openshift-adp
spec:
  backupName: daily-payments-backup-20260825020000
  includedNamespaces: ["payments-prod"]
  restorePVs: true
  existingResourcePolicy: update

oc get restore payments-prod-restore -n openshift-adp -o jsonpath='{.status.phase}'
velero backup describe daily-payments-backup-20260825020000
oc get restore payments-prod-restore -n openshift-adp -o jsonpath='{.status.warnings}'

A Restore object references one specific completed Backup by name — velero backup describe (or the equivalent oc get backup) is the way to list available backups and confirm exactly which point-in-time snapshot a given restore will recover to, the same "know precisely what you're restoring to before running it" discipline the etcd restore section applied to cluster-level recovery, now applied to namespace-level, data-inclusive recovery.

The Built-In Monitoring Stack#

Every OpenShift cluster ships a fully pre-integrated Prometheus, Alertmanager, and Grafana stack (via the monitoring Cluster Operator from Part 1's roster) covering the platform's own components by default — no separate installation or wiring required, a direct payoff of Part 1's "curated, jointly-tested platform" framing.

oc get pods -n openshift-monitoring
oc get prometheusrule -n openshift-monitoring
oc get alertmanager main -n openshift-monitoring -o yaml
oc get routes -n openshift-monitoring

This default stack monitors the platform itself (API server latency, etcd health, node conditions, Cluster Operator status) — it deliberately does not scrape arbitrary user-workload namespaces out of the box, which is exactly the gap the next section closes.

User Workload Monitoring#

Enabling user workload monitoring extends the same Prometheus/Alertmanager model to application teams' own namespaces, without requiring cluster-admin privileges for day-to-day use once enabled:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

This deploys a separate Prometheus instance and Thanos Ruler into openshift-user-workload-monitoring — deliberately isolated from the platform's own monitoring Prometheus, so a runaway or misconfigured application-level PrometheusRule can't degrade the platform team's own visibility into cluster health, the same isolation principle Part 2 applied to multi-tenancy generally, now applied to the monitoring stack itself.

Querying Metrics Directly#

Both Prometheus instances are reachable through the web console's Observe → Metrics tab, or directly via the CLI for scripting/automation needs:

oc exec -n openshift-monitoring prometheus-k8s-0 -c prometheus -- \
  curl -s 'http://localhost:9090/api/v1/query?query=up{job="apiserver"}'

oc get route prometheus-k8s -n openshift-monitoring -o jsonpath='{.spec.host}'
oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}'

TOKEN=$(oc create token prometheus-k8s -n openshift-monitoring)
curl -s -H "Authorization: Bearer $TOKEN" \
  --data-urlencode 'query=sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace)' \
  https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query

The thanos-querier route/service is worth knowing specifically: it federates queries across both the platform Prometheus and user-workload Prometheus instances transparently, meaning a single PromQL query issued against Thanos Querier can correlate a platform-level metric (node CPU pressure) against a user-workload metric (an application's own request rate) in one query, without needing to know or care which of the two separate Prometheus instances actually collected each series.

A Worked Example: A Custom Alert for a User Workload#

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web
  namespace: payments-prod
  labels: { k8s-app: web }
spec:
  selector:
    matchLabels: { app: web }
  endpoints:
    - port: metrics
      interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: web-error-rate
  namespace: payments-prod
spec:
  groups:
    - name: web
      rules:
        - alert: WebHighErrorRate
          expr: |
            sum(rate(http_requests_total{job="web",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="web"}[5m])) > 0.05
          for: 10m
          labels: { severity: warning }
          annotations:
            summary: "web error rate above 5% for 10 minutes"

The ServiceMonitor tells user-workload Prometheus what to scrape (any Service matching the label selector, on the named port); the PrometheusRule defines the actual alerting logic — both objects live entirely within the application team's own namespace, editable with the edit/admin RBAC role from Part 2, with no platform-team involvement required for a team to instrument and alert on their own service once user workload monitoring is enabled cluster-wide.

Routing the Alert Somewhere a Human Will See It#

A firing PrometheusRule alert is only useful once it actually reaches a human — a namespaced AlertmanagerConfig object routes a team's own alerts to their own notification channel, without touching the platform's shared Alertmanager configuration at all:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: web-alerts
  namespace: payments-prod
spec:
  route:
    receiver: payments-slack
    groupBy: ["alertname"]
  receivers:
    - name: payments-slack
      slackConfigs:
        - apiURL: { name: slack-webhook-secret, key: url }
          channel: "#payments-alerts"

This namespaced object is what makes user workload monitoring genuinely self-service end to end — a team can define what to scrape (ServiceMonitor), what counts as a problem (PrometheusRule), and where to send the resulting alert (AlertmanagerConfig), entirely within their own namespace's RBAC boundary, with the platform team's only remaining role being having enabled enableUserWorkload: true once, cluster-wide.

Grafana Dashboards for Custom Metrics#

The console's built-in Grafana instance can also visualize any metric either Prometheus instance collects, including a team's own custom application metrics — a ConfigMap labeled console.openshift.io/dashboard: "true" in openshift-config-managed is the mechanism for adding a custom dashboard directly into the console's own Dashboards view, alongside the built-in platform dashboards, rather than requiring a separately-hosted Grafana instance a team has to stand up and secure themselves.

The Built-In Logging Stack: Loki and the Cluster Logging Operator#

Metrics (Prometheus) answer "how much/how often"; logs answer "what exactly happened, in this specific request." The Cluster Logging Operator, paired with a LokiStack (via the separate Loki Operator), is OpenShift's current-generation built-in logging stack — a materially lighter-weight architecture than the older Elasticsearch-based stack it replaced, and one worth knowing the current object model for rather than out-of-date ClusterLogging-only documentation.

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.small
  storage:
    secret: { name: loki-s3-secret, type: s3 }
  storageClassName: gp3-csi
  tenants: { mode: openshift-logging }
---
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  serviceAccount: { name: collector }
  outputs:
    - name: default-lokistack
      type: lokiStack
      lokiStack: { target: { name: logging-loki, namespace: openshift-logging } }
  pipelines:
    - name: app-logs
      inputRefs: ["application"]
      outputRefs: ["default-lokistack"]

ClusterLogForwarder is the object worth focusing on: it defines named pipelines, each routing a specific log input (application, infrastructure, or audit — three built-in log classes every cluster produces) to one or more outputs, which can be the in-cluster LokiStack, or an external system (Splunk, an external Elasticsearch, Kafka) — meaning a single forwarder definition can simultaneously send application logs to Loki for day-to-day querying while also forwarding audit logs to a separate, compliance-mandated external SIEM, without running two separate collection agents to achieve it.

Querying Logs With LogQL#

Loki's query language, LogQL, is deliberately close to PromQL in shape — a label-selector stream expression, optionally piped through filters and aggregations, rather than a full-text search index:

{ kubernetes_namespace_name="payments-prod", kubernetes_container_name="web" }
  |= "error"
  | json
  | line_format "{{.message}}"

sum(rate({kubernetes_namespace_name="payments-prod"} |= "error" [5m])) by (kubernetes_pod_name)

{ log_type="audit" } | json | user_username="jane.doe"

The first query streams raw log lines matching the namespace/container labels, filtered to lines containing "error," then parses each line as JSON and reformats the output; the second aggregates the rate of matching log lines per Pod over a 5-minute window — the same rate-based reasoning PromQL applies to metrics, applied here directly to log volume, letting a team alert on "this Pod's error-log rate just spiked" using the same mental model as a metrics-based alert, through the web console's Observe → Logs tab or the logcli CLI tool directly.

OLM Day-2 Patterns: Channels and Approval Strategies Revisited#

Part 1 introduced OLM's Subscription/InstallPlan model; the Day-2 discipline worth adding here is treating channel and installPlanApproval as ongoing operational levers, not one-time install choices.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cert-manager-operator
  namespace: openshift-operators
spec:
  channel: stable-v1
  installPlanApproval: Manual

A platform team's actual Day-2 discipline is periodically reviewing every installed Operator's pending InstallPlans (oc get installplan --all-namespaces), approving them deliberately after checking the target CSV's release notes — exactly the same "review before approving, using installPlanApproval: Manual" pattern Part 1 introduced, now framed as a recurring operational habit rather than a one-time install-time decision, alongside the OpenShift cluster's own upgrade cadence from earlier in this chapter, since an incompatible Operator version is one of the most common real causes of a cluster upgrade going wrong.

The Actual Approval Workflow#

oc get installplan -n openshift-operators

oc get installplan install-abc12 -n openshift-operators -o jsonpath='{.spec.clusterServiceVersionNames}'

oc get subscription cert-manager-operator -n openshift-operators -o jsonpath='{.status.installedCSV}'

# After reviewing the target CSV's release notes:
oc patch installplan install-abc12 -n openshift-operators \
  --type merge -p '{"spec":{"approved":true}}'

oc get csv -n openshift-operators
oc get csv -n openshift-operators -o jsonpath='{.items[*].status.phase}'

The InstallPlan sits in a pending, unapproved state indefinitely until this patch is applied — nothing times out or auto-approves it, which is exactly the point of installPlanApproval: Manual: the Operator's new version simply waits for a human decision, with no risk of an unreviewed change silently applying itself after some delay.

Putting It Together: A Day-2 Operations Runbook#

Every mechanism this chapter covered composes into a standing operational rhythm a platform team actually runs, on some regular cadence, for the life of the cluster:

Daily/automated:
  - etcd backup (cluster-backup.sh, scheduled)
  - OADP application backup (Velero Schedule)
  - Monitor ClusterOperator health, Alertmanager routing
  - Review firing alerts and LogQL error-rate queries for anomalies

Before any upgrade:
  - Fresh etcd backup, confirmed restorable
  - Every OLM Operator's Subscription channel/version compatibility checked
  - Review Cincinnati conditional-update warnings (Part 1)
  - Confirm no PodDisruptionBudget cluster-wide currently allows zero disruptions

Ongoing:
  - Review pending InstallPlans, approve deliberately
  - Review MachineAutoscaler bounds against actual usage trends
  - Confirm MachineHealthCheck coverage on every MachineSet
  - Track MachineHealthCheck remediation rate/zone concentration as its own metric
  - Periodic oc adm prune images (Part 4) and log-retention review
  - Test-restore an etcd backup and an OADP backup onto a disposable cluster
  - Review self-provisioning and quota defaults against actual Project growth (Part 2)
  - Confirm every user-workload-monitoring ServiceMonitor still points at a live Service
  - Rotate the Sealed Secrets key (Part 4) on its own documented schedule

Nothing in this runbook is exotic — it's the accumulated discipline every mechanism in this chapter individually justified, assembled into the actual recurring practice that keeps a cluster healthy for years rather than merely for its first successful rollout.

Quick Reference: Key Terms From This Chapter#

Common Mistakes and Interview Traps#

Worked Practice Problems#

1. A cluster upgrade has been "in progress" for over an hour with no ClusterOperator reporting Degraded. What are the first two things to check before assuming something has failed?#

First, check oc get mcp to confirm whether the relevant MachineConfigPool(s) are genuinely still Updating — a large worker node count rolling one node at a time can legitimately take a long time, and this alone can fully explain the apparent stall with nothing actually broken. Second, check oc get pdb --all-namespaces for an overly strict PodDisruptionBudget blocking the MCO's drain step on a specific node — exactly the failure mode this chapter's own trenches story describes, which produces no ClusterOperator-level degradation at all since nothing about the operators themselves is unhealthy, only the node-drain step waiting on a constraint it's correctly refusing to violate.

2. A team enables the cluster autoscaler by creating a ClusterAutoscaler object, but reports that the cluster never scales up even when Pods are stuck Pending due to insufficient resources. What's missing?#

The team is very likely missing a MachineAutoscaler targeting the specific MachineSet(s) they want scaled — a ClusterAutoscaler alone defines cluster-wide policy (overall node limits, scale-down timing) but has no MachineSet it's actually authorized to adjust without a MachineAutoscaler bound to it. The fix is creating at least one MachineAutoscaler with scaleTargetRef pointing at the relevant MachineSet(s) and sensible minReplicas/maxReplicas bounds — only then does the ClusterAutoscaler have anything it's permitted to scale in response to unschedulable Pods.

3. A platform team's disaster-recovery plan only includes scheduled etcd backups. A production incident destroys a PersistentVolume backing a critical stateful application. What does the etcd backup restore, and what does it not?#

Restoring from the etcd backup recovers the cluster's own configuration state — every Kubernetes object definition, including the PersistentVolumeClaim object referencing the now-destroyed volume — but it does not recover the actual data that lived on that volume, since etcd never stored the data itself, only the cluster's metadata about it. The team's disaster-recovery plan has a real, concrete gap: OADP (or an equivalent Velero-based backup covering volume snapshots) is the missing piece specifically for this failure class, and this incident is exactly the scenario that gap predicts — the fix going forward is adding scheduled OADP backups for any namespace with genuinely critical persistent data, not merely relying on etcd backup to cover disaster recovery as a whole.

4. A security team asks why audit logs need to go to an external SIEM while application logs stay in the in-cluster LokiStack, and whether this requires running two separate logging agents. What's the actual answer?#

It requires no second agent — a single ClusterLogForwarder definition can route different log inputs (application, infrastructure, audit) to different outputs within the same object, meaning the existing Vector-based collection agent forwards application logs to the in-cluster LokiStack for day-to-day debugging while simultaneously forwarding audit logs to an external SIEM output for compliance retention, both from the same underlying collected log stream. The distinction the security team is really asking about is a routing/pipeline configuration choice within one forwarder definition, not a requirement for separate collection infrastructure per destination.

5. A platform team needs to replace one specific worker node whose disk has developed early SMART warnings, without waiting for it to become unhealthy enough to trigger MachineHealthCheck remediation on its own. What's the precise, controlled way to do this?#

The precise way is annotating the specific Machine backing that node with machine.openshift.io/delete-machine="true", then scaling its MachineSet down by one and back up by one (or simply down and up, letting the annotation steer which one is removed) — this proactively removes exactly the flagged machine rather than leaving the choice to the MachineSet controller's default newest-first heuristic, which has no awareness of the SMART warning at all and might just as easily remove a perfectly healthy, newer machine instead. This is meaningfully more controlled than waiting for MachineHealthCheck to eventually intervene, since that mechanism only acts once the node's Ready condition itself degrades, which a disk nearing failure may not yet be doing.

6. A team wants a single alert that fires when an application's real end-user error rate is high, correlated against whether the underlying nodes are also under memory pressure, without standing up a separate observability tool. Is this achievable with the built-in stack, and how?#

Yes — this is exactly what Thanos Querier's federation across the platform and user-workload Prometheus instances is for: a single PromQL expression can reference both a user-workload metric (the application's own http_requests_total error-rate series, scraped via the team's own ServiceMonitor) and a platform-level metric (node memory pressure, scraped by the platform's own Prometheus) in one query, since Thanos Querier presents both as one federated queryable source. The alert itself would still be authored as a namespaced PrometheusRule in the application's own namespace, referencing both series in its expr — no separate observability tool or manual metric-forwarding pipeline is required, since the federation is a built-in property of the stack this chapter described, not something the team needs to wire up themselves.

Summary and Series Wrap-Up#

This chapter closed the operational loop every earlier chapter's architecture made possible: the CVO's tested release payload (Part 1) becomes a real, repeatable upgrade procedure with known failure modes and known fixes; the Machine API extends the same declarative model to the cluster's own infrastructure, with the ClusterAutoscaler/MachineAutoscaler pair and MachineHealthCheck turning capacity management and hardware-failure remediation from manual, paged-at-3am tasks into automated, policy-driven ones; etcd backup and OADP together cover the two genuinely distinct disaster-recovery failure classes — cluster configuration and application data — that a mature Day-2 practice needs both of, not just one; and the built-in monitoring and logging stacks give a platform team (and, via user workload monitoring, every application team) the observability into all of the above without needing to stand up or integrate any of it themselves.

Across all five parts, this series followed one consistent throughline, first named explicitly in Part 1 and revisited at every layer since: OpenShift's core bet is that a curated, jointly-tested, operator-driven platform is worth its opinionation specifically when an organization's actual pain is coordinating and securing many teams or many clusters at once — not a universal improvement over vanilla Kubernetes for every team at every scale. Part 1's architecture, Part 2's default-restrictive multi-tenancy, Part 3's integrated networking and optional service mesh, Part 4's built-in delivery tooling, and this chapter's Day-2 automation are five expressions of the identical trade, each worth adopting exactly as far as an organization's real, current requirements justify it, and no further — the same honest, requirement-driven judgment call this series has asked the reader to make explicitly at every layer, rather than treating any of it as automatically "more enterprise" and therefore unconditionally better.

Cross-reference: this catalog's Kubernetes Deep Dive series covers etcd's Raft consensus, compaction, and defragmentation mechanics, and cluster upgrade/rollout concepts from a vanilla-Kubernetes vantage point — both chapters' depth applies unchanged to the OpenShift-managed etcd and CVO-driven upgrades this chapter covered operationally. Its own Observability domain content (metrics, logging, tracing fundamentals) is likewise worth revisiting for the platform-agnostic half of what this chapter's monitoring and logging sections build on.

Sources consulted for this chapter: Red Hat's OpenShift Container Platform Updating Clusters documentation, the Backup and Restore documentation (etcd disaster recovery, OADP), the Machine Management documentation (Machine API, autoscaling, MachineHealthCheck), the Monitoring Stack for Red Hat OpenShift documentation (user workload monitoring), and Red Hat's OpenShift Logging (LokiStack, ClusterLogForwarder) documentation and release notes.