Backup & Disaster Recovery for Workloads

Assumes you're comfortable with Part 4's etcd backup and disaster-recovery drill and Part 3's CSI VolumeSnapshot mechanics — this chapter goes deep on the workload/PersistentVolume backup layer those parts only touched in passing, rather than re-explaining etcd internals.

Table of Contents#

1. Why This Part Exists
2. The Two Layers of Kubernetes Disaster Recovery
3. Velero Architecture: BackupStorageLocation, VolumeSnapshotLocation, and the Backup Controller
4. CSI Snapshot Integration — How Velero Actually Protects a PersistentVolume
5. File-System Backup with Kopia — When CSI Snapshots Aren't an Option
6. Choosing CSI Snapshots vs. File-System Backup — A Decision Framework
7. Application-Consistent Backups: Pre and Post Hooks for Stateful Workloads
8. Backup Scheduling and Retention Policies
9. The Backup Object Lifecycle
10. Monitoring and Alerting for Backup Health
11. Consistency Across Namespaces — the Multi-Service Backup Problem
12. Restoring Into the Same Cluster — Namespace Mapping and Resource Policies
13. Cross-Cluster and Cross-Region Restore Patterns
14. The GitOps Interaction Problem — Suspending Reconciliation During Restore
15. RTO and RPO for Kubernetes Workloads — Turning SLAs into Backup Design
16. Disaster Recovery Drills — Practicing Restore Before You Need It
17. Securing Backup Data — Encryption, Access, and Secrets
18. Backup Storage Cost and Lifecycle Management
19. Multi-Cluster Fleet Backup Strategy
20. A Full Worked Scenario: Recovering checkout From an Accidental Namespace Deletion
21. A Full Worked Scenario: Regional Failover for catalog-service
22. A Full Worked Scenario: Running a Quarterly Disaster Recovery Drill
23. Part 18 CLI Cheat Sheet
24. Common Mistakes and Interview Traps
25. Worked Practice Problems
26. Summary and What's Next

Why This Part Exists#

Every earlier part of this series that touched backup stopped at etcd — Part 4's disaster-recovery drill and Part 15's pre-upgrade snapshot both protect the control plane's own state, and neither one protects a single byte of the actual data your applications generate. An etcd snapshot restores every Deployment, Service, and ConfigMap object definition exactly as they were at snapshot time — but it says nothing about what's actually sitting inside a PersistentVolume. If checkout-orders-db's underlying disk is destroyed, corrupted, or simply deleted by an operator running the wrong kubectl delete pvc in the wrong terminal tab, restoring etcd from a backup brings back a PersistentVolumeClaim object pointing at a volume that no longer physically exists. The order history itself is gone, and no control-plane backup was ever going to prevent that.

This chapter is about the layer etcd backup cannot reach: workload-level backup and disaster recovery, centered on Velero, the de facto standard tool for this job in the Kubernetes ecosystem. The throughline system gains one concrete new piece here — checkout-orders-db, a Postgres StatefulSet backing checkout-service's order history in the checkout namespace — alongside the existing catalog-service (namespace catalog) and inventory-service (namespace inventory) from earlier parts, giving this chapter real stateful data to actually lose and recover throughout its worked examples.

The Two Layers of Kubernetes Disaster Recovery#

A cluster owner needs two genuinely separate recovery capabilities, and conflating them is the single most common gap this chapter exists to close. One layer recovers the control plane's own record of what should exist; the other recovers the actual data those objects reference.

The two layers are complementary, not competing — a genuinely complete disaster recovery posture needs both, and neither substitutes for the other.

Velero Architecture: BackupStorageLocation, VolumeSnapshotLocation, and the Backup Controller#

Velero itself runs as an ordinary in-cluster Deployment — a controller watching its own set of CRDs (Backup, Restore, Schedule, BackupStorageLocation) and reconciling them the same "observe, compare, act" way every controller in this series has worked since Part 1.

Everything Velero produces — object manifests and PV data alike — ends up in the same durable object storage bucket, which is exactly what makes the cross-cluster restore pattern later in this chapter possible.

BackupStorageLocation (BSL) defines where the object manifests and metadata for every backup are stored — almost always an S3-compatible bucket. VolumeSnapshotLocation (VSL) configures where the volume snapshot data lives, since some storage providers keep snapshot metadata in a separate provider-specific location from the bucket holding Kubernetes object YAML.

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: platform-velero-backups
    prefix: prod-cluster
  config:
    region: us-east-1
---
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  config:
    region: us-east-1

CSI Snapshot Integration — How Velero Actually Protects a PersistentVolume#

Part 3 introduced VolumeSnapshot/VolumeSnapshotClass as a manual, standalone mechanism — Velero's CSI integration is that exact same mechanism, triggered automatically as part of a Backup, with the resulting snapshot's metadata tracked as a Backup-owned resource rather than something an operator creates by hand.

The actual bytes of the volume never pass through the Velero pod itself here — Velero only orchestrates the storage provider's own native snapshot mechanism and records the result.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: checkout-orders-db-2026-08-27
  namespace: velero
spec:
  includedNamespaces: ["checkout"]
  snapshotVolumes: true          # enables CSI snapshot for matched PVCs
  storageLocation: default
  volumeSnapshotLocations: ["default"]
  ttl: 720h0m0s                  # 30-day retention — see Backup Scheduling below

Because a CSI snapshot never scans the volume's actual filesystem — it asks the storage layer to snapshot the underlying block device directly — this is almost always both faster and less disruptive than a file-level backup, especially for a large volume, and it's the correct default whenever the underlying StorageClass's CSI driver supports VolumeSnapshot at all.

File-System Backup with Kopia — When CSI Snapshots Aren't an Option#

Not every volume type supports CSI snapshots — a hostPath volume, a CSI driver that never implemented the snapshot extension, or a storage backend without native snapshot support at all leaves CSI-based backup unavailable, which is exactly the gap Velero's File System Backup (FSB) fills.

FSB runs as a node-agent DaemonSet, mounting the same volume the workload uses and copying its file contents directly into the backup's object storage location, using Kopia as the uploader.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-service
  namespace: catalog
spec:
  template:
    metadata:
      annotations:
        # Opts this pod's "catalog-images" volume into file-system backup
        backup.velero.io/backup-volumes: catalog-images
    spec:
      containers:
        - name: catalog-service
          volumeMounts:
            - name: catalog-images
              mountPath: /data/images

From the Trenches: A platform team backed up a large, slow-changing catalog-images PVC using file-system backup because "it's just images, CSI snapshots seemed like overkill." Backup duration crept from 12 minutes to over 3 hours across six months as the volume grew, eventually blowing past the backup window and colliding with the next scheduled backup. The immediate cause was FSB's need to walk the entire filesystem tree on every run regardless of how little had actually changed; the underlying condition was that nobody had revisited the CSI-vs-FSB choice as the volume's size profile changed — a decision made once, for a small volume, was never re-evaluated as the workload it protected grew by two orders of magnitude.

Choosing CSI Snapshots vs. File-System Backup — A Decision Framework#

The two approaches aren't interchangeable defaults — the right choice depends on what the underlying storage actually supports and how the backup needs to behave operationally.

CSI snapshot duration stays roughly flat regardless of volume size because it's a storage-layer operation, not a file-by-file copy — the gap widens dramatically as volumes grow, exactly the pattern that caught the team in the previous section's callout.

Application-Consistent Backups: Pre and Post Hooks for Stateful Workloads#

A CSI snapshot captures the volume's on-disk state at the instant it's taken — for a database mid-write, that instant might land between a transaction's data write and its log write, producing a crash-consistent but not application-consistent snapshot. Velero's backup hooks run arbitrary commands inside a container immediately before and after the snapshot, giving the application a chance to reach a clean, quiescent state first.

apiVersion: v1
kind: Pod
metadata:
  name: checkout-orders-db-0
  namespace: checkout
  annotations:
    pre.hook.backup.velero.io/command: '["/bin/bash", "-c", "psql -U postgres -c \"SELECT pg_backup_start(''velero'')\""]'
    pre.hook.backup.velero.io/timeout: 30s
    post.hook.backup.velero.io/command: '["/bin/bash", "-c", "psql -U postgres -c \"SELECT pg_backup_stop()\""]'

Backup Scheduling and Retention Policies#

A one-off Backup object is a manual snapshot; a Schedule object is what actually gets a team a durable, ongoing recovery posture — generating a fresh Backup on a cron cadence with its own retention (ttl).

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: checkout-daily
  namespace: velero
spec:
  schedule: "0 3 * * *"          # 03:00 UTC daily
  template:
    includedNamespaces: ["checkout"]
    snapshotVolumes: true
    storageLocation: default
    ttl: 720h0m0s                # 30 days

From the Trenches: A team set every namespace's Schedule to a uniform 7-day ttl "to keep storage costs predictable," including checkout-orders-db. A billing dispute six weeks later needed the order state from 40 days prior — long past every retention window, with the underlying CSI snapshots already expired and reclaimed by the storage provider. The immediate cause was a retention policy set once and never revisited per-workload; the underlying condition was treating "backup retention" as a single global knob instead of a decision that should map to each workload's actual recovery and compliance requirements, the same tiering shown in the table above.

The Backup Object Lifecycle#

A Backup object moves through a defined set of phases from creation to eventual expiry — worth knowing precisely, since "why is my backup stuck" is a genuinely common on-call question this maps directly to.

A PartiallyFailed backup is genuinely usable for restore in most cases — it's worth checking exactly which items failed (velero backup describe --details) rather than treating the whole run as worthless.

Monitoring and Alerting for Backup Health#

The previous section's tip — alert on backup outcome, not just occurrence — needs real metrics and alerting rules behind it, since "check velero backup get manually every morning" doesn't scale past a handful of namespaces and reliably gets skipped the first busy week.

Velero exposes its own Prometheus metrics endpoint, the same observability pattern Part 10 established generally for scraping any in-cluster component's /metrics endpoint.

# PrometheusRule — alert if checkout's backup schedule hasn't produced
# a successful backup in over 26 hours (a daily schedule plus margin)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: velero-backup-health
  namespace: velero
spec:
  groups:
    - name: velero-backup-health
      rules:
        - alert: BackupStale
          expr: (time() - velero_backup_last_successful_timestamp{schedule="checkout-daily"}) > 93600
          for: 15m
          labels: { severity: critical }
          annotations:
            summary: "checkout-daily has not produced a successful backup in over 26 hours"
        - alert: BackupPartialFailure
          expr: increase(velero_backup_partial_failure_total[1h]) > 0
          labels: { severity: warning }
          annotations:
            summary: "A recent backup completed with partial failures — check velero backup describe --details"

Consistency Across Namespaces — the Multi-Service Backup Problem#

checkout-service, catalog-service, and inventory-service each live in their own namespace with their own independently-scheduled Backup, and that independence — genuinely useful for per-team ownership and retention tuning — creates a real, easy-to-miss gap: a restore spanning all three namespaces can bring each one back from a slightly different point in time, not one shared, mutually consistent moment.

Each individual namespace's backup is internally valid — the inconsistency only exists at the boundary between namespaces, which is exactly why it's easy to miss when reviewing backups one namespace at a time.

From the Trenches: A regional-failover drill restored checkout and catalog from backups taken 20 minutes apart, and the resulting standby environment showed checkout-service order confirmations referencing catalog SKUs that, in the restored catalog namespace, didn't exist yet — they'd been added in the 20-minute gap. The immediate cause was two independently-scheduled backups with no coordination; the underlying condition was that nobody had ever explicitly decided whether checkout and catalog constituted one consistency domain or two, so each team's backup schedule was tuned purely for its own namespace's convenience with zero awareness of the cross-namespace assumption the application itself depends on.

Restoring Into the Same Cluster — Namespace Mapping and Resource Policies#

The simplest restore case — recovering a namespace back into the same cluster it was backed up from — is also where the resource-conflict behavior matters most, since some of the namespace's objects may still exist.

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: catalog-restore-2026-08-27
  namespace: velero
spec:
  backupName: catalog-daily-20260827030000
  includedNamespaces: ["catalog"]
  existingResourcePolicy: update    # update objects that already exist, don't skip them

Namespace mapping lets a restore land in a different namespace name than the one it was backed up from — genuinely useful for restoring a production backup into a staging namespace for validation without touching production at all:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: catalog-restore-to-staging
  namespace: velero
spec:
  backupName: catalog-daily-20260827030000
  includedNamespaces: ["catalog"]
  namespaceMapping:
    catalog: catalog-staging-restore-test

Cross-Cluster and Cross-Region Restore Patterns#

Because every Backup's manifests and volume snapshot data live in ordinary object storage rather than inside the originating cluster, a completely separate cluster — in a different region, a different cloud account, or even a different provider for file-system-backed volumes — can restore from that same bucket with zero coordination from the original cluster at all.

The standby cluster in Region B needs nothing from Region A beyond read access to the same bucket — this is the property that makes Velero a genuine regional-failover tool, not just a same-cluster undo button.

The GitOps Interaction Problem — Suspending Reconciliation During Restore#

A cluster running Argo CD or Flux has a second controller that also reconciles the exact objects Velero is trying to restore — and if that controller is still running during a restore, it can undo the restore within seconds of it completing.

This isn't a Velero bug — Argo CD and Velero are both, correctly, doing exactly what they're each designed to do; the conflict only exists because nobody told one of them to stand down first.

The fix is explicit: suspend GitOps reconciliation for the affected scope before restoring, and resume it only after confirming the restored state is actually what you want kept.

# Argo CD — skip reconciliation for the affected Application during restore
kubectl annotate application catalog-service -n argocd \
  argocd.argoproj.io/skip-reconcile="true" --overwrite

# Flux — suspend the Kustomization/HelmRelease covering the restored namespace
flux suspend kustomization catalog-service

# ... perform the Velero restore, verify it, then resume:
flux resume kustomization catalog-service

RTO and RPO for Kubernetes Workloads — Turning SLAs into Backup Design#

Recovery Time Objective (RTO) and Recovery Point Objective (RPO) are the two numbers that should actually drive every backup-design decision in this chapter, rather than backup cadence being chosen arbitrarily. RTO is how long a workload can be down before the business impact is unacceptable; RPO is how much recent data loss is acceptable if a disaster hits between backups.

The genuinely important point this table makes concrete: Velero's own backup cadence is only one lever, and for the tightest RPO targets it isn't sufficient on its own. A 5-minute RPO for checkout-orders-db isn't achievable by scheduling Velero backups every 5 minutes (the operational overhead and snapshot churn would be substantial) — it's achieved by pairing infrequent Velero backups (for full-namespace disaster recovery) with the database's own continuous replication mechanism (streaming replication to a standby, independent of Kubernetes entirely) for the tight-RPO data-loss case specifically.

Disaster Recovery Drills — Practicing Restore Before You Need It#

Part 4's etcd drill established the core principle for the control-plane layer — a backup that's never been test-restored is a hypothesis, not a verified recovery capability — and it applies with equal force here, arguably more so given how many more moving parts a workload restore has (CSI drivers, hooks, GitOps suspension, namespace mapping) than a single etcdctl snapshot restore command.

Quarterly DR drill checklist:
- [ ] Pick a real backup from the last 24 hours, not a specially-prepared "known good" one
- [ ] Restore into an isolated drill namespace or drill cluster, never production
- [ ] Suspend GitOps reconciliation for the drill scope before restoring
- [ ] Time the actual restore, end to end, against the workload's stated RTO
- [ ] Verify application health post-restore, not just "the objects exist"
- [ ] Verify data integrity specifically — row counts, a checksum, a known test record
- [ ] Document any gap between the drill's actual RTO and the target, and file a follow-up
- [ ] Rotate which namespace/workload is drilled each quarter — don't always drill the easy one

From the Trenches: A platform team ran quarterly DR drills faithfully for two years, always restoring the catalog namespace because it was fast and reliable — a comfortable, low-risk drill to report as "green" every quarter. When checkout-orders-db actually needed restoring after a storage-layer incident, the pre-backup hook's pg_backup_start() command had been silently failing for months due to an unrelated credentials rotation, meaning every recent backup was crash-consistent only, not the application-consistent backup the team believed they had. The immediate cause was a hook failure with no alerting on hook success/failure specifically; the underlying condition was a DR drill program that exercised the same easy, well-behaved workload repeatedly instead of rotating through every workload's real recovery path, which is exactly the rotation the checklist above now enforces.

The verification step deserves automation, not just a checklist item — the quarterly drill worked scenario later in this chapter finds exactly this gap in practice: a fast, well-within-RTO restore followed by a slow, entirely manual verification pass. A small verification script, run automatically as the last step of every drill (and, ideally, wired into the drill's own CI pipeline rather than run by hand), closes that gap directly:

#!/usr/bin/env bash
# post-restore-verify.sh — run immediately after a drill restore completes
set -euo pipefail

NAMESPACE="$1"
EXPECTED_CHECKSUM_FILE="$2"

kubectl wait --for=condition=Ready pod -l app="${NAMESPACE}-service" -n "$NAMESPACE" --timeout=120s

# Compare a known, pre-drill data checksum against the restored state
kubectl exec -n "$NAMESPACE" deploy/"${NAMESPACE}-service" -- \
  ./scripts/checksum-data.sh > /tmp/restored-checksum.txt

if diff -q "$EXPECTED_CHECKSUM_FILE" /tmp/restored-checksum.txt > /dev/null; then
  echo "PASS: restored data matches expected checksum"
else
  echo "FAIL: restored data diverges from expected checksum — investigate before closing the drill"
  exit 1
fi

Securing Backup Data — Encryption, Access, and Secrets#

A Velero backup bucket is, by construction, a complete off-cluster copy of a namespace's Secret objects along with everything else — anyone with read access to that bucket has read access to every credential those Secrets hold, independent of the in-cluster RBAC controls Part 11 covered in depth.

Backup Storage Cost and Lifecycle Management#

Backup storage cost is easy to ignore while a namespace is small and expensive to discover only once a fleet-wide bill review flags it — the same growth pattern that quietly turned this chapter's file-system backup callout into a three-hour job applies just as much to storage spend as to backup duration.

CSI snapshots for block storage are typically billed incrementally (only the changed blocks since the previous snapshot), which keeps steady-state cost far lower than the full volume size might suggest — but that economy disappears if retention (ttl) is set generously "just in case" across every tier, since every retained snapshot in the chain has to be kept until nothing downstream still depends on it.

# S3 lifecycle policy on the Velero backup bucket — move backups older
# than 30 days to cheaper, slower-retrieval storage, and expire outright
# past the compliance-driven retention ceiling
{
  "Rules": [
    {
      "ID": "velero-backup-tiering",
      "Status": "Enabled",
      "Filter": { "Prefix": "prod-cluster/backups/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 2555 }
    }
  ]
}

Multi-Cluster Fleet Backup Strategy#

Part 13 covered fleet management as the escape-hatch beyond single-cluster multi-tenancy — backup strategy needs the same fleet-level thinking, since a per-cluster, independently-configured Velero installation across a growing fleet quietly becomes as hard to audit as the per-cluster policy drift Part 13 warned about generally.

# Helm values shared across every cluster's Velero install via the
# fleet's GitOps pipeline — the per-cluster bucket prefix is the
# ONLY value that should ever differ between clusters
configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: platform-velero-backups
      config:
        region: us-east-1
      # prefix is templated per-cluster by the GitOps pipeline,
      # e.g. "prod-us-east-1", "prod-eu-west-1", "staging-us-east-1"
      prefix: "{{ .Values.clusterName }}"
schedules:
  checkout-daily:
    schedule: "0 3 * * *"
    template:
      ttl: "720h"
      includedNamespaces: ["checkout"]

Templating everything except the bucket prefix through one shared Helm release, deployed identically to every cluster in the fleet, is what actually prevents the "which cluster's Velero config is the odd one out" question from ever needing to be asked — the same principle Part 13 applied to policy-engine configuration generally, applied here specifically to backup infrastructure.

A Full Worked Scenario: Recovering checkout From an Accidental Namespace Deletion#

An engineer, intending to delete a long-abandoned checkout-canary-test namespace, runs kubectl delete namespace checkout instead — every Deployment, Service, ConfigMap, and PVC in the real production checkout namespace begins terminating immediately.

1. Stop the bleeding first. Confirm no automation (a CI pipeline, a GitOps controller) will recreate the namespace in a half-restored state while the real restore is being prepared — suspend the relevant GitOps Application/Kustomization per the earlier section, immediately.
2. Identify the most recent good Backup. velero backup get confirms the last scheduled checkout-daily backup completed successfully roughly 90 minutes before the incident.
3. Restore into the same cluster, same namespace name, since the namespace itself no longer exists to conflict with:
   Copy

   velero restore create checkout-recovery-20260827 \
     --from-backup checkout-daily-20260827030000 \
     --include-namespaces checkout

4. Verify PVC data specifically, not just object existence — checkout-orders-db's CSI-snapshot-restored volume needs Postgres to actually start and serve queries cleanly, confirmed against the pre-backup hook's consistency guarantee from earlier in this chapter.
5. Accept the RPO gap explicitly. Any order placed in the roughly 90 minutes between the last backup and the deletion is genuinely lost from this restore path alone — this is exactly why checkout-orders-db's RTO/RPO table earlier in this chapter called for continuous database replication as a second, faster recovery layer alongside Velero, not Velero backup cadence alone.
6. Resume GitOps reconciliation only after confirming the restored checkout namespace matches the intended current desired state, not a stale one.

A Full Worked Scenario: Regional Failover for catalog-service#

us-east-1 suffers a multi-hour regional outage affecting the primary cluster entirely — catalog-service needs to come back up in us-west-2 on the standby cluster provisioned per this chapter's cross-region pattern.

Step 7 deserves its own emphasis: fail-back is not simply "run the same restore in reverse" once the primary region recovers. By the time us-east-1 is healthy again, the standby cluster in us-west-2 has been serving live production traffic and has genuinely newer data than the last backup restored into it — failing back naively would discard everything written during the failover window. A real fail-back requires either a fresh backup taken from the standby and restored into the recovered primary, or a period of bidirectional data reconciliation specific to catalog-service's own data model — this is a deliberate, planned operation, never an automated reversal of the failover steps above.

A Full Worked Scenario: Running a Quarterly Disaster Recovery Drill#

Bringing this chapter's full checklist together for a real quarterly drill against inventory-service, rotating away from the easier catalog namespace per the earlier "From the Trenches" lesson.

The drill surfaces a real finding worth narrating: the restore itself completes in 9 minutes, comfortably inside the 30-minute RTO, but verifying inventory counts against the pre-drill checksum takes another 40 minutes because no automated verification script existed — someone had to manually query and compare counts across a dozen product categories. The actual finding isn't "the restore is too slow," it's "the verification step has no tooling," which becomes this quarter's concrete follow-up: build an automated post-restore verification script before the next drill, rather than re-discovering the same manual bottleneck every quarter.

Part 18 CLI Cheat Sheet#

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A team's etcd backups (Part 4) are healthy and regularly tested, but a storage-layer incident destroys the underlying disk backing checkout-orders-db's PVC. After restoring etcd from the most recent snapshot, the PersistentVolumeClaim object exists again but the application still can't read any order history. What's actually missing, and why didn't the etcd restore fix it?

Answer: etcd only stores API object definitions — restoring it recreates the PersistentVolumeClaim object's YAML exactly as it was, but that object is just a pointer to a physical volume, and the physical volume's actual data was destroyed, not the pointer to it. Recovering the real order data requires a workload-level backup tool (Velero, with CSI snapshots or file-system backup covering that PVC) taken before the incident — etcd backup and workload backup protect two structurally different things, and having one healthy says nothing about the other.

Problem 2: A Backup for the checkout namespace shows Completed, but three months later a restore from it produces a checkout-orders-db that fails to start Postgres cleanly, with corruption errors in the database log. The backup hooks were configured correctly. What's the most likely explanation?

Answer: A Completed phase for the overall Backup object confirms Velero successfully captured and uploaded everything it was told to capture — it does not independently verify the data itself is internally consistent at the application level beyond what the pre/post hooks already ensured at backup time. The most likely explanation is either a hook that appeared to succeed but didn't actually complete its intended action (a pg_backup_start() call that silently failed, as in this chapter's DR-drill "From the Trenches" callout) or an underlying storage-layer bug in the CSI snapshot itself — either way, this is precisely the gap a DR drill's data-integrity verification step (checksums, known test records) is designed to catch, which a "backup completed" status alone cannot.

Problem 3: A platform team configures cross-region restore for catalog-service, tests it successfully in a drill, and considers the DR posture complete. During a real regional outage, the restore into the standby cluster fails because the standby's StorageClass references a CSI driver parameter specific to a region that no longer matches. What should the team have caught, and where does the fix belong?

Answer: Velero only restores what it captured — application objects and PV data — it never provisions or validates cluster-scoped infrastructure like StorageClass definitions, CRDs, or IngressClass objects that a restored workload depends on. The standby cluster's baseline infrastructure needs to be provisioned and kept in sync independently, typically through the same GitOps/IaC pipeline used for the primary cluster, and validated as part of the DR drill itself — not assumed to already be correct because the Velero restore step succeeded in isolation during an earlier, less complete test.

Summary and What's Next#

Workload-level backup and disaster recovery is a genuinely distinct discipline from the etcd backup Part 4 and Part 15 already covered — protecting PersistentVolume data and giving teams a real cross-cluster, cross-region recovery path that etcd snapshots structurally cannot provide. Velero's CSI snapshot and file-system backup mechanisms, application-consistent hooks, and shared-object-storage architecture combine to make that recovery path practical, but only when RTO/RPO targets are set deliberately per workload, when GitOps reconciliation is suspended correctly during a restore, and when the recovery procedure is actually drilled — rotating across real workloads, not just the easiest one — rather than assumed to work because a backup shows Completed.

Part 19 turns to a different kind of infrastructure entirely: building the custom controllers and Operators this series has referenced conceptually since Part 4, but now at the actual client-go/controller-runtime mechanics level — informers, workqueues, reconciler code, CRD versioning in practice, and how to test a controller before it ever reconciles a real cluster.