# Kubernetes Deep Dive — Part 18: Backup & Disaster Recovery for Workloads

> **Series:** Kubernetes Deep Dive (18 of 19)
> **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane
> **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects
> **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI)
> **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators
> **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE
> **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes
> **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth
> **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway
> **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem
> **Part 10:** `10-troubleshooting-kubernetes.md` — Troubleshooting Kubernetes, Systematically
> **Part 11:** `11-kubernetes-security-deep-dive.md` — Kubernetes Security Deep Dive (CKS-Aligned)
> **Part 12:** `12-autoscaling-hpa-vpa-keda.md` — Autoscaling: HPA, VPA, KEDA & Cluster Autoscaling
> **Part 13:** `13-multi-tenancy-and-cluster-sharing.md` — Multi-Tenancy & Cluster Sharing at Scale
> **Part 14:** `14-ai-ml-workloads-on-kubernetes.md` — Running AI/ML Workloads on Kubernetes
> **Part 15:** `15-cluster-upgrades-and-lifecycle.md` — Cluster Upgrades & Lifecycle Management
> **Part 16:** `16-capacity-planning.md` — Capacity Planning for Kubernetes Clusters
> **Part 17:** `17-cilium-ebpf-networking.md` — Cilium & eBPF Networking Deep Dive
> **Part 18:** This file — Backup & Disaster Recovery for Workloads
> **Part 19:** `19-building-custom-controllers-and-operators.md` — Building Custom Controllers and Operators
> **Questions:** `questions.md`

Assumes you're comfortable with Part 4's etcd backup and disaster-recovery drill and Part 3's CSI
VolumeSnapshot mechanics — this chapter goes deep on the workload/PersistentVolume backup layer those parts
only touched in passing, rather than re-explaining etcd internals.

## Table of Contents

1. [Why This Part Exists](#why-this-part-exists)
2. [The Two Layers of Kubernetes Disaster Recovery](#the-two-layers-of-kubernetes-disaster-recovery)
3. [Velero Architecture: BackupStorageLocation, VolumeSnapshotLocation, and the Backup Controller](#velero-architecture-backupstoragelocation-volumesnapshotlocation-and-the-backup-controller)
4. [CSI Snapshot Integration — How Velero Actually Protects a PersistentVolume](#csi-snapshot-integration--how-velero-actually-protects-a-persistentvolume)
5. [File-System Backup with Kopia — When CSI Snapshots Aren't an Option](#file-system-backup-with-kopia--when-csi-snapshots-arent-an-option)
6. [Choosing CSI Snapshots vs. File-System Backup — A Decision Framework](#choosing-csi-snapshots-vs-file-system-backup--a-decision-framework)
7. [Application-Consistent Backups: Pre and Post Hooks for Stateful Workloads](#application-consistent-backups-pre-and-post-hooks-for-stateful-workloads)
8. [Backup Scheduling and Retention Policies](#backup-scheduling-and-retention-policies)
9. [The Backup Object Lifecycle](#the-backup-object-lifecycle)
10. [Monitoring and Alerting for Backup Health](#monitoring-and-alerting-for-backup-health)
11. [Consistency Across Namespaces — the Multi-Service Backup Problem](#consistency-across-namespaces--the-multi-service-backup-problem)
12. [Restoring Into the Same Cluster — Namespace Mapping and Resource Policies](#restoring-into-the-same-cluster--namespace-mapping-and-resource-policies)
13. [Cross-Cluster and Cross-Region Restore Patterns](#cross-cluster-and-cross-region-restore-patterns)
14. [The GitOps Interaction Problem — Suspending Reconciliation During Restore](#the-gitops-interaction-problem--suspending-reconciliation-during-restore)
15. [RTO and RPO for Kubernetes Workloads — Turning SLAs into Backup Design](#rto-and-rpo-for-kubernetes-workloads--turning-slas-into-backup-design)
16. [Disaster Recovery Drills — Practicing Restore Before You Need It](#disaster-recovery-drills--practicing-restore-before-you-need-it)
17. [Securing Backup Data — Encryption, Access, and Secrets](#securing-backup-data--encryption-access-and-secrets)
18. [Backup Storage Cost and Lifecycle Management](#backup-storage-cost-and-lifecycle-management)
19. [Multi-Cluster Fleet Backup Strategy](#multi-cluster-fleet-backup-strategy)
20. [A Full Worked Scenario: Recovering `checkout` From an Accidental Namespace Deletion](#a-full-worked-scenario-recovering-checkout-from-an-accidental-namespace-deletion)
21. [A Full Worked Scenario: Regional Failover for `catalog-service`](#a-full-worked-scenario-regional-failover-for-catalog-service)
22. [A Full Worked Scenario: Running a Quarterly Disaster Recovery Drill](#a-full-worked-scenario-running-a-quarterly-disaster-recovery-drill)
23. [Part 18 CLI Cheat Sheet](#part-18-cli-cheat-sheet)
24. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
25. [Worked Practice Problems](#worked-practice-problems)
26. [Summary and What's Next](#summary-and-whats-next)

---

## Why This Part Exists

**Every earlier part of this series that touched backup stopped at etcd — Part 4's disaster-recovery drill
and Part 15's pre-upgrade snapshot both protect the control plane's own state, and neither one protects a
single byte of the actual data your applications generate.** An etcd snapshot restores every `Deployment`,
`Service`, and `ConfigMap` object definition exactly as they were at snapshot time — but it says nothing
about what's actually sitting inside a `PersistentVolume`. If `checkout-orders-db`'s underlying disk is
destroyed, corrupted, or simply deleted by an operator running the wrong `kubectl delete pvc` in the wrong
terminal tab, restoring etcd from a backup brings back a `PersistentVolumeClaim` object pointing at a volume
that no longer physically exists. The order history itself is gone, and no control-plane backup was ever
going to prevent that.

This chapter is about the layer etcd backup cannot reach: workload-level backup and disaster recovery,
centered on Velero, the de facto standard tool for this job in the Kubernetes ecosystem. The throughline
system gains one concrete new piece here — `checkout-orders-db`, a Postgres `StatefulSet` backing
`checkout-service`'s order history in the `checkout` namespace — alongside the existing `catalog-service`
(namespace `catalog`) and `inventory-service` (namespace `inventory`) from earlier parts, giving this
chapter real stateful data to actually lose and recover throughout its worked examples.

> [!NOTE]
> Velero is not the only tool in this space (Kasten K10 and Trilio are notable commercial alternatives with
> broadly similar architecture), but it's open source, CNCF-adjacent, and the tool most platform teams reach
> for first — this chapter uses it as the concrete reference implementation while keeping the underlying
> concepts (BackupStorageLocation-style abstractions, CSI snapshot integration, application-consistent
> hooks) portable to whichever tool a given organization actually standardizes on.

## The Two Layers of Kubernetes Disaster Recovery

**A cluster owner needs two genuinely separate recovery capabilities, and conflating them is the single most
common gap this chapter exists to close.** One layer recovers the control plane's own record of what should
exist; the other recovers the actual data those objects reference.

```mermaid
flowchart TD
    Disaster(("Something goes wrong")) --> Type{"What was lost?"}
    Type -->|"Control plane state:<br/>every object definition<br/>in etcd is gone/corrupted"| CP["Layer 1: etcd backup/restore<br/>(Part 4, Part 15)"]
    Type -->|"Application data:<br/>a PV, a namespace,<br/>or specific objects"| WL["Layer 2: workload backup/restore<br/>(this chapter — Velero)"]

    CP --> CPScope["Restores: every API object<br/>exactly as it existed —<br/>Deployments, Services, RBAC,<br/>the whole control-plane record"]
    CP --> CPGap["Does NOT restore:<br/>PersistentVolume contents,<br/>data inside a running database"]

    WL --> WLScope["Restores: selected namespaces/<br/>objects PLUS the actual<br/>PersistentVolume data,<br/>into the SAME or a<br/>DIFFERENT cluster"]
    WL --> WLGap["Does NOT restore:<br/>control-plane components<br/>themselves — assumes a<br/>working API server already<br/>exists to restore into"]

    classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    class Disaster crit
    class CP,WL info
    class CPScope,WLScope ok
    class CPGap,WLGap crit
```

*The two layers are complementary, not competing — a genuinely complete disaster recovery posture needs
both, and neither substitutes for the other.*

| | etcd backup/restore (Part 4/15) | Velero (this chapter) |
|---|---|---|
| Restores | Every object definition stored in etcd | Selected namespaces/objects, plus PV data |
| Granularity | All-or-nothing — restores the entire control-plane state as a snapshot | Namespace- or label-selector-scoped — can restore one team's workloads without touching another's |
| Works on managed Kubernetes (EKS/GKE/AKS)? | No — the cloud provider manages etcd directly; you have no snapshot access | Yes — Velero runs as a normal workload and never touches etcd |
| Restores into a different cluster? | Effectively no — an etcd snapshot is tied to the exact cluster it came from | Yes — this is one of Velero's core, designed-for use cases |
| Protects PersistentVolume data | No | Yes — via CSI snapshots or file-system backup |

> [!IMPORTANT]
> On any managed Kubernetes offering, Velero-style workload backup is not just a nice-to-have layered on top
> of etcd backup — it's the *only* layer you have any control over at all, since the cloud provider never
> exposes raw etcd snapshot access on a managed control plane. Teams migrating from a self-managed cluster
> (where Part 4's etcd drill was the primary DR story) to EKS/GKE/AKS sometimes carry over an etcd-centric
> mental model that simply doesn't apply anymore.

## Velero Architecture: BackupStorageLocation, VolumeSnapshotLocation, and the Backup Controller

**Velero itself runs as an ordinary in-cluster Deployment — a controller watching its own set of CRDs
(`Backup`, `Restore`, `Schedule`, `BackupStorageLocation`) and reconciling them the same "observe, compare,
act" way every controller in this series has worked since Part 1.**

```mermaid
flowchart LR
    User["Operator or Schedule"] -->|"kubectl apply<br/>a Backup object"| API["API Server"]
    API --> Ctrl["Velero server<br/>(Deployment)"]
    Ctrl -->|"reads objects to back up"| APIres[("Cluster API objects")]
    Ctrl -->|"triggers PV snapshot"| CSI["CSI snapshot-controller"]
    Ctrl -->|"uploads manifests + data"| BSL[["BackupStorageLocation:<br/>object storage<br/>(S3 / GCS / Azure Blob)"]]
    CSI -->|"snapshot metadata"| VSL[["VolumeSnapshotLocation"]]
    VSL --> BSL

    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef accent fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    class Ctrl accent
    class BSL,VSL ok
    class APIres,CSI info
```

*Everything Velero produces — object manifests and PV data alike — ends up in the same durable object
storage bucket, which is exactly what makes the cross-cluster restore pattern later in this chapter possible.*

`BackupStorageLocation` (BSL) defines where the object manifests and metadata for every backup are stored —
almost always an S3-compatible bucket. `VolumeSnapshotLocation` (VSL) configures where the *volume* snapshot
data lives, since some storage providers keep snapshot metadata in a separate provider-specific location
from the bucket holding Kubernetes object YAML.

```yaml
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: platform-velero-backups
    prefix: prod-cluster
  config:
    region: us-east-1
---
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  config:
    region: us-east-1
```

| Object | Scope | Holds |
|---|---|---|
| `BackupStorageLocation` | Cluster-wide (in the `velero` namespace) | Where Kubernetes object manifests/metadata for every backup live |
| `VolumeSnapshotLocation` | Cluster-wide | Provider-specific config for where PV snapshot data lives |
| `Backup` | One backup run | A record of exactly what was captured, referencing both locations above |
| `Schedule` | A recurring policy | Generates a new `Backup` object on a cron schedule (covered later this chapter) |

> [!NOTE]
> Since Velero v1.14, CSI snapshot support ships built into the core Velero binary — the separate
> `velero-plugin-for-csi` add-on that earlier versions required is no longer a separate install step. If
> you're following an older tutorial or internal runbook that still references installing that plugin
> manually, confirm your actual Velero version before assuming the extra step is still needed.

## CSI Snapshot Integration — How Velero Actually Protects a PersistentVolume

**Part 3 introduced `VolumeSnapshot`/`VolumeSnapshotClass` as a manual, standalone mechanism — Velero's CSI
integration is that exact same mechanism, triggered automatically as part of a `Backup`, with the resulting
snapshot's metadata tracked as a `Backup`-owned resource rather than something an operator creates by hand.**

```mermaid
sequenceDiagram
    participant Op as Operator
    participant Velero as Velero server
    participant Snap as snapshot-controller
    participant CSI as CSI driver
    participant Store as Object storage

    Op->>Velero: Create Backup (includes checkout namespace)
    Velero->>Velero: Discover PVCs bound to<br/>PVs in checkout-orders-db
    Velero->>Snap: Create VolumeSnapshot for<br/>each discovered PVC
    Snap->>CSI: Request storage-level snapshot
    CSI-->>Snap: Snapshot ready (VolumeSnapshotContent)
    Velero->>Store: Upload object manifests +<br/>VolumeSnapshot metadata
    Note over Store: A single Backup now contains<br/>both the Deployment/StatefulSet<br/>YAML AND a pointer to the<br/>storage-level PV snapshot
```

*The actual bytes of the volume never pass through the Velero pod itself here — Velero only orchestrates
the storage provider's own native snapshot mechanism and records the result.*

```yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: checkout-orders-db-2026-08-27
  namespace: velero
spec:
  includedNamespaces: ["checkout"]
  snapshotVolumes: true          # enables CSI snapshot for matched PVCs
  storageLocation: default
  volumeSnapshotLocations: ["default"]
  ttl: 720h0m0s                  # 30-day retention — see Backup Scheduling below
```

**Because a CSI snapshot never scans the volume's actual filesystem** — it asks the storage layer to
snapshot the underlying block device directly — this is almost always both faster and less disruptive than
a file-level backup, especially for a large volume, and it's the correct default whenever the underlying
`StorageClass`'s CSI driver supports `VolumeSnapshot` at all.

## File-System Backup with Kopia — When CSI Snapshots Aren't an Option

**Not every volume type supports CSI snapshots — a `hostPath` volume, a CSI driver that never implemented
the snapshot extension, or a storage backend without native snapshot support at all leaves CSI-based backup
unavailable, which is exactly the gap Velero's File System Backup (FSB) fills.**

FSB runs as a `node-agent` DaemonSet, mounting the same volume the workload uses and copying its file
contents directly into the backup's object storage location, using Kopia as the uploader.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-service
  namespace: catalog
spec:
  template:
    metadata:
      annotations:
        # Opts this pod's "catalog-images" volume into file-system backup
        backup.velero.io/backup-volumes: catalog-images
    spec:
      containers:
        - name: catalog-service
          volumeMounts:
            - name: catalog-images
              mountPath: /data/images
```

> [!NOTE]
> Velero's original file-system uploader, Restic, is deprecated and disabled for new backups in current
> Velero releases — **Kopia is the default uploader for FSB today.** If an internal runbook or older blog
> post references `restic.velero.io/backup-volumes` as the annotation key, that's the pre-deprecation form;
> current Velero accepts the same annotation name but always routes through Kopia underneath regardless of
> which key name is used, so functionally the distinction mostly matters for understanding what's actually
> running, not for writing new manifests.

> **From the Trenches:** A platform team backed up a large, slow-changing `catalog-images` PVC using file-system
> backup because "it's just images, CSI snapshots seemed like overkill." Backup duration crept from 12 minutes
> to over 3 hours across six months as the volume grew, eventually blowing past the backup window and
> colliding with the next scheduled backup. The immediate cause was FSB's need to walk the entire filesystem
> tree on every run regardless of how little had actually changed; the underlying condition was that nobody
> had revisited the CSI-vs-FSB choice as the volume's size profile changed — a decision made once, for a
> small volume, was never re-evaluated as the workload it protected grew by two orders of magnitude.

## Choosing CSI Snapshots vs. File-System Backup — A Decision Framework

**The two approaches aren't interchangeable defaults — the right choice depends on what the underlying
storage actually supports and how the backup needs to behave operationally.**

| Choose | When |
|---|---|
| CSI snapshot | The `StorageClass`'s CSI driver supports `VolumeSnapshot` (true for essentially every major cloud block-storage driver: EBS, Persistent Disk, Azure Disk) |
| File-system backup (Kopia) | The volume type has no CSI snapshot support (`hostPath`, some legacy NFS setups), or you need portability across genuinely different storage backends between backup and restore targets |
| File-system backup (Kopia) | You need to restore into a cluster on a different cloud provider or storage class entirely — a CSI snapshot is tied to its originating storage provider's format |

```mermaid
xychart-beta
    title "Approximate Backup Duration by Volume Size"
    x-axis ["10 GB", "100 GB", "500 GB", "2 TB"]
    y-axis "Backup duration (minutes)" 0 --> 180
    bar "CSI snapshot" [1, 2, 3, 5]
    bar "File-system backup (Kopia)" [3, 25, 90, 160]
```

*CSI snapshot duration stays roughly flat regardless of volume size because it's a storage-layer operation,
not a file-by-file copy — the gap widens dramatically as volumes grow, exactly the pattern that caught the
team in the previous section's callout.*

> [!TIP]
> Default to CSI snapshots for every volume where the driver supports them, and reserve file-system backup
> specifically for the cases the decision table above actually calls for — not as a universal fallback
> "because it works everywhere." A blanket FSB-for-everything policy trades away CSI's speed and lower
> operational overhead for a portability benefit most backups never actually need.

## Application-Consistent Backups: Pre and Post Hooks for Stateful Workloads

**A CSI snapshot captures the volume's on-disk state at the instant it's taken — for a database mid-write,
that instant might land between a transaction's data write and its log write, producing a crash-consistent
but not application-consistent snapshot.** Velero's backup hooks run arbitrary commands inside a container
immediately before and after the snapshot, giving the application a chance to reach a clean, quiescent state
first.

```mermaid
sequenceDiagram
    participant Velero
    participant Pod as checkout-orders-db pod
    participant Storage as CSI storage layer

    Velero->>Pod: exec pre-hook:<br/>psql -c "SELECT pg_backup_start()"
    Note over Pod: Postgres flushes WAL,<br/>enters backup-safe mode
    Velero->>Storage: Trigger CSI VolumeSnapshot
    Storage-->>Velero: Snapshot complete
    Velero->>Pod: exec post-hook:<br/>psql -c "SELECT pg_backup_stop()"
    Note over Pod: Postgres exits backup mode,<br/>resumes normal writes
```

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: checkout-orders-db-0
  namespace: checkout
  annotations:
    pre.hook.backup.velero.io/command: '["/bin/bash", "-c", "psql -U postgres -c \"SELECT pg_backup_start(''velero'')\""]'
    pre.hook.backup.velero.io/timeout: 30s
    post.hook.backup.velero.io/command: '["/bin/bash", "-c", "psql -U postgres -c \"SELECT pg_backup_stop()\""]'
```

> [!WARNING]
> A snapshot taken without hooks against a live, actively-written database isn't necessarily unusable —
> most modern databases (Postgres included) can recover from a crash-consistent snapshot the same way they'd
> recover from an actual power loss, replaying their write-ahead log on startup. But that recovery path is
> slower, less predictable, and in rare cases can surface corruption the WAL replay can't fully repair.
> **Treat hooks as mandatory for any stateful workload where recovery time and certainty both matter** —
> which, for `checkout-orders-db` specifically, is every workload in this chapter's throughline.

## Backup Scheduling and Retention Policies

**A one-off `Backup` object is a manual snapshot; a `Schedule` object is what actually gets a team a durable,
ongoing recovery posture — generating a fresh `Backup` on a cron cadence with its own retention (`ttl`).**

```yaml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: checkout-daily
  namespace: velero
spec:
  schedule: "0 3 * * *"          # 03:00 UTC daily
  template:
    includedNamespaces: ["checkout"]
    snapshotVolumes: true
    storageLocation: default
    ttl: 720h0m0s                # 30 days
```

| Tier | Cadence | Typical `ttl` | Fits |
|---|---|---|---|
| Hourly | Every hour | 48h | High-write-rate, low-tolerance-for-loss data (`checkout-orders-db`) |
| Daily | Once daily | 30 days | Most application namespaces (`catalog`, `inventory`) |
| Weekly | Once weekly | 90 days | Slower-changing reference data, compliance-driven retention |
| Monthly (long-term) | Once monthly | 1-7 years | Regulatory/audit retention requirements, independent of operational recovery needs |

> **From the Trenches:** A team set every namespace's `Schedule` to a uniform 7-day `ttl` "to keep storage
> costs predictable," including `checkout-orders-db`. A billing dispute six weeks later needed the order
> state from 40 days prior — long past every retention window, with the underlying CSI snapshots already
> expired and reclaimed by the storage provider. The immediate cause was a retention policy set once and
> never revisited per-workload; the underlying condition was treating "backup retention" as a single global
> knob instead of a decision that should map to each workload's actual recovery and compliance requirements,
> the same tiering shown in the table above.

## The Backup Object Lifecycle

**A `Backup` object moves through a defined set of phases from creation to eventual expiry — worth knowing
precisely, since "why is my backup stuck" is a genuinely common on-call question this maps directly to.**

```mermaid
stateDiagram-v2
    [*] --> New
    New --> InProgress: Velero controller<br/>picks it up
    InProgress --> Completed: All objects and<br/>volumes backed up<br/>successfully
    InProgress --> PartiallyFailed: Some items failed,<br/>backup still usable
    InProgress --> Failed: Backup could not<br/>complete at all
    Completed --> Deleting: ttl expired, or<br/>manual deletion requested
    PartiallyFailed --> Deleting: ttl expired
    Deleting --> [*]

    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c
    classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c
    classDef muted fill:#eaeef1,stroke:#c3ccd4,color:#10161c
    class Completed ok
    class PartiallyFailed warn
    class Failed crit
    class Deleting,New muted
```

*A `PartiallyFailed` backup is genuinely usable for restore in most cases — it's worth checking exactly
which items failed (`velero backup describe --details`) rather than treating the whole run as worthless.*

> [!TIP]
> Alert specifically on `Failed` and `PartiallyFailed` backups, not just on "did the `Schedule` fire" —
> a `Schedule` firing successfully and creating a `Backup` object says nothing about whether that backup
> actually completed successfully. A surprising number of real backup gaps are discovered only when a
> restore is attempted, precisely because nobody was alerting on backup *outcome*, only backup *occurrence*.

## Monitoring and Alerting for Backup Health

**The previous section's tip — alert on backup outcome, not just occurrence — needs real metrics and
alerting rules behind it, since "check `velero backup get` manually every morning" doesn't scale past a
handful of namespaces and reliably gets skipped the first busy week.**

Velero exposes its own Prometheus metrics endpoint, the same observability pattern Part 10 established
generally for scraping any in-cluster component's `/metrics` endpoint.

| Metric | What it reveals |
|---|---|
| `velero_backup_success_total` / `velero_backup_failure_total` | Raw success/failure counts, per schedule — the base signal for any alerting rule |
| `velero_backup_duration_seconds` | Backup duration trend — the same metric the file-system-backup "From the Trenches" callout earlier in this chapter should have been alerting on as it crept from minutes to hours |
| `velero_backup_partial_failure_total` | Backups that completed but with some items failing — easy to miss if only watching for hard failures |
| `velero_backup_last_successful_timestamp` | Age of the most recent successful backup per schedule — the single most important metric for catching a schedule that's silently stopped producing usable backups |

```yaml
# PrometheusRule — alert if checkout's backup schedule hasn't produced
# a successful backup in over 26 hours (a daily schedule plus margin)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: velero-backup-health
  namespace: velero
spec:
  groups:
    - name: velero-backup-health
      rules:
        - alert: BackupStale
          expr: (time() - velero_backup_last_successful_timestamp{schedule="checkout-daily"}) > 93600
          for: 15m
          labels: { severity: critical }
          annotations:
            summary: "checkout-daily has not produced a successful backup in over 26 hours"
        - alert: BackupPartialFailure
          expr: increase(velero_backup_partial_failure_total[1h]) > 0
          labels: { severity: warning }
          annotations:
            summary: "A recent backup completed with partial failures — check velero backup describe --details"
```

> [!TIP]
> `velero_backup_last_successful_timestamp` staleness is the single highest-value alert in this table to
> get right first — it catches the exact class of silent failure the earlier DR-drill "From the Trenches"
> callout describes (a hook quietly failing for months while the `Schedule` itself kept firing and producing
> `Backup` objects that looked superficially fine). A team with only one Velero alert configured should make
> it this one.

## Consistency Across Namespaces — the Multi-Service Backup Problem

**`checkout-service`, `catalog-service`, and `inventory-service` each live in their own namespace with their
own independently-scheduled `Backup`, and that independence — genuinely useful for per-team ownership and
retention tuning — creates a real, easy-to-miss gap: a restore spanning all three namespaces can bring each
one back from a slightly different point in time, not one shared, mutually consistent moment.**

```mermaid
sequenceDiagram
    participant CheckoutB as checkout Backup
    participant CatalogB as catalog Backup
    participant InventoryB as inventory Backup

    Note over CheckoutB: Backup taken 03:00
    Note over CatalogB: Backup taken 03:15
    Note over InventoryB: Backup taken 02:45

    Note over CheckoutB,InventoryB: An order placed at 02:50, referencing a<br/>catalog item added at 03:05, restores into<br/>a state where checkout remembers the order<br/>but catalog has never heard of that item yet
```

*Each individual namespace's backup is internally valid — the inconsistency only exists at the boundary
**between** namespaces, which is exactly why it's easy to miss when reviewing backups one namespace at a
time.*

| Approach | How it addresses cross-namespace consistency |
|---|---|
| A single `Backup`/`Schedule` including all related namespaces together (`includedNamespaces: [checkout, catalog, inventory]`) | All three snapshot at the same instant — the simplest fix, at the cost of coupling their backup cadence and retention together |
| Independent per-namespace backups, restored together deliberately with awareness of the time gap | Preserves per-team backup ownership, but requires the restoring operator to actively reason about and communicate the consistency gap, rather than assuming a clean restore |
| Event-sourced reconciliation after restore (e.g., replaying an order/inventory event log to re-derive consistent state) | Addresses the gap after the fact rather than preventing it — genuinely useful for `inventory-service`'s specific event-driven data model, not a general substitute for the first two options |

> [!TIP]
> Group namespaces into a single `Backup`/`Schedule` whenever they represent one transactionally-related
> business domain — `checkout`, `catalog`, and `inventory` together form one meaningful "can this whole
> flow be trusted after a restore" boundary, even though they're separate namespaces for RBAC and
> deployment-ownership reasons covered back in Part 13. Namespace boundaries drawn for multi-tenancy and
> ownership reasons don't automatically align with the boundaries that matter for backup consistency, and
> conflating the two is the root of this section's whole problem.

> **From the Trenches:** A regional-failover drill restored `checkout` and `catalog` from backups taken 20
> minutes apart, and the resulting standby environment showed checkout-service order confirmations
> referencing catalog SKUs that, in the restored catalog namespace, didn't exist yet — they'd been added in
> the 20-minute gap. The immediate cause was two independently-scheduled backups with no coordination; the
> underlying condition was that nobody had ever explicitly decided whether `checkout` and `catalog`
> constituted one consistency domain or two, so each team's backup schedule was tuned purely for its own
> namespace's convenience with zero awareness of the cross-namespace assumption the application itself
> depends on.

## Restoring Into the Same Cluster — Namespace Mapping and Resource Policies

**The simplest restore case — recovering a namespace back into the same cluster it was backed up from — is
also where the resource-conflict behavior matters most, since some of the namespace's objects may still
exist.**

```yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: catalog-restore-2026-08-27
  namespace: velero
spec:
  backupName: catalog-daily-20260827030000
  includedNamespaces: ["catalog"]
  existingResourcePolicy: update    # update objects that already exist, don't skip them
```

| `existingResourcePolicy` | Behavior when an object already exists in the target |
|---|---|
| (unset, default) | Skip the object entirely — the pre-existing version wins |
| `update` | Overwrite the existing object's spec with the backed-up version |

**Namespace mapping** lets a restore land in a *different* namespace name than the one it was backed up
from — genuinely useful for restoring a production backup into a staging namespace for validation without
touching production at all:

```yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: catalog-restore-to-staging
  namespace: velero
spec:
  backupName: catalog-daily-20260827030000
  includedNamespaces: ["catalog"]
  namespaceMapping:
    catalog: catalog-staging-restore-test
```

## Cross-Cluster and Cross-Region Restore Patterns

**Because every `Backup`'s manifests and volume snapshot data live in ordinary object storage rather than
inside the originating cluster, a completely separate cluster — in a different region, a different cloud
account, or even a different provider for file-system-backed volumes — can restore from that same bucket
with zero coordination from the original cluster at all.**

```mermaid
flowchart TD
    subgraph RegionA["Region A — primary"]
        ClusterA["Primary cluster"] -->|"scheduled backups"| Bucket
    end
    Bucket[["Shared S3 bucket<br/>platform-velero-backups"]]
    subgraph RegionB["Region B — standby"]
        ClusterB["Standby cluster<br/>(same BackupStorageLocation<br/>config, different cluster)"]
    end
    Bucket -->|"velero restore create<br/>--from-backup"| ClusterB

    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef accent fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    class ClusterA info
    class Bucket accent
    class ClusterB ok
```

*The standby cluster in Region B needs nothing from Region A beyond read access to the same bucket — this
is the property that makes Velero a genuine regional-failover tool, not just a same-cluster undo button.*

> [!IMPORTANT]
> Cross-cluster restore only carries what Velero actually captured — cluster-scoped, provider-specific
> resources it doesn't manage (a `StorageClass` referencing a region-specific CSI driver parameter, an
> `IngressClass` tied to a load balancer controller only installed in one region) need to already exist,
> correctly configured, in the target cluster *before* the restore runs, or the restored namespace's objects
> will reference infrastructure that isn't there. A standby cluster's baseline configuration (StorageClasses,
> CRDs, cluster-wide policy) should be provisioned and kept in sync independently of Velero — typically via
> the same GitOps/IaC pipeline that provisions the primary, not as something Velero is expected to recreate.

## The GitOps Interaction Problem — Suspending Reconciliation During Restore

**A cluster running Argo CD or Flux has a second controller that also reconciles the exact objects Velero is
trying to restore — and if that controller is still running during a restore, it can undo the restore within
seconds of it completing.**

```mermaid
sequenceDiagram
    participant Velero
    participant API as API Server
    participant Argo as Argo CD (still reconciling)

    Velero->>API: Restore creates catalog-service<br/>Deployment (old, backed-up spec)
    Argo->>API: Notices catalog-service drifted<br/>from Git's current desired state
    Argo->>API: Reconciles — overwrites the<br/>restored Deployment back to<br/>whatever Git currently says
    Note over API: The restore appeared to<br/>succeed, then silently<br/>reverted moments later
```

*This isn't a Velero bug — Argo CD and Velero are both, correctly, doing exactly what they're each designed
to do; the conflict only exists because nobody told one of them to stand down first.*

The fix is explicit: suspend GitOps reconciliation for the affected scope before restoring, and resume it
only after confirming the restored state is actually what you want kept.

```bash
# Argo CD — skip reconciliation for the affected Application during restore
kubectl annotate application catalog-service -n argocd \
  argocd.argoproj.io/skip-reconcile="true" --overwrite

# Flux — suspend the Kustomization/HelmRelease covering the restored namespace
flux suspend kustomization catalog-service

# ... perform the Velero restore, verify it, then resume:
flux resume kustomization catalog-service
```

> [!CAUTION]
> Deciding what "the restored state should stay" actually means requires real judgment, not just resuming
> GitOps blindly the moment the restore command exits successfully — if Git's current desired state is
> actually correct and the restore was only meant to recover *data* (the order rows in
> `checkout-orders-db`), resuming reconciliation immediately is right. If the restore is also meant to roll
> back a recent, bad deployment, resuming GitOps immediately will silently re-apply the very change you just
> rolled back. Know which kind of restore you're doing before touching the GitOps suspend/resume switch.

## RTO and RPO for Kubernetes Workloads — Turning SLAs into Backup Design

**Recovery Time Objective (RTO) and Recovery Point Objective (RPO) are the two numbers that should actually
drive every backup-design decision in this chapter, rather than backup cadence being chosen arbitrarily.**
RTO is how long a workload can be down before the business impact is unacceptable; RPO is how much recent
data loss is acceptable if a disaster hits between backups.

| Service | RPO target | RTO target | What that implies for backup design |
|---|---|---|---|
| `checkout-orders-db` | 5 minutes | 15 minutes | Continuous WAL streaming to a standby, not just periodic snapshots — a daily Velero backup alone can't hit a 5-minute RPO |
| `catalog-service` | 1 hour | 30 minutes | Hourly Velero `Schedule` with CSI snapshots; catalog data changes slowly enough that hourly loss is tolerable |
| `inventory-service` | 15 minutes | 30 minutes | Sub-hourly Velero schedule plus event-sourced inventory adjustments replayed from a message log as a secondary safety net |
| `recommendations` (Part 14) | 24 hours | 4 hours | Daily backup is sufficient — a stale recommendation model is a quality issue, not a data-loss incident |

**The genuinely important point this table makes concrete: Velero's own backup cadence is only *one* lever,
and for the tightest RPO targets it isn't sufficient on its own.** A 5-minute RPO for `checkout-orders-db`
isn't achievable by scheduling Velero backups every 5 minutes (the operational overhead and snapshot churn
would be substantial) — it's achieved by pairing infrequent Velero backups (for full-namespace disaster
recovery) with the database's own continuous replication mechanism (streaming replication to a standby,
independent of Kubernetes entirely) for the tight-RPO data-loss case specifically.

> [!TIP]
> Assign an explicit RTO/RPO pair to every namespace *before* designing its backup schedule, not after —
> working backward from "what backup cadence feels reasonable" to an RPO number is how teams end up
> discovering their actual recovery point was much worse than stakeholders assumed, during an incident
> rather than during planning.

## Disaster Recovery Drills — Practicing Restore Before You Need It

**Part 4's etcd drill established the core principle for the control-plane layer — a backup that's never
been test-restored is a hypothesis, not a verified recovery capability — and it applies with equal force
here, arguably more so given how many more moving parts a workload restore has (CSI drivers, hooks,
GitOps suspension, namespace mapping) than a single `etcdctl snapshot restore` command.**

```markdown
Quarterly DR drill checklist:
- [ ] Pick a real backup from the last 24 hours, not a specially-prepared "known good" one
- [ ] Restore into an isolated drill namespace or drill cluster, never production
- [ ] Suspend GitOps reconciliation for the drill scope before restoring
- [ ] Time the actual restore, end to end, against the workload's stated RTO
- [ ] Verify application health post-restore, not just "the objects exist"
- [ ] Verify data integrity specifically — row counts, a checksum, a known test record
- [ ] Document any gap between the drill's actual RTO and the target, and file a follow-up
- [ ] Rotate which namespace/workload is drilled each quarter — don't always drill the easy one
```

> **From the Trenches:** A platform team ran quarterly DR drills faithfully for two years, always restoring
> the `catalog` namespace because it was fast and reliable — a comfortable, low-risk drill to report as
> "green" every quarter. When `checkout-orders-db` actually needed restoring after a storage-layer incident,
> the pre-backup hook's `pg_backup_start()` command had been silently failing for months due to an
> unrelated credentials rotation, meaning every recent backup was crash-consistent only, not the
> application-consistent backup the team believed they had. The immediate cause was a hook failure with no
> alerting on hook success/failure specifically; the underlying condition was a DR drill program that
> exercised the same easy, well-behaved workload repeatedly instead of rotating through every workload's
> real recovery path, which is exactly the rotation the checklist above now enforces.

**The verification step deserves automation, not just a checklist item** — the quarterly drill worked
scenario later in this chapter finds exactly this gap in practice: a fast, well-within-RTO restore followed
by a slow, entirely manual verification pass. A small verification script, run automatically as the last
step of every drill (and, ideally, wired into the drill's own CI pipeline rather than run by hand), closes
that gap directly:

```bash
#!/usr/bin/env bash
# post-restore-verify.sh — run immediately after a drill restore completes
set -euo pipefail

NAMESPACE="$1"
EXPECTED_CHECKSUM_FILE="$2"

kubectl wait --for=condition=Ready pod -l app="${NAMESPACE}-service" -n "$NAMESPACE" --timeout=120s

# Compare a known, pre-drill data checksum against the restored state
kubectl exec -n "$NAMESPACE" deploy/"${NAMESPACE}-service" -- \
  ./scripts/checksum-data.sh > /tmp/restored-checksum.txt

if diff -q "$EXPECTED_CHECKSUM_FILE" /tmp/restored-checksum.txt > /dev/null; then
  echo "PASS: restored data matches expected checksum"
else
  echo "FAIL: restored data diverges from expected checksum — investigate before closing the drill"
  exit 1
fi
```

> [!TIP]
> Wire the drill's timing and verification result into the same fleet observability pipeline used for
> production backup alerting (the previous "Monitoring and Alerting" section) rather than a separate,
> manually-maintained spreadsheet of drill results — a drill history that lives in the same dashboard as
> real backup health makes a slipping RTO trend visible over time instead of rediscovered fresh each quarter.

## Securing Backup Data — Encryption, Access, and Secrets

**A Velero backup bucket is, by construction, a complete off-cluster copy of a namespace's `Secret` objects
along with everything else — anyone with read access to that bucket has read access to every credential
those Secrets hold, independent of the in-cluster RBAC controls Part 11 covered in depth.**

| Control | Why it matters here specifically |
|---|---|
| Enable server-side encryption on the backup bucket (SSE-S3/SSE-KMS or the cloud-equivalent) | The bucket is a durable, long-lived copy of every Secret in every backed-up namespace — encryption at rest is not optional for this data class |
| Restrict bucket IAM/access policy to only the Velero service identity and named break-glass operators | Backup buckets are a common, under-scrutinized path to bulk credential exposure — they don't get the same RBAC review as in-cluster access |
| Exclude highly sensitive namespaces from a shared BSL, or use a separate BSL with tighter access | Not every team's backups need to be readable by the same set of platform operators |
| Rotate any credential that was present in a Secret at backup time, after any suspected backup-bucket exposure | A leaked backup is functionally equivalent to a leaked Secret for every credential it contains, at the point in time it was taken |

> [!WARNING]
> A backup bucket is frequently excluded from the security review process applied to the cluster itself,
> precisely because it "isn't the cluster" — but it holds the exact same Secret data, often for a longer
> retention window than the Secrets ever exist for inside the cluster. Treat backup-bucket access control
> with the same rigor Part 11 applied to in-cluster Secret RBAC, not as an afterthought bolted on once
> backups already exist.

## Backup Storage Cost and Lifecycle Management

**Backup storage cost is easy to ignore while a namespace is small and expensive to discover only once a
fleet-wide bill review flags it — the same growth pattern that quietly turned this chapter's file-system
backup callout into a three-hour job applies just as much to storage spend as to backup duration.**

CSI snapshots for block storage are typically billed incrementally (only the changed blocks since the
previous snapshot), which keeps steady-state cost far lower than the full volume size might suggest — but
that economy disappears if retention (`ttl`) is set generously "just in case" across every tier, since every
retained snapshot in the chain has to be kept until nothing downstream still depends on it.

```yaml
# S3 lifecycle policy on the Velero backup bucket — move backups older
# than 30 days to cheaper, slower-retrieval storage, and expire outright
# past the compliance-driven retention ceiling
{
  "Rules": [
    {
      "ID": "velero-backup-tiering",
      "Status": "Enabled",
      "Filter": { "Prefix": "prod-cluster/backups/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 2555 }
    }
  ]
}
```

| Lever | Effect |
|---|---|
| Tiering `ttl` per workload (this chapter's earlier table) | The single biggest cost lever — a namespace defaulted to a long `ttl` "for safety" pays for retention it likely never needed |
| Object storage lifecycle transitions (Standard → Infrequent Access → Glacier/cold) | Cuts long-term retention cost for backups kept mainly for compliance, rarely for actual restore |
| Incremental CSI snapshot billing | Already the storage provider's default behavior for most block-storage snapshot mechanisms — no action needed, but worth confirming for a less common storage backend |
| Deleting orphaned `Backup` objects for decommissioned namespaces | A namespace deleted from the cluster doesn't automatically stop its historical backups from continuing to consume storage until their own `ttl` expires |

> [!WARNING]
> Setting `Expiration` in an object-storage lifecycle policy independently of Velero's own `ttl` on the
> `Backup` object can create a dangerous mismatch: if the bucket lifecycle policy deletes the underlying
> data *before* Velero's own `ttl` expires, `velero backup get` can still list a `Backup` as valid and
> `Completed` long after its actual restorable data is already gone from the bucket. Keep the two retention
> mechanisms in sync deliberately, or manage retention through Velero's own `ttl` exclusively and leave the
> bucket lifecycle policy to handle only the storage-class tiering transitions, not expiration.

## Multi-Cluster Fleet Backup Strategy

**Part 13 covered fleet management as the escape-hatch beyond single-cluster multi-tenancy — backup strategy
needs the same fleet-level thinking, since a per-cluster, independently-configured Velero installation
across a growing fleet quietly becomes as hard to audit as the per-cluster policy drift Part 13 warned
about generally.**

| Fleet-level concern | Approach |
|---|---|
| Consistent BSL/VSL configuration across every cluster | Provision Velero itself via the same GitOps/Helm pipeline used for every other fleet-wide add-on, not a per-cluster manual install |
| Knowing which cluster owns which backup | A distinct bucket prefix per cluster (as shown in this chapter's `BackupStorageLocation` example) — never a shared, unprefixed bucket across the fleet |
| Auditing that every cluster's schedules are actually running | A fleet-wide dashboard aggregating `Backup` object status across clusters (via a fleet observability tool, Part 13's multi-tenant observability pattern extended fleet-wide) rather than checking each cluster individually |
| Standby-cluster capacity for the cross-region pattern | Provisioned and kept warm (or at minimum, provisionable within the target RTO) independently of any single primary cluster's health |

```yaml
# Helm values shared across every cluster's Velero install via the
# fleet's GitOps pipeline — the per-cluster bucket prefix is the
# ONLY value that should ever differ between clusters
configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: platform-velero-backups
      config:
        region: us-east-1
      # prefix is templated per-cluster by the GitOps pipeline,
      # e.g. "prod-us-east-1", "prod-eu-west-1", "staging-us-east-1"
      prefix: "{{ .Values.clusterName }}"
schedules:
  checkout-daily:
    schedule: "0 3 * * *"
    template:
      ttl: "720h"
      includedNamespaces: ["checkout"]
```

**Templating everything except the bucket prefix through one shared Helm release, deployed identically to
every cluster in the fleet, is what actually prevents the "which cluster's Velero config is the odd one
out" question from ever needing to be asked** — the same principle Part 13 applied to policy-engine
configuration generally, applied here specifically to backup infrastructure.

## A Full Worked Scenario: Recovering `checkout` From an Accidental Namespace Deletion

**An engineer, intending to delete a long-abandoned `checkout-canary-test` namespace, runs
`kubectl delete namespace checkout` instead — every Deployment, Service, ConfigMap, and PVC in the real
production `checkout` namespace begins terminating immediately.**

1. **Stop the bleeding first.** Confirm no automation (a CI pipeline, a GitOps controller) will recreate the
   namespace in a half-restored state while the real restore is being prepared — suspend the relevant
   GitOps `Application`/`Kustomization` per the earlier section, immediately.
2. **Identify the most recent good `Backup`.** `velero backup get` confirms the last scheduled
   `checkout-daily` backup completed successfully roughly 90 minutes before the incident.
3. **Restore into the same cluster, same namespace name**, since the namespace itself no longer exists to
   conflict with:
   ```bash
   velero restore create checkout-recovery-20260827 \
     --from-backup checkout-daily-20260827030000 \
     --include-namespaces checkout
   ```
4. **Verify PVC data specifically**, not just object existence — `checkout-orders-db`'s CSI-snapshot-restored
   volume needs Postgres to actually start and serve queries cleanly, confirmed against the pre-backup hook's
   consistency guarantee from earlier in this chapter.
5. **Accept the RPO gap explicitly.** Any order placed in the roughly 90 minutes between the last backup and
   the deletion is genuinely lost from this restore path alone — this is exactly why `checkout-orders-db`'s
   RTO/RPO table earlier in this chapter called for continuous database replication as a *second*, faster
   recovery layer alongside Velero, not Velero backup cadence alone.
6. **Resume GitOps reconciliation** only after confirming the restored `checkout` namespace matches the
   intended current desired state, not a stale one.

## A Full Worked Scenario: Regional Failover for `catalog-service`

**`us-east-1` suffers a multi-hour regional outage affecting the primary cluster entirely — `catalog-service`
needs to come back up in `us-west-2` on the standby cluster provisioned per this chapter's cross-region
pattern.**

```mermaid
flowchart TD
    S1["1. Declare regional<br/>failover — primary<br/>cluster confirmed<br/>unreachable"] --> S2["2. Confirm standby cluster's<br/>baseline (StorageClasses,<br/>CRDs, IngressClass) is<br/>already provisioned<br/>and current"]
    S2 --> S3["3. velero restore create<br/>--from-backup, against<br/>the shared BackupStorageLocation"]
    S3 --> S4["4. Verify catalog-service<br/>pods healthy and PVC<br/>data intact"]
    S4 --> S5["5. Re-point DNS/traffic<br/>(Gateway API, Part 8/9)<br/>to the standby cluster"]
    S5 --> S6["6. Accept catalog data loss<br/>up to the last completed<br/>backup's RPO window"]
    S6 --> S7["7. When us-east-1 recovers,<br/>plan a deliberate<br/>fail-back — never an<br/>automatic one"]

    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c
    class S1,S7 warn
    class S2,S3,S4,S5 info
    class S6 ok
```

**Step 7 deserves its own emphasis: fail-back is not simply "run the same restore in reverse" once the
primary region recovers.** By the time `us-east-1` is healthy again, the standby cluster in `us-west-2` has
been serving live production traffic and has genuinely newer data than the last backup restored into it —
failing back naively would discard everything written during the failover window. A real fail-back requires
either a fresh backup taken *from the standby* and restored into the recovered primary, or a period of
bidirectional data reconciliation specific to `catalog-service`'s own data model — this is a deliberate,
planned operation, never an automated reversal of the failover steps above.

## A Full Worked Scenario: Running a Quarterly Disaster Recovery Drill

**Bringing this chapter's full checklist together for a real quarterly drill against `inventory-service`,
rotating away from the easier `catalog` namespace per the earlier "From the Trenches" lesson.**

```mermaid
flowchart TD
    D1["1. Select inventory namespace<br/>for this quarter's drill<br/>(rotation, not the usual pick)"] --> D2["2. Pick the most recent<br/>real scheduled backup —<br/>not a hand-crafted one"]
    D2 --> D3["3. Provision an isolated<br/>drill namespace/cluster,<br/>never touching production"]
    D3 --> D4["4. Suspend GitOps for<br/>the drill scope"]
    D4 --> D5["5. Restore, timing the<br/>full operation against<br/>inventory-service's<br/>30-minute RTO target"]
    D5 --> D6["6. Verify inventory counts<br/>against a known checksum<br/>taken before the drill"]
    D6 --> D7["7. Document actual RTO vs<br/>target, file a follow-up<br/>for any gap found"]
    D7 --> D8["8. Tear down the drill<br/>environment, confirm no<br/>drill artifacts leak into<br/>production monitoring"]

    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef accent fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    class D1,D8 accent
    class D2,D3,D4 info
    class D5,D6,D7 ok
```

**The drill surfaces a real finding worth narrating**: the restore itself completes in 9 minutes, comfortably
inside the 30-minute RTO, but verifying inventory counts against the pre-drill checksum takes another 40
minutes because no automated verification script existed — someone had to manually query and compare counts
across a dozen product categories. The actual finding isn't "the restore is too slow," it's "the *verification
step* has no tooling," which becomes this quarter's concrete follow-up: build an automated post-restore
verification script before the next drill, rather than re-discovering the same manual bottleneck every
quarter.

## Part 18 CLI Cheat Sheet

| Command | Purpose |
|---|---|
| `velero backup create <name> --include-namespaces <ns>` | Trigger an ad hoc, one-off backup |
| `velero backup get` | List backups and their phase (`Completed`, `PartiallyFailed`, `Failed`) |
| `velero backup describe <name> --details` | See exactly which items succeeded/failed in a backup |
| `velero backup logs <name>` | Full backup operation log, for diagnosing a `Failed`/`PartiallyFailed` run |
| `velero restore create --from-backup <backup-name>` | Restore from a specific backup |
| `velero restore describe <name> --details` | Diagnose a partial or failed restore |
| `velero schedule get` | List all configured `Schedule` objects and their cron expressions |
| `kubectl get volumesnapshot -A` | Inspect CSI VolumeSnapshot objects Velero created directly |
| `flux suspend kustomization <name>` / `flux resume kustomization <name>` | Suspend/resume Flux reconciliation around a restore |

## Common Mistakes and Interview Traps

| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Treating etcd backup as sufficient disaster recovery | etcd backup restores object definitions only, never PersistentVolume data | Layer Velero (or an equivalent) for workload/PV-level recovery, especially on managed Kubernetes where etcd access doesn't exist at all |
| Backing up a live database with no pre/post hooks | A CSI snapshot alone may be only crash-consistent, not application-consistent | Use backup hooks to quiesce the application before the snapshot and resume it after |
| Setting one uniform retention (`ttl`) across every namespace | Different workloads have genuinely different RPO/compliance retention needs | Tier retention per workload, per this chapter's tiering table, driven by real RTO/RPO targets |
| Restoring into a cluster with GitOps still actively reconciling | The GitOps controller can silently revert the restore within seconds | Suspend GitOps reconciliation for the affected scope before restoring, resume deliberately after |
| Assuming a `Schedule` firing means backups are succeeding | A `Schedule` creating `Backup` objects says nothing about whether those backups actually complete | Alert on `Backup` phase (`Failed`/`PartiallyFailed`) specifically, not just schedule execution |
| Drilling the same easy namespace every quarter | Doesn't exercise the workloads with the riskiest, least-tested recovery paths | Rotate which namespace/workload is drilled, and verify data integrity, not just object existence |
| Treating a fail-back as the failover steps run in reverse | Data written to the standby during failover is real and would be discarded by a naive reversal | Treat fail-back as its own deliberate operation requiring fresh backup/reconciliation from the standby |

## Worked Practice Problems

**Problem 1**: A team's etcd backups (Part 4) are healthy and regularly tested, but a storage-layer incident
destroys the underlying disk backing `checkout-orders-db`'s PVC. After restoring etcd from the most recent
snapshot, the `PersistentVolumeClaim` object exists again but the application still can't read any order
history. What's actually missing, and why didn't the etcd restore fix it?

*Answer*: etcd only stores API object definitions — restoring it recreates the `PersistentVolumeClaim`
object's YAML exactly as it was, but that object is just a pointer to a physical volume, and the physical
volume's actual data was destroyed, not the pointer to it. Recovering the real order data requires a
workload-level backup tool (Velero, with CSI snapshots or file-system backup covering that PVC) taken
*before* the incident — etcd backup and workload backup protect two structurally different things, and
having one healthy says nothing about the other.

**Problem 2**: A `Backup` for the `checkout` namespace shows `Completed`, but three months later a restore
from it produces a `checkout-orders-db` that fails to start Postgres cleanly, with corruption errors in the
database log. The backup hooks were configured correctly. What's the most likely explanation?

*Answer*: A `Completed` phase for the overall `Backup` object confirms Velero successfully captured and
uploaded everything it was told to capture — it does not independently verify the *data itself* is
internally consistent at the application level beyond what the pre/post hooks already ensured at backup
time. The most likely explanation is either a hook that appeared to succeed but didn't actually complete
its intended action (a `pg_backup_start()` call that silently failed, as in this chapter's DR-drill "From
the Trenches" callout) or an underlying storage-layer bug in the CSI snapshot itself — either way, this is
precisely the gap a DR drill's data-integrity verification step (checksums, known test records) is designed
to catch, which a "backup completed" status alone cannot.

**Problem 3**: A platform team configures cross-region restore for `catalog-service`, tests it successfully
in a drill, and considers the DR posture complete. During a real regional outage, the restore into the
standby cluster fails because the standby's `StorageClass` references a CSI driver parameter specific to a
region that no longer matches. What should the team have caught, and where does the fix belong?

*Answer*: Velero only restores what it captured — application objects and PV data — it never provisions or
validates cluster-scoped infrastructure like `StorageClass` definitions, CRDs, or `IngressClass` objects
that a restored workload depends on. The standby cluster's baseline infrastructure needs to be provisioned
and kept in sync independently, typically through the same GitOps/IaC pipeline used for the primary cluster,
and validated as part of the DR drill itself — not assumed to already be correct because the Velero restore
step succeeded in isolation during an earlier, less complete test.

## Summary and What's Next

Workload-level backup and disaster recovery is a genuinely distinct discipline from the etcd backup Part 4
and Part 15 already covered — protecting PersistentVolume data and giving teams a real cross-cluster,
cross-region recovery path that etcd snapshots structurally cannot provide. Velero's CSI snapshot and
file-system backup mechanisms, application-consistent hooks, and shared-object-storage architecture combine
to make that recovery path practical, but only when RTO/RPO targets are set deliberately per workload, when
GitOps reconciliation is suspended correctly during a restore, and when the recovery procedure is actually
drilled — rotating across real workloads, not just the easiest one — rather than assumed to work because a
backup shows `Completed`.

Part 19 turns to a different kind of infrastructure entirely: building the custom controllers and Operators
this series has referenced conceptually since Part 4, but now at the actual `client-go`/`controller-runtime`
mechanics level — informers, workqueues, reconciler code, CRD versioning in practice, and how to test a
controller before it ever reconciles a real cluster.