# Databases & Storage Reliability — Part 1: Replication & Failover > **Series:** Databases & Storage Reliability (1 of 8) > **Part 1:** This file — Replication & Failover > **Part 2:** `02-sharding-and-partitioning.md` — Sharding & Partitioning > **Part 3:** `03-backup-recovery-and-durability.md` — Backup, Recovery & Durability > **Part 4:** `04-sql-fundamentals.md` — SQL Language Fundamentals > **Part 5:** `05-nosql-database-types.md` — NoSQL Database Types > **Part 6:** `06-mysql-in-depth.md` — MySQL In Depth > **Part 7:** `07-dynamodb-in-depth.md` — DynamoDB In Depth > **Part 8:** `08-mongodb-in-depth.md` — MongoDB In Depth > **Questions:** `questions.md` ## Table of Contents 1. [Why Databases Get Their Own Deep Dive](#why-databases-get-their-own-deep-dive) 2. [What Replication Actually Is](#what-replication-actually-is) 3. [Synchronous vs Asynchronous Replication](#synchronous-vs-asynchronous-replication) 4. [Semi-Synchronous Replication — A Practical Middle Ground](#semi-synchronous-replication--a-practical-middle-ground) 5. [How Replication Actually Works Under the Hood](#how-replication-actually-works-under-the-hood) 6. [Replication Lag, Revisited](#replication-lag-revisited) 7. [Failover — What Happens When the Primary Dies](#failover--what-happens-when-the-primary-dies) 8. [Automatic vs Manual Failover](#automatic-vs-manual-failover) 9. [The Split-Brain Problem](#the-split-brain-problem) 10. [Leader Election and Consensus](#leader-election-and-consensus) 11. [A Full Worked Failover, Start to Finish](#a-full-worked-failover-start-to-finish) 12. [Multi-Region Replication](#multi-region-replication) 13. [Common Mistakes](#common-mistakes) 14. [Worked Practice Problems](#worked-practice-problems) 15. [Summary and What's Next](#summary-and-whats-next) --- ## Why Databases Get Their Own Deep Dive Databases are the single hardest thing to scale, replicate, and recover reliably in almost any system — every other tutorial in this course has treated them as "the stateful thing behind the app servers." This series goes inside that box. Everything here builds directly on ideas already introduced: the CAP theorem and consistency models (Reliability & Architecture Patterns), read replicas (Capacity Planning & Performance), and etcd's own use of consensus (Kubernetes Deep Dive) — this tutorial applies those same ideas specifically, deeply, to the database layer itself. --- ## What Replication Actually Is **Replication** means keeping a copy (or several copies) of a database's data on separate servers, kept continuously in sync with the original. ```mermaid graph LR Primary["Primary
(accepts writes)"] -->|"continuously streams
every change"| Replica1["Replica 1"] Primary -->|"continuously streams
every change"| Replica2["Replica 2"] ``` **Simple analogy:** think of a shared team document with "track changes" turned on, being continuously mirrored to several backup copies the instant any edit happens — if the original document is ever lost, one of those mirrors already has (almost) everything, ready to become the new original. **Why this matters, tying directly back to earlier tutorials:** replication is the foundation of two completely different things covered elsewhere in this course — **read scaling** (Capacity Planning & Performance, Part 1 — spreading read traffic across replicas) and **availability/disaster recovery** (this Part — having a ready backup if the primary dies). The same underlying mechanism serves both purposes. --- ## Synchronous vs Asynchronous Replication This is one of the single most important, most commonly tested database concepts — a direct, concrete application of the CAP theorem and PACELC tradeoffs from the Reliability & Architecture Patterns series. ```mermaid sequenceDiagram participant App participant Primary participant Replica Note over App,Replica: SYNCHRONOUS replication App->>Primary: Write X = 5 Primary->>Replica: Send the change Replica-->>Primary: Confirmed - I have it too Primary-->>App: OK (only AFTER replica confirms) ``` ```mermaid sequenceDiagram participant App participant Primary participant Replica Note over App,Replica: ASYNCHRONOUS replication App->>Primary: Write X = 5 Primary-->>App: OK (immediately - doesn't wait) Primary->>Replica: Send the change
(happens shortly after) ``` | | Synchronous | Asynchronous | |---|---|---| | When does the write "succeed"? | Only after the replica confirms it too | Immediately, before the replica even receives it | | Data loss risk if primary dies right after a write? | None — the replica already has it | Real — the replica might not have gotten it yet | | Write latency | Higher — waits for a round trip to the replica | Lower — doesn't wait at all | | Real-world use | Financial transactions, anything where losing even one write is unacceptable | The vast majority of general-purpose replication, including most read replicas | **This is exactly PACELC's "Else" branch (Latency vs. Consistency) from the Reliability & Architecture Patterns series, made completely concrete:** synchronous replication sacrifices latency to guarantee zero data loss; asynchronous replication sacrifices a small, real risk of data loss to get lower latency. **Neither choice is universally "correct" — it's a deliberate tradeoff, sized to how bad losing a recent write would actually be for that specific system.** --- ## Semi-Synchronous Replication — A Practical Middle Ground A genuinely useful, real-world compromise worth knowing by name. ```mermaid graph TD SemiSync["Semi-synchronous:
wait for confirmation from
AT LEAST ONE replica
(not ALL of them), then
consider the write
successful"] --> Why["Balances: SOME real
protection against data
loss (at least one copy
definitely has it) WITHOUT
paying the full latency
cost of waiting for EVERY
replica"] ``` **Why this is a strong, senior-level interview answer to "how would you configure replication for a payments database":** "I'd lean toward semi-synchronous replication — requiring confirmation from at least one replica before acknowledging a write gives real protection against losing a committed transaction if the primary suddenly dies, without paying the latency cost (and added fragility, if a replica becomes unreachable) of waiting for every single replica to confirm, which pure synchronous replication requires." --- ## How Replication Actually Works Under the Hood Almost every relational database implements replication using the same core mechanism: a **write-ahead log (WAL)** — every change is first written to a sequential log file *before* it's applied to the actual data, and that log is what gets streamed to replicas. ```mermaid flowchart TD Write["A write happens"] --> WAL["FIRST, recorded in the
Write-Ahead Log (WAL)
— a sequential, append-
only record of every
change"] WAL --> Apply["THEN applied to the
actual data files"] WAL --> Stream["The SAME WAL entries are
streamed to replicas,
which replay them in
the EXACT same order"] ``` ```bash # PostgreSQL: set up a replica by taking a base backup, then # streaming WAL changes continuously from that point forward pg_basebackup -h primary-host -D /var/lib/postgresql/data \ -U replicator -P --wal-method=stream # Check replication status/lag from the primary psql -c "SELECT client_addr, state, sent_lsn, replay_lsn FROM pg_stat_replication;" ``` **Why the WAL mechanism matters, tying directly to Part 3 of this series:** the exact same write-ahead log used for replication is *also* what makes point-in-time recovery possible (covered in Part 3) — a database can be restored to any specific moment by replaying WAL entries up to that exact point. Replication and recovery share the same underlying technical foundation. --- ## Replication Lag, Revisited Already introduced in the Capacity Planning & Performance series (Part 1) as the tradeoff behind read replicas — here's the deeper, operational picture. ```mermaid graph TD Lag["Replication lag: the delay
between a write landing on
the primary and that SAME
write becoming visible on
a replica"] --> Causes["Common causes: network
latency, replica falling
behind under heavy WRITE
volume, replica running
slower hardware, a long-
running query blocking
replay on the replica"] ``` ```bash # Check replication lag in seconds (PostgreSQL) psql -c "SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;" ``` **Why monitoring replication lag as a first-class metric matters, connecting directly to the Monitoring Methodologies and Observability series:** rising replication lag is a genuine **Saturation**-style leading indicator (USE method) — it tells you a replica is falling behind *before* it becomes so far behind that failing over to it would mean losing a meaningful, noticeable amount of recent data. A well-designed alert on replication lag (using the same burn-rate-style thinking from the Observability series) gives operators time to intervene before an unplanned failover forces the issue. --- ## Failover — What Happens When the Primary Dies **Failover** is the process of promoting a replica to become the new primary when the original primary becomes unavailable — directly extending the failover concepts from the Reliability & Architecture Patterns series (Part 1), applied specifically to databases. ```mermaid sequenceDiagram participant Monitor as Health Monitor participant Primary participant Replica Monitor->>Primary: Health check Primary--xMonitor: No response (down!) Monitor->>Monitor: Confirms via repeated
checks (avoid a false alarm
from one blip) Monitor->>Replica: Promote to become
the NEW primary Replica->>Replica: Stops replaying WAL
from the old primary,
starts accepting WRITES
directly Monitor->>Monitor: Updates routing/DNS/
Service (Kubernetes series)
to point at the NEW primary ``` **Why promotion isn't instant, and why this matters practically:** the replica needs to fully finish applying every WAL entry it's already received before it can safely start accepting new writes as a primary — if it started accepting writes while still catching up, new writes could be applied out of order relative to the replication stream it was still processing, risking data corruption. --- ## Automatic vs Manual Failover The exact same tradeoff pattern seen throughout this course, applied specifically to databases. ```mermaid graph TD Auto["AUTOMATIC failover:
a monitoring system
detects primary failure
and promotes a replica
WITHOUT human approval"] --> AutoNote["✅ Much FASTER recovery
(no waiting for a human
to wake up and act)
❌ Real risk of a FALSE
POSITIVE (a brief network
blip triggers an
unnecessary failover) —
and risk of split-brain
(below) if not implemented
carefully"] Manual["MANUAL failover: a human
confirms the primary is
truly down before
promoting a replica"] --> ManualNote["✅ Avoids false-positive
failovers entirely
❌ SLOWER recovery — bound
by how fast a human can
be paged and respond
(Incident Management
series)"] ``` **Real-world tools worth knowing by name, since this is a genuinely common, specific interview topic:** **Patroni** (a widely used automatic failover manager for PostgreSQL, built on top of a distributed consensus store like etcd or Consul) and **Orchestrator** (a similar, widely used tool for MySQL) — both exist specifically to implement automatic failover *safely*, using leader election (below) to avoid the split-brain problem that naive automatic failover risks. --- ## The Split-Brain Problem A genuinely important, high-stakes failure mode, and one of the most commonly tested "what could go wrong" database interview questions. ```mermaid graph TD A["A network partition
happens — the OLD primary
can't be reached by the
monitoring system, but is
actually STILL RUNNING and
STILL accepting writes
from clients on ITS side
of the partition"] --> B["The monitoring system,
believing the primary is
DEAD, promotes a REPLICA
to become a NEW primary"] B --> C["🚨 SPLIT-BRAIN: TWO
primaries now BOTH
accepting writes,
independently, with NO
way to reconcile them"] ``` **Why this is so dangerous, worth stating explicitly:** once two primaries have independently accepted different, conflicting writes, there is **no clean, automatic way to merge them back together** — whichever writes happened on the "losing" side during the split are typically lost or require painful, manual, application-specific reconciliation. This is precisely the CAP theorem's Consistency-vs-Availability dilemma from the Reliability & Architecture Patterns series, playing out in its most damaging, concrete real-world form. ```mermaid graph LR A["Prevention technique 1:
FENCING — actively cut off
the old primary's ability
to accept writes (e.g. via
STONITH - 'Shoot The Other
Node In The Head') BEFORE
promoting a new one"] --> Safe1["Guarantees only ONE
primary can ever accept
writes at a time"] B["Prevention technique 2:
Require a QUORUM (majority)
of monitoring/consensus
nodes to agree the primary
is truly down before ANY
promotion happens"] --> Safe2["A minority partition can
never unilaterally decide
to promote a new primary"] ``` --- ## Leader Election and Consensus This directly reuses the exact same Raft/quorum-based consensus concepts from etcd (Kubernetes Deep Dive series) — a genuinely important connection worth stating explicitly in an interview. ```mermaid graph TD Cluster["A cluster of monitoring/
consensus nodes (e.g.
Patroni instances, backed
by etcd)"] --> Quorum["Requires a MAJORITY to
agree before declaring the
primary dead AND before
confirming a new leader"] Quorum --> SameAsEtcd["EXACTLY the same quorum
principle as etcd's own
Raft consensus (Kubernetes
Deep Dive series, Part 4) —
a MINORITY partition can
never make a unilateral
decision"] ``` **A strong, senior-level interview line connecting multiple parts of this course together:** "Modern automatic database failover tools like Patroni don't invent their own ad hoc 'is the primary down' logic — they build on the exact same quorum-based consensus principle etcd itself uses for Kubernetes's own control plane. A minority of nodes, even if they genuinely believe the primary is unreachable, can never unilaterally promote a new leader — this is precisely what prevents split-brain." --- ## A Full Worked Failover, Start to Finish Tying every concept in this Part together into one complete, realistic story. ```mermaid sequenceDiagram participant Patroni as Patroni Cluster
(3 nodes, quorum-based) participant Primary as Old Primary participant Replica as Replica (candidate) participant DNS as Service/DNS Primary--xPatroni: Primary stops responding
to health checks Patroni->>Patroni: Requires MAJORITY of
Patroni nodes to agree
primary is truly down
(avoids false positive) Patroni->>Primary: Attempts to FENCE the
old primary (prevent it
from accepting writes,
even if it comes back) Patroni->>Replica: Confirms replica has
fully applied all
received WAL entries Patroni->>Replica: Promotes it to
new PRIMARY Patroni->>DNS: Updates routing to
point at the new primary Note over Primary,DNS: Total failover time:
typically seconds to
low tens of seconds,
with a SAFE, quorum-
based process throughout ``` --- ## Multi-Region Replication A brief but important extension — replicating not just across servers in one data center, but across entire geographic regions, directly connecting to the multi-region discussion in the Reliability & Architecture Patterns series (Part 1). ```mermaid graph TD US["US Primary"] -->|"Replication across a
LONG-DISTANCE network
link — real, meaningfully
higher latency"| EU["EU Replica"] ``` **Why synchronous replication becomes genuinely impractical across regions, worth stating explicitly:** the speed of light itself imposes a real, physical latency floor on any cross-continental network round trip (commonly 60-150ms+ depending on distance) — requiring every single write to wait for that full round trip before acknowledging would make the database's write latency unacceptably slow for most applications. **This is exactly why multi-region database replication is almost always asynchronous**, accepting some real risk of losing the most recent writes in a true regional disaster, in exchange for acceptable local write latency the rest of the time. --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Using pure asynchronous replication for a payments/financial database | Real risk of silently losing a committed transaction if the primary dies right after acknowledging it | Use synchronous or semi-synchronous replication for data where losing even one recent write is unacceptable | | Implementing automatic failover without a quorum-based consensus mechanism | Genuine risk of split-brain — a network partition can trigger two simultaneously-active primaries | Use a proven tool (Patroni, Orchestrator) built on quorum-based consensus, not ad hoc health-check logic | | Promoting a replica before confirming it's fully caught up on WAL replay | Risks data corruption from writes applied out of order | Always confirm full WAL replay completion before promotion | | No monitoring/alerting on replication lag | A replica can silently fall dangerously far behind, only discovered during an actual failover — the worst possible moment | Alert on replication lag as a first-class, Saturation-style leading indicator | | Assuming multi-region replication can be synchronous "for extra safety" | Physical network latency across regions makes synchronous replication impractically slow for most write-heavy applications | Use asynchronous replication across regions, and explicitly size the acceptable data-loss window this introduces | | Treating manual failover as always safer than automatic | Manual failover is bound by human response time (Incident Management series), which can mean much longer real downtime for a well-understood, common failure mode | Weigh the specific tradeoff — automatic failover, done safely with quorum-based consensus, is often the right choice for well-understood failure patterns | --- ## Worked Practice Problems **Problem 1:** A team configures pure asynchronous replication for their primary financial ledger database. During an incident, the primary crashes hard immediately after acknowledging a batch of customer transactions. What's the risk, and how would semi-synchronous replication have changed the outcome? *Answer:* With pure asynchronous replication, those acknowledged transactions may never have actually reached any replica before the crash — meaning they're genuinely, silently lost, even though the application received a successful confirmation for them. With semi-synchronous replication (requiring at least one replica to confirm before acknowledging), the same crash would still leave at least one replica with those transactions durably recorded, since the primary would never have acknowledged them to the application until that replica confirmed receipt — completely eliminating this specific class of silent data loss, at the cost of somewhat higher write latency. **Problem 2:** A monitoring system, running as a single instance with no quorum mechanism, detects the primary database is unreachable (due to a brief network partition, not an actual crash) and immediately promotes a replica. Fifteen seconds later, the network partition heals and the old primary — which had continued accepting writes the whole time — is reachable again. What's the resulting failure mode, and what should have been different about the failover design? *Answer:* This is a textbook split-brain scenario — for those 15 seconds, both the old primary (still genuinely running and accepting writes on its side of the partition) and the newly-promoted replica were simultaneously acting as independent primaries, accepting potentially conflicting writes with no way to reconcile them afterward. The failover design should have used a quorum-based consensus mechanism (like Patroni backed by etcd) requiring a majority of monitoring nodes to agree the primary was truly down, and should have actively fenced the old primary (preventing it from accepting further writes) as part of the promotion process, rather than a single monitoring instance making a unilateral decision based on its own, possibly-partitioned view of the world. **Problem 3:** A company wants to replicate their primary US database to a new EU region for disaster recovery. An engineer proposes synchronous replication "to guarantee zero data loss." What would you push back on, and what would you recommend instead? *Answer:* Synchronous replication across a US-to-EU link would require every single write to wait for a full cross-continental network round trip (commonly well over 100ms) before being acknowledged — this would make write latency for the primary database unacceptably slow for most real applications, essentially trading acceptable performance for a guarantee that's disproportionate to the actual risk being protected against. I'd recommend asynchronous replication instead, explicitly documenting and communicating the resulting RPO (how much data could be lost in a true regional disaster — covered fully in Part 3), which is almost always the correct, standard tradeoff for genuinely long-distance, cross-region replication. --- ## Summary and What's Next - **Synchronous replication** guarantees zero data loss but costs real write latency; **asynchronous replication** is fast but risks losing the most recent writes if the primary fails; **semi-synchronous** is a practical middle ground requiring only one replica's confirmation. - Nearly all relational database replication is built on the **write-ahead log (WAL)** — the same mechanism that also enables point-in-time recovery, covered in Part 3. - **Replication lag** should be monitored as a first-class, Saturation-style leading indicator, not discovered for the first time during an actual failover. - **Failover** promotes a replica to primary when the original becomes unavailable — but promotion must wait for full WAL replay completion to avoid data corruption. - **Automatic failover** is faster but risks false positives and split-brain if not built on proper quorum-based consensus; **manual failover** avoids that risk but is bound by human response time. - **Split-brain** — two simultaneously-active primaries accepting conflicting writes — is one of the most dangerous database failure modes, and is specifically what quorum-based leader election (the same principle as etcd's own Raft consensus) and fencing exist to prevent. - **Multi-region replication is almost always asynchronous**, since the physical latency of cross-continental network round trips makes synchronous replication impractical for most real applications. **Continue to Part 2** (`02-sharding-and-partitioning.md`) to cover the other major database scaling technique — splitting data itself across multiple independent databases (sharding), and the real challenges that come with it.