Replication & Failover

Table of Contents#

1. Why Databases Get Their Own Deep Dive
2. What Replication Actually Is
3. Synchronous vs Asynchronous Replication
4. Semi-Synchronous Replication — A Practical Middle Ground
5. How Replication Actually Works Under the Hood
6. Replication Lag, Revisited
7. Failover — What Happens When the Primary Dies
8. Automatic vs Manual Failover
9. The Split-Brain Problem
10. Leader Election and Consensus
11. A Full Worked Failover, Start to Finish
12. Multi-Region Replication
13. Common Mistakes
14. Worked Practice Problems
15. Summary and What's Next

---

Why Databases Get Their Own Deep Dive#

Databases are the single hardest thing to scale, replicate, and recover reliably in almost any system — every other tutorial in this course has treated them as "the stateful thing behind the app servers." This series goes inside that box. Everything here builds directly on ideas already introduced: the CAP theorem and consistency models (Reliability & Architecture Patterns), read replicas (Capacity Planning & Performance), and etcd's own use of consensus (Kubernetes Deep Dive) — this tutorial applies those same ideas specifically, deeply, to the database layer itself.

---

What Replication Actually Is#

Replication means keeping a copy (or several copies) of a database's data on separate servers, kept continuously in sync with the original.

Diagram

Simple analogy: think of a shared team document with "track changes" turned on, being continuously mirrored to several backup copies the instant any edit happens — if the original document is ever lost, one of those mirrors already has (almost) everything, ready to become the new original.

Why this matters, tying directly back to earlier tutorials: replication is the foundation of two completely different things covered elsewhere in this course — read scaling (Capacity Planning & Performance, Part 1 — spreading read traffic across replicas) and availability/disaster recovery (this Part — having a ready backup if the primary dies). The same underlying mechanism serves both purposes.

---

Synchronous vs Asynchronous Replication#

This is one of the single most important, most commonly tested database concepts — a direct, concrete application of the CAP theorem and PACELC tradeoffs from the Reliability & Architecture Patterns series.

Diagram

Diagram

This is exactly PACELC's "Else" branch (Latency vs. Consistency) from the Reliability & Architecture Patterns series, made completely concrete: synchronous replication sacrifices latency to guarantee zero data loss; asynchronous replication sacrifices a small, real risk of data loss to get lower latency. Neither choice is universally "correct" — it's a deliberate tradeoff, sized to how bad losing a recent write would actually be for that specific system.

---

Semi-Synchronous Replication — A Practical Middle Ground#

A genuinely useful, real-world compromise worth knowing by name.

Diagram

Why this is a strong, senior-level interview answer to "how would you configure replication for a payments database": "I'd lean toward semi-synchronous replication — requiring confirmation from at least one replica before acknowledging a write gives real protection against losing a committed transaction if the primary suddenly dies, without paying the latency cost (and added fragility, if a replica becomes unreachable) of waiting for every single replica to confirm, which pure synchronous replication requires."

---

How Replication Actually Works Under the Hood#

Almost every relational database implements replication using the same core mechanism: a write-ahead log (WAL) — every change is first written to a sequential log file before it's applied to the actual data, and that log is what gets streamed to replicas.

Diagram

Copy
# PostgreSQL: set up a replica by taking a base backup, then
# streaming WAL changes continuously from that point forward
pg_basebackup -h primary-host -D /var/lib/postgresql/data \
  -U replicator -P --wal-method=stream

# Check replication status/lag from the primary
psql -c "SELECT client_addr, state, sent_lsn, replay_lsn FROM pg_stat_replication;"

Why the WAL mechanism matters, tying directly to Part 3 of this series: the exact same write-ahead log used for replication is also what makes point-in-time recovery possible (covered in Part 3) — a database can be restored to any specific moment by replaying WAL entries up to that exact point. Replication and recovery share the same underlying technical foundation.

---

Replication Lag, Revisited#

Already introduced in the Capacity Planning & Performance series (Part 1) as the tradeoff behind read replicas — here's the deeper, operational picture.

Diagram

Copy
# Check replication lag in seconds (PostgreSQL)
psql -c "SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;"

Why monitoring replication lag as a first-class metric matters, connecting directly to the Monitoring Methodologies and Observability series: rising replication lag is a genuine Saturation-style leading indicator (USE method) — it tells you a replica is falling behind before it becomes so far behind that failing over to it would mean losing a meaningful, noticeable amount of recent data. A well-designed alert on replication lag (using the same burn-rate-style thinking from the Observability series) gives operators time to intervene before an unplanned failover forces the issue.

---

Failover — What Happens When the Primary Dies#

Failover is the process of promoting a replica to become the new primary when the original primary becomes unavailable — directly extending the failover concepts from the Reliability & Architecture Patterns series (Part 1), applied specifically to databases.

Diagram

Why promotion isn't instant, and why this matters practically: the replica needs to fully finish applying every WAL entry it's already received before it can safely start accepting new writes as a primary — if it started accepting writes while still catching up, new writes could be applied out of order relative to the replication stream it was still processing, risking data corruption.

---

Automatic vs Manual Failover#

The exact same tradeoff pattern seen throughout this course, applied specifically to databases.

Diagram

Real-world tools worth knowing by name, since this is a genuinely common, specific interview topic: Patroni (a widely used automatic failover manager for PostgreSQL, built on top of a distributed consensus store like etcd or Consul) and Orchestrator (a similar, widely used tool for MySQL) — both exist specifically to implement automatic failover safely, using leader election (below) to avoid the split-brain problem that naive automatic failover risks.

---

The Split-Brain Problem#

A genuinely important, high-stakes failure mode, and one of the most commonly tested "what could go wrong" database interview questions.

Diagram

Why this is so dangerous, worth stating explicitly: once two primaries have independently accepted different, conflicting writes, there is no clean, automatic way to merge them back together — whichever writes happened on the "losing" side during the split are typically lost or require painful, manual, application-specific reconciliation. This is precisely the CAP theorem's Consistency-vs-Availability dilemma from the Reliability & Architecture Patterns series, playing out in its most damaging, concrete real-world form.

Diagram

---

Leader Election and Consensus#

This directly reuses the exact same Raft/quorum-based consensus concepts from etcd (Kubernetes Deep Dive series) — a genuinely important connection worth stating explicitly in an interview.

Diagram

A strong, senior-level interview line connecting multiple parts of this course together: "Modern automatic database failover tools like Patroni don't invent their own ad hoc 'is the primary down' logic — they build on the exact same quorum-based consensus principle etcd itself uses for Kubernetes's own control plane. A minority of nodes, even if they genuinely believe the primary is unreachable, can never unilaterally promote a new leader — this is precisely what prevents split-brain."

---

A Full Worked Failover, Start to Finish#

Tying every concept in this Part together into one complete, realistic story.

Diagram

---

Multi-Region Replication#

A brief but important extension — replicating not just across servers in one data center, but across entire geographic regions, directly connecting to the multi-region discussion in the Reliability & Architecture Patterns series (Part 1).

Diagram

Why synchronous replication becomes genuinely impractical across regions, worth stating explicitly: the speed of light itself imposes a real, physical latency floor on any cross-continental network round trip (commonly 60-150ms+ depending on distance) — requiring every single write to wait for that full round trip before acknowledging would make the database's write latency unacceptably slow for most applications. This is exactly why multi-region database replication is almost always asynchronous, accepting some real risk of losing the most recent writes in a true regional disaster, in exchange for acceptable local write latency the rest of the time.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A team configures pure asynchronous replication for their primary financial ledger database. During an incident, the primary crashes hard immediately after acknowledging a batch of customer transactions. What's the risk, and how would semi-synchronous replication have changed the outcome?

Answer: With pure asynchronous replication, those acknowledged transactions may never have actually reached any replica before the crash — meaning they're genuinely, silently lost, even though the application received a successful confirmation for them. With semi-synchronous replication (requiring at least one replica to confirm before acknowledging), the same crash would still leave at least one replica with those transactions durably recorded, since the primary would never have acknowledged them to the application until that replica confirmed receipt — completely eliminating this specific class of silent data loss, at the cost of somewhat higher write latency.

Problem 2: A monitoring system, running as a single instance with no quorum mechanism, detects the primary database is unreachable (due to a brief network partition, not an actual crash) and immediately promotes a replica. Fifteen seconds later, the network partition heals and the old primary — which had continued accepting writes the whole time — is reachable again. What's the resulting failure mode, and what should have been different about the failover design?

Answer: This is a textbook split-brain scenario — for those 15 seconds, both the old primary (still genuinely running and accepting writes on its side of the partition) and the newly-promoted replica were simultaneously acting as independent primaries, accepting potentially conflicting writes with no way to reconcile them afterward. The failover design should have used a quorum-based consensus mechanism (like Patroni backed by etcd) requiring a majority of monitoring nodes to agree the primary was truly down, and should have actively fenced the old primary (preventing it from accepting further writes) as part of the promotion process, rather than a single monitoring instance making a unilateral decision based on its own, possibly-partitioned view of the world.

Problem 3: A company wants to replicate their primary US database to a new EU region for disaster recovery. An engineer proposes synchronous replication "to guarantee zero data loss." What would you push back on, and what would you recommend instead?

Answer: Synchronous replication across a US-to-EU link would require every single write to wait for a full cross-continental network round trip (commonly well over 100ms) before being acknowledged — this would make write latency for the primary database unacceptably slow for most real applications, essentially trading acceptable performance for a guarantee that's disproportionate to the actual risk being protected against. I'd recommend asynchronous replication instead, explicitly documenting and communicating the resulting RPO (how much data could be lost in a true regional disaster — covered fully in Part 3), which is almost always the correct, standard tradeoff for genuinely long-distance, cross-region replication.

---

Summary and What's Next#

• Synchronous replication guarantees zero data loss but costs real write latency; asynchronous replication is fast but risks losing the most recent writes if the primary fails; semi-synchronous is a practical middle ground requiring only one replica's confirmation.
• Nearly all relational database replication is built on the write-ahead log (WAL) — the same mechanism that also enables point-in-time recovery, covered in Part 3.
• Replication lag should be monitored as a first-class, Saturation-style leading indicator, not discovered for the first time during an actual failover.
• Failover promotes a replica to primary when the original becomes unavailable — but promotion must wait for full WAL replay completion to avoid data corruption.
• Automatic failover is faster but risks false positives and split-brain if not built on proper quorum-based consensus; manual failover avoids that risk but is bound by human response time.
• Split-brain — two simultaneously-active primaries accepting conflicting writes — is one of the most dangerous database failure modes, and is specifically what quorum-based leader election (the same principle as etcd's own Raft consensus) and fencing exist to prevent.
• Multi-region replication is almost always asynchronous, since the physical latency of cross-continental network round trips makes synchronous replication impractical for most real applications.

Continue to Part 2 (02-sharding-and-partitioning.md) to cover the other major database scaling technique — splitting data itself across multiple independent databases (sharding), and the real challenges that come with it.