Scaling Strategies

Table of Contents#

1. Why Capacity Planning Is Its Own Discipline
2. Vertical Scaling — Give It a Bigger Box
3. Horizontal Scaling — Give It More Boxes
4. Vertical vs Horizontal — The Full Comparison
5. Stateless vs Stateful — Why Scaling Isn't the Same for Everything
6. Scaling Databases — The Hard Part
7. Read Replicas — Scaling Reads Separately From Writes
8. Caching — The Cheapest Scaling Trick That Exists
9. Vertical Scaling's Hidden Ceiling
10. Scaling the Whole Stack Together
11. Common Mistakes
12. Worked Practice Problems
13. Summary and What's Next

---

Why Capacity Planning Is Its Own Discipline#

Everything in this course so far has been about making a system reliable once it's built and running. Capacity planning asks a different, earlier question: will this system actually have enough horsepower to handle the traffic it's going to get — today, and six months from now?

A simple way to picture it: imagine running a coffee shop. Reliability engineering is making sure the espresso machine doesn't break down. Capacity planning is deciding, in advance, whether you need one register or five for the Monday morning rush — get it wrong, and even a perfectly working espresso machine can't save you from a line out the door.

Diagram

---

Vertical Scaling — Give It a Bigger Box#

Vertical scaling (also called "scaling up") means making a single machine more powerful — more CPU, more memory, faster disks.

Diagram

Simple analogy: if one cashier can't keep up, vertical scaling is like replacing them with someone who can ring up items twice as fast — same one person, just more capable.

• Pro: simple. No architectural changes needed — the application usually doesn't even know it happened.
• Con: there's a hard ceiling. Eventually, no bigger machine exists to buy, and even before that, bigger machines get disproportionately expensive.

---

Horizontal Scaling — Give It More Boxes#

Horizontal scaling (also called "scaling out") means adding more machines, and spreading the work across all of them — directly reusing the load balancing concepts from the Reliability & Architecture Patterns series.

Diagram

Simple analogy: instead of one super-fast cashier, open three more registers. Each one is normal speed, but together they handle far more customers than one, no matter how fast, ever could alone.

• Pro: in principle, no ceiling — keep adding machines as demand grows. Also improves availability as a side effect (more redundancy, per the Reliability series).
• Con: genuinely harder to build. The application has to be designed to run correctly across many machines at once — this is where "stateless vs stateful" (below) becomes the whole story.

---

Vertical vs Horizontal — The Full Comparison#

A strong, senior-level interview line: "I'd default to horizontal scaling for anything that can be made stateless, since it avoids a hard ceiling and improves availability for free. I'd only reach for vertical scaling first when the component genuinely can't be easily distributed — classically, a single primary database — or as a quick, low-effort stopgap while a proper horizontal redesign is planned."

---

Stateless vs Stateful — Why Scaling Isn't the Same for Everything#

This distinction determines almost everything about how hard horizontal scaling actually is.

Diagram

A concrete example of the difference: a typical web API server that just processes a request and returns a response, with no memory of the previous request, is stateless — you can run 50 identical copies behind a load balancer with zero extra thought (this is exactly the active-active pattern from the Reliability & Architecture Patterns series). A database holding customer orders is stateful — you can't just spin up a second, empty copy and expect it to somehow already have all the existing orders in it.

The practical takeaway, worth stating explicitly in an interview: "The first thing I'd check before proposing horizontal scaling for any component is whether it's actually stateless. If it holds state, scaling it horizontally requires a real design decision about how that state gets shared or partitioned — it's never just 'add more copies.'"

---

Scaling Databases — The Hard Part#

Databases are the classic hard case, because they're inherently stateful — the whole point of a database is to hold onto data reliably.

Diagram

This tutorial covers the first two (vertical scaling and read replicas) in depth, since they're the more commonly-reached-for first steps. Sharding — splitting data across multiple independent databases — is covered in full depth in the dedicated Databases & Storage Reliability tutorial (topic 9), since it deserves its own complete treatment.

---

Read Replicas — Scaling Reads Separately From Writes#

A very common, practical insight: most applications read data far more often than they write it (think: a social media feed — millions of reads for every single post written). Read replicas exploit this by copying data from a primary database to one or more read-only copies, and directing read traffic to the replicas instead of the primary.

Diagram

Why this is such a common, high-leverage first move: it directly attacks the most common real bottleneck (read volume) without needing the much harder work of splitting write data across multiple databases (sharding). It's often the single cheapest, highest-ROI capacity improvement available for a read-heavy application.

The real cost, worth naming explicitly, and directly connecting back to the CAP theorem tutorial: replication isn't instant — there's a small delay ("replication lag") between a write landing on the primary and that write becoming visible on a replica. This is exactly the eventual consistency tradeoff from the CAP Theorem tutorial in the Reliability & Architecture Patterns series: read replicas trade a small amount of consistency (a replica might briefly serve slightly stale data) for a large amount of read capacity and availability.

Diagram

A genuinely important, practical mitigation worth naming: for any read that must reflect a user's own very-recent write (e.g., "I just placed this order, why doesn't it show up?"), route that specific read to the primary instead of a replica — directly reusing the "read-your-writes consistency" pattern from the CAP Theorem tutorial.

---

Caching — The Cheapest Scaling Trick That Exists#

Before reaching for read replicas or sharding, the single cheapest capacity win available to almost any system is simply not re-doing expensive work that was already done recently.

Diagram

Copy
# A simple, common caching pattern using Redis (the "cache-aside" pattern)
# Pseudocode showing the actual logic:

def get_product(product_id):
    cached = redis.get(f"product:{product_id}")
    if cached:
        return cached                      # cache HIT — fast, no DB hit at all

    product = db.query("SELECT * FROM products WHERE id = ?", product_id)
    redis.set(f"product:{product_id}", product, ex=300)  # cache for 5 minutes
    return product                          # cache MISS — slower, but only once
                                             # every 5 minutes per product

The single most important tradeoff to name explicitly, since it's a favorite interview follow-up: cache invalidation — if the underlying data changes, how does the cache know to update? A cache that serves stale data for too long can cause real, confusing bugs, but a cache that's invalidated too aggressively provides very little benefit at all. This exact tension is famously referenced by the old programming joke: "There are only two hard things in computer science: cache invalidation and naming things."

---

Vertical Scaling's Hidden Ceiling#

Worth spelling out concretely, since "just get a bigger machine" sounds infinitely repeatable but genuinely isn't.

Diagram

A strong interview point: "Vertical scaling isn't wrong as a first move — it's simple and buys real time. But it's a strategy with a shelf life, not a permanent solution, and the cost curve gets worse well before you hit the actual hardware ceiling. I'd treat hitting diminishing returns on vertical scaling as the trigger to start seriously investing in horizontal scaling (or, for a database specifically, read replicas and eventually sharding), rather than waiting until there's truly no bigger box left to buy."

---

Scaling the Whole Stack Together#

A genuinely important, easy-to-miss point: scaling one layer of a system without checking the others just moves the bottleneck, it doesn't remove it — directly connecting back to the USE method from the Monitoring Methodologies series (always check every resource, not just the one you assumed was the problem).

Diagram

A concrete, memorable interview line: "Scaling is only as good as your weakest link — adding 10x more application servers just means 10x more traffic arrives at whatever's downstream of them. I'd always model the whole request path end-to-end (app servers, caches, databases, external dependencies) before scaling any single piece, using exactly the USE-method checklist from the Monitoring Methodologies series, to make sure I'm not just relocating the bottleneck instead of actually fixing it."

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A team scales their application servers from 5 to 50 instances ahead of an expected traffic surge, but the surge still causes a full outage. Investigation shows the app servers themselves were healthy the whole time. What's the most likely explanation?

Answer: The bottleneck almost certainly moved downstream — most likely the database (or another shared dependency) couldn't handle 10x the traffic that 50 app servers were now capable of generating, even though it was never a problem at the old scale of 5 servers. This is a textbook case of scaling one layer without checking the rest of the request path — the fix is to model and load-test the entire path end-to-end (including the database, caches, and any external dependencies) before scaling any single layer in isolation.

Problem 2: A product page's data is cached with a 24-hour TTL to reduce database load, but customer support starts getting complaints that price updates don't show up for hours after being changed. What's the tradeoff at play, and how would you fix it?

Answer: This is the cache invalidation tradeoff — a long TTL maximizes database load reduction but risks serving meaningfully stale data for a long time; a short TTL keeps data fresher but provides much less caching benefit. The better fix here isn't necessarily just shortening the TTL (which sacrifices most of the caching benefit) — it's adding event-based invalidation: when a price is actually updated, explicitly clear or update that specific product's cache entry immediately, rather than waiting for a fixed TTL to expire. This gets both freshness (for the rare case something actually changes) and efficiency (for the common case where it hasn't).

Problem 3: An application currently reads and writes to a single database instance, which is becoming a bottleneck primarily due to read traffic (95% of queries are reads). What's the lowest-effort, highest-leverage next step, and why not jump straight to sharding?

Answer: Read replicas — since 95% of the load is reads, directing that traffic to one or more read replicas (leaving only the 5% write traffic on the primary) directly targets the actual bottleneck with comparatively low implementation effort. Sharding would also technically solve the capacity problem, but it's a much larger architectural undertaking (splitting data across independent databases, rewriting queries to route to the correct shard) that's disproportionate to a problem that's overwhelmingly read-driven — read replicas get most of the benefit for a fraction of the engineering cost, and sharding can be reserved for if/when write volume itself eventually becomes the bottleneck.

---

Summary and What's Next#

• Capacity planning asks a distinct question from reliability engineering: not just "does it work correctly," but "is there enough of it to handle real demand, now and in the future."
• Vertical scaling (bigger machine) is simple but has a real cost/hardware ceiling; horizontal scaling (more machines) has no practical ceiling but requires real design work, especially for anything stateful.
• Stateless services scale horizontally almost for free; stateful services (databases especially) require an explicit design decision about how state is shared or partitioned.
• Read replicas are usually the highest-leverage first step for a read-heavy database bottleneck, trading a small amount of consistency (replication lag) for a large amount of read capacity — directly reusing the CAP theorem tradeoffs from the Reliability & Architecture Patterns series.
• Caching is often the cheapest capacity win of all, but always requires a deliberate invalidation strategy — an uninvalidated cache causes stale-data bugs, and an over-invalidated one provides little real benefit.
• Scaling only helps if you scale the actual bottleneck — scaling one layer without checking the rest of the request path just relocates the problem downstream, exactly the lesson from the USE method in the Monitoring Methodologies series.

Continue to Part 2 (02-queuing-theory-and-capacity-math.md) to learn the actual math behind capacity planning — Little's Law, queuing theory, and how to forecast how much capacity you'll actually need.