Companion question bank for the 3-part tutorial series in this folder: 01-golden-signals-and-red.md, 02-use-method-and-metrics.md, 03-applying-methodologies.md.

Think of these three methodologies as three different pairs of glasses for looking at the same system — each one makes a different kind of problem obvious. Answers below are kept short on purpose; use the analogies and worked examples from the tutorials to expand them out loud.

---

Part 1 Questions: Golden Signals & RED

Conceptual#

1. What are the Four Golden Signals?#

Latency, Traffic, Errors, Saturation. Simple way to remember it: Latency and Errors are "what the user feels right now," Traffic is "how much demand there is," and Saturation is "how close we are to falling over." From Google's SRE book — "if you can only measure four things about a user-facing system, measure these."

2. What's the difference between explicit and implicit errors?#

Explicit errors are loud — HTTP 5xx, exceptions, crashes. Implicit errors are sneaky — a response that says "200 OK" but is secretly wrong (an error page returned with a success status code, or a response that's technically correct but way too slow). If you only watch status codes, implicit errors slip right past you.

3. Why should you track latency separately for successful vs. failed requests?#

Picture 100 requests: 90 take 200ms, 10 fail instantly in 2ms. Blend them together and the average looks great (~180ms) — but 10% of your users just got an error! A fast failure quietly drags the average down and hides the real story. Always keep success latency and failure latency as two separate lines.

4. What is saturation, and why is it called a "leading indicator"?#

Saturation is "how full is the tank" — CPU run queue, connection pool usage, message queue depth. It's a leading indicator because it climbs first: load rises → saturation climbs → then latency climbs (things start queueing) → only then do errors appear (timeouts). If you only alert on errors, you always find out last, after the user already had a bad time.

5. What's the difference between utilization and saturation?#

Utilization is "how busy is it right now" (CPU at 80%). Saturation is "how much extra work is stuck waiting" (12 processes in line for 8 CPU cores). A resource can be 100% utilized and perfectly healthy (no line forming), or only 60% utilized on average but still saturated during bursts (a line forms, then clears, then forms again). They sound similar but answer different questions — don't mix them up.

6. What is the RED method and who created it?#

Created by Tom Wilkie. RED = Rate, Errors, Duration — basically Golden Signals with Saturation removed, purpose-built for microservices. Think of it as "the three things every single service, no matter what it does, can report about itself using one shared piece of instrumentation code."

7. Why did RED drop Saturation compared to Golden Signals?#

Because Saturation is resource-specific (CPU for one service, a connection pool for another) — there's no one generic way to measure it that works identically for every service. Rate/Errors/Duration, on the other hand, can all come from one shared middleware wrapped around any HTTP handler, which is exactly why service meshes can generate RED dashboards automatically for every service with zero code changes.

8. Should 4xx client errors count toward RED's "Errors" metric?#

Generally no — a 400/404 usually means the client sent something wrong, not that the service is broken. Folding them into your error rate would falsely blame the service. Track 4xx separately (useful for spotting API misuse) and keep 5xx (or genuine semantic failures) as the number that drives SLOs and alerts.

9. Why is RED such a natural fit for microservices and service meshes?#

Because each of the three metrics maps to one counter + one histogram, wrapped around every HTTP call the same way. A service mesh sidecar (Istio, Linkerd) sits in front of every service anyway (for routing/mTLS), so it can emit these metrics automatically — every team gets a matching RED dashboard for free, with zero app code changes.

Applied / Scenario#

10. Users report intermittent slowness on checkout. Walk me through your investigation using RED.#

Open the checkout service's RED dashboard: is Rate normal (rules out a traffic spike)? Are Errors near zero (rules out outright failures)? Is Duration (p99, not the average!) actually elevated? If Duration is spiking while Rate and Errors look fine, that tells me the service itself is slow to respond — time to go look at why (that's where USE comes in, covered in Part 2).

11. Errors and Latency are both spiking, but Traffic is completely flat. What does that tell you?#

It rules out "we got overloaded" — if traffic didn't increase, this isn't a capacity problem. Something else changed on its own: a bad deploy, a downstream dependency failing, a cert expiring, a disk filling up. First move: check the deploy timeline and dependency health, not "let's add more servers."

12. How would you adapt RED for a Kafka consumer that doesn't handle HTTP requests at all?#

Translate the shape, not the literal metric names: Rate → messages consumed/sec, Errors → failed/dead-lettered messages as a % of consumed, Duration → per-message processing time as a histogram (still use percentiles!). Showing you understand the pattern generalizes — not just memorized HTTP metric names — is the real signal here.

---

Part 2 Questions: USE Method, Metric Types & Percentiles

Conceptual#

13. What is the USE method and who created it?#

Created by Brendan Gregg. USE = Utilization, Saturation, Errors — a simple three-question checklist you run against any resource (CPU, disk, memory, a connection pool): how busy is it, is work backing up waiting for it, and is it throwing errors. Designed specifically to be run methodically during a live incident so you don't tunnel-vision on your favorite metric and miss the real bottleneck.

14. Walk through USE for a database connection pool.#

Utilization: active_connections / max_connections. Saturation: how many requests are stuck waiting for a free connection. Errors: connection refused/timeout counts. This exact three-question pattern works for CPU, disk, network, memory, thread pools — it's the same checklist everywhere, which is the whole point.

15. Your dashboard shows CPU utilization at only 60%, but the service is still slow under load. What's your hypothesis?#

Utilization alone hides bursts — 60% average could easily be "100% for half the time, idle the other half," with a queue forming during those spikes. I'd check Saturation specifically: run queue length, thread pool queue depth, DB connection wait time. It's also entirely possible CPU isn't the bottleneck at all — I'd run the full USE checklist (CPU → memory → disk → network → app-level resources) rather than assume.

16. Why does Gregg recommend running USE as a strict top-to-bottom checklist instead of just checking whatever seems obvious?#

Because under incident pressure, people gravitate to the metric they're most familiar with and stop looking too early. A fixed checklist (CPU, memory, disk, network, then app-level resources) forces completeness and catches the bottleneck even when it's somewhere unexpected.

17. What's the difference between a counter, a gauge, a histogram, and a summary?#

• Counter: only ever goes up (total requests served). Never read it raw — always wrap it in rate().
• Gauge: goes up and down freely, a snapshot right now (memory used, queue depth). Read it directly — never rate() a gauge, the math doesn't make sense.
• Histogram: sorts observations into buckets, lets you compute percentiles after the fact, and — the important part — can be summed across many machines.
• Summary: pre-computes percentiles on each machine individually. Cheaper, but those per-machine numbers can't be validly combined into one fleet-wide number.

18. Histograms vs summaries — which would you choose for a Prometheus-based service running on 50 pods, and why?#

Histograms. Prometheus can add up the bucket counts from all 50 pods and then compute one real, valid p99 for the whole fleet. Summaries compute a separate p99 on each pod — and you mathematically cannot average 50 different p99s into a meaningful fleet-wide p99. It's the single most common "gotcha" question about metric types, so know it cold.

19. Why shouldn't you rely on average latency alone?#

Imagine 100 requests: 99 take 10ms, 1 takes 5000ms. The average comes out to about 60ms — looks totally healthy! But the p99 (5000ms) shows you the real, painful outlier. Averages are great at hiding exactly the thing you most need to see.

20. Why does tail latency (p99) matter more at large scale than small scale?#

At 1 million requests/day, a p99 of 3 seconds means 10,000 requests every day experience 3+ seconds of pain — that's a real, headline-worthy problem, not a rounding error. At 1,000 requests/day the same p99 is only 10 unhappy users — easy to shrug off. Same percentage, wildly different real-world weight. (This is the core idea behind Google's well-known "The Tail at Scale" paper.)

21. A request fans out to 20 backend calls in parallel, and the response can't return until all 20 finish. Each backend has a 1% chance of being "slow" (that's its own p99). What's the odds the OVERALL request hits a slow call somewhere?#

Roughly 1 − 0.99²⁰ ≈ 18% — nearly 1 in 5 requests gets caught by some slow call, even though every individual service looks perfectly healthy on its own dashboard. Fan-out amplifies tail latency. This is exactly why techniques like hedged requests (fire a backup request if the first one's taking too long) exist.

22. How do you choose histogram bucket boundaries, and what happens if you choose badly?#

Pick boundaries around your actual SLO threshold and space them roughly exponentially (10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s...). If your SLO cares about "under 300ms" but your nearest buckets are 100ms and 500ms, histogram_quantile() has to guess/interpolate across that huge gap — right where accuracy matters most. Too many buckets costs storage/cardinality; too few costs accuracy.

Applied / Scenario#

23. vmstat 1 shows CPU at 45% utilization but a run queue of 16 on an 8-core box. Is CPU a bottleneck?#

Yes — 16 processes want to run, only 8 cores exist, so 8 are always waiting their turn even though the "45% busy" number looks fine on average. This is saturation hiding behind a deceptively low utilization number — a textbook USE-method catch.

24. A summary metric type is used for request duration on a service that just scaled from 3 to 40 pods. What's wrong, and what would you change?#

Per-pod p99s from a summary can't be combined into one true fleet-wide p99 — it was borderline-ok-ish at 3 pods (eyeball 3 lines) but breaks down completely at 40. Switch to a histogram so Prometheus can sum bucket counts across all 40 pods and compute one mathematically valid aggregate percentile.

---

Part 3 Questions: Combining Methodologies & Real Investigations

Conceptual#

25. How would you decide whether to use RED or USE for a given thing you're monitoring?#

Ask: "does this handle discrete requests?" → RED. "Is this a resource with a capacity limit?" → USE. In practice you rarely pick just one for an entire system — you layer them: RED per service, USE per resource underneath it, and a Golden-Signals-style rollup on top for a "is the product healthy" view non-engineers can read.

26. Describe a layered observability dashboard architecture, top to bottom.#

Layer 1: a top-level Golden-Signals-style dashboard for "is the product healthy right now." Layer 2: per-service RED dashboards (auto-generated, ideally, via a service mesh) telling you which service is degraded. Layer 3: per-resource USE dashboards telling you why — which specific resource is the bottleneck. Layer 4: deep tracing/profiling tools for pinpointing the exact line of code or query. Incident response typically moves top-down through these layers.

27. In one sentence, how do RED and USE work together during an investigation?#

RED narrows down WHERE the problem is (which service/endpoint); USE narrows down WHY (which resource, and what kind of problem — too busy, backed up, or erroring); deep tracing narrows down the exact WHAT (which code path or query).

28. Why does an organization with 100+ microservices need a standard dashboard template, not just "good enough" dashboards per team?#

Because during a cross-team incident, the person responding is often unfamiliar with the internals of whatever dependency is misbehaving. A consistent, predictable layout (same panel order, same metric names) lets them navigate someone else's service dashboard productively under time pressure — without having to learn that team's personal conventions mid-incident.

29. How do RED and USE metrics connect back to SLIs, SLOs, and error budgets?#

RED's Errors/Duration numbers are literally what becomes the SLI (good events / valid events) — that gets compared against the SLO and drives error-budget burn-rate math. USE's Saturation metrics are used differently: as a leading-indicator alert that fires before the SLI/error budget is actually hit, buying the on-call team a head start.

Applied / Scenario — Full Investigation Walkthrough#

30. Checkout is slow. RED shows normal Rate, near-zero Errors, but Duration p99 tripled. You check the database and query execution time looks totally normal. What did you likely miss?#

"Query execution time is normal" only rules out slow queries — it doesn't rule out the rest of the USE checklist for that database, specifically Saturation: connection pool exhaustion, lock contention, or requests just waiting in line for a free connection. In a real worked example, CPU looked fine (40%) while the connection pool was sitting at 98/100 — the actual bottleneck was requests waiting to get a connection, not anything about the query itself.

31. A memory graph is flat and healthy for weeks, then suddenly jumps to 100% and OOMs within minutes — no gradual ramp. How does this change your hypothesis compared to a slow, linear memory leak?#

A gradual linear ramp usually points to a leak (something never gets released, and it builds up over time). A sudden jump instead suggests a discrete trigger — an unusually large request/batch job, a traffic spike overwhelming a cache, or a recent deploy hitting a rare code path. I'd check for a recent deploy and any unusual traffic right before the jump, rather than hunting for a slow leak — the shape of the graph itself points the investigation in a different direction.

32. What's the difference between Golden Signals/RED/USE and DORA (Four Keys) metrics?#

Golden Signals/RED/USE measure runtime system health — is the system working well right now. DORA metrics (deployment frequency, lead time for changes, change failure rate, time to restore service) measure delivery/engineering process performance — how fast and safely the team ships changes. They're complementary but answer completely different questions — don't conflate them if asked to distinguish.

---

Quick-Fire / Rapid Recall#