Combining Methodologies, Real Dashboards & Worked Incidents

Table of Contents#

1. Recap: The Three Methodologies Side by Side
2. Choosing the Right Methodology — A Decision Framework
3. The Layered Dashboard Architecture
4. Full Worked Incident: Checkout Slowness
5. Full Worked Incident: The Mystery Memory Leak
6. Combining RED and USE in a Single Investigation
7. Building an Organization-Wide Dashboard Standard
8. Connecting Methodologies to SLOs and Error Budgets
9. Methodologies and Alerting Design
10. Case Study: How Netflix, Google, and Amazon Approach This
11. Beyond the Big Three — Other Methodologies Worth Knowing
12. Common Mistakes When Combining Methodologies
13. Worked Practice Problems
14. Summary — The Complete Monitoring Methodologies Series

---

Recap: The Three Methodologies Side by Side#

Diagram

The senior-level synthesis, worth stating verbatim in an interview: "These aren't competing methodologies — they're complementary lenses at different layers. I'd use RED dashboards per microservice for day-to-day service health, USE dashboards for the underlying infrastructure to catch resource bottlenecks before they cause user-facing symptoms, and roll the most critical signals up into a Golden-Signals-style top-level view that non-SRE stakeholders can read at a glance without needing to understand connection pools or run queues."

---

Choosing the Right Methodology — A Decision Framework#

Diagram

A common interview trap is treating this as "pick one methodology for the whole system." The correct answer is almost always "use multiple, at different layers" — see the layered architecture below.

---

The Layered Dashboard Architecture#

A realistic, mature observability stack for a mid-to-large microservices system, from the top-level executive view down to raw kernel metrics:

Diagram

Interview framing: "During an incident, I move top-down through these layers — Layer 1 tells me something is wrong and roughly how bad; Layer 2 (RED) tells me which service; Layer 3 (USE) tells me why, at the resource level; Layer 4 is where I go if USE doesn't immediately reveal an obvious bottleneck and I need to trace the actual code path or query."

---

Full Worked Incident: Checkout Slowness#

A complete, narrated walkthrough — this exact shape ("tell me how you'd debug X") is extremely common in SRE interviews, and having a fully worked example ready to adapt on the fly is high-value prep.

The Page#

03:14 UTC — PagerDuty alert: "checkout-service: burn rate 18x over 1h/5m windows — SLO at risk."

Diagram

The Investigation, Narrated#

1. Layer 1 check: confirms it's scoped to checkout-service specifically — other services' SLOs are healthy, ruling out a platform-wide issue (e.g., a shared load balancer or DNS problem).
2. Layer 2 (RED) check on checkout-service: Rate is flat (rules out a traffic spike/overload scenario), Errors are near-zero (requests aren't failing, they're just slow), Duration p99 has spiked dramatically. Checking recent deploys turns up nothing in the last 6 hours, ruling out "bad deploy" as the immediate trigger.
3. Layer 3 (USE) check on the dependency: this is the key pivot — since checkout-service's own RED metrics show slowness with no obvious internal cause, the investigation moves to its direct dependency, the database. CPU utilization on the DB looks totally normal (40%) — a common false trail that could mislead someone who stops checking here. But the connection pool saturation metric (98/100 connections in use) tells the real story, confirmed by a rising connection-timeout error count.
4. Layer 4 (tracing) check: confirms precisely where the 4.2 seconds is going — not query execution time, but time spent waiting in line for a connection to even become available. This distinguishes "the database is slow" from "the database is fine, but we've configured too few connections for current demand" — a critical distinction that determines the fix.

The Fix and the Postmortem Connection#

Root cause: a slow-leaking connection (from an unrelated recent change to a retry-handling code path, deployed 2 days earlier — outside the "last 6 hours" deploy window the on-call initially checked, an important lesson) was failing to release connections back to the pool under a specific error condition, gradually starving the pool over ~48 hours until it finally saturated. Immediate mitigation: restart the service (releases the leaked connections) and increase pool size as a buffer. The postmortem's real action item: fix the leak in the retry-handling code, and — tying back to the SRE Fundamentals series — add a Saturation-based alert on DB connection pool usage (e.g., page at 80% sustained) so this class of issue is caught proactively next time, well before it becomes a full SLO-breaching incident.

This worked example demonstrates the full stack working together: Layer 1 scoped it, Layer 2 (RED) ruled out load/deploy/errors and pinpointed duration as the anomaly, Layer 3 (USE) found the actual resource bottleneck (which utilization alone would have hidden — CPU looked fine!), and Layer 4 confirmed the precise mechanism. This is exactly the kind of structured, multi-layer reasoning interviewers are listening for.

---

Full Worked Incident: The Mystery Memory Leak#

A second, shorter worked example emphasizing a different pattern — a slow-building resource issue with no discrete triggering event.

Diagram

Key teaching point from this example: a linear, steadily-climbing saturation graph (rather than a sudden jump) is a recognizable signature worth naming explicitly in an interview — it strongly suggests a resource leak (memory, file descriptors, connections) rather than a sudden load spike or a discrete bad deploy. Recognizing this shape of the graph, not just the raw numbers, is a genuinely useful diagnostic skill.

---

Combining RED and USE in a Single Investigation#

A general pattern, worth internalizing as a repeatable heuristic beyond just the two worked examples above:

Diagram

A one-line summary worth memorizing for interviews: "RED narrows the search from 'the whole system' to one service or endpoint. USE narrows it further from 'that service' to a specific resource. Deep tracing/profiling narrows it from 'that resource' to the exact line of code or query causing it."

---

Building an Organization-Wide Dashboard Standard#

A practical, often-overlooked topic: at scale, every team building dashboards their own way is itself a reliability risk (nobody can quickly navigate an unfamiliar team's dashboard during a cross-team incident). Mature orgs enforce a dashboard template standard.

Diagram

Why this matters for interviews: if asked "how would you scale observability practices across 100+ microservice teams," a strong answer includes enforcing a shared dashboard/metric-naming template (often via a shared instrumentation library or service mesh, so it's automatic rather than relying on every team remembering to follow a style guide), because during a cross-team incident, an on-call engineer unfamiliar with a dependency's internals still needs to be able to open its dashboard and immediately understand it.

---

Connecting Methodologies to SLOs and Error Budgets#

This ties the entire Monitoring Methodologies series back to the SRE Fundamentals series — an integration interviewers explicitly look for, since disconnected knowledge of each topic in isolation is a weaker signal than seeing how they compose.

Diagram

The concrete link: a service's RED "Errors" and "Duration" metrics are literally what gets plugged into the SLI formula from the SRE Fundamentals series (good events / valid events), which then drives the SLO comparison and error-budget burn-rate calculation. USE's Saturation metrics, meanwhile, are what you'd use for proactive, leading-indicator alerts that fire before the RED-derived SLI actually degrades — giving on-call a head start, exactly as demonstrated in the Golden Signals cascade-timing diagram in Part 1.

---

Methodologies and Alerting Design#

A brief bridge to the Observability tutorial (topic 4), which covers alerting design in full depth — worth previewing the connection here:

---

Case Study: How Netflix, Google, and Amazon Approach This#

Grounding the methodologies in real, named industry practice is a strong way to demonstrate depth beyond textbook definitions.

• Google: originated the Golden Signals framing in the SRE book; internally, Google's monitoring philosophy (via tools like Monarch, their internal time-series database) emphasizes symptom-based alerting tied directly to SLOs, exactly as described throughout this series.
• Netflix: publicly known for popularizing chaos engineering (Chaos Monkey and the broader Simian Army) specifically to validate that their USE-style resource resilience (auto-scaling, failover) actually works under real failure conditions, not just in theory — connecting monitoring methodology to proactive resilience testing (covered further in the Reliability & Architecture Patterns and Incident Management tutorials).
• Amazon: well known for operating at a scale where "the tail at scale" effects (fan-out amplifying p99 latency, covered in Part 2) are a first-order design concern — publicly documented techniques like hedged requests and careful SLA-tiering per internal service trace directly back to this exact class of problem.

---

Beyond the Big Three — Other Methodologies Worth Knowing#

While Golden Signals, RED, and USE cover the vast majority of interview questions, a few additional frameworks are worth being aware of by name, in case they come up:

Diagram

• DORA / Four Keys metrics (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Time to Restore Service) are a distinct, complementary framework — they measure delivery/DevOps performance, not runtime system health, so don't conflate them with Golden Signals/RED/USE if asked to distinguish. They're covered further in the Automation, CI/CD & GitOps tutorial.
• Business-level SLIs (orders per minute, signups per minute, active user count) are sometimes layered on top of Layer 1 in the dashboard architecture above — useful because a sudden drop in business metrics can be an earlier signal of a subtle problem (e.g., a broken button that returns 200 OK but doesn't actually submit the form) than any purely technical RED/USE metric would catch.

---

Common Mistakes When Combining Methodologies#

---

Worked Practice Problems#

Problem 1: An on-call engineer sees checkout-service's RED dashboard showing normal Rate, near-zero Errors, but Duration p99 has tripled. They immediately conclude "the database must be slow" and start investigating query performance, but query execution times (from DB-side monitoring) look completely normal. What did they likely miss, and what should they check next?

Answer: They jumped straight to "database is slow" without following the full USE checklist on the dependency — query execution time being normal doesn't rule out the database layer entirely; it specifically rules out slow queries, but not connection pool exhaustion, lock contention, or network latency between the service and the DB. They should check USE's Saturation dimension specifically (connection pool usage, lock wait time) rather than stopping at Utilization/query-execution-time alone — this is exactly the pattern from the worked "Checkout Slowness" incident above, where CPU utilization looked fine but connection pool saturation was the real story.

Problem 2: You're designing the org-wide dashboard standard for a company with 150 microservices owned by 30 different teams. What's the single most important design principle, and why?

Answer: Consistency/uniformity across every service's dashboard — same layout, same panel order, same metric naming conventions — ideally enforced automatically via a shared instrumentation library or service mesh rather than a style guide teams might not follow. The reasoning: during a cross-team incident, the responding engineer is very likely unfamiliar with the internals of whatever dependency is implicated, and a consistent, predictable dashboard layout lets them navigate it productively anyway, without needing to learn that team's bespoke conventions under time pressure.

Problem 3: A memory-usage graph for a service shows a perfectly flat, healthy line for weeks, then suddenly jumps straight to 100% and OOMs within minutes, with no gradual ramp beforehand. How does this shape change your root-cause hypothesis compared to the gradual-ramp memory leak example earlier in this tutorial?

Answer: A sudden jump (rather than a gradual linear ramp) suggests a discrete triggering event rather than a slow accumulating leak — likely candidates: a single request/batch job that loaded an unusually large payload into memory all at once, a sudden traffic spike overwhelming an in-memory cache, or a recent deploy introducing a bug that allocates a large structure under a specific, rarely-hit condition. I'd immediately check for a recent deploy and for any unusual traffic pattern or large request right before the jump, rather than looking for a slow leak — the shape of the graph directly points the investigation in a different direction.

---

Summary — The Complete Monitoring Methodologies Series#

• Golden Signals, RED, and USE are complementary, not competing — they operate at different layers (top-level system health, per-service request handling, per-resource capacity) and answer different questions (is it healthy, where's the problem, why is it happening).
• A mature observability stack is layered: business/SLO dashboards → per-service RED → per-resource USE → deep tracing/profiling, and incident investigation typically flows top-down through these layers.
• RED narrows "where," USE narrows "why," tracing narrows "exactly what."
• Real incident investigation benefits from recognizing graph shapes (gradual ramp = likely leak; sudden jump = likely discrete trigger), not just current values.
• These methodologies directly feed the SLI/SLO/error-budget machinery from the SRE Fundamentals series — RED metrics typically are the SLI; USE Saturation metrics provide proactive, leading-indicator alerting ahead of actual SLO impact.
• At organizational scale, enforcing a consistent dashboard standard (via shared instrumentation/service mesh) matters as much as picking the right methodology, since cross-team incident response depends on dashboards being navigable by people unfamiliar with a given service's internals.
• Related but distinct frameworks worth knowing by name: DORA/Four Keys (delivery performance, not runtime health) and business-level SLIs (can catch certain classes of subtle bugs earlier than pure technical metrics).

This completes the Monitoring Methodologies series. See questions.md in this folder for the full interview question bank covering all three parts.