# Monitoring Methodologies — Part 3: Combining Methodologies, Real Dashboards & Worked Incidents > **Series:** Monitoring Methodologies (3 of 3) > **Part 1:** `01-golden-signals-and-red.md` — Golden Signals + RED Method > **Part 2:** `02-use-method-and-metrics.md` — USE Method + Metric Types + Percentiles > **Part 3:** This file — Combining methodologies, real dashboards, worked incidents > **Questions:** `questions.md` ## Table of Contents 1. [Recap: The Three Methodologies Side by Side](#recap-the-three-methodologies-side-by-side) 2. [Choosing the Right Methodology — A Decision Framework](#choosing-the-right-methodology--a-decision-framework) 3. [The Layered Dashboard Architecture](#the-layered-dashboard-architecture) 4. [Full Worked Incident: Checkout Slowness](#full-worked-incident-checkout-slowness) 5. [Full Worked Incident: The Mystery Memory Leak](#full-worked-incident-the-mystery-memory-leak) 6. [Combining RED and USE in a Single Investigation](#combining-red-and-use-in-a-single-investigation) 7. [Building an Organization-Wide Dashboard Standard](#building-an-organization-wide-dashboard-standard) 8. [Connecting Methodologies to SLOs and Error Budgets](#connecting-methodologies-to-slos-and-error-budgets) 9. [Methodologies and Alerting Design](#methodologies-and-alerting-design) 10. [Case Study: How Netflix, Google, and Amazon Approach This](#case-study-how-netflix-google-and-amazon-approach-this) 11. [Beyond the Big Three — Other Methodologies Worth Knowing](#beyond-the-big-three--other-methodologies-worth-knowing) 12. [Common Mistakes When Combining Methodologies](#common-mistakes-when-combining-methodologies) 13. [Worked Practice Problems](#worked-practice-problems) 14. [Summary — The Complete Monitoring Methodologies Series](#summary--the-complete-monitoring-methodologies-series) --- ## Recap: The Three Methodologies Side by Side ```mermaid graph TB subgraph "Golden Signals (Google, 2016)" GS1[Latency] GS2[Traffic] GS3[Errors] GS4[Saturation] end subgraph "RED (Tom Wilkie, 2015) — for services" R1["Rate ≈ Traffic"] R2["Errors ≈ Errors"] R3["Duration ≈ Latency"] end subgraph "USE (Brendan Gregg, 2012) — for resources" U1[Utilization] U2["Saturation ≈ Saturation"] U3["Errors ≈ Errors"] end ``` | | Golden Signals | RED | USE | |---|---|---|---| | **Focus** | General service health | Request-driven services | Physical/logical resources | | **Metrics** | Latency, Traffic, Errors, Saturation | Rate, Errors, Duration | Utilization, Saturation, Errors | | **Typical scope** | Top-level SLO/service dashboard | Per-microservice dashboard | Per-node/per-resource dashboard | | **Auto-instrumentable?** | Partially (needs resource-specific knowledge for Saturation) | Yes, via generic middleware/service mesh | Partially (Linux tools cover common resources; app-level needs custom instrumentation) | | **What it deliberately omits** | — | Explicit Saturation | Explicit Latency/Duration | | **Best answers the question** | "Is this system healthy overall?" | "Is this specific service serving requests well?" | "Is this specific resource about to become a bottleneck?" | **The senior-level synthesis, worth stating verbatim in an interview:** *"These aren't competing methodologies — they're complementary lenses at different layers. I'd use RED dashboards per microservice for day-to-day service health, USE dashboards for the underlying infrastructure to catch resource bottlenecks before they cause user-facing symptoms, and roll the most critical signals up into a Golden-Signals-style top-level view that non-SRE stakeholders can read at a glance without needing to understand connection pools or run queues."* --- ## Choosing the Right Methodology — A Decision Framework ```mermaid flowchart TD Start{What are you monitoring?} --> Q1{Is it a service that
handles discrete requests?} Q1 -->|Yes| RED["Use RED
(Rate, Errors, Duration)"] Q1 -->|No| Q2{Is it a physical or logical
resource with a capacity limit?} Q2 -->|Yes| USE["Use USE
(Utilization, Saturation, Errors)"] Q2 -->|No| Q3{"Is it an async/batch workload
(queue consumer, cron job, pipeline)?"} Q3 -->|Yes| Adapted["Adapt RED's pattern:
throughput, failure rate,
processing-time distribution"] Q3 -->|No| Q4{Building a new service
with no historical baseline yet,
or explaining reliability to
a non-technical stakeholder?} Q4 -->|Yes| GS["Start with Golden Signals
as the general framework"] ``` **A common interview trap** is treating this as "pick one methodology for the whole system." The correct answer is almost always "use multiple, at different layers" — see the layered architecture below. --- ## The Layered Dashboard Architecture A realistic, mature observability stack for a mid-to-large microservices system, from the top-level executive view down to raw kernel metrics: ```mermaid graph TD L1["Layer 1: Business/SLO Dashboard
(Golden-Signals-style, for leadership & on-call triage)"] --> L2 L2["Layer 2: Per-Service RED Dashboards
(one per microservice, auto-generated
from service mesh sidecars)"] --> L3 L3["Layer 3: Per-Resource USE Dashboards
(nodes, databases, message queues,
connection pools)"] --> L4 L4["Layer 4: Deep Diagnostic Tools
(distributed tracing, flame graphs,
eBPF/bpftrace, log search)"] ``` | Layer | Audience | Primary Question Answered | Refresh Cadence | |---|---|---|---| | Layer 1 — Business/SLO | Leadership, on-call first responders | "Is the product healthy right now?" | Real-time, always visible | | Layer 2 — Per-Service RED | Service-owning engineers, on-call | "Which service is degraded?" | Real-time | | Layer 3 — Per-Resource USE | SRE/infra engineers | "Why is that service degraded — what resource is the bottleneck?" | Real-time | | Layer 4 — Deep diagnostics | Whoever's actively debugging | "What exact code path/query/syscall is the problem?" | On-demand, investigative | **Interview framing:** "During an incident, I move top-down through these layers — Layer 1 tells me *something* is wrong and roughly how bad; Layer 2 (RED) tells me *which service*; Layer 3 (USE) tells me *why*, at the resource level; Layer 4 is where I go if USE doesn't immediately reveal an obvious bottleneck and I need to trace the actual code path or query." --- ## Full Worked Incident: Checkout Slowness A complete, narrated walkthrough — this exact shape ("tell me how you'd debug X") is extremely common in SRE interviews, and having a fully worked example ready to adapt on the fly is high-value prep. ### The Page *03:14 UTC — PagerDuty alert: "checkout-service: burn rate 18x over 1h/5m windows — SLO at risk."* ```mermaid sequenceDiagram participant Alert as Alert participant OnCall as On-Call SRE participant L1 as Layer 1: SLO Dashboard participant L2 as Layer 2: RED (checkout-service) participant L3 as Layer 3: USE (checkout-db) participant L4 as Layer 4: Query tracing Alert->>OnCall: Pages at 03:14 UTC OnCall->>L1: Check top-level dashboard L1-->>OnCall: checkout-service SLO in breach,
other services look normal OnCall->>L2: Check checkout-service RED dashboard L2-->>OnCall: Rate: normal. Errors: near-zero.
Duration p99: 4.2s (was 180ms) — huge spike OnCall->>OnCall: Correlate with Traffic —
flat, not a load spike. Check recent deploys — none in 6h. OnCall->>L3: Check USE for checkout-service's
dependencies (checkout-db) L3-->>OnCall: DB CPU: 40% (normal).
DB connection pool: 98/100 used (near saturation!)
DB errors: connection timeout count rising OnCall->>L4: Pull distributed trace for a slow request L4-->>OnCall: Trace shows requests spending 3.9s of
their 4.2s total waiting to ACQUIRE a DB connection,
not executing queries OnCall->>OnCall: Root cause hypothesis: connection pool
exhaustion, NOT a slow query or DB overload ``` ### The Investigation, Narrated 1. **Layer 1 check**: confirms it's scoped to `checkout-service` specifically — other services' SLOs are healthy, ruling out a platform-wide issue (e.g., a shared load balancer or DNS problem). 2. **Layer 2 (RED) check on checkout-service**: Rate is flat (rules out a traffic spike/overload scenario), Errors are near-zero (requests aren't *failing*, they're just slow), Duration p99 has spiked dramatically. Checking recent deploys turns up nothing in the last 6 hours, ruling out "bad deploy" as the immediate trigger. 3. **Layer 3 (USE) check on the dependency**: this is the key pivot — since checkout-service's own RED metrics show slowness with no obvious internal cause, the investigation moves to its direct dependency, the database. CPU utilization on the DB looks totally normal (40%) — a common false trail that could mislead someone who stops checking here. But the **connection pool saturation** metric (98/100 connections in use) tells the real story, confirmed by a rising connection-timeout error count. 4. **Layer 4 (tracing) check**: confirms precisely *where* the 4.2 seconds is going — not query execution time, but time spent waiting in line for a connection to even become available. This distinguishes "the database is slow" from "the database is fine, but we've configured too few connections for current demand" — a critical distinction that determines the fix. ### The Fix and the Postmortem Connection Root cause: a slow-leaking connection (from an unrelated recent change to a retry-handling code path, deployed 2 days earlier — outside the "last 6 hours" deploy window the on-call initially checked, an important lesson) was failing to release connections back to the pool under a specific error condition, gradually starving the pool over ~48 hours until it finally saturated. Immediate mitigation: restart the service (releases the leaked connections) and increase pool size as a buffer. The postmortem's real action item: fix the leak in the retry-handling code, and — tying back to the SRE Fundamentals series — add a **Saturation-based alert on DB connection pool usage** (e.g., page at 80% sustained) so this class of issue is caught proactively next time, well before it becomes a full SLO-breaching incident. **This worked example demonstrates the full stack working together**: Layer 1 scoped it, Layer 2 (RED) ruled out load/deploy/errors and pinpointed *duration* as the anomaly, Layer 3 (USE) found the actual resource bottleneck (which utilization alone would have hidden — CPU looked fine!), and Layer 4 confirmed the precise mechanism. This is exactly the kind of structured, multi-layer reasoning interviewers are listening for. --- ## Full Worked Incident: The Mystery Memory Leak A second, shorter worked example emphasizing a different pattern — a slow-building resource issue with no discrete triggering event. ```mermaid flowchart TD A["Alert: recommendation-service pods
restarting every ~6 hours (CrashLoopBackOff)"] --> B["USE — Kubernetes pod layer:
Utilization: memory usage climbing
steadily within each pod's lifetime"] B --> C["USE — Errors: OOMKilled events
correlate exactly with each restart"] C --> D["Saturation: memory usage graph shows
a clean linear ramp from pod start
to OOM — classic leak signature,
not a sudden spike"] D --> E["Layer 4: heap profiler attached to a
live pod confirms an ever-growing
in-memory cache with no eviction policy"] E --> F["Fix: add TTL-based eviction to the cache;
add a memory-usage-trend alert
(not just an absolute threshold)
to catch this class of issue earlier next time"] ``` **Key teaching point from this example:** a *linear, steadily-climbing* saturation graph (rather than a sudden jump) is a recognizable signature worth naming explicitly in an interview — it strongly suggests a resource leak (memory, file descriptors, connections) rather than a sudden load spike or a discrete bad deploy. Recognizing this *shape* of the graph, not just the raw numbers, is a genuinely useful diagnostic skill. --- ## Combining RED and USE in a Single Investigation A general pattern, worth internalizing as a repeatable heuristic beyond just the two worked examples above: ```mermaid flowchart LR A["RED tells you WHERE
(which service, which endpoint)"] --> B["USE tells you WHY
(which resource, what kind of
problem: utilization, saturation,
or errors)"] B --> C["Tracing/profiling tells you
EXACTLY WHAT
(which code path, query,
or syscall)"] ``` **A one-line summary worth memorizing for interviews:** *"RED narrows the search from 'the whole system' to one service or endpoint. USE narrows it further from 'that service' to a specific resource. Deep tracing/profiling narrows it from 'that resource' to the exact line of code or query causing it."* --- ## Building an Organization-Wide Dashboard Standard A practical, often-overlooked topic: at scale, every team building dashboards their own way is itself a reliability risk (nobody can quickly navigate an unfamiliar team's dashboard during a cross-team incident). Mature orgs enforce a **dashboard template standard**. ```mermaid graph TD Standard["Org-Wide RED Dashboard Template
(every service MUST have one, same layout)"] --> S1["Row 1: Rate (always top-left)"] Standard --> S2["Row 2: Errors (always top-right)"] Standard --> S3["Row 3: Duration p50/p95/p99 (always bottom)"] Standard --> S4["Standard variables: service, route,
environment — same names everywhere"] ``` **Why this matters for interviews:** if asked "how would you scale observability practices across 100+ microservice teams," a strong answer includes **enforcing a shared dashboard/metric-naming template** (often via a shared instrumentation library or service mesh, so it's automatic rather than relying on every team remembering to follow a style guide), because during a cross-team incident, an on-call engineer unfamiliar with a dependency's internals still needs to be able to open its dashboard and immediately understand it. --- ## Connecting Methodologies to SLOs and Error Budgets This ties the entire Monitoring Methodologies series back to the SRE Fundamentals series — an integration interviewers explicitly look for, since disconnected knowledge of each topic in isolation is a weaker signal than seeing how they compose. ```mermaid flowchart TD RED["RED Duration/Errors metrics"] --> SLI["Become the SLI
(good events / valid events)"] SLI --> SLO["Measured against the SLO"] SLO --> Budget["Error budget consumption
calculated from RED data"] Budget --> BurnRate["Burn rate = RED error rate ÷
SLO-implied allowed error rate"] BurnRate --> Alert["Multi-window burn-rate alert fires"] USE["USE Saturation metrics"] --> Leading["Used as a LEADING indicator —
alert BEFORE the SLI/error budget
is actually impacted"] ``` **The concrete link:** a service's RED "Errors" and "Duration" metrics are literally what gets plugged into the SLI formula from the SRE Fundamentals series (`good events / valid events`), which then drives the SLO comparison and error-budget burn-rate calculation. USE's Saturation metrics, meanwhile, are what you'd use for **proactive, leading-indicator alerts** that fire *before* the RED-derived SLI actually degrades — giving on-call a head start, exactly as demonstrated in the Golden Signals cascade-timing diagram in Part 1. --- ## Methodologies and Alerting Design A brief bridge to the Observability tutorial (topic 4), which covers alerting design in full depth — worth previewing the connection here: | Signal Source | Typical Alert Design | |---|---| | RED Errors/Duration (→ SLI) | **Burn-rate alerts** — multi-window, tied to error budget consumption | | USE Saturation | **Trend/threshold alerts** — e.g., "connection pool >80% for 5+ minutes," fired as a leading indicator, often a lower-urgency page or ticket rather than an immediate SEV1 | | USE Errors (disk/network/memory hardware-level) | **Threshold alerts on rare events** — e.g., any OOM-kill, any rising SMART error count — these should almost always page, since they're rare and specific enough that false-positive risk is low | | RED Rate | Rarely alerted on directly — more often used as **correlating context** displayed alongside an Errors/Duration alert, to help the responder quickly determine "was this a load spike?" | --- ## Case Study: How Netflix, Google, and Amazon Approach This Grounding the methodologies in real, named industry practice is a strong way to demonstrate depth beyond textbook definitions. - **Google**: originated the Golden Signals framing in the SRE book; internally, Google's monitoring philosophy (via tools like Monarch, their internal time-series database) emphasizes symptom-based alerting tied directly to SLOs, exactly as described throughout this series. - **Netflix**: publicly known for popularizing chaos engineering (Chaos Monkey and the broader Simian Army) specifically to validate that their USE-style resource resilience (auto-scaling, failover) actually works under real failure conditions, not just in theory — connecting monitoring methodology to proactive resilience testing (covered further in the Reliability & Architecture Patterns and Incident Management tutorials). - **Amazon**: well known for operating at a scale where "the tail at scale" effects (fan-out amplifying p99 latency, covered in Part 2) are a first-order design concern — publicly documented techniques like hedged requests and careful SLA-tiering per internal service trace directly back to this exact class of problem. --- ## Beyond the Big Three — Other Methodologies Worth Knowing While Golden Signals, RED, and USE cover the vast majority of interview questions, a few additional frameworks are worth being aware of by name, in case they come up: ```mermaid graph TD Other[Other Notable Frameworks] --> W["Working Set metrics
(memory-specific, tracks actively
used pages, not just allocated)"] Other --> TSE["TSE / Four Keys
(DORA metrics — deployment frequency,
lead time, change failure rate, MTTR —
measures delivery performance,
not runtime reliability)"] Other --> Business["Business-level SLIs
(orders/min, signups/min —
sometimes called 'Business Golden Signals')"] ``` - **DORA / Four Keys metrics** (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Time to Restore Service) are a distinct, complementary framework — they measure **delivery/DevOps performance**, not runtime system health, so don't conflate them with Golden Signals/RED/USE if asked to distinguish. They're covered further in the Automation, CI/CD & GitOps tutorial. - **Business-level SLIs** (orders per minute, signups per minute, active user count) are sometimes layered on top of Layer 1 in the dashboard architecture above — useful because a sudden drop in business metrics can be an *earlier* signal of a subtle problem (e.g., a broken button that returns 200 OK but doesn't actually submit the form) than any purely technical RED/USE metric would catch. --- ## Common Mistakes When Combining Methodologies | Mistake | Why It's Wrong | Fix | |---|---|---| | Treating RED and USE as competing/redundant | They answer different questions (where vs. why) at different layers | Use both, layered, as shown in the dashboard architecture | | Building only Layer 1 (top-level) dashboards, no per-service RED | Great for "is something wrong" but useless for "what exactly and where" | Ensure every service has a RED dashboard, ideally auto-generated | | Investigating USE metrics before checking RED | Wastes time checking resources before confirming which service/endpoint is actually affected | Always narrow with RED first, then use USE to explain why | | Ignoring the "shape" of a saturation graph (sudden vs. gradual) | Misses a free diagnostic clue — gradual linear ramps suggest leaks; sudden jumps suggest load spikes or discrete bad changes | Always look at the graph's shape, not just its current value | | No shared dashboard template across teams | Slows cross-team incident response when responders can't navigate unfamiliar dashboards | Enforce an org-wide RED/USE dashboard standard, ideally auto-generated | | Forgetting to check "did anything deploy recently" broadly enough (e.g., only checking the last few hours) | Slow leaks or subtle bugs can originate from changes days earlier, as in the worked incident above | Check a wider deploy window, especially for gradually-building issues | --- ## Worked Practice Problems **Problem 1:** An on-call engineer sees checkout-service's RED dashboard showing normal Rate, near-zero Errors, but Duration p99 has tripled. They immediately conclude "the database must be slow" and start investigating query performance, but query execution times (from DB-side monitoring) look completely normal. What did they likely miss, and what should they check next? *Answer:* They jumped straight to "database is slow" without following the full USE checklist on the dependency — query *execution* time being normal doesn't rule out the database layer entirely; it specifically rules out slow *queries*, but not connection pool exhaustion, lock contention, or network latency between the service and the DB. They should check USE's Saturation dimension specifically (connection pool usage, lock wait time) rather than stopping at Utilization/query-execution-time alone — this is exactly the pattern from the worked "Checkout Slowness" incident above, where CPU utilization looked fine but connection pool saturation was the real story. **Problem 2:** You're designing the org-wide dashboard standard for a company with 150 microservices owned by 30 different teams. What's the single most important design principle, and why? *Answer:* Consistency/uniformity across every service's dashboard — same layout, same panel order, same metric naming conventions — ideally enforced automatically via a shared instrumentation library or service mesh rather than a style guide teams might not follow. The reasoning: during a cross-team incident, the responding engineer is very likely unfamiliar with the internals of whatever dependency is implicated, and a consistent, predictable dashboard layout lets them navigate it productively anyway, without needing to learn that team's bespoke conventions under time pressure. **Problem 3:** A memory-usage graph for a service shows a perfectly flat, healthy line for weeks, then suddenly jumps straight to 100% and OOMs within minutes, with no gradual ramp beforehand. How does this shape change your root-cause hypothesis compared to the gradual-ramp memory leak example earlier in this tutorial? *Answer:* A sudden jump (rather than a gradual linear ramp) suggests a **discrete triggering event** rather than a slow accumulating leak — likely candidates: a single request/batch job that loaded an unusually large payload into memory all at once, a sudden traffic spike overwhelming an in-memory cache, or a recent deploy introducing a bug that allocates a large structure under a specific, rarely-hit condition. I'd immediately check for a recent deploy and for any unusual traffic pattern or large request right before the jump, rather than looking for a slow leak — the *shape* of the graph directly points the investigation in a different direction. --- ## Summary — The Complete Monitoring Methodologies Series - **Golden Signals, RED, and USE are complementary, not competing** — they operate at different layers (top-level system health, per-service request handling, per-resource capacity) and answer different questions (is it healthy, where's the problem, why is it happening). - A mature observability stack is **layered**: business/SLO dashboards → per-service RED → per-resource USE → deep tracing/profiling, and incident investigation typically flows top-down through these layers. - **RED narrows "where," USE narrows "why," tracing narrows "exactly what."** - Real incident investigation benefits from recognizing **graph shapes** (gradual ramp = likely leak; sudden jump = likely discrete trigger), not just current values. - These methodologies directly feed the **SLI/SLO/error-budget** machinery from the SRE Fundamentals series — RED metrics typically *are* the SLI; USE Saturation metrics provide proactive, leading-indicator alerting ahead of actual SLO impact. - At organizational scale, **enforcing a consistent dashboard standard** (via shared instrumentation/service mesh) matters as much as picking the right methodology, since cross-team incident response depends on dashboards being navigable by people unfamiliar with a given service's internals. - Related but distinct frameworks worth knowing by name: **DORA/Four Keys** (delivery performance, not runtime health) and **business-level SLIs** (can catch certain classes of subtle bugs earlier than pure technical metrics). This completes the **Monitoring Methodologies** series. See `questions.md` in this folder for the full interview question bank covering all three parts.