# Monitoring Methodologies — Part 2: The USE Method, Metric Types & Percentiles > **Series:** Monitoring Methodologies (2 of 3) > **Part 1:** `01-golden-signals-and-red.md` — Golden Signals + RED Method > **Part 2:** This file — USE Method + Metric Types + Percentiles > **Part 3:** `03-applying-methodologies.md` — Combining methodologies, real dashboards, worked incidents > **Questions:** `questions.md` ## Table of Contents 1. [The USE Method — Origin and Definition](#the-use-method--origin-and-definition) 2. [USE in Depth: Utilization](#use-in-depth-utilization) 3. [USE in Depth: Saturation](#use-in-depth-saturation) 4. [USE in Depth: Errors](#use-in-depth-errors) 5. [USE Applied to Every Major Resource Type](#use-applied-to-every-major-resource-type) 6. [Running USE as a Systematic Incident Checklist](#running-use-as-a-systematic-incident-checklist) 7. [USE on Linux — The Actual Commands](#use-on-linux--the-actual-commands) 8. [USE for Cloud/Managed Resources](#use-for-cloudmanaged-resources) 9. [Metric Types — The Foundation Under Every Methodology](#metric-types--the-foundation-under-every-methodology) 10. [Counters, In Depth](#counters-in-depth) 11. [Gauges, In Depth](#gauges-in-depth) 12. [Histograms, In Depth](#histograms-in-depth) 13. [Summaries, In Depth](#summaries-in-depth) 14. [Histogram vs Summary — The Interview-Favorite Gotcha](#histogram-vs-summary--the-interview-favorite-gotcha) 15. [Percentiles — Why Averages Lie](#percentiles--why-averages-lie) 16. [The Mathematics of Tail Latency at Scale](#the-mathematics-of-tail-latency-at-scale) 17. [Choosing Histogram Buckets — A Practical Guide](#choosing-histogram-buckets--a-practical-guide) 18. [Common Mistakes With USE and Metric Types](#common-mistakes-with-use-and-metric-types) 19. [Worked Practice Problems](#worked-practice-problems) 20. [Summary and What's Next](#summary-and-whats-next) --- ## The USE Method — Origin and Definition Coined by **Brendan Gregg** (formerly of Sun/Oracle, Netflix, and now Intel — a well-known Linux performance engineer) around 2012, USE is designed for **systematically diagnosing resource bottlenecks**: CPU, memory, disk, network, and any resource-like abstraction (thread pools, connection pools, file descriptors, locks). ```mermaid graph TD USE[USE Method] --> U[Utilization] USE --> S[Saturation] USE --> E[Errors] U --> U1["% of time the resource was busy
servicing work"] S --> S1["Amount of queued work the resource
couldn't service immediately"] E --> E1["Count of error events for this resource —
e.g. disk I/O errors, retransmits, OOM kills"] ``` Gregg's own framing, worth quoting: *"For every resource, check utilization, saturation, and errors."* It was explicitly designed as a **checklist to run methodically during a live performance investigation**, precisely because under incident pressure, engineers tend to jump straight to their favorite/most-familiar metric and miss an obvious bottleneck elsewhere. USE forces completeness. --- ## USE in Depth: Utilization The percentage of time the resource was busy doing useful work, over a measurement interval. ``` Utilization = (time resource was busy) / (total elapsed time) × 100% ``` **Important subtlety:** utilization is often reported as an *average over an interval* (e.g., "CPU was 80% busy over the last minute"), which can hide short bursts of 100% utilization within that interval. This is exactly the same "averages hide spikes" problem covered later for latency — it applies to utilization metrics too. ```mermaid graph LR A["1-minute average: 50% CPU utilization"] --> B["Could mean: steady 50% the whole minute"] A --> C["Could ALSO mean: 100% for 30 seconds,
0% for 30 seconds"] B -.-> Same["Same average, VERY different reality"] C -.-> Same ``` --- ## USE in Depth: Saturation The amount of extra work queued that the resource couldn't service immediately — the degree to which demand is exceeding capacity **right now**. ``` Saturation examples: - CPU: run queue length (processes waiting for a core) - Memory: swap activity, page fault rate - Disk: I/O queue depth, average wait time in queue - Network: interface transmit/receive queue drops - Connection pool: requests blocked waiting for a free connection ``` As covered in Part 1, saturation is the metric most directly correlated with **actual user-visible degradation** — a resource can be heavily utilized without being saturated (efficient, no queueing) or moderately utilized while still being saturated during bursts (inefficient/bursty, queueing happens anyway). --- ## USE in Depth: Errors The count of error events associated with the resource — distinct from application-level errors (which RED already covers). USE errors are specifically about **the resource itself failing or degrading**. ```mermaid graph TD E[USE-Method Errors] --> E1["Disk: read/write I/O errors,
reallocated sectors (SMART data)"] E --> E2["Network: CRC errors, retransmits,
dropped packets"] E --> E3["Memory: ECC correctable/uncorrectable
errors, OOM-killer invocations"] E --> E4["CPU: machine check exceptions (MCEs),
thermal throttling events"] ``` **Interview-relevant point:** these are often *the earliest, most underrated warning signs* of impending hardware failure or resource exhaustion — a disk throwing a rising rate of correctable I/O errors, or a host whose OOM-killer has fired twice this week, is telling you something is wrong well before Utilization or Saturation alone would show it clearly. A comprehensive USE check always includes this errors dimension, not just the two "how busy/how backed-up" numbers. --- ## USE Applied to Every Major Resource Type The full worked table — this is worth memorizing the *shape* of (three columns, one row per resource), since interviewers frequently ask "walk me through USE for [some resource]" live. | Resource | Utilization | Saturation | Errors | |---|---|---|---| | **CPU** | `%busy` (100% − idle%) | Run queue length; load average > core count | Machine check exceptions, thermal throttling | | **Memory** | `%used` of total RAM | Swap in/out activity, page fault rate | OOM-killer invocations, ECC errors | | **Disk I/O** | `%busy` doing I/O (iostat `%util`) | I/O queue depth, average wait time (`await`) | Read/write I/O errors, SMART reallocated sectors | | **Network interface** | Throughput vs. link capacity (e.g., 8Gbps of 10Gbps) | TX/RX queue drops, interface buffer overflows | CRC errors, retransmits, collisions | | **DB connection pool** | `active_connections / max_connections` | Requests waiting/blocked for a free connection | Connection refused/timeout errors | | **Thread pool** | `busy_threads / max_threads` | Task queue depth (tasks waiting for a thread) | Rejected task exceptions | | **File descriptors** | `open_fds / max_fds (ulimit)` | N/A (typically hard-fails rather than queues) | "Too many open files" errors | | **Message queue (Kafka, etc.)** | Broker disk/network throughput vs. capacity | Consumer lag (messages waiting to be consumed) | Failed produce/consume requests, rebalance storms | --- ## Running USE as a Systematic Incident Checklist Gregg designed USE to be run **top-to-bottom, resource by resource**, specifically because it's exhaustive enough to prevent tunnel vision during a stressful live incident. ```mermaid flowchart TD Start["Incident: system is slow,
no obvious errors in app logs"] --> CPU{"CPU:
U/S/E checked?"} CPU --> MEM{"Memory:
U/S/E checked?"} MEM --> DISK{"Disk:
U/S/E checked?"} DISK --> NET{"Network:
U/S/E checked?"} NET --> APP{"App-level resources:
thread pool, DB pool,
file descriptors, locks"} APP --> Found{Bottleneck identified?} Found -->|Yes| Fix[Address that specific
resource — scale it,
fix the leak, tune the limit] Found -->|No| Deeper["Go deeper: per-process breakdown,
kernel tracing (perf, eBPF/bpftrace),
flame graphs"] ``` **Why this order (CPU → Memory → Disk → Network → App-level) is a reasonable default:** it roughly follows "how likely is this to be the bottleneck, and how cheap is it to check" — CPU and memory are checked first because they're fastest to check (`top`/`htop` gives you both instantly) and are common bottlenecks; disk and network follow because they require slightly more specialized tools; app-level resources (connection pools, thread pools) come last because they're service-specific and require knowing that particular application's internals. **Strong interview answer template** for "how would you debug a slow server with no obvious app-level errors": *"I'd run the USE checklist systematically — CPU utilization/saturation/errors, then memory, then disk I/O, then network, then app-level resources like connection pools and thread pools — rather than guessing based on whatever metric I happen to check first. This is specifically designed to prevent tunnel vision under incident pressure."* --- ## USE on Linux — The Actual Commands Being able to name real tools, not just the abstract methodology, signals hands-on experience. ```mermaid graph TD USE[USE Checklist] --> CPU["CPU:
mpstat -P ALL 1
vmstat 1 (r column = run queue)
top / htop"] USE --> MEM["Memory:
free -m
vmstat 1 (si/so = swap activity)
dmesg | grep -i 'killed process' (OOM)"] USE --> DISK["Disk:
iostat -xz 1 (%util, await, svctm)
smartctl -a /dev/sda (error counts)"] USE --> NET["Network:
sar -n DEV 1 (throughput)
ip -s link (errors, drops)
ss -s (socket summary)"] USE --> DEEP["Deeper tracing:
perf top / perf record
bpftrace / eBPF tools
strace for syscall-level detail"] ``` | Resource | Command | What to Look At | |---|---|---| | CPU utilization | `mpstat -P ALL 1` | `%usr + %sys`, per-core skew | | CPU saturation | `vmstat 1` | `r` column (runnable processes) vs. core count | | Memory utilization | `free -m` | `used` vs `total`, `available` | | Memory saturation | `vmstat 1` | `si`/`so` (swap in/out) — any non-zero is a red flag | | Memory errors | `dmesg \| grep -i oom` | OOM-killer invocations | | Disk utilization | `iostat -xz 1` | `%util` column | | Disk saturation | `iostat -xz 1` | `await` (ms waiting), queue depth | | Disk errors | `smartctl -a /dev/sdX` | Reallocated sector count, pending sectors | | Network utilization | `sar -n DEV 1` | rxkB/s, txkB/s vs. link capacity | | Network saturation | `ip -s link` | Dropped packets on the interface | | Network errors | `ip -s link` / `netstat -s` | CRC errors, retransmit segments | This is essentially **Brendan Gregg's own "Linux Performance Analysis in 60 Seconds"** checklist, which is a commonly referenced real-world artifact worth knowing exists — mentioning it by name is a strong signal in an interview. --- ## USE for Cloud/Managed Resources USE doesn't only apply to raw Linux boxes — it generalizes cleanly to managed cloud resources, which is an important adaptation to mention since most modern SRE work happens on managed infrastructure, not bare metal. | Managed Resource | Utilization | Saturation | Errors | |---|---|---|---| | **RDS/Cloud SQL instance** | CPU/memory utilization (CloudWatch/Cloud Monitoring metric) | `DatabaseConnections` near max, `ReplicaLag` growing | Failed connection attempts, deadlock count | | **Kubernetes node** | `kubectl top node` CPU/memory | Pod eviction events, pending pods due to insufficient resources | `OOMKilled` container restarts | | **Kubernetes pod** | Container CPU/memory vs. requests/limits | CPU throttling (`container_cpu_cfs_throttled_seconds_total`) | Restart count, `CrashLoopBackOff` | | **Managed load balancer** | Requests vs. provisioned capacity units | Surge queue length (e.g., AWS ALB `SurgeQueueLength`) | 5xx from the LB itself (distinct from backend 5xx) | | **Serverless function (Lambda)** | Concurrent executions vs. account concurrency limit | Throttled invocation count | Function error rate, timeout count | **A great interview line:** "USE isn't just a bare-metal/VM concept — it generalizes to any resource with a capacity limit, including managed cloud services. For a Kubernetes pod, for instance, Utilization is CPU/memory usage against its requests/limits, Saturation shows up as CPU throttling, and Errors shows up as OOMKilled restarts." --- ## Metric Types — The Foundation Under Every Methodology Golden Signals, RED, and USE are all built on top of a small set of underlying **metric types** — this maps directly onto the Prometheus data model, the de facto standard in modern SRE tooling. ```mermaid graph LR M[Metric Types] --> C[Counter] M --> G[Gauge] M --> H[Histogram] M --> S[Summary] C --> C1["Monotonically increasing
(only goes up, or resets to 0)"] G --> G1["Goes up AND down —
a point-in-time snapshot value"] H --> H1["Buckets of observations —
enables percentile math AFTER
collection, aggregatable"] S --> S1["Pre-calculated quantiles
client-side — NOT aggregatable"] ``` --- ## Counters, In Depth A counter only ever increases (or resets to zero, typically on process restart). You never read a counter's raw value directly for monitoring purposes — you always apply `rate()` (or `increase()`) to get a meaningful per-second (or per-interval) value. ```promql # WRONG — raw counter value is meaningless on its own http_requests_total # RIGHT — rate of increase per second, over a 5-minute smoothing window rate(http_requests_total[5m]) ``` **Why counters reset, and why that's fine:** when a process restarts, its in-memory counter resets to 0. Prometheus's `rate()` function is specifically designed to detect this reset (a value going *down* is interpreted as a restart, not negative traffic) and correctly compute the rate across the discontinuity — this is a well-known Prometheus behavior worth knowing exists, even if you don't need to explain the exact internal algorithm. Typical counter examples: `http_requests_total`, `errors_total`, `bytes_sent_total`. --- ## Gauges, In Depth A gauge is a point-in-time value that can go up or down freely — it represents "the current state of something," not "the accumulated total of something." ```promql # Gauges are queried directly — no rate() needed memory_usage_bytes queue_depth active_connections ``` Typical gauge examples: `memory_usage_bytes`, `queue_depth`, `temperature_celsius`, `active_connections`, `cpu_usage_percent`. **Interview trap:** applying `rate()` to a gauge is almost always a mistake — `rate()` assumes monotonic increase (a counter), so applying it to something that legitimately goes up and down (a gauge) produces nonsensical results. Knowing which metric type you're looking at *before* choosing your query function is a basic but frequently-tested competency. --- ## Histograms, In Depth A histogram samples observations (typically request durations, but also response sizes or any other continuous value) into a set of configurable buckets, and exposes: 1. A count per bucket (`_bucket{le="0.1"}`, `_bucket{le="0.5"}`, etc. — cumulative, "less than or equal to") 2. A total sum of all observed values (`_sum`) 3. A total count of observations (`_count`) ``` http_request_duration_seconds_bucket{le="0.1"} 240 http_request_duration_seconds_bucket{le="0.5"} 490 http_request_duration_seconds_bucket{le="1.0"} 500 http_request_duration_seconds_bucket{le="+Inf"} 500 http_request_duration_seconds_sum 125.4 http_request_duration_seconds_count 500 ``` Percentiles are computed **after the fact**, at query time, using `histogram_quantile()`: ```promql histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le) ) ``` ### The Critical Property: Aggregatability Because histogram buckets are just counts, you can **sum them across many instances** (e.g., all 50 pods of a service) and *then* compute a valid, statistically meaningful percentile for the whole fleet. This is the single biggest reason histograms are generally preferred over summaries (next section) in a horizontally-scaled environment. ```mermaid flowchart TD P1["Pod 1: bucket counts"] --> Sum["sum() across all pods
(valid — buckets are just counts)"] P2["Pod 2: bucket counts"] --> Sum P3["Pod 3: bucket counts"] --> Sum Sum --> Quantile["histogram_quantile()
→ mathematically valid
cluster-wide p99"] ``` --- ## Summaries, In Depth A summary computes quantiles **client-side, at instrumentation time**, using a streaming algorithm, and exposes the pre-calculated quantile values directly. ``` http_request_duration_seconds{quantile="0.5"} 0.043 http_request_duration_seconds{quantile="0.9"} 0.187 http_request_duration_seconds{quantile="0.99"} 0.923 http_request_duration_seconds_sum 125.4 http_request_duration_seconds_count 500 ``` **Advantages:** cheaper to query (no `histogram_quantile()` math needed at query time — the number is already computed) and doesn't require you to guess good bucket boundaries in advance. **The critical disadvantage:** those pre-computed quantiles are **per-instance** and **cannot be validly combined across instances**. You cannot average 50 different p99 values from 50 pods and get a meaningful fleet-wide p99 — percentiles simply don't aggregate that way mathematically. --- ## Histogram vs Summary — The Interview-Favorite Gotcha This exact comparison is one of the highest-yield "gotcha" questions in SRE/observability interviews — expect it in some form almost every time monitoring internals come up. ```mermaid graph TD Q["Need cluster-wide percentiles
across many horizontally-scaled instances?"] --> Hist["Use HISTOGRAM
✅ buckets aggregate correctly
via sum() then histogram_quantile()"] Q2["Only need per-instance percentiles,
and query cost/cardinality is a concern?"] --> Summ["Summary MAY be acceptable
⚠️ but still can't combine across instances"] ``` | | Histogram | Summary | |---|---|---| | Where quantiles are computed | Server-side, at query time (`histogram_quantile()`) | Client-side, at instrumentation time | | Can aggregate across instances? | **Yes** — bucket counts sum validly | **No** — pre-computed quantiles cannot be meaningfully combined | | Requires choosing bucket boundaries upfront? | Yes — a real design decision (see below) | No | | Query cost | Slightly higher (computation at query time) | Lower (value already computed) | | Recommended default for horizontally-scaled services | **Yes, almost always** | Rarely — mainly single-instance/low-cardinality cases | **Model interview answer:** "For a service running 50 replicas, I'd use a histogram, not a summary — because Prometheus can sum histogram bucket counts across all 50 instances and then compute a mathematically valid cluster-wide p99 with `histogram_quantile()`. A summary computes p99 independently on each of the 50 pods, and those 50 independent p99 values can't be correctly combined into one true cluster-wide p99 — averaging them, for instance, is not the same number you'd get from the raw combined data." --- ## Percentiles — Why Averages Lie This is one of the single highest-yield SRE interview topics across the entire monitoring domain — expect a direct question on it in nearly every SRE interview loop. ### The Core Problem A single very slow outlier request can be completely hidden by many fast ones when averaged. ```mermaid graph TD A["100 requests:
99 take 10ms, 1 takes 5000ms"] --> B["Average = (99×10 + 5000) / 100
≈ 59.9ms — looks totally fine!"] A --> C["p99 = 5000ms
reveals the real outlier"] B -.->|"Hides the problem entirely"| X["❌ Misleading"] C -.->|"Surfaces the exact problem"| Y["✅ Actionable"] ``` ### Common Percentiles Used in SRE, and What Each Is For | Percentile | Meaning | Typical Use | |---|---|---| | **p50 (median)** | Half of requests are faster than this | "Typical" experience — good for understanding normal-case UX | | **p90** | 90% of requests are faster than this | General health check, less noisy than p99 | | **p95** | 95% of requests are faster than this | Very common SLO target — a reasonable balance of strictness vs. noise | | **p99** | 99% of requests are faster than this | Tail-latency SLO — catches the pain that "vocal unhappy users" actually feel | | **p99.9** | 99.9% of requests are faster than this | Needed at very large scale — at millions of requests/day, even 0.1% is thousands of real unhappy users | --- ## The Mathematics of Tail Latency at Scale **Why p99 matters more as scale grows** — a concrete, quantified argument worth memorizing: ```mermaid graph TD A["Service handles 1,000,000
requests/day, p99 = 3 seconds"] --> B["1% of 1,000,000 = 10,000
requests/day experience 3+ second latency"] B --> C["At small scale (1,000 req/day),
that same p99 would mean only
10 unhappy users/day — often ignorable"] B --> D["At large scale, 10,000/day is a
MATERIAL, headline-worthy UX problem —
not statistical noise"] ``` This exact insight — "the tail at scale" — was popularized in a well-known paper by Jeffrey Dean and Luiz André Barroso (Google), *"The Tail at Scale"* (2013, Communications of the ACM). Being able to cite this by name in an interview is a strong signal of genuine depth beyond memorized definitions. ### A Second, Compounding Effect: Fan-Out Amplifies Tail Latency If a single user request internally triggers calls to, say, 20 backend microservices in parallel, and the *overall* response can't return until **all 20** have responded, then the overall p99 latency is driven by the **slowest of the 20**, not any individual service's own p99. ```mermaid graph TD User[User Request] --> FanOut["Fans out to 20 backend calls
in parallel"] FanOut --> S1["Service 1: p99 = 100ms"] FanOut --> S2["Service 2: p99 = 100ms"] FanOut --> Sn["... 18 more services,
each p99 = 100ms"] S1 --> Wait["Overall response waits for
the SLOWEST of all 20"] S2 --> Wait Sn --> Wait Wait --> Result["Effective p99 for the user is
MUCH worse than 100ms —
with 20 independent 1%-chance
slow calls, P(at least one is slow)
≈ 1 − 0.99²⁰ ≈ 18%!"] ``` If each of 20 independent backend calls has a 1% chance of being "slow" (i.e., that's the definition of each one's own p99), the probability that **at least one** of the 20 is slow on any given user request is roughly `1 − 0.99²⁰ ≈ 18%` — meaning nearly 1 in 5 user requests experiences *some* tail-latency-affected sub-call, even though each individual service looks perfectly healthy at "only" a 1% tail rate. This is a genuinely advanced, high-signal point to raise when discussing microservice architecture tradeoffs, and it's one of the core motivating arguments in the "Tail at Scale" paper for techniques like **hedged requests** (send a duplicate request to a second backend if the first hasn't responded within some threshold, and use whichever comes back first). --- ## Choosing Histogram Buckets — A Practical Guide A real, practical skill often glossed over: histogram buckets must be chosen *in advance* at instrumentation time, and a poor choice silently degrades your percentile accuracy. ```mermaid graph TD A["Poorly chosen buckets:
[1s, 10s, 100s]"] --> B["Real p99 might be 340ms,
but your buckets can only tell you
'it's somewhere under 1 second' —
no useful resolution!"] C["Well chosen buckets:
[10ms, 25ms, 50ms, 100ms,
250ms, 500ms, 1s, 2.5s, 5s]"] --> D["Resolution matches the actual
range of real request latencies —
accurate percentile estimates"] ``` **Practical guidance:** - Choose bucket boundaries based on your **actual SLO thresholds** — if your SLO cares about "under 400ms," make sure a bucket boundary sits at or near 400ms so `histogram_quantile()` can accurately tell you whether you're above or below it. - Use roughly **exponential spacing** (e.g., 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s) to get reasonable resolution across a wide dynamic range without an excessive number of buckets. - Too many buckets increases **cardinality** and storage/query cost; too few buckets loses percentile accuracy. This is a genuine engineering tradeoff, not a "just add more buckets" situation. - Some Prometheus client libraries support **native histograms** (a newer feature) which use dynamically-sized buckets automatically, reducing the need to hand-tune boundaries — worth mentioning if you're familiar with recent Prometheus developments. --- ## Common Mistakes With USE and Metric Types | Mistake | Why It's Wrong | Fix | |---|---|---| | Only checking Utilization, skipping Saturation | Misses bursty/queueing behavior that averages-based utilization hides | Always check Saturation-specific metrics (queue depth, wait time), not just %busy | | Applying `rate()` to a gauge | Produces nonsensical output — gauges aren't monotonic | Query gauges directly; only apply `rate()`/`increase()` to counters | | Using a summary for a horizontally-scaled service that needs fleet-wide percentiles | Per-instance quantiles can't be validly combined | Use a histogram instead | | Choosing histogram buckets arbitrarily, not based on real SLO thresholds | Percentile accuracy silently degrades right where it matters most | Pick bucket boundaries around your actual SLO targets | | Treating a single "average latency" panel as sufficient | Hides outliers that dominate real user pain, especially at scale | Always pair with p95/p99 (and p99.9 at high volume) | | Running USE only for CPU/memory, skipping disk/network/app-level resources | Tunnel vision — misses the actual bottleneck if it's elsewhere | Run the full USE checklist systematically, every time | --- ## Worked Practice Problems **Problem 1:** `vmstat 1` shows CPU utilization steady at 45%, but the `r` (run queue) column shows a value of 16 on an 8-core machine. What does this indicate, and is the CPU actually a bottleneck? *Answer:* Yes — despite "only" 45% utilization, a run queue of 16 on 8 cores means, on average, 8 processes are ready to run but have no free core available (16 total runnable, 8 cores can service them concurrently, leaving 8 waiting). This is **saturation without high utilization** — a classic case where looking at utilization alone would wrongly conclude "CPU is fine, there's headroom." The 45% average is likely hiding bursty spikes to 100%+ effective demand. CPU is very much a bottleneck here. **Problem 2:** Your team migrated a service's Duration metric from a Summary to a Histogram ahead of a planned horizontal scale-out from 3 to 40 replicas. Why was this the right call, and what would have broken if they hadn't? *Answer:* Summaries compute quantiles per-instance and can't be aggregated across replicas. At 3 replicas, a team might get away with eyeballing 3 separate p99 lines, but at 40 replicas that becomes unreadable, and worse, any attempt to "average the p99s together" to get a single fleet-wide number would be mathematically invalid — it would neither equal the true fleet-wide p99 nor reliably trend in the same direction as it. Migrating to a histogram lets Prometheus correctly sum bucket counts across all 40 instances and compute one valid, accurate fleet-wide percentile via `histogram_quantile()`. **Problem 3:** A histogram's buckets are `[0.1, 0.5, 1, 5]` seconds. Your actual SLO is "99% of requests under 300ms." What's wrong with this bucket choice, and how would you fix it? *Answer:* There's no bucket boundary near 300ms — the closest boundaries are 100ms and 500ms, a 5x gap that straddles the actual SLO threshold. `histogram_quantile()` linearly interpolates within a bucket, so the estimated p99 near 300ms could be meaningfully inaccurate. Fix: add a bucket boundary at (or very near) 0.3s specifically, e.g., `[0.05, 0.1, 0.2, 0.3, 0.5, 1, 2.5, 5]`, so the percentile calculation has real resolution exactly where the SLO decision boundary is. --- ## Summary and What's Next - **USE** (Utilization, Saturation, Errors) is a systematic, resource-focused checklist — run it top-to-bottom (CPU → memory → disk → network → app-level resources) during incidents to avoid tunnel vision. - Utilization and Saturation are **distinct** — a resource can look fine on average utilization while still saturating during bursts; always check queue-depth/wait-time metrics specifically, not just %busy. - USE generalizes cleanly beyond bare metal to **managed cloud resources** (RDS connections, Kubernetes CPU throttling, Lambda concurrency limits). - The four core **Prometheus metric types** are Counter (monotonic, use `rate()`), Gauge (point-in-time, query directly), Histogram (bucketed, aggregatable, query-time percentiles), and Summary (pre-computed, cheap, but NOT aggregatable across instances). - **Prefer histograms over summaries** for any horizontally-scaled service where you need fleet-wide percentiles. - **Never trust average latency** — always use percentiles, because averages systematically hide the tail-latency outliers that real unhappy users actually experience, and this effect gets *worse*, not better, as scale and fan-out increase (see "The Tail at Scale," Dean & Barroso, 2013). - Histogram **bucket boundaries are a real design decision** — align them with your actual SLO thresholds for accurate percentile estimates exactly where they matter. **Continue to Part 3** (`03-applying-methodologies.md`) to see RED and USE combined in real dashboards, a full worked incident walkthrough, and a head-to-head comparison of all three methodologies.