The USE Method, Metric Types & Percentiles

Table of Contents#

1. The USE Method — Origin and Definition
2. USE in Depth: Utilization
3. USE in Depth: Saturation
4. USE in Depth: Errors
5. USE Applied to Every Major Resource Type
6. Running USE as a Systematic Incident Checklist
7. USE on Linux — The Actual Commands
8. USE for Cloud/Managed Resources
9. Metric Types — The Foundation Under Every Methodology
10. Counters, In Depth
11. Gauges, In Depth
12. Histograms, In Depth
13. Summaries, In Depth
14. Histogram vs Summary — The Interview-Favorite Gotcha
15. Percentiles — Why Averages Lie
16. The Mathematics of Tail Latency at Scale
17. Choosing Histogram Buckets — A Practical Guide
18. Common Mistakes With USE and Metric Types
19. Worked Practice Problems
20. Summary and What's Next

---

The USE Method — Origin and Definition#

Coined by Brendan Gregg (formerly of Sun/Oracle, Netflix, and now Intel — a well-known Linux performance engineer) around 2012, USE is designed for systematically diagnosing resource bottlenecks: CPU, memory, disk, network, and any resource-like abstraction (thread pools, connection pools, file descriptors, locks).

Diagram

Gregg's own framing, worth quoting: "For every resource, check utilization, saturation, and errors." It was explicitly designed as a checklist to run methodically during a live performance investigation, precisely because under incident pressure, engineers tend to jump straight to their favorite/most-familiar metric and miss an obvious bottleneck elsewhere. USE forces completeness.

---

USE in Depth: Utilization#

The percentage of time the resource was busy doing useful work, over a measurement interval.

Utilization = (time resource was busy) / (total elapsed time) × 100%

Important subtlety: utilization is often reported as an average over an interval (e.g., "CPU was 80% busy over the last minute"), which can hide short bursts of 100% utilization within that interval. This is exactly the same "averages hide spikes" problem covered later for latency — it applies to utilization metrics too.

Diagram

---

USE in Depth: Saturation#

The amount of extra work queued that the resource couldn't service immediately — the degree to which demand is exceeding capacity right now.

Saturation examples:
- CPU: run queue length (processes waiting for a core)
- Memory: swap activity, page fault rate
- Disk: I/O queue depth, average wait time in queue
- Network: interface transmit/receive queue drops
- Connection pool: requests blocked waiting for a free connection

As covered in Part 1, saturation is the metric most directly correlated with actual user-visible degradation — a resource can be heavily utilized without being saturated (efficient, no queueing) or moderately utilized while still being saturated during bursts (inefficient/bursty, queueing happens anyway).

---

USE in Depth: Errors#

The count of error events associated with the resource — distinct from application-level errors (which RED already covers). USE errors are specifically about the resource itself failing or degrading.

Diagram

Interview-relevant point: these are often the earliest, most underrated warning signs of impending hardware failure or resource exhaustion — a disk throwing a rising rate of correctable I/O errors, or a host whose OOM-killer has fired twice this week, is telling you something is wrong well before Utilization or Saturation alone would show it clearly. A comprehensive USE check always includes this errors dimension, not just the two "how busy/how backed-up" numbers.

---

USE Applied to Every Major Resource Type#

The full worked table — this is worth memorizing the shape of (three columns, one row per resource), since interviewers frequently ask "walk me through USE for [some resource]" live.

---

Running USE as a Systematic Incident Checklist#

Gregg designed USE to be run top-to-bottom, resource by resource, specifically because it's exhaustive enough to prevent tunnel vision during a stressful live incident.

Diagram

Why this order (CPU → Memory → Disk → Network → App-level) is a reasonable default: it roughly follows "how likely is this to be the bottleneck, and how cheap is it to check" — CPU and memory are checked first because they're fastest to check (top/htop gives you both instantly) and are common bottlenecks; disk and network follow because they require slightly more specialized tools; app-level resources (connection pools, thread pools) come last because they're service-specific and require knowing that particular application's internals.

Strong interview answer template for "how would you debug a slow server with no obvious app-level errors": "I'd run the USE checklist systematically — CPU utilization/saturation/errors, then memory, then disk I/O, then network, then app-level resources like connection pools and thread pools — rather than guessing based on whatever metric I happen to check first. This is specifically designed to prevent tunnel vision under incident pressure."

---

USE on Linux — The Actual Commands#

Being able to name real tools, not just the abstract methodology, signals hands-on experience.

Diagram

This is essentially Brendan Gregg's own "Linux Performance Analysis in 60 Seconds" checklist, which is a commonly referenced real-world artifact worth knowing exists — mentioning it by name is a strong signal in an interview.

---

USE for Cloud/Managed Resources#

USE doesn't only apply to raw Linux boxes — it generalizes cleanly to managed cloud resources, which is an important adaptation to mention since most modern SRE work happens on managed infrastructure, not bare metal.

A great interview line: "USE isn't just a bare-metal/VM concept — it generalizes to any resource with a capacity limit, including managed cloud services. For a Kubernetes pod, for instance, Utilization is CPU/memory usage against its requests/limits, Saturation shows up as CPU throttling, and Errors shows up as OOMKilled restarts."

---

Metric Types — The Foundation Under Every Methodology#

Golden Signals, RED, and USE are all built on top of a small set of underlying metric types — this maps directly onto the Prometheus data model, the de facto standard in modern SRE tooling.

Diagram

---

Counters, In Depth#

A counter only ever increases (or resets to zero, typically on process restart). You never read a counter's raw value directly for monitoring purposes — you always apply rate() (or increase()) to get a meaningful per-second (or per-interval) value.

Copy
# WRONG — raw counter value is meaningless on its own
http_requests_total

# RIGHT — rate of increase per second, over a 5-minute smoothing window
rate(http_requests_total[5m])

Why counters reset, and why that's fine: when a process restarts, its in-memory counter resets to 0. Prometheus's rate() function is specifically designed to detect this reset (a value going down is interpreted as a restart, not negative traffic) and correctly compute the rate across the discontinuity — this is a well-known Prometheus behavior worth knowing exists, even if you don't need to explain the exact internal algorithm.

Typical counter examples: http_requests_total, errors_total, bytes_sent_total.

---

Gauges, In Depth#

A gauge is a point-in-time value that can go up or down freely — it represents "the current state of something," not "the accumulated total of something."

Copy
# Gauges are queried directly — no rate() needed
memory_usage_bytes
queue_depth
active_connections

Typical gauge examples: memory_usage_bytes, queue_depth, temperature_celsius, active_connections, cpu_usage_percent.

Interview trap: applying rate() to a gauge is almost always a mistake — rate() assumes monotonic increase (a counter), so applying it to something that legitimately goes up and down (a gauge) produces nonsensical results. Knowing which metric type you're looking at before choosing your query function is a basic but frequently-tested competency.

---

Histograms, In Depth#

A histogram samples observations (typically request durations, but also response sizes or any other continuous value) into a set of configurable buckets, and exposes:

1. A count per bucket (_bucket{le="0.1"}, _bucket{le="0.5"}, etc. — cumulative, "less than or equal to")
2. A total sum of all observed values (_sum)
3. A total count of observations (_count)

http_request_duration_seconds_bucket{le="0.1"} 240
http_request_duration_seconds_bucket{le="0.5"} 490
http_request_duration_seconds_bucket{le="1.0"} 500
http_request_duration_seconds_bucket{le="+Inf"} 500
http_request_duration_seconds_sum 125.4
http_request_duration_seconds_count 500

Percentiles are computed after the fact, at query time, using histogram_quantile():

Copy
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

The Critical Property: Aggregatability#

Because histogram buckets are just counts, you can sum them across many instances (e.g., all 50 pods of a service) and then compute a valid, statistically meaningful percentile for the whole fleet. This is the single biggest reason histograms are generally preferred over summaries (next section) in a horizontally-scaled environment.

Diagram

---

Summaries, In Depth#

A summary computes quantiles client-side, at instrumentation time, using a streaming algorithm, and exposes the pre-calculated quantile values directly.

http_request_duration_seconds{quantile="0.5"} 0.043
http_request_duration_seconds{quantile="0.9"} 0.187
http_request_duration_seconds{quantile="0.99"} 0.923
http_request_duration_seconds_sum 125.4
http_request_duration_seconds_count 500

Advantages: cheaper to query (no histogram_quantile() math needed at query time — the number is already computed) and doesn't require you to guess good bucket boundaries in advance.

The critical disadvantage: those pre-computed quantiles are per-instance and cannot be validly combined across instances. You cannot average 50 different p99 values from 50 pods and get a meaningful fleet-wide p99 — percentiles simply don't aggregate that way mathematically.

---

Histogram vs Summary — The Interview-Favorite Gotcha#

This exact comparison is one of the highest-yield "gotcha" questions in SRE/observability interviews — expect it in some form almost every time monitoring internals come up.

Diagram

Model interview answer: "For a service running 50 replicas, I'd use a histogram, not a summary — because Prometheus can sum histogram bucket counts across all 50 instances and then compute a mathematically valid cluster-wide p99 with histogram_quantile(). A summary computes p99 independently on each of the 50 pods, and those 50 independent p99 values can't be correctly combined into one true cluster-wide p99 — averaging them, for instance, is not the same number you'd get from the raw combined data."

---

Percentiles — Why Averages Lie#

This is one of the single highest-yield SRE interview topics across the entire monitoring domain — expect a direct question on it in nearly every SRE interview loop.

The Core Problem#

A single very slow outlier request can be completely hidden by many fast ones when averaged.

Diagram

Common Percentiles Used in SRE, and What Each Is For#

---

The Mathematics of Tail Latency at Scale#

Why p99 matters more as scale grows — a concrete, quantified argument worth memorizing:

Diagram

This exact insight — "the tail at scale" — was popularized in a well-known paper by Jeffrey Dean and Luiz André Barroso (Google), "The Tail at Scale" (2013, Communications of the ACM). Being able to cite this by name in an interview is a strong signal of genuine depth beyond memorized definitions.

A Second, Compounding Effect: Fan-Out Amplifies Tail Latency#

If a single user request internally triggers calls to, say, 20 backend microservices in parallel, and the overall response can't return until all 20 have responded, then the overall p99 latency is driven by the slowest of the 20, not any individual service's own p99.

Diagram

If each of 20 independent backend calls has a 1% chance of being "slow" (i.e., that's the definition of each one's own p99), the probability that at least one of the 20 is slow on any given user request is roughly 1 − 0.99²⁰ ≈ 18% — meaning nearly 1 in 5 user requests experiences some tail-latency-affected sub-call, even though each individual service looks perfectly healthy at "only" a 1% tail rate. This is a genuinely advanced, high-signal point to raise when discussing microservice architecture tradeoffs, and it's one of the core motivating arguments in the "Tail at Scale" paper for techniques like hedged requests (send a duplicate request to a second backend if the first hasn't responded within some threshold, and use whichever comes back first).

---

Choosing Histogram Buckets — A Practical Guide#

A real, practical skill often glossed over: histogram buckets must be chosen in advance at instrumentation time, and a poor choice silently degrades your percentile accuracy.

Diagram

Practical guidance:

• Choose bucket boundaries based on your actual SLO thresholds — if your SLO cares about "under 400ms," make sure a bucket boundary sits at or near 400ms so histogram_quantile() can accurately tell you whether you're above or below it.
• Use roughly exponential spacing (e.g., 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s) to get reasonable resolution across a wide dynamic range without an excessive number of buckets.
• Too many buckets increases cardinality and storage/query cost; too few buckets loses percentile accuracy. This is a genuine engineering tradeoff, not a "just add more buckets" situation.
• Some Prometheus client libraries support native histograms (a newer feature) which use dynamically-sized buckets automatically, reducing the need to hand-tune boundaries — worth mentioning if you're familiar with recent Prometheus developments.

---

Common Mistakes With USE and Metric Types#

---

Worked Practice Problems#

Problem 1: vmstat 1 shows CPU utilization steady at 45%, but the r (run queue) column shows a value of 16 on an 8-core machine. What does this indicate, and is the CPU actually a bottleneck?

Answer: Yes — despite "only" 45% utilization, a run queue of 16 on 8 cores means, on average, 8 processes are ready to run but have no free core available (16 total runnable, 8 cores can service them concurrently, leaving 8 waiting). This is saturation without high utilization — a classic case where looking at utilization alone would wrongly conclude "CPU is fine, there's headroom." The 45% average is likely hiding bursty spikes to 100%+ effective demand. CPU is very much a bottleneck here.

Problem 2: Your team migrated a service's Duration metric from a Summary to a Histogram ahead of a planned horizontal scale-out from 3 to 40 replicas. Why was this the right call, and what would have broken if they hadn't?

Answer: Summaries compute quantiles per-instance and can't be aggregated across replicas. At 3 replicas, a team might get away with eyeballing 3 separate p99 lines, but at 40 replicas that becomes unreadable, and worse, any attempt to "average the p99s together" to get a single fleet-wide number would be mathematically invalid — it would neither equal the true fleet-wide p99 nor reliably trend in the same direction as it. Migrating to a histogram lets Prometheus correctly sum bucket counts across all 40 instances and compute one valid, accurate fleet-wide percentile via histogram_quantile().

Problem 3: A histogram's buckets are [0.1, 0.5, 1, 5] seconds. Your actual SLO is "99% of requests under 300ms." What's wrong with this bucket choice, and how would you fix it?

Answer: There's no bucket boundary near 300ms — the closest boundaries are 100ms and 500ms, a 5x gap that straddles the actual SLO threshold. histogram_quantile() linearly interpolates within a bucket, so the estimated p99 near 300ms could be meaningfully inaccurate. Fix: add a bucket boundary at (or very near) 0.3s specifically, e.g., [0.05, 0.1, 0.2, 0.3, 0.5, 1, 2.5, 5], so the percentile calculation has real resolution exactly where the SLO decision boundary is.

---

Summary and What's Next#

• USE (Utilization, Saturation, Errors) is a systematic, resource-focused checklist — run it top-to-bottom (CPU → memory → disk → network → app-level resources) during incidents to avoid tunnel vision.
• Utilization and Saturation are distinct — a resource can look fine on average utilization while still saturating during bursts; always check queue-depth/wait-time metrics specifically, not just %busy.
• USE generalizes cleanly beyond bare metal to managed cloud resources (RDS connections, Kubernetes CPU throttling, Lambda concurrency limits).
• The four core Prometheus metric types are Counter (monotonic, use rate()), Gauge (point-in-time, query directly), Histogram (bucketed, aggregatable, query-time percentiles), and Summary (pre-computed, cheap, but NOT aggregatable across instances).
• Prefer histograms over summaries for any horizontally-scaled service where you need fleet-wide percentiles.
• Never trust average latency — always use percentiles, because averages systematically hide the tail-latency outliers that real unhappy users actually experience, and this effect gets worse, not better, as scale and fan-out increase (see "The Tail at Scale," Dean & Barroso, 2013).
• Histogram bucket boundaries are a real design decision — align them with your actual SLO thresholds for accurate percentile estimates exactly where they matter.

Continue to Part 3 (03-applying-methodologies.md) to see RED and USE combined in real dashboards, a full worked incident walkthrough, and a head-to-head comparison of all three methodologies.