# Monitoring Methodologies — Part 1: The Four Golden Signals & the RED Method

> **Series:** Monitoring Methodologies (1 of 3)
> **Part 1:** This file — Golden Signals + RED Method
> **Part 2:** `02-use-method-and-metrics.md` — USE Method + Metric Types + Percentiles
> **Part 3:** `03-applying-methodologies.md` — Combining methodologies, real dashboards, worked incidents
> **Questions:** `questions.md`

## Table of Contents

1. [Why Methodologies Matter](#why-methodologies-matter)
2. [The Observability Pyramid — Where Methodologies Fit](#the-observability-pyramid--where-methodologies-fit)
3. [The Four Golden Signals — Origin and Overview](#the-four-golden-signals--origin-and-overview)
4. [Signal 1: Latency](#signal-1-latency)
5. [Signal 2: Traffic](#signal-2-traffic)
6. [Signal 3: Errors](#signal-3-errors)
7. [Signal 4: Saturation](#signal-4-saturation)
8. [Utilization vs Saturation — The Classic Trap](#utilization-vs-saturation--the-classic-trap)
9. [How the Four Signals Interact During an Incident](#how-the-four-signals-interact-during-an-incident)
10. [The RED Method — Origin and Definition](#the-red-method--origin-and-definition)
11. [RED in Depth: Rate](#red-in-depth-rate)
12. [RED in Depth: Errors](#red-in-depth-errors)
13. [RED in Depth: Duration](#red-in-depth-duration)
14. [Instrumenting RED With Prometheus](#instrumenting-red-with-prometheus)
15. [RED and Service Meshes](#red-and-service-meshes)
16. [Building a RED Dashboard — A Full Worked Example](#building-a-red-dashboard--a-full-worked-example)
17. [What RED Deliberately Leaves Out](#what-red-deliberately-leaves-out)
18. [Common Mistakes With Golden Signals and RED](#common-mistakes-with-golden-signals-and-red)
19. [Worked Practice Problems](#worked-practice-problems)
20. [Summary and What's Next](#summary-and-whats-next)

---

## Why Methodologies Matter

Any sufficiently large production system emits *thousands* of possible metrics — CPU counters, per-endpoint latencies, queue depths, cache hit ratios, garbage collection pauses, connection pool stats, and on and on. Without a framework for deciding what to actually watch, teams fall into one of two failure modes:

```mermaid
graph TD
    A[No monitoring methodology] --> B["Failure Mode 1:<br/>Dashboard sprawl<br/>200 graphs, nobody looks at any of them"]
    A --> C["Failure Mode 2:<br/>Under-monitoring<br/>Only 'is it up' is tracked,<br/>miss the leading indicator that<br/>predicted the outage"]
```

Monitoring methodologies are **checklists that guarantee you're watching the right categories of signal** for a given kind of system, so you neither drown in noise nor miss the metric that would have given you a 20-minute head start on an incident.

```mermaid
graph TD
    M[Monitoring Methodologies] --> GS["Golden Signals<br/>(Google SRE book, 2016)"]
    M --> RED["RED Method<br/>(Tom Wilkie, ~2015)"]
    M --> USE["USE Method<br/>(Brendan Gregg, ~2012)"]

    GS --> GSuse["General-purpose starting point<br/>for ANY user-facing service"]
    RED --> REDuse["Specialized for request-driven<br/>services (APIs, microservices)"]
    USE --> USEuse["Specialized for resource-driven<br/>components (CPU, disk, queues)"]
```

**Interview framing you should lead with:** these three are not competing — they're complementary lenses at different layers of the stack. A mature observability setup uses **RED per service** and **USE per underlying resource**, with **Golden Signals** as the unifying mental model that ties both into a top-level "is this system healthy" story for stakeholders who don't want to see 200 graphs.

---

## The Observability Pyramid — Where Methodologies Fit

Before diving into each methodology, it helps to place them in the broader observability stack (covered fully in the Observability tutorial, topic 4) — this context prevents confusing "monitoring methodology" with "observability pillar," a common conflation in interviews.

```mermaid
graph TD
    Data["Raw Telemetry:<br/>Metrics, Logs, Traces"] --> Method["Monitoring Methodologies<br/>(WHAT to measure):<br/>Golden Signals / RED / USE"]
    Method --> Dash["Dashboards<br/>(HOW to visualize it)"]
    Dash --> Alert["Alerting<br/>(WHEN to page a human)"]
    Alert --> Response["Incident Response<br/>(WHAT to do about it)"]
```

Metrics/logs/traces are the raw *data*. Golden Signals/RED/USE are the *methodology* for deciding which slices of that data matter. Dashboards are how you *look* at the chosen slices. Alerting decides *when* a human needs to be interrupted based on those slices (often tied directly to error-budget burn rate, as covered in the SRE Fundamentals series). This tutorial focuses entirely on the middle layer: **what to measure and why**.

---

## The Four Golden Signals — Origin and Overview

From Google's *Site Reliability Engineering* book (Chapter 6, "Monitoring Distributed Systems"): "If you can only measure four metrics of your user-facing system, focus on these four."

```mermaid
graph LR
    GS[Four Golden Signals] --> L[Latency]
    GS --> T[Traffic]
    GS --> E[Errors]
    GS --> S[Saturation]

    L --> L1["Time to service a request<br/>(split success vs failure latency!)"]
    T --> T1["Demand on the system<br/>e.g. requests/sec, sessions, bandwidth"]
    E --> E1["Rate of failed requests —<br/>explicit (5xx) + implicit (wrong content)"]
    S --> S1["How 'full' the system is —<br/>CPU, memory, queue depth, connection pool"]
```

### Why These Four, Specifically?

The SRE book's reasoning: these four signals collectively answer the questions that matter most to **users** and to **capacity planning**, in the fewest possible metrics. Notice the deliberate ordering — Latency and Errors describe **what the user is currently experiencing**; Traffic describes **how much demand exists**; Saturation describes **how close the system is to falling over**, which is predictive rather than descriptive.

```mermaid
graph TD
    Q1["What is the user experiencing right now?"] --> Latency
    Q1 --> Errors
    Q2["How much demand is there?"] --> Traffic
    Q3["How close are we to falling over?"] --> Saturation
```

---

## Signal 1: Latency

The time it takes to service a request.

### The Critical Nuance: Split Success From Failure Latency

This is the single most commonly tested nuance about latency in interviews. If you average success and failure latency together, a fast-failing request (e.g., an instant circuit-breaker rejection that returns in 2ms) will **pull your average latency down** — making the service look fast while users are actually receiving errors.

```mermaid
graph TD
    A["100 requests: 90 succeed at 200ms,<br/>10 fail instantly at 2ms (circuit breaker)"] --> B["Blended average latency:<br/>≈180.2ms — looks OK!"]
    A --> C["Success-only latency: 200ms<br/>Error rate: 10% — the REAL story"]
    B -.->|"Hides the real problem"| X["❌ Misleading"]
    C -.->|"Surfaces both dimensions separately"| Y["✅ Actionable"]
```

**Rule to state explicitly in an interview:** "Always track latency segmented by outcome — success latency and failure latency as separate time series, never blended — because blending hides exactly the failure mode you most need to see."

### Latency Should Be a Distribution, Not a Single Number

Covered in depth in Part 2, but worth flagging here: latency should be instrumented as a **histogram** so you can query percentiles (p50, p95, p99), not just an average. A single "average latency" number is one of the most misleading metrics in all of monitoring — a handful of very slow outlier requests can be completely invisible in an average while dominating the actual experience of unlucky users.

---

## Signal 2: Traffic

A measure of demand on the system.

| System Type | Typical Traffic Metric |
|---|---|
| Web service / API | HTTP requests per second |
| Database | Queries/transactions per second |
| Streaming/video service | Concurrent sessions, bandwidth (Mbps/Gbps) |
| Message queue | Messages published/consumed per second |
| Batch pipeline | Jobs scheduled/started per hour |

### Why Traffic Matters Beyond Just "How Busy Are We"

Traffic is the essential **correlating signal** for the other three. When errors or latency spike, the very first question is: **"did traffic change too?"**

```mermaid
flowchart TD
    Spike[Error rate spike detected] --> Q{Did traffic also spike?}
    Q -->|Yes, traffic spiked too| A["Likely cause: overload —<br/>system couldn't handle the surge<br/>→ check Saturation next"]
    Q -->|No, traffic was flat/normal| B["Likely cause: something changed<br/>independently of load —<br/>a bad deploy, a dependency failure,<br/>a config change → check recent deploys"]
```

This single correlation check — "is this a load problem or a change problem?" — is one of the fastest, highest-value triage steps in any incident, and is a great answer to "what's the first thing you check when errors spike?"

---

## Signal 3: Errors

The rate of requests that fail.

### Explicit vs Implicit Errors

```mermaid
graph TD
    Errors[Types of Errors] --> Explicit[Explicit Errors]
    Errors --> Implicit[Implicit Errors]

    Explicit --> E1["HTTP 5xx status codes"]
    Explicit --> E2["Unhandled exceptions / stack traces"]
    Explicit --> E3["Non-zero process exit codes"]

    Implicit --> I1["HTTP 200 but wrong/error content<br/>(a classic gotcha — status code lies)"]
    Implicit --> I2["Response violates a policy<br/>e.g. 30s response when SLA promises 2s"]
    Implicit --> I3["Silent data corruption —<br/>'succeeded' but produced wrong result"]
```

**Interview gotcha to know by name:** a service that catches an internal error and returns `HTTP 200` with an error message embedded in the JSON body (instead of a proper 5xx) will look perfectly healthy on any dashboard that only counts status codes. This is exactly why synthetic monitoring/canary checks that validate *response content*, not just status code, matter — and why "errors" as a Golden Signal should be defined at the *semantic* level (did the user get what they wanted), not just the *protocol* level (did the HTTP layer report success).

### Error Rate as a Ratio, Not a Raw Count

Always express error rate as a **percentage of total traffic**, not a raw count — 50 errors means something completely different at 100 requests/sec (50% error rate, severe) versus 1,000,000 requests/sec (0.005% error rate, likely noise). This ties directly back to the SLI framing from the SRE Fundamentals series: `errors / valid_events`.

---

## Signal 4: Saturation

How "full" your service is — the resource most constrained. Often service-specific: CPU/memory for compute-bound services, available threads/connections for I/O-bound services, queue depth for async systems.

### Saturation as a Leading Indicator

```mermaid
sequenceDiagram
    participant Load as Incoming Load
    participant Sat as Saturation (e.g. CPU%, queue depth)
    participant Lat as Latency
    participant Err as Errors

    Load->>Sat: Traffic increases
    Sat->>Sat: Climbs toward capacity limit
    Note over Sat: 🟡 Early warning zone —<br/>nothing user-visible YET
    Sat->>Lat: Queueing begins, requests wait longer
    Note over Lat: 🟠 Users start noticing slowness
    Lat->>Err: Timeouts start triggering hard failures
    Note over Err: 🔴 Users now see outright errors
```

**This sequence is the single most important argument for saturation-based alerting**, and a very strong interview answer to "how would you design proactive alerting instead of purely reactive alerting": if you only alert on Errors, you are, by definition, always finding out *after* users are already impacted. Alerting on a Saturation trend (e.g., "connection pool usage has been climbing steadily for 10 minutes and is projected to hit 100% in 15 more") lets on-call intervene — scale up, shed load, fail over — **before** the Latency and Errors signals ever degrade.

### Common Saturation Metrics by System Type

| System | Saturation Metric |
|---|---|
| Web server / API | Thread pool utilization, active connection count vs. max |
| Database | Connection pool usage, active query count vs. max_connections |
| Message queue | Consumer lag, queue depth relative to processing rate |
| Compute node | CPU run queue length (not just %busy — see next section) |
| Memory-bound service | Available heap headroom, GC pause frequency/duration trending up |

---

## Utilization vs Saturation — The Classic Trap

This distinction is one of the most frequently mis-explained concepts by candidates, so it's worth its own dedicated section.

- **Utilization**: the percentage of a resource's capacity currently in use, at a point in time (e.g., "CPU is at 80% busy").
- **Saturation**: the amount of *extra, queued* work the resource can't currently service — i.e., demand exceeding capacity right now (e.g., "the run queue has 12 processes waiting for a CPU core that's already busy").

```mermaid
graph TD
    A["Resource at 100% utilization,<br/>steady, predictable load,<br/>no queue building"] --> A1["✅ Healthy — fully utilized,<br/>NOT saturated"]
    B["Resource at 60% utilization,<br/>but bursty/spiky load causes<br/>periodic queue buildup"] --> B1["⚠️ Can still be saturated<br/>during the bursts, even though<br/>average utilization looks moderate"]
```

**Why this distinction matters practically:** a system running at 100% CPU utilization with zero queueing is often *efficient*, not broken — it's using every cycle you paid for with no waste. A system at 60% average CPU utilization that periodically saturates during traffic bursts is the more dangerous one, because average-based dashboards will make it look "fine" while users intermittently experience real degradation. **Always monitor saturation-specific metrics (queue depth, wait time) in addition to raw utilization percentages** — this is a direct, concrete answer if asked "what's a mistake people make when only watching CPU%?"

---

## How the Four Signals Interact During an Incident

A realistic worked incident, showing how the four signals typically evolve together:

```mermaid
gantt
    dateFormat X
    axisFormat %M min
    title Signal Evolution During a Traffic-Spike Incident (minutes from start)
    section Traffic
    Steady baseline           :a1, 0, 10
    Traffic spikes 5x         :crit, a2, 10, 5
    section Saturation
    Healthy (30% pool usage)  :b1, 0, 12
    Climbing toward 100%      :crit, b2, 12, 4
    section Latency
    Normal p99 (150ms)        :c1, 0, 14
    p99 climbing (queueing)   :crit, c2, 14, 3
    section Errors
    Near-zero error rate      :d1, 0, 16
    Timeouts spike             :crit, d2, 16, 2
```

Notice the **cascade timing**: Traffic spikes first (t=10) → Saturation climbs a few minutes later as the extra load accumulates (t=12) → Latency starts degrading as queueing kicks in (t=14) → Errors only appear last, once queued requests start timing out (t=16). **An alert fired on Saturation at t=12 would give roughly 4 minutes of head start compared to waiting for the Errors signal at t=16** — a concrete, quantified argument for why Saturation-based alerting matters, useful to cite verbatim in an interview.

---

## The RED Method — Origin and Definition

Coined by **Tom Wilkie** (at Weaveworks, later Grafana Labs) around 2015, specifically for monitoring **microservices**. RED takes the Golden Signals and specializes them for anything that is fundamentally "a thing that handles requests" — dropping Saturation (which is more naturally a property of the underlying *resource* the service runs on, not the service's own request-handling logic).

```mermaid
graph TD
    RED[RED Method] --> R[Rate]
    RED --> E[Errors]
    RED --> D[Duration]

    R --> R1["Requests per second<br/>the service is handling"]
    E --> E1["Failed requests per second,<br/>or as a % of total"]
    D --> D1["Distribution of request durations —<br/>ALWAYS as a histogram/percentiles,<br/>never a bare average"]
```

### Why RED Dropped Saturation

Tom Wilkie's own reasoning (widely cited): in a microservices world with dozens or hundreds of independently deployed services, you want a **uniform, auto-instrumentable dashboard template per service**. Rate/Errors/Duration can all be derived from a single generic HTTP middleware wrapping every service identically — no per-service knowledge of "what resource does this service care about" is required. Saturation, by contrast, is inherently resource-specific (CPU for one service, connection pool for another, queue depth for a third) and doesn't generalize the same way — which is exactly why USE (Part 2) exists as the complementary method for that layer.

```mermaid
graph LR
    A["Golden Signals<br/>(4 signals, one per service,<br/>needs resource-specific knowledge)"] -->|"Wilkie's specialization<br/>for microservices"| B["RED<br/>(3 signals, fully generic,<br/>auto-instrumentable via middleware)"]
```

---

## RED in Depth: Rate

Requests per second the service is handling — the "traffic" analog from Golden Signals, but scoped specifically to *this one service's* inbound request volume.

```promql
# Requests per second, per service, over a 5-minute smoothing window
sum(rate(http_requests_total[5m])) by (service)
```

Rate is typically broken down further by dimension for triage:

```promql
# Rate broken down by route and method
sum(rate(http_requests_total[5m])) by (service, route, method)
```

---

## RED in Depth: Errors

Failed requests per second, or as a fraction of total requests.

```promql
# Raw error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)

# Error rate as a percentage of total traffic (the more useful form)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
```

**Design decision worth naming in an interview:** should client errors (4xx) count as "errors" for RED purposes? Typically **no** — a 404 or 400 usually reflects bad client input, not a service-side failure, and counting them as "errors" would falsely implicate the service for problems it didn't cause. Some teams track 4xx separately as a *different* signal (useful for API misuse detection) without folding it into the RED error rate used for SLOs/alerting.

---

## RED in Depth: Duration

Distribution of request durations — the "latency" analog, always expressed via a histogram to enable percentile queries.

```promql
# p99 latency, using histogram_quantile over a histogram metric
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)
```

**Never report Duration as a raw average** — see the Latency section above and the full percentile treatment in Part 2. RED's "Duration" is explicitly meant to be read as a distribution (p50/p90/p99), not a single blended number.

---

## Instrumenting RED With Prometheus

A concrete, minimal instrumentation pattern (conceptually language-agnostic, shown as pseudocode middleware):

```python
# Pseudocode: HTTP middleware instrumenting RED metrics
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["service", "route", "method", "status"]
)
REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "HTTP request duration",
    ["service", "route", "method"],
    buckets=[0.01, 0.05, 0.1, 0.2, 0.5, 1, 2, 5]
)

def middleware(request, next_handler):
    start = now()
    response = next_handler(request)
    duration = now() - start

    REQUEST_COUNT.labels(
        service=SERVICE_NAME,
        route=request.route,
        method=request.method,
        status=response.status_code
    ).inc()

    REQUEST_DURATION.labels(
        service=SERVICE_NAME,
        route=request.route,
        method=request.method
    ).observe(duration)

    return response
```

```mermaid
flowchart LR
    Req[Incoming Request] --> Mid[Middleware/Instrumentation<br/>wraps every route uniformly]
    Mid --> Handler[Actual route handler logic]
    Handler --> Mid
    Mid --> Counter["http_requests_total{service,route,method,status}"]
    Mid --> Hist["http_request_duration_seconds_bucket{service,route,method,le}"]
    Counter --> RateQ[Rate query]
    Counter --> ErrQ[Errors query]
    Hist --> DurQ[Duration/percentile query]
```

The key architectural point: **because this middleware is generic**, every single service in the fleet — regardless of what it actually does internally — automatically gets an identical RED dashboard the moment it adopts the shared middleware/library. This is what makes RED so operationally cheap to roll out organization-wide compared to hand-crafting bespoke dashboards per service.

---

## RED and Service Meshes

In practice, most large microservice fleets don't rely on every team remembering to add RED middleware manually — they get it **automatically from a service mesh**.

```mermaid
graph TD
    subgraph "Pod A"
    App1[Application Container] <--> Sidecar1[Envoy/Linkerd-proxy Sidecar]
    end
    subgraph "Pod B"
    App2[Application Container] <--> Sidecar2[Envoy/Linkerd-proxy Sidecar]
    end

    Sidecar1 <-->|"mTLS + RED metrics<br/>emitted automatically"| Sidecar2
    Sidecar1 --> Metrics[Prometheus scrapes<br/>sidecar-emitted RED metrics]
    Sidecar2 --> Metrics
```

In a service mesh like **Istio** or **Linkerd**, every pod gets a sidecar proxy that intercepts all inbound/outbound traffic. Because *all* traffic flows through this proxy, RED metrics (request rate, error rate, duration histograms) are emitted **automatically for every service in the mesh, with zero application code changes**. This is one of the most commonly cited practical benefits of service meshes in interviews — "what observability benefit does a service mesh give you for free?" → automatic, uniform RED metrics and distributed tracing headers, without instrumenting each service's code individually.

---

## Building a RED Dashboard — A Full Worked Example

A realistic single-service RED dashboard layout (as you'd build in Grafana):

```mermaid
graph TD
    Dash["checkout-service RED Dashboard"] --> Row1["Row 1: Rate"]
    Dash --> Row2["Row 2: Errors"]
    Dash --> Row3["Row 3: Duration"]

    Row1 --> R1a["Panel: req/sec, split by route"]
    Row1 --> R1b["Panel: req/sec, split by status class (2xx/4xx/5xx)"]

    Row2 --> R2a["Panel: error rate % over time"]
    Row2 --> R2b["Panel: error rate by route (find the worst offender)"]

    Row3 --> R3a["Panel: p50/p95/p99 latency lines, overlaid"]
    Row3 --> R3b["Panel: latency heatmap (full distribution over time)"]
```

**Why overlay p50/p95/p99 on the same panel instead of separate ones:** it lets you instantly see *tail divergence* — if p50 is flat but p99 is climbing, that's a strong, immediate visual signal that a subset of requests (maybe hitting a specific slow code path, or a specific unhealthy backend instance) are degrading while the bulk of traffic is unaffected — exactly the kind of pattern an average would hide entirely.

---

## What RED Deliberately Leaves Out

Being able to articulate RED's *limitations*, not just its definition, is what separates a strong interview answer from a memorized one.

| What RED Doesn't Cover | Why It Matters | What Covers It Instead |
|---|---|---|
| Resource saturation (CPU, memory, disk, connection pools) | A service can have perfect RED metrics while the node it runs on is about to fall over | USE method (Part 2) |
| Business-level correctness (was the *data* right, not just the HTTP status) | A 200 response with a wrong total charged to a customer looks perfectly healthy in RED | Custom business-logic SLIs / semantic monitoring |
| Anything about asynchronous/non-request-driven work (batch jobs, queue consumers) | RED assumes a request/response shape; a Kafka consumer or nightly batch job doesn't fit that shape | Different metric shapes — consumer lag, job completion/freshness SLIs |
| Dependency health (is a downstream service healthy) | RED for Service A only tells you about Service A's own request handling, not why it's slow (which might be a downstream call) | Distributed tracing; RED applied *per-dependency call*, not just per-inbound-request |

---

## Common Mistakes With Golden Signals and RED

| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Blending success and failure latency into one number | Hides the exact failure mode (fast failures pull the average down) | Segment latency by outcome, always |
| Reporting latency as a bare average | Hides tail outliers that real unhappy users experience | Use histograms + percentiles (p50/p95/p99) |
| Counting 4xx client errors the same as 5xx service errors | Falsely implicates the service for client-caused problems | Track 4xx separately; only 5xx (or semantic failures) drive SLOs |
| Treating status-code-200 as automatically "success" | Misses implicit errors (200 + wrong content) | Validate response semantics in synthetic checks, not just status codes |
| Only building RED dashboards, no USE dashboards | Misses resource bottlenecks entirely — RED tells you *where*, not *why* | Pair RED (per service) with USE (per resource) — see Part 2/3 |
| Not correlating Traffic with Errors/Latency during triage | Slower root-causing — miss the "was this a load problem or a change problem" split | Always check Traffic first when Errors/Latency spike |

---

## Worked Practice Problems

**Problem 1:** A service's average latency dashboard shows a flat 180ms all day, but customer complaints about slowness are increasing. What's your first hypothesis, and what would you check?

*Answer:* The average is likely hiding tail latency growth — a small but growing subset of requests could be getting much slower while the bulk stay fast, keeping the average deceptively flat (especially if slow requests are a small percentage of total volume). I'd immediately pull up the p99 (and p99.9 if traffic volume is high) latency panel instead of the average, and check whether it diverges from p50 — that divergence is the signal the average is hiding.

**Problem 2:** During an incident, Errors and Latency are both spiking, but Traffic is flat — completely unchanged from baseline. What does this rule in/out as a likely cause?

*Answer:* This rules out a pure overload/capacity scenario (traffic didn't increase, so it's not "too much demand for the resources we have"). It points toward something that changed independently of load: a recent deploy, a downstream dependency failure, a config change, or a resource that degraded on its own (e.g., a disk filling up, a certificate expiring). The next step is checking the deploy timeline and dependency health, not scaling up capacity.

**Problem 3:** You're asked to add RED-style monitoring to a Kafka consumer service that doesn't handle HTTP requests at all. How would you adapt the RED method?

*Answer:* Map the concepts to the async equivalent: "Rate" becomes messages consumed per second; "Errors" becomes failed/DLQ'd message processing rate (as a % of consumed); "Duration" becomes per-message processing time as a histogram (still percentile-based). This shows the *pattern* generalizes even though the literal HTTP-shaped metrics don't apply — a key sign of understanding the method rather than memorizing the metric names.

---

## Summary and What's Next

- **Golden Signals** (Latency, Traffic, Errors, Saturation) are the general-purpose starting checklist for any user-facing system — Latency/Errors describe current user experience, Traffic describes demand, Saturation is the leading/predictive indicator.
- Always segment **Latency by outcome** (success vs failure) and always use **percentiles, never averages**.
- **Saturation** climbs before Latency and Errors do — alerting on it buys real lead time before users are impacted; a worked incident timeline shows this can be several minutes of head start.
- **Utilization ≠ Saturation** — a resource can be fully utilized and healthy, or moderately utilized and still saturated during bursts. Monitor both.
- **RED** (Rate, Errors, Duration) is Golden Signals specialized for request-driven services, deliberately dropping Saturation because it doesn't generalize the same way across arbitrary services — this is exactly why it pairs with the resource-focused USE method.
- RED's generic, middleware-based instrumentation is why **service meshes can auto-generate RED dashboards for every service with zero app code changes**.
- RED has real blind spots: resource saturation, business-logic correctness, and anything non-request-shaped (batch/async) — know these limits, don't oversell RED as sufficient on its own.

**Continue to Part 2** (`02-use-method-and-metrics.md`) for the USE method (the resource-side complement to RED), the four Prometheus metric types, and a full treatment of percentiles vs averages.
