The Four Golden Signals & the RED Method

Table of Contents#

1. Why Methodologies Matter
2. The Observability Pyramid — Where Methodologies Fit
3. The Four Golden Signals — Origin and Overview
4. Signal 1: Latency
5. Signal 2: Traffic
6. Signal 3: Errors
7. Signal 4: Saturation
8. Utilization vs Saturation — The Classic Trap
9. How the Four Signals Interact During an Incident
10. The RED Method — Origin and Definition
11. RED in Depth: Rate
12. RED in Depth: Errors
13. RED in Depth: Duration
14. Instrumenting RED With Prometheus
15. RED and Service Meshes
16. Building a RED Dashboard — A Full Worked Example
17. What RED Deliberately Leaves Out
18. Common Mistakes With Golden Signals and RED
19. Worked Practice Problems
20. Summary and What's Next

---

Why Methodologies Matter#

Any sufficiently large production system emits thousands of possible metrics — CPU counters, per-endpoint latencies, queue depths, cache hit ratios, garbage collection pauses, connection pool stats, and on and on. Without a framework for deciding what to actually watch, teams fall into one of two failure modes:

Diagram

Monitoring methodologies are checklists that guarantee you're watching the right categories of signal for a given kind of system, so you neither drown in noise nor miss the metric that would have given you a 20-minute head start on an incident.

Diagram

Interview framing you should lead with: these three are not competing — they're complementary lenses at different layers of the stack. A mature observability setup uses RED per service and USE per underlying resource, with Golden Signals as the unifying mental model that ties both into a top-level "is this system healthy" story for stakeholders who don't want to see 200 graphs.

---

The Observability Pyramid — Where Methodologies Fit#

Before diving into each methodology, it helps to place them in the broader observability stack (covered fully in the Observability tutorial, topic 4) — this context prevents confusing "monitoring methodology" with "observability pillar," a common conflation in interviews.

Diagram

Metrics/logs/traces are the raw data. Golden Signals/RED/USE are the methodology for deciding which slices of that data matter. Dashboards are how you look at the chosen slices. Alerting decides when a human needs to be interrupted based on those slices (often tied directly to error-budget burn rate, as covered in the SRE Fundamentals series). This tutorial focuses entirely on the middle layer: what to measure and why.

---

The Four Golden Signals — Origin and Overview#

From Google's Site Reliability Engineering book (Chapter 6, "Monitoring Distributed Systems"): "If you can only measure four metrics of your user-facing system, focus on these four."

Diagram

Why These Four, Specifically?#

The SRE book's reasoning: these four signals collectively answer the questions that matter most to users and to capacity planning, in the fewest possible metrics. Notice the deliberate ordering — Latency and Errors describe what the user is currently experiencing; Traffic describes how much demand exists; Saturation describes how close the system is to falling over, which is predictive rather than descriptive.

Diagram

---

Signal 1: Latency#

The time it takes to service a request.

The Critical Nuance: Split Success From Failure Latency#

This is the single most commonly tested nuance about latency in interviews. If you average success and failure latency together, a fast-failing request (e.g., an instant circuit-breaker rejection that returns in 2ms) will pull your average latency down — making the service look fast while users are actually receiving errors.

Diagram

Rule to state explicitly in an interview: "Always track latency segmented by outcome — success latency and failure latency as separate time series, never blended — because blending hides exactly the failure mode you most need to see."

Latency Should Be a Distribution, Not a Single Number#

Covered in depth in Part 2, but worth flagging here: latency should be instrumented as a histogram so you can query percentiles (p50, p95, p99), not just an average. A single "average latency" number is one of the most misleading metrics in all of monitoring — a handful of very slow outlier requests can be completely invisible in an average while dominating the actual experience of unlucky users.

---

Signal 2: Traffic#

A measure of demand on the system.

Why Traffic Matters Beyond Just "How Busy Are We"#

Traffic is the essential correlating signal for the other three. When errors or latency spike, the very first question is: "did traffic change too?"

Diagram

This single correlation check — "is this a load problem or a change problem?" — is one of the fastest, highest-value triage steps in any incident, and is a great answer to "what's the first thing you check when errors spike?"

---

Signal 3: Errors#

The rate of requests that fail.

Explicit vs Implicit Errors#

Diagram

Interview gotcha to know by name: a service that catches an internal error and returns HTTP 200 with an error message embedded in the JSON body (instead of a proper 5xx) will look perfectly healthy on any dashboard that only counts status codes. This is exactly why synthetic monitoring/canary checks that validate response content, not just status code, matter — and why "errors" as a Golden Signal should be defined at the semantic level (did the user get what they wanted), not just the protocol level (did the HTTP layer report success).

Error Rate as a Ratio, Not a Raw Count#

Always express error rate as a percentage of total traffic, not a raw count — 50 errors means something completely different at 100 requests/sec (50% error rate, severe) versus 1,000,000 requests/sec (0.005% error rate, likely noise). This ties directly back to the SLI framing from the SRE Fundamentals series: errors / valid_events.

---

Signal 4: Saturation#

How "full" your service is — the resource most constrained. Often service-specific: CPU/memory for compute-bound services, available threads/connections for I/O-bound services, queue depth for async systems.

Saturation as a Leading Indicator#

Diagram

This sequence is the single most important argument for saturation-based alerting, and a very strong interview answer to "how would you design proactive alerting instead of purely reactive alerting": if you only alert on Errors, you are, by definition, always finding out after users are already impacted. Alerting on a Saturation trend (e.g., "connection pool usage has been climbing steadily for 10 minutes and is projected to hit 100% in 15 more") lets on-call intervene — scale up, shed load, fail over — before the Latency and Errors signals ever degrade.

Common Saturation Metrics by System Type#

---

Utilization vs Saturation — The Classic Trap#

This distinction is one of the most frequently mis-explained concepts by candidates, so it's worth its own dedicated section.

• Utilization: the percentage of a resource's capacity currently in use, at a point in time (e.g., "CPU is at 80% busy").
• Saturation: the amount of extra, queued work the resource can't currently service — i.e., demand exceeding capacity right now (e.g., "the run queue has 12 processes waiting for a CPU core that's already busy").

Diagram

Why this distinction matters practically: a system running at 100% CPU utilization with zero queueing is often efficient, not broken — it's using every cycle you paid for with no waste. A system at 60% average CPU utilization that periodically saturates during traffic bursts is the more dangerous one, because average-based dashboards will make it look "fine" while users intermittently experience real degradation. Always monitor saturation-specific metrics (queue depth, wait time) in addition to raw utilization percentages — this is a direct, concrete answer if asked "what's a mistake people make when only watching CPU%?"

---

How the Four Signals Interact During an Incident#

A realistic worked incident, showing how the four signals typically evolve together:

Diagram

Notice the cascade timing: Traffic spikes first (t=10) → Saturation climbs a few minutes later as the extra load accumulates (t=12) → Latency starts degrading as queueing kicks in (t=14) → Errors only appear last, once queued requests start timing out (t=16). An alert fired on Saturation at t=12 would give roughly 4 minutes of head start compared to waiting for the Errors signal at t=16 — a concrete, quantified argument for why Saturation-based alerting matters, useful to cite verbatim in an interview.

---

The RED Method — Origin and Definition#

Coined by Tom Wilkie (at Weaveworks, later Grafana Labs) around 2015, specifically for monitoring microservices. RED takes the Golden Signals and specializes them for anything that is fundamentally "a thing that handles requests" — dropping Saturation (which is more naturally a property of the underlying resource the service runs on, not the service's own request-handling logic).

Diagram

Why RED Dropped Saturation#

Tom Wilkie's own reasoning (widely cited): in a microservices world with dozens or hundreds of independently deployed services, you want a uniform, auto-instrumentable dashboard template per service. Rate/Errors/Duration can all be derived from a single generic HTTP middleware wrapping every service identically — no per-service knowledge of "what resource does this service care about" is required. Saturation, by contrast, is inherently resource-specific (CPU for one service, connection pool for another, queue depth for a third) and doesn't generalize the same way — which is exactly why USE (Part 2) exists as the complementary method for that layer.

Diagram

---

RED in Depth: Rate#

Requests per second the service is handling — the "traffic" analog from Golden Signals, but scoped specifically to this one service's inbound request volume.

Copy
# Requests per second, per service, over a 5-minute smoothing window
sum(rate(http_requests_total[5m])) by (service)

Rate is typically broken down further by dimension for triage:

Copy
# Rate broken down by route and method
sum(rate(http_requests_total[5m])) by (service, route, method)

---

RED in Depth: Errors#

Failed requests per second, or as a fraction of total requests.

Copy
# Raw error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)

# Error rate as a percentage of total traffic (the more useful form)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

Design decision worth naming in an interview: should client errors (4xx) count as "errors" for RED purposes? Typically no — a 404 or 400 usually reflects bad client input, not a service-side failure, and counting them as "errors" would falsely implicate the service for problems it didn't cause. Some teams track 4xx separately as a different signal (useful for API misuse detection) without folding it into the RED error rate used for SLOs/alerting.

---

RED in Depth: Duration#

Distribution of request durations — the "latency" analog, always expressed via a histogram to enable percentile queries.

Copy
# p99 latency, using histogram_quantile over a histogram metric
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

Never report Duration as a raw average — see the Latency section above and the full percentile treatment in Part 2. RED's "Duration" is explicitly meant to be read as a distribution (p50/p90/p99), not a single blended number.

---

Instrumenting RED With Prometheus#

A concrete, minimal instrumentation pattern (conceptually language-agnostic, shown as pseudocode middleware):

Copy
# Pseudocode: HTTP middleware instrumenting RED metrics
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["service", "route", "method", "status"]
)
REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "HTTP request duration",
    ["service", "route", "method"],
    buckets=[0.01, 0.05, 0.1, 0.2, 0.5, 1, 2, 5]
)

def middleware(request, next_handler):
    start = now()
    response = next_handler(request)
    duration = now() - start

    REQUEST_COUNT.labels(
        service=SERVICE_NAME,
        route=request.route,
        method=request.method,
        status=response.status_code
    ).inc()

    REQUEST_DURATION.labels(
        service=SERVICE_NAME,
        route=request.route,
        method=request.method
    ).observe(duration)

    return response

Diagram

The key architectural point: because this middleware is generic, every single service in the fleet — regardless of what it actually does internally — automatically gets an identical RED dashboard the moment it adopts the shared middleware/library. This is what makes RED so operationally cheap to roll out organization-wide compared to hand-crafting bespoke dashboards per service.

---

RED and Service Meshes#

In practice, most large microservice fleets don't rely on every team remembering to add RED middleware manually — they get it automatically from a service mesh.

Diagram

In a service mesh like Istio or Linkerd, every pod gets a sidecar proxy that intercepts all inbound/outbound traffic. Because all traffic flows through this proxy, RED metrics (request rate, error rate, duration histograms) are emitted automatically for every service in the mesh, with zero application code changes. This is one of the most commonly cited practical benefits of service meshes in interviews — "what observability benefit does a service mesh give you for free?" → automatic, uniform RED metrics and distributed tracing headers, without instrumenting each service's code individually.

---

Building a RED Dashboard — A Full Worked Example#

A realistic single-service RED dashboard layout (as you'd build in Grafana):

Diagram

Why overlay p50/p95/p99 on the same panel instead of separate ones: it lets you instantly see tail divergence — if p50 is flat but p99 is climbing, that's a strong, immediate visual signal that a subset of requests (maybe hitting a specific slow code path, or a specific unhealthy backend instance) are degrading while the bulk of traffic is unaffected — exactly the kind of pattern an average would hide entirely.

---

What RED Deliberately Leaves Out#

Being able to articulate RED's limitations, not just its definition, is what separates a strong interview answer from a memorized one.

---

Common Mistakes With Golden Signals and RED#

---

Worked Practice Problems#

Problem 1: A service's average latency dashboard shows a flat 180ms all day, but customer complaints about slowness are increasing. What's your first hypothesis, and what would you check?

Answer: The average is likely hiding tail latency growth — a small but growing subset of requests could be getting much slower while the bulk stay fast, keeping the average deceptively flat (especially if slow requests are a small percentage of total volume). I'd immediately pull up the p99 (and p99.9 if traffic volume is high) latency panel instead of the average, and check whether it diverges from p50 — that divergence is the signal the average is hiding.

Problem 2: During an incident, Errors and Latency are both spiking, but Traffic is flat — completely unchanged from baseline. What does this rule in/out as a likely cause?

Answer: This rules out a pure overload/capacity scenario (traffic didn't increase, so it's not "too much demand for the resources we have"). It points toward something that changed independently of load: a recent deploy, a downstream dependency failure, a config change, or a resource that degraded on its own (e.g., a disk filling up, a certificate expiring). The next step is checking the deploy timeline and dependency health, not scaling up capacity.

Problem 3: You're asked to add RED-style monitoring to a Kafka consumer service that doesn't handle HTTP requests at all. How would you adapt the RED method?

Answer: Map the concepts to the async equivalent: "Rate" becomes messages consumed per second; "Errors" becomes failed/DLQ'd message processing rate (as a % of consumed); "Duration" becomes per-message processing time as a histogram (still percentile-based). This shows the pattern generalizes even though the literal HTTP-shaped metrics don't apply — a key sign of understanding the method rather than memorizing the metric names.

---

Summary and What's Next#

• Golden Signals (Latency, Traffic, Errors, Saturation) are the general-purpose starting checklist for any user-facing system — Latency/Errors describe current user experience, Traffic describes demand, Saturation is the leading/predictive indicator.
• Always segment Latency by outcome (success vs failure) and always use percentiles, never averages.
• Saturation climbs before Latency and Errors do — alerting on it buys real lead time before users are impacted; a worked incident timeline shows this can be several minutes of head start.
• Utilization ≠ Saturation — a resource can be fully utilized and healthy, or moderately utilized and still saturated during bursts. Monitor both.
• RED (Rate, Errors, Duration) is Golden Signals specialized for request-driven services, deliberately dropping Saturation because it doesn't generalize the same way across arbitrary services — this is exactly why it pairs with the resource-focused USE method.
• RED's generic, middleware-based instrumentation is why service meshes can auto-generate RED dashboards for every service with zero app code changes.
• RED has real blind spots: resource saturation, business-logic correctness, and anything non-request-shaped (batch/async) — know these limits, don't oversell RED as sufficient on its own.

Continue to Part 2 (02-use-method-and-metrics.md) for the USE method (the resource-side complement to RED), the four Prometheus metric types, and a full treatment of percentiles vs averages.