The Three Pillars & Prometheus

Table of Contents#

1. Monitoring vs Observability — Not the Same Thing
2. The Three Pillars, Explained Simply
3. Metrics — The Pillar of Numbers
4. Logs — The Pillar of Stories
5. Traces — The Pillar of Journeys
6. How the Three Pillars Work Together
7. Structured vs Unstructured Logging
8. Log Levels — Getting Them Right
9. The Cardinality Problem
10. Prometheus — Architecture Overview
11. Pull vs Push — A Deliberate Design Choice
12. PromQL — The Query Language, By Example
13. Service Discovery and Scrape Configs
14. Prometheus's Limitations — Know Them
15. Grafana — Turning Numbers Into Pictures
16. Common Mistakes
17. Worked Practice Problems
18. Summary and What's Next

---

Monitoring vs Observability — Not the Same Thing#

These two words get used interchangeably a lot, but there's a real, useful distinction, and interviewers like to test whether you know it.

Diagram

A simple analogy: monitoring is like a smoke detector — it watches for a specific thing you already know to worry about (smoke) and alerts you. Observability is like being a detective who can walk into a room after something went wrong and, using whatever evidence is available, figure out exactly what happened — even for a situation nobody predicted in advance.

The one-line interview answer: "Monitoring tells you something is wrong. Observability lets you figure out why, even for a problem you never explicitly thought to watch for — by combining metrics, logs, and traces so you can ask new, specific questions after the fact."

---

The Three Pillars, Explained Simply#

Diagram

A quick mental shortcut: metrics tell you something's wrong, logs tell you the details of what happened, traces tell you where in the system it happened.

---

Metrics — The Pillar of Numbers#

A metric is a number, measured repeatedly over time, usually with some labels attached (like which service, which endpoint). Cheap to store, cheap to query, great for dashboards and alerting — but they only tell you the aggregate picture, not the story of any one specific request.

Diagram

Strength: cheap enough to keep for months, fast enough to power real-time dashboards and alerts, and this is the exact material covered in the Monitoring Methodologies series (Golden Signals, RED, USE, counters/gauges/histograms).

Weakness: metrics are aggregated on purpose, which means they throw away detail — you can't ask a metric "show me exactly what happened for user #12345's request."

---

Logs — The Pillar of Stories#

A log is a timestamped record of a specific event, usually with a human-readable (or structured) description of what happened.

2026-06-01T14:02:17Z ERROR checkout-service: payment gateway timeout
  user_id=12345 order_id=98765 gateway_latency_ms=5023 trace_id=abc123

Strength: full detail about one specific thing that happened — the exact error message, the exact user, the exact values involved.

Weakness: expensive to store at high volume (every single request could generate several log lines), and searching through them at scale requires a proper log aggregation system (not just grep-ing a file on one server) — which is exactly what tools like Elasticsearch/OpenSearch, Loki, or Splunk exist for.

---

Traces — The Pillar of Journeys#

A trace follows one single request as it travels through multiple services, showing exactly how long each step took.

Diagram

Strength: shows you EXACTLY where time is going, for one specific slow or failed request — in the example above, it's immediately obvious the Payment Service (180ms out of a 450ms total) is where most of the time went, something an average latency metric alone would never show you.

Weakness: requires every service in the call chain to participate (propagate a shared trace ID) — one un-instrumented service in the middle creates a blind spot in the trace, called a "trace gap." Covered fully in Part 2.

---

How the Three Pillars Work Together#

The real power comes from jumping between pillars during an investigation — this exact workflow is one of the best things to describe when asked "walk me through how you'd debug an incident."

Diagram

The key technical glue that makes step 2 → step 3 possible: the trace_id. If every log line includes the trace ID of the request that generated it, you can jump directly from "this specific trace was slow" to "show me every log line associated with that exact trace" — instead of manually searching through millions of unrelated log lines. This connective tissue is exactly what modern observability platforms (and OpenTelemetry, covered in Part 2) are built around.

---

Structured vs Unstructured Logging#

Diagram

Why this matters practically: an unstructured log line is written for a human eyeball scanning one file. A structured log line (typically JSON) is written for a machine to search, filter, and aggregate across millions of lines instantly — this is what actually makes a log aggregation platform useful at real scale. Nearly every mature engineering org mandates structured logging for exactly this reason.

---

Log Levels — Getting Them Right#

A common, practical mistake worth naming: logging everything at INFO (or worse, ERROR) regardless of actual severity. This makes log volume enormous (expensive) and makes it impossible to filter for what actually matters during an incident — if everything is "important," nothing is. A good rule of thumb: could a human ignore this log line 99% of the time without missing anything critical? If yes, it's probably DEBUG or INFO, not WARN/ERROR.

---

The Cardinality Problem#

This is one of the highest-yield "gotcha" topics in observability interviews — a genuinely common, expensive mistake in real production systems.

Cardinality = the number of unique combinations of label values a metric can have.

Diagram

Why this actually breaks things, not just "uses more disk": every unique combination of label values creates a brand new, separate time series that Prometheus (or whatever your metrics backend is) has to track and store forever (until it ages out). Adding a high-cardinality label like user_id, request_id, email, or a raw, unbounded URL path (with IDs embedded in it, e.g., /orders/98765 instead of /orders/:id) can turn a metric that should have a few hundred time series into one with millions — this can genuinely crash or severely degrade a metrics backend, not just make it slightly slower.

The fix: never put a high-cardinality value in a metric label. If you need to correlate by user_id, that's exactly what logs and traces are for (they're built to handle high-cardinality, per-event detail) — metrics are specifically the aggregated, low-cardinality pillar, by design. This distinction — "logs/traces for high cardinality, metrics for low cardinality" — is one of the best, most concrete answers to "what's the difference between these three pillars" that you can give in an interview.

Diagram

---

Prometheus — Architecture Overview#

Prometheus is the de facto standard open-source metrics system in modern SRE work — understanding its architecture (not just its query language) is a common interview expectation.

Diagram

The Key Components#

---

Pull vs Push — A Deliberate Design Choice#

This is a genuinely common interview question: "why does Prometheus pull metrics instead of having services push them?"

Diagram

The nuanced, complete interview answer: "Prometheus deliberately chose pull, mainly because a failed scrape is itself meaningful signal — 'I couldn't reach this target' is valuable information, not just missing data. Pull also keeps configuration centralized (Prometheus decides what to scrape, rather than every service needing to know where to push to) and makes local testing trivial. The tradeoff is short-lived batch jobs that finish before a scrape cycle would ever catch them — Prometheus handles this specific case with an add-on component called the Pushgateway, where such jobs push their final result once, and Prometheus scrapes the Pushgateway itself."

---

PromQL — The Query Language, By Example#

A working knowledge of real PromQL syntax is expected in most SRE interviews — even if not asked to write it live, being able to read and explain a query is common.

Copy
# 1. Raw counter value (rarely useful on its own)
http_requests_total

# 2. Rate of increase per second, over a 5-minute window
rate(http_requests_total[5m])

# 3. Same, but summed across all instances, grouped by service
sum(rate(http_requests_total[5m])) by (service)

# 4. Error rate as a percentage of total traffic
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

# 5. p99 latency from a histogram
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# 6. Alert-style expression: is error rate above 5% for the last 5 minutes?
(
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  /
  sum(rate(http_requests_total[5m])) by (service)
) > 0.05

Key PromQL Concepts to Know by Name#

---

Service Discovery and Scrape Configs#

In a static environment, you might list scrape targets by hand. In a dynamic environment (autoscaling, Kubernetes), Prometheus needs to automatically discover what to scrape as instances come and go.

Diagram

Why this matters practically: in a Kubernetes environment where pods are constantly being created and destroyed (deploys, autoscaling, restarts), manually maintaining a static list of scrape targets is completely impractical. Prometheus's Kubernetes service discovery integration solves this by continuously watching the Kubernetes API and automatically updating its scrape target list — this is one of the reasons Prometheus became the dominant metrics tool specifically in the Kubernetes ecosystem.

---

Prometheus's Limitations — Know Them#

A senior-level answer doesn't just praise a tool — it knows its real weaknesses too.

Interview-ready line: "Prometheus is excellent for what it's designed for — real-time, short-to-medium-term metrics with a rich query language — but out of the box it's a single node with no built-in HA or long-term storage, so most production setups pair it with something like Thanos or Cortex/Mimir for durability, long retention, and global querying across multiple Prometheus instances."

---

Grafana — Turning Numbers Into Pictures#

Grafana itself doesn't store any data — it's a visualization and dashboarding layer that queries data sources (Prometheus being the most common, but also supports Loki for logs, and many others).

Diagram

Worth knowing: Grafana can also fire its own alerts (Grafana Alerting), overlapping somewhat with Prometheus's own Alertmanager — teams pick one primary alerting path (usually Prometheus + Alertmanager for metric-based alerts) to avoid duplicate/confusing alert routing, a real, practical decision worth mentioning if asked about alerting architecture (covered fully in Part 3).

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A team adds http_requests_total{status, method, route, user_id} to track per-user request counts. A week later, their Prometheus server starts running out of memory and becomes unresponsive. What happened, and how would you fix it?

Answer: Cardinality explosion — user_id can have millions of unique values, and each unique combination of label values creates a brand-new time series Prometheus has to track. With potentially millions of users, this metric alone could be generating millions of time series, overwhelming Prometheus's memory. Fix: remove user_id from the metric label entirely; if per-user analysis is genuinely needed, get it from logs or traces (which are built to handle high-cardinality, per-event data) instead of a metric.

Problem 2: During an incident, a metric-based alert tells you checkout-service p99 latency spiked at 14:02. You want to find out exactly what was slow for one specific affected user's request. What's the fastest path using all three pillars?

Answer: Pull a distributed trace for one of the slow requests around 14:02 (traces are covered in depth in Part 2) — this shows exactly which internal service call took the most time. Then use that trace's trace_id to filter logs specifically down to that one request's log lines across every service it touched, instead of manually searching through unrelated logs. This "metric found it → trace narrowed it → logs explain it" workflow is the standard cross-pillar investigation pattern.

Problem 3: Your team wants to keep 2 years of metric history for capacity planning trend analysis, but your single Prometheus server only retains 15 days locally by default and occasionally runs out of disk. What would you recommend?

Answer: Pair Prometheus with a long-term storage backend like Thanos, Cortex, or Grafana Mimir — these systems let Prometheus continue handling real-time scraping and short-term local queries as usual, while offloading older data to cheaper, more scalable object storage (like S3) for long-term retention and cross-instance/global querying, without needing to grow the local Prometheus disk indefinitely.

---

Summary and What's Next#

• Monitoring watches for problems you already knew to expect; Observability lets you investigate problems you didn't anticipate, by combining metrics, logs, and traces.
• Metrics are cheap, aggregated numbers over time — great for dashboards/alerts, but they discard per-event detail.
• Logs are detailed, timestamped records of specific events — expensive at scale, best used structured (JSON) for reliable searching.
• Traces follow one request's journey across services, showing exactly where time went — but require every service in the chain to participate.
• The real power comes from linking the three pillars together via a shared trace_id, letting you jump from "a metric alert fired" → "look at a trace for the affected request" → "pull exact logs for that trace" in one smooth investigation.
• Cardinality is a critical, often-tested concept: never put high-cardinality data (user IDs, request IDs) in a metric label — that belongs in logs/traces, which are designed to handle it.
• Prometheus uses a deliberate pull model (a failed scrape is itself useful signal, and configuration stays centralized), with PromQL as its query language, and Alertmanager handling alert routing/deduplication separately from the core server.
• Prometheus is not inherently highly available or built for long-term storage out of the box — real production setups typically pair it with Thanos, Cortex, or Mimir.

Continue to Part 2 (02-distributed-tracing-and-opentelemetry.md) for a full deep dive into distributed tracing — spans, context propagation, sampling, and OpenTelemetry, the modern standard tying all three pillars together.