# Observability — Part 1: The Three Pillars & Prometheus > **Series:** Observability (1 of 3) > **Part 1:** This file — Metrics, Logs, Traces & Prometheus > **Part 2:** `02-distributed-tracing-and-opentelemetry.md` — Distributed Tracing & OpenTelemetry > **Part 3:** `03-alerting-design.md` — Alerting Design & Burn-Rate Alerts > **Questions:** `questions.md` ## Table of Contents 1. [Monitoring vs Observability — Not the Same Thing](#monitoring-vs-observability--not-the-same-thing) 2. [The Three Pillars, Explained Simply](#the-three-pillars-explained-simply) 3. [Metrics — The Pillar of Numbers](#metrics--the-pillar-of-numbers) 4. [Logs — The Pillar of Stories](#logs--the-pillar-of-stories) 5. [Traces — The Pillar of Journeys](#traces--the-pillar-of-journeys) 6. [How the Three Pillars Work Together](#how-the-three-pillars-work-together) 7. [Structured vs Unstructured Logging](#structured-vs-unstructured-logging) 8. [Log Levels — Getting Them Right](#log-levels--getting-them-right) 9. [The Cardinality Problem](#the-cardinality-problem) 10. [Prometheus — Architecture Overview](#prometheus--architecture-overview) 11. [Pull vs Push — A Deliberate Design Choice](#pull-vs-push--a-deliberate-design-choice) 12. [PromQL — The Query Language, By Example](#promql--the-query-language-by-example) 13. [Service Discovery and Scrape Configs](#service-discovery-and-scrape-configs) 14. [Prometheus's Limitations — Know Them](#prometheus-limitations--know-them) 15. [Grafana — Turning Numbers Into Pictures](#grafana--turning-numbers-into-pictures) 16. [Common Mistakes](#common-mistakes) 17. [Worked Practice Problems](#worked-practice-problems) 18. [Summary and What's Next](#summary-and-whats-next) --- ## Monitoring vs Observability — Not the Same Thing These two words get used interchangeably a lot, but there's a real, useful distinction, and interviewers like to test whether you know it. ```mermaid graph TD Mon["Monitoring:
watching for problems
you ALREADY knew to
expect"] --> MonEx["Example: 'alert me if
CPU goes above 90%'
— you had to think of
this in advance"] Obs["Observability:
being able to ask NEW
questions about a
problem you DIDN'T
anticipate"] --> ObsEx["Example: 'why did THIS
specific user's checkout
fail at 3:14pm?'
— you couldn't have
pre-built a dashboard
for this exact question"] ``` **A simple analogy:** monitoring is like a smoke detector — it watches for a specific thing you already know to worry about (smoke) and alerts you. Observability is like being a detective who can walk into a room after something went wrong and, using whatever evidence is available, figure out exactly what happened — even for a situation nobody predicted in advance. **The one-line interview answer:** "Monitoring tells you *something* is wrong. Observability lets you figure out *why*, even for a problem you never explicitly thought to watch for — by combining metrics, logs, and traces so you can ask new, specific questions after the fact." --- ## The Three Pillars, Explained Simply ```mermaid graph TD O[Observability] --> M[Metrics] O --> L[Logs] O --> T[Traces] M --> M1["Numbers over time.
'WHAT is happening, in aggregate?'"] L --> L1["Timestamped text records.
'WHAT exactly happened,
in detail, for one event?'"] T --> T1["The path one request took
through many services.
'WHERE did the time go,
for THIS specific request?'"] ``` A quick mental shortcut: **metrics tell you something's wrong, logs tell you the details of what happened, traces tell you where in the system it happened.** --- ## Metrics — The Pillar of Numbers A metric is a number, measured repeatedly over time, usually with some labels attached (like which service, which endpoint). Cheap to store, cheap to query, great for dashboards and alerting — but they only tell you the *aggregate* picture, not the story of any one specific request. ```mermaid graph LR A["http_requests_total{status='500', service='checkout'} = 47"] --> B["Tells you: SOMETHING is
failing, 47 times so far"] B --> C["Does NOT tell you: WHICH
user, WHAT they were doing,
or WHY it failed"] ``` **Strength:** cheap enough to keep for months, fast enough to power real-time dashboards and alerts, and this is the exact material covered in the Monitoring Methodologies series (Golden Signals, RED, USE, counters/gauges/histograms). **Weakness:** metrics are aggregated on purpose, which means they throw away detail — you can't ask a metric "show me exactly what happened for user #12345's request." --- ## Logs — The Pillar of Stories A log is a timestamped record of a specific event, usually with a human-readable (or structured) description of what happened. ``` 2026-06-01T14:02:17Z ERROR checkout-service: payment gateway timeout user_id=12345 order_id=98765 gateway_latency_ms=5023 trace_id=abc123 ``` **Strength:** full detail about one specific thing that happened — the exact error message, the exact user, the exact values involved. **Weakness:** expensive to store at high volume (every single request could generate several log lines), and searching through them at scale requires a proper log aggregation system (not just `grep`-ing a file on one server) — which is exactly what tools like Elasticsearch/OpenSearch, Loki, or Splunk exist for. --- ## Traces — The Pillar of Journeys A trace follows **one single request** as it travels through multiple services, showing exactly how long each step took. ```mermaid gantt dateFormat X axisFormat %Lms title A Trace: One Checkout Request's Journey (milliseconds) section API Gateway Total request :a1, 0, 450 section Auth Service Verify token :a2, 5, 20 section Cart Service Fetch cart items :a3, 25, 80 section Inventory Service Check stock :a4, 105, 150 section Payment Service Process payment (SLOW!) :crit, a5, 255, 180 section Notification Service Send confirmation email :a6, 435, 15 ``` **Strength:** shows you EXACTLY where time is going, for one specific slow or failed request — in the example above, it's immediately obvious the Payment Service (180ms out of a 450ms total) is where most of the time went, something an average latency metric alone would never show you. **Weakness:** requires every service in the call chain to participate (propagate a shared trace ID) — one un-instrumented service in the middle creates a blind spot in the trace, called a "trace gap." Covered fully in Part 2. --- ## How the Three Pillars Work Together The real power comes from **jumping between pillars** during an investigation — this exact workflow is one of the best things to describe when asked "walk me through how you'd debug an incident." ```mermaid flowchart TD A["1. A METRIC alert fires:
checkout-service p99
latency is elevated"] --> B["2. Look at a TRACE for one
of the slow requests —
see exactly which internal
step (e.g. Payment Service
call) is the slow one"] B --> C["3. Pull LOGS specifically
from Payment Service,
filtered by the trace ID
from step 2, to see the
exact error/detail"] C --> D["4. Root cause found:
logs show 'connection pool
exhausted' at the exact
moment the trace showed
the slowdown"] ``` **The key technical glue that makes step 2 → step 3 possible: the `trace_id`.** If every log line includes the trace ID of the request that generated it, you can jump directly from "this specific trace was slow" to "show me every log line associated with that exact trace" — instead of manually searching through millions of unrelated log lines. This connective tissue is exactly what modern observability platforms (and OpenTelemetry, covered in Part 2) are built around. --- ## Structured vs Unstructured Logging ```mermaid graph TD Unstruct["Unstructured log:
'User 12345 failed to
checkout order 98765
after 5023ms'"] --> UProb["❌ Hard to search/filter
reliably at scale — you're
stuck doing fragile text
pattern matching"] Struct["Structured log (JSON):
{user_id: 12345,
order_id: 98765,
latency_ms: 5023,
event: 'checkout_failed'}"] --> SGood["✅ Easy to filter, aggregate,
and query precisely —
'show me all checkout_failed
events where latency_ms > 3000'"] ``` **Why this matters practically:** an unstructured log line is written for a human eyeball scanning one file. A structured log line (typically JSON) is written for a *machine* to search, filter, and aggregate across millions of lines instantly — this is what actually makes a log aggregation platform useful at real scale. Nearly every mature engineering org mandates structured logging for exactly this reason. --- ## Log Levels — Getting Them Right | Level | When to Use | Example | |---|---|---| | **DEBUG** | Fine-grained detail, useful only when actively debugging | "Cache lookup for key X took 2ms" | | **INFO** | Normal, expected events worth recording | "User 12345 logged in" | | **WARN** | Something unexpected happened, but the system recovered | "Retry succeeded after 1 failed attempt" | | **ERROR** | An operation failed and needs attention | "Payment gateway call failed after all retries" | | **FATAL/CRITICAL** | The service itself can't continue running | "Failed to connect to required config store on startup" | **A common, practical mistake worth naming:** logging everything at `INFO` (or worse, `ERROR`) regardless of actual severity. This makes log volume enormous (expensive) and makes it impossible to filter for what actually matters during an incident — if everything is "important," nothing is. A good rule of thumb: **could a human ignore this log line 99% of the time without missing anything critical?** If yes, it's probably DEBUG or INFO, not WARN/ERROR. --- ## The Cardinality Problem This is one of the highest-yield "gotcha" topics in observability interviews — a genuinely common, expensive mistake in real production systems. **Cardinality** = the number of unique combinations of label values a metric can have. ```mermaid graph TD A["Good: http_requests_total{status, method, route}"] --> A1["status: ~5 values
method: ~5 values
route: ~50 values
→ manageable cardinality
(~1,250 combinations)"] B["BAD: http_requests_total{status, method, route, user_id}"] --> B1["user_id: potentially
MILLIONS of unique values
→ cardinality EXPLOSION —
millions of unique time
series created"] ``` **Why this actually breaks things, not just "uses more disk":** every unique combination of label values creates a brand new, separate time series that Prometheus (or whatever your metrics backend is) has to track and store *forever* (until it ages out). Adding a high-cardinality label like `user_id`, `request_id`, `email`, or a raw, unbounded URL path (with IDs embedded in it, e.g., `/orders/98765` instead of `/orders/:id`) can turn a metric that should have a few hundred time series into one with millions — this can genuinely crash or severely degrade a metrics backend, not just make it slightly slower. **The fix:** never put a high-cardinality value in a metric label. If you need to correlate by `user_id`, that's exactly what **logs and traces** are for (they're built to handle high-cardinality, per-event detail) — metrics are specifically the *aggregated, low-cardinality* pillar, by design. **This distinction — "logs/traces for high cardinality, metrics for low cardinality" — is one of the best, most concrete answers to "what's the difference between these three pillars" that you can give in an interview.** ```mermaid graph LR Q["Does this piece of data have
a huge/unbounded number
of possible values?
(user ID, request ID, email)"] -->|Yes| Log["Put it in LOGS or
TRACE attributes,
NOT a metric label"] Q -->|"No (small, fixed set —
status code, region, service name)"| Metric["Safe to use as a
METRIC label"] ``` --- ## Prometheus — Architecture Overview Prometheus is the de facto standard open-source metrics system in modern SRE work — understanding its architecture (not just its query language) is a common interview expectation. ```mermaid graph TD subgraph "Your Services" App1["Service A
exposes /metrics"] App2["Service B
exposes /metrics"] end Prom["Prometheus Server"] -->|"scrapes (pulls) /metrics
every N seconds"| App1 Prom -->|"scrapes"| App2 Prom --> TSDB["Local Time-Series
Database (TSDB)"] Prom --> Rules["Alerting/Recording
Rules Engine"] Rules --> AM["Alertmanager
(routing, grouping,
deduplication)"] AM --> Notify["Notifications:
PagerDuty, Slack, email"] Prom --> Graf["Grafana
(queries Prometheus
for dashboards)"] ``` ### The Key Components | Component | Job | |---|---| | **Exporters / instrumented apps** | Expose a `/metrics` HTTP endpoint with current metric values in a simple text format | | **Prometheus Server** | Periodically "scrapes" (pulls) `/metrics` from every configured target, storing the results | | **TSDB (Time-Series Database)** | Prometheus's own efficient local storage engine for time-stamped metric data | | **PromQL** | The query language used to ask questions of the stored data | | **Alertmanager** | A separate component that receives firing alerts from Prometheus and handles routing, grouping, silencing, and deduplication before sending notifications | | **Grafana** (external) | The most common tool for building dashboards on top of Prometheus data | --- ## Pull vs Push — A Deliberate Design Choice This is a genuinely common interview question: "why does Prometheus pull metrics instead of having services push them?" ```mermaid graph TD Pull["PULL model (Prometheus):
Prometheus reaches OUT
and asks each service
'give me your current
metrics' on a schedule"] --> PullPro["✅ Prometheus can tell
immediately if a target
is unreachable (a failed
scrape IS itself useful
information — 'this service
might be down')
✅ Simpler service discovery —
one central place decides
what to scrape
✅ Easy to test locally —
just curl the /metrics endpoint"] Push["PUSH model (e.g. StatsD):
Each service actively SENDS
its metrics to a central
collector"] --> PushPro["✅ Works well for very
short-lived jobs that might
finish before a scrape
would ever happen (e.g.
a batch job) — this is
Prometheus's own
acknowledged blind spot,
solved via the Pushgateway"] ``` **The nuanced, complete interview answer:** "Prometheus deliberately chose pull, mainly because a failed scrape is itself meaningful signal — 'I couldn't reach this target' is valuable information, not just missing data. Pull also keeps configuration centralized (Prometheus decides what to scrape, rather than every service needing to know where to push to) and makes local testing trivial. The tradeoff is short-lived batch jobs that finish before a scrape cycle would ever catch them — Prometheus handles this specific case with an add-on component called the Pushgateway, where such jobs push their final result once, and Prometheus scrapes the Pushgateway itself." --- ## PromQL — The Query Language, By Example A working knowledge of real PromQL syntax is expected in most SRE interviews — even if not asked to write it live, being able to read and explain a query is common. ```promql # 1. Raw counter value (rarely useful on its own) http_requests_total # 2. Rate of increase per second, over a 5-minute window rate(http_requests_total[5m]) # 3. Same, but summed across all instances, grouped by service sum(rate(http_requests_total[5m])) by (service) # 4. Error rate as a percentage of total traffic sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) # 5. p99 latency from a histogram histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service) ) # 6. Alert-style expression: is error rate above 5% for the last 5 minutes? ( sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) ) > 0.05 ``` ### Key PromQL Concepts to Know by Name | Concept | What It Means | |---|---| | **Instant vector** | A query result with one value per matching time series, at a single point in time | | **Range vector** | A query result with a *range* of values over a time window (e.g., `[5m]`) — usually fed into a function like `rate()` | | **`rate()`** | Per-second average rate of increase for a counter, over the given window — handles counter resets automatically | | **`sum() by (...)`** | Aggregates across multiple time series, grouped by the given labels | | **`histogram_quantile()`** | Computes an approximate percentile from histogram bucket data | --- ## Service Discovery and Scrape Configs In a static environment, you might list scrape targets by hand. In a dynamic environment (autoscaling, Kubernetes), Prometheus needs to automatically discover what to scrape as instances come and go. ```mermaid graph TD K8s["Kubernetes API"] -->|"Prometheus watches for
pods/services with the right
annotations/labels"| Prom["Prometheus
Service Discovery"] Prom --> Targets["Automatically builds and
updates the list of
scrape targets as pods
are created/destroyed"] ``` **Why this matters practically:** in a Kubernetes environment where pods are constantly being created and destroyed (deploys, autoscaling, restarts), manually maintaining a static list of scrape targets is completely impractical. Prometheus's Kubernetes service discovery integration solves this by continuously watching the Kubernetes API and automatically updating its scrape target list — this is one of the reasons Prometheus became the dominant metrics tool specifically in the Kubernetes ecosystem. --- ## Prometheus's Limitations — Know Them A senior-level answer doesn't just praise a tool — it knows its real weaknesses too. | Limitation | Why It's a Real Problem | Common Solution | |---|---|---| | Single-node storage (by default) | A single Prometheus server's local TSDB doesn't scale indefinitely, and isn't itself highly available | Remote-write to a scalable long-term storage backend (e.g., Thanos, Cortex, Mimir), or run Prometheus in a federated/sharded setup | | Not built for long-term retention | Local disk storage isn't ideal for keeping years of historical data | Same solutions as above — Thanos/Cortex/Mimir add long-term, cheaper object storage | | No built-in high availability | If the one Prometheus server dies, you lose metrics collection until it's back | Run 2 identical Prometheus instances scraping the same targets in parallel (simple but wasteful), or use one of the HA-focused storage layers above | | High-cardinality data can overwhelm it | As covered above — this is a design constraint, not a bug | Strict cardinality discipline; some teams add cardinality-limiting proxies in front of ingestion | | Pull model struggles with very short-lived jobs | A batch job that finishes in 2 seconds might never get scraped | Pushgateway (with the caveat that it should be used sparingly, per Prometheus's own documentation) | **Interview-ready line:** "Prometheus is excellent for what it's designed for — real-time, short-to-medium-term metrics with a rich query language — but out of the box it's a single node with no built-in HA or long-term storage, so most production setups pair it with something like Thanos or Cortex/Mimir for durability, long retention, and global querying across multiple Prometheus instances." --- ## Grafana — Turning Numbers Into Pictures Grafana itself doesn't store any data — it's a visualization and dashboarding layer that queries data sources (Prometheus being the most common, but also supports Loki for logs, and many others). ```mermaid graph LR Prom[Prometheus] --> Graf[Grafana] Loki["Loki
(logs)"] --> Graf Graf --> Dash["Dashboards
(panels, alerts,
annotations)"] ``` **Worth knowing:** Grafana can also fire its own alerts (Grafana Alerting), overlapping somewhat with Prometheus's own Alertmanager — teams pick one primary alerting path (usually Prometheus + Alertmanager for metric-based alerts) to avoid duplicate/confusing alert routing, a real, practical decision worth mentioning if asked about alerting architecture (covered fully in Part 3). --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Using `user_id` or `request_id` as a metric label | Causes cardinality explosion — can crash or badly degrade the metrics backend | Keep high-cardinality data in logs/traces, not metric labels | | Logging everything at INFO or ERROR regardless of severity | Makes log volume unmanageable and drowns out what actually matters during an incident | Use log levels deliberately — DEBUG/INFO for routine, WARN/ERROR for things needing attention | | Treating "monitoring" and "observability" as synonyms in an interview answer | Misses the actual distinction interviewers are testing for | Monitoring = watching for known problems; Observability = being able to investigate unknown ones | | Assuming a single Prometheus server is inherently highly available | It's not, by default | Mention the pairing with Thanos/Cortex/Mimir, or dual-scraping, for HA and long retention | | Unstructured, free-text logs at scale | Hard to search/filter reliably; fragile pattern matching | Structured (JSON) logging, filterable by field | | Not propagating a shared trace ID between logs and traces | Loses the ability to jump from "this trace was slow" to "show me the exact logs for it" — the most powerful cross-pillar workflow | Ensure every log line includes the current trace ID | --- ## Worked Practice Problems **Problem 1:** A team adds `http_requests_total{status, method, route, user_id}` to track per-user request counts. A week later, their Prometheus server starts running out of memory and becomes unresponsive. What happened, and how would you fix it? *Answer:* Cardinality explosion — `user_id` can have millions of unique values, and each unique combination of label values creates a brand-new time series Prometheus has to track. With potentially millions of users, this metric alone could be generating millions of time series, overwhelming Prometheus's memory. Fix: remove `user_id` from the metric label entirely; if per-user analysis is genuinely needed, get it from logs or traces (which are built to handle high-cardinality, per-event data) instead of a metric. **Problem 2:** During an incident, a metric-based alert tells you checkout-service p99 latency spiked at 14:02. You want to find out exactly what was slow for one specific affected user's request. What's the fastest path using all three pillars? *Answer:* Pull a distributed trace for one of the slow requests around 14:02 (traces are covered in depth in Part 2) — this shows exactly which internal service call took the most time. Then use that trace's `trace_id` to filter logs specifically down to that one request's log lines across every service it touched, instead of manually searching through unrelated logs. This "metric found it → trace narrowed it → logs explain it" workflow is the standard cross-pillar investigation pattern. **Problem 3:** Your team wants to keep 2 years of metric history for capacity planning trend analysis, but your single Prometheus server only retains 15 days locally by default and occasionally runs out of disk. What would you recommend? *Answer:* Pair Prometheus with a long-term storage backend like Thanos, Cortex, or Grafana Mimir — these systems let Prometheus continue handling real-time scraping and short-term local queries as usual, while offloading older data to cheaper, more scalable object storage (like S3) for long-term retention and cross-instance/global querying, without needing to grow the local Prometheus disk indefinitely. --- ## Summary and What's Next - **Monitoring** watches for problems you already knew to expect; **Observability** lets you investigate problems you didn't anticipate, by combining metrics, logs, and traces. - **Metrics** are cheap, aggregated numbers over time — great for dashboards/alerts, but they discard per-event detail. - **Logs** are detailed, timestamped records of specific events — expensive at scale, best used structured (JSON) for reliable searching. - **Traces** follow one request's journey across services, showing exactly where time went — but require every service in the chain to participate. - The real power comes from **linking the three pillars together** via a shared `trace_id`, letting you jump from "a metric alert fired" → "look at a trace for the affected request" → "pull exact logs for that trace" in one smooth investigation. - **Cardinality** is a critical, often-tested concept: never put high-cardinality data (user IDs, request IDs) in a metric label — that belongs in logs/traces, which are designed to handle it. - **Prometheus** uses a deliberate **pull** model (a failed scrape is itself useful signal, and configuration stays centralized), with **PromQL** as its query language, and **Alertmanager** handling alert routing/deduplication separately from the core server. - Prometheus is **not inherently highly available or built for long-term storage** out of the box — real production setups typically pair it with Thanos, Cortex, or Mimir. **Continue to Part 2** (`02-distributed-tracing-and-opentelemetry.md`) for a full deep dive into distributed tracing — spans, context propagation, sampling, and OpenTelemetry, the modern standard tying all three pillars together.