Companion question bank for the 3-part tutorial series in this folder: 01-three-pillars-and-prometheus.md, 02-distributed-tracing-and-opentelemetry.md, 03-alerting-design.md.

Answers are kept short and plain — expand out loud using the analogies and diagrams from the tutorials.

---

Part 1 Questions: Three Pillars & Prometheus

Conceptual#

1. What's the difference between monitoring and observability?#

Monitoring watches for problems you already knew to expect — like a smoke detector waiting for smoke. Observability lets you investigate a problem you never specifically planned for, by combining metrics, logs, and traces to ask new questions after the fact, like a detective working a scene.

2. Describe the three pillars of observability in one sentence each.#

Metrics: numbers over time, cheap, great for "is something wrong" at a glance. Logs: detailed, timestamped records of specific events — the "what exactly happened" story. Traces: the path one request took across many services — "where did the time go."

3. What's the single best answer for "what's the difference between metrics and logs," in terms of cardinality?#

Metrics are meant for low-cardinality, aggregated data (a small, fixed set of label values) — cheap to store and query for years. Logs (and traces) are meant for high-cardinality, per-event detail (like a specific user ID or request ID) — more expensive, but designed to handle that kind of uniqueness.

4. What is the cardinality problem, and why is it dangerous, not just "wasteful"?#

Cardinality is the number of unique label-value combinations a metric can have. Adding a high-cardinality label (like user_id) creates a separate time series for every unique value — potentially millions of them — which can genuinely crash or badly degrade a metrics backend like Prometheus, not just make it a bit slower.

5. What's the difference between structured and unstructured logging?#

Unstructured: free text meant for a human eyeball ("User 12345 failed checkout"). Structured: a consistent format like JSON meant for a machine to search and filter precisely ({user_id: 12345, event: "checkout_failed"}). Structured logging is what actually makes a log aggregation system useful at real scale.

6. Name the standard log levels and when to use each.#

DEBUG (fine detail, only useful when actively debugging), INFO (normal, expected events), WARN (something unexpected but the system recovered), ERROR (an operation failed and needs attention), FATAL/CRITICAL (the service itself can't keep running).

7. Why does Prometheus pull metrics instead of having services push them?#

A failed scrape is itself useful information — "I couldn't reach this target" tells you something might be down. Pull also keeps config centralized (Prometheus decides what to scrape) and makes local testing trivial (just curl the endpoint). The tradeoff: very short-lived jobs might finish before a scrape ever happens — solved with an add-on called the Pushgateway.

8. Explain the difference between a counter, a gauge, and a histogram (quick recap).#

Counter: only goes up, use rate() to get a per-second value. Gauge: goes up and down, query directly, never rate() it. Histogram: buckets observations so you can compute percentiles after the fact, and — importantly — sum across many machines to get a valid fleet-wide percentile.

9. Is a single Prometheus server highly available and good for long-term storage out of the box?#

No — it's a single node with local storage by default, no built-in HA. Real production setups typically pair it with something like Thanos, Cortex, or Grafana Mimir for durability, long retention, and querying across multiple Prometheus instances.

Applied / Scenario#

10. A team adds user_id as a metric label and a week later Prometheus starts running out of memory. What happened, and what's the fix?#

Cardinality explosion — millions of possible user_id values means millions of separate time series being tracked. Fix: remove user_id from the metric label; if per-user detail is genuinely needed, get it from logs or traces instead, which are built to handle that kind of high-cardinality data.

11. During an incident, a metric alert fires. What's the fastest path to root cause using all three pillars together?#

Pull a trace for one of the affected requests to see exactly which internal step was slow. Use that trace's ID to filter logs down to only that specific request's log lines, instead of searching through unrelated noise. This "metric found it, trace narrowed it, logs explain it" flow is the standard cross-pillar workflow.

---

Part 2 Questions: Distributed Tracing & OpenTelemetry

Conceptual#

12. What's the difference between a trace and a span?#

A trace is the entire journey of one request, start to finish, identified by one shared trace ID. A span is one individual unit of work within that journey (like "the Auth service checked the token"), with its own start time, duration, and ID. A trace is made up of many spans forming a tree.

13. Why is the "longest span" in a trace waterfall not automatically the root cause?#

Because it might just be a parent span whose time is almost entirely explained by a slow child underneath it — like a "Payment" span that looks slow but turns out to be 90% waiting on a third-party bank API call, which is a completely different fix than optimizing your own Payment code.

14. How does a trace actually stay connected across separate services (context propagation)?#

When Service A calls Service B, it includes the current trace ID (and its own span as "parent") in an outgoing header, typically called traceparent (part of the W3C Trace Context standard). Service B reads it and starts its own span tagged with the same trace ID and marked as a child. If any hop fails to forward this header, the trace breaks at that point — a "trace gap."

15. Why do message queues commonly cause trace gaps?#

Because trace context needs to be explicitly written into the message payload/headers by the producer and explicitly read back out by the consumer — this doesn't happen automatically the way it often does for simple HTTP-to-HTTP calls with common auto-instrumentation. Any queue hop that skips this breaks the trace at that exact point.

16. Why can't you trace 100% of production requests at real scale?#

Storage cost and per-request performance overhead — capturing full detail for every single request at high volume gets expensive fast, and most of it would never even get looked at.

17. Compare head-based and tail-based sampling.#

Head-based: decide whether to keep a trace right at the start, before knowing how it turns out (e.g., a flat random 1%) — cheap, but can easily miss the specific rare, interesting failure by bad luck. Tail-based: buffer the full trace temporarily and decide after seeing the outcome — always keep errors/slow requests, sample the "boring" fast ones lightly — guarantees you catch what you actually need, at the cost of more infrastructure.

18. What is OpenTelemetry, and why does it matter?#

A vendor-neutral, open standard (merging the earlier OpenTracing and OpenCensus projects) for generating metrics, logs, and traces using one shared API. Before it, app code was often tightly coupled to one specific vendor's tracing library — with OpenTelemetry, you can switch observability backends by changing configuration, not by re-instrumenting your whole codebase.

19. What does the OpenTelemetry Collector do, and why is it a separate component?#

It sits between your instrumented apps and wherever telemetry data ultimately needs to go, receiving data over one standard protocol (OTLP) and centrally handling batching, filtering, sampling, and routing to one or more backends. This decouples "how my app generates telemetry" from "where it ends up," so you can change vendors or add processing logic in one central place instead of touching every app.

20. What's the difference between automatic and manual instrumentation?#

Automatic: an agent/library hooks into common frameworks (HTTP servers, DB clients) with zero code changes, giving broad baseline coverage for free. Manual: you explicitly wrap specific business logic you care about in your own spans, for detail automatic instrumentation can't know is meaningful.

Applied / Scenario#

21. A trace shows a root span of 800ms with one child "DatabaseQuery" span taking 750ms. A teammate wants to add an index. What would you check first?#

Whether that 750ms is genuine query execution time or includes time spent waiting to acquire a connection from a pool — those have completely different fixes. I'd check if the span has further breakdown or attributes distinguishing "connection wait" from "query execution" before agreeing an index is the right fix.

22. Your team uses a flat 1% head-based sampling rate, and during a recent incident almost none of the failed requests were captured in any trace. What would you recommend?#

Move to tail-based sampling (or a hybrid) so that any request which errors or is unusually slow is always kept, instead of relying on a random 1% chance to happen to catch it. This requires briefly buffering full trace data before the keep/discard decision, but it directly solves the exact gap experienced.

---

Part 3 Questions: Alerting Design & Burn-Rate Alerts

Conceptual#

23. What's the actual goal of a good alerting system?#

Page a human only when a human genuinely needs to do something right now. If the typical response to a page is "yeah I saw it, nothing to do," that alert shouldn't have paged anyone — too many alerts is just as much a failure as too few.

24. What's the difference between symptom-based and cause-based alerting, and which should you page on?#

Symptom-based: what the user is actually experiencing right now (error rate, latency vs SLO) — always worth investigating. Cause-based: internal system metrics (CPU, memory) that might or might not actually be hurting anyone. Page on symptoms; use cause-based metrics only for diagnosis after a symptom alert has already fired.

25. What is alert fatigue, and why is it a genuinely serious problem, not just an annoyance?#

Too many low-value alerts desensitize the on-call engineer, who starts reflexively ignoring or delaying pages — including the real ones, because a genuinely critical alert looks just like all the noise around it. This directly and measurably extends real outages; it's a real, documented cause of prolonged incidents, not just a comfort issue.

26. List the four properties of a good alert.#

Actionable (a human can do something about it), urgent (needs attention now, not eventually), real (not a flaky false positive), and clear (tells you what's wrong, ideally links a runbook).

27. Why isn't a simple fixed threshold (like "error rate > 1%") good enough for alerting?#

It has two opposite failure modes: too sensitive for brief blips (a 30-second spike during a deploy pages someone for nothing), and too insensitive for a slow leak (a sustained 0.5% error rate never crosses 1%, but can fully drain the error budget over the SLO window if ignored). It's also just an arbitrary number, not tied to your actual reliability target.

28. What is burn rate, and how does it fix the fixed-threshold problem?#

Burn rate = actual error rate ÷ the error rate your SLO actually allows. It's automatically calibrated to your own service's real reliability target instead of an arbitrary number, and it distinguishes a brief small blip from a sustained problem that will genuinely exhaust the budget.

29. Explain the "two-window trick" for burn-rate alerting, and why it's needed.#

Require BOTH a short window (e.g., 5 minutes, for fast detection) AND a long window (e.g., 1 hour, to confirm it's sustained, not just a blip) to simultaneously show a high burn rate before paging. A short window alone is noisy; a long window alone is too slow to detect a genuinely severe outage. Requiring both together gets fast detection with built-in noise filtering.

30. What does a 14.4x burn rate over 1 hour + 5 minutes actually mean, and where does that specific number come from?#

It means the current error rate is 14.4 times what the SLO allows — at that pace, a 28-day error budget would be fully consumed in about 2 days if sustained. The number isn't arbitrary — it's derived from "what fraction of the total budget would this burn rate consume within this specific time window" (Google's published pattern targets roughly 2% of a 28-day budget within a 1-hour window).

31. Why should every alert link to a runbook?#

Without one, every responder has to start investigating completely from scratch, even for a problem the team has seen and solved many times before — wasting critical minutes, especially at 3 AM when memory and judgment are impaired. If the first diagnostic steps are always the same, that's toil that should be automated or at least codified into a linked runbook.

32. What do deduplication, grouping, and silencing each solve?#

Deduplication: the same still-true alert condition shouldn't re-page repeatedly. Grouping: multiple different alerts caused by one root cause (e.g., a database outage triggering errors in five dependent services) get bundled into one notification instead of five separate pages. Silencing: deliberately suppressing alerts during planned, expected disruption (like a maintenance window) so it doesn't generate unnecessary pages.

33. How do you know if your alerting system is actually healthy, as an ongoing practice?#

Track the percentage of pages that were genuinely actionable (the responder actually did something as a result) over time. A declining actionable percentage is a concrete, measurable signal that specific alert rules need tuning — the same "measure the process itself" discipline used for postmortem action items.

Applied / Scenario#

34. On-call was paged 15 times last week, but only 2 pages required real action. What's happening, and what would you check first?#

Classic alert fatigue in the making — a ~13% actionable rate is a clear signal the alerting rules are miscalibrated. I'd check whether these are single-window, threshold-based alerts (likely too sensitive to brief blips) instead of proper multi-window burn-rate alerts, since the two-window trick exists specifically to filter out exactly this kind of short-lived noise.

35. Design a "Critical" burn-rate alert tier conceptually for a service with a 99.95% SLO over 28 days.#

Page if the error rate is at least ~14.4x the SLO's allowed rate (0.05% × 14.4 ≈ 0.72% actual error rate), sustained over both a short window (e.g., 5 minutes) and a longer window (e.g., 1 hour) simultaneously. This should be an immediate, high-urgency page, since at that pace the full 28-day budget would be consumed in roughly 2 days.

36. A database outage causes five different microservices to each fire their own "high error rate" alert within the same minute, generating five separate pages to one on-call engineer. How would you fix this?#

Configure Alertmanager grouping so alerts sharing a common cause (e.g., a shared dependency label, or simply firing within the same short time window across related services) get bundled into a single notification listing all five affected services, rather than five separate, disorienting pages for what is genuinely one incident.

---

Quick-Fire / Rapid Recall#