Distributed Tracing & OpenTelemetry

Table of Contents#

1. Why Distributed Tracing Exists
2. The Building Blocks: Traces and Spans
3. Parent-Child Relationships and the Trace Tree
4. Context Propagation — How the Trace Follows the Request
5. Reading a Real Trace Waterfall
6. Span Attributes, Events, and Tags
7. Sampling — Why You Can't Trace Everything
8. Sampling Strategies, Compared
9. OpenTelemetry — The Modern Standard
10. The OpenTelemetry Architecture
11. Instrumentation: Automatic vs Manual
12. Where Traces Go: Jaeger, Zipkin, and Backends
13. Trace Gaps — The Silent Blind Spot
14. A Full Worked Trace Investigation
15. Common Mistakes
16. Worked Practice Problems
17. Summary and What's Next

---

Why Distributed Tracing Exists#

Picture a single user clicking "checkout" on a website. Behind that one click, a request might bounce through 15 different microservices — auth, cart, inventory, pricing, payment, notifications, and more — before the user finally sees "Order confirmed!"

If that click feels slow, which of those 15 services is actually the slow one?

Diagram

Without tracing, you'd have to guess, or manually cross-reference timestamps across 8 different services' separate logs — slow, painful, and error-prone. Distributed tracing solves exactly this problem: it follows one single request across every service it touches and shows you, visually, exactly where the time went.

---

The Building Blocks: Traces and Spans#

Diagram

• A trace is the complete record of one request's end-to-end journey, identified by a single, unique trace ID shared by every piece of it.
• A span is one individual unit of work within that journey — e.g., "the Auth Service checked the token," or "the database ran this specific query." Each span has its own start time, end time (so, a duration), and a unique span ID.

Simple analogy: think of a trace as an entire relay race, and each span as one runner's individual leg of the race. The trace ID is the race itself; each span ID is one specific runner's timed segment. Looking at all the spans together tells you exactly which runner (which service) was slow.

---

Parent-Child Relationships and the Trace Tree#

Spans aren't just a flat list — they form a tree, showing which operation triggered which other operation.

Diagram

Why the tree structure matters: it tells you not just "what was slow" but "what caused what." In this example, the Payment span looks slow (180ms) — but the tree reveals that almost all of that time (165ms) was actually spent waiting on an external Bank API call, which is a child of the Payment span. That's a critical distinction: it tells you the bottleneck isn't your own Payment Service's code — it's a third-party dependency, which completely changes what the fix should be (e.g., add a timeout/circuit breaker around that call, rather than trying to "optimize" Payment Service code that was never the problem).

---

Context Propagation — How the Trace Follows the Request#

This is the actual mechanical magic that makes distributed tracing work across separate services (separate processes, possibly separate machines) — and it's a favorite, specific interview question.

Diagram

The core mechanism, in plain terms: when Service A calls Service B, it includes the current trace ID (and its own current span ID, as the "parent") in the outgoing request — typically as an HTTP header. Service B reads that header, starts its own new span, but tags it as belonging to the same trace and as a child of Service A's span. This is called context propagation, and the specific, now-standardized header format for it is called traceparent (part of the W3C Trace Context standard — worth knowing this exact name).

Why this matters practically: if even one service in the chain fails to read and forward this header (e.g., because it wasn't instrumented, or a message queue in between doesn't carry headers through), the trace breaks at that point — this is called a trace gap, covered later in this tutorial, and it's one of the most common real-world tracing headaches.

---

Reading a Real Trace Waterfall#

Trace visualization tools (Jaeger, Zipkin, and most APM products) display a trace as a "waterfall" — a horizontal bar chart where each span is a bar, positioned and sized by its start time and duration.

Diagram

How to read this, in plain terms: bars further right started later; longer bars took more time; a bar nested underneath another (like BankAPI under Payment) is a child operation happening as part of the parent's work. The single longest bar isn't necessarily the root cause — you have to look at nested children to find where the time is actually being spent, exactly like the Payment/BankAPI example above.

---

Span Attributes, Events, and Tags#

A span isn't just "a start time and a duration" — it can carry rich, structured metadata, which is what makes traces so useful for debugging, not just for measuring speed.

Diagram

This is exactly where per-request, high-cardinality detail belongs — recall from Part 1 that metrics should never carry a user_id label (cardinality explosion), but a trace span attribute can absolutely carry user_id=12345, because traces are specifically designed to handle unique, per-event detail. This distinction ("cardinality-safe data → metrics; unique, per-request detail → traces/logs") is one of the cleanest ways to demonstrate you understand how the three pillars from Part 1 actually divide responsibility.

---

Sampling — Why You Can't Trace Everything#

Here's a real, practical problem: if your service handles 1,000,000 requests per second, generating and storing a full, detailed trace for every single one would be enormously expensive — both in overhead added to every request, and in storage cost for all that trace data.

Diagram

The tradeoff to state explicitly: sampling trades completeness for cost/performance. The real engineering question is how to sample smartly, so you don't accidentally throw away the exact trace you needed.

---

Sampling Strategies, Compared#

Diagram

Head-Based Sampling#

The decision ("will I record this trace?") is made at the very beginning of the request — usually a simple probability, like "trace 1% of all requests, chosen randomly."

• Pro: cheap and simple — no need to hold onto trace data for requests you'll ultimately discard.
• Con: completely blind to outcome — a random 1% sample might easily miss the exact slow or failed request you actually cared about, purely by bad luck.

Tail-Based Sampling#

Every request's full trace data is temporarily buffered, and the decision to permanently keep it is made after the request finishes — based on what actually happened (e.g., "keep it if it was slow, or if it errored, or if it's just a normal fast request, keep only a small random sample of those").

Diagram

• Pro: guarantees you actually capture the interesting cases (errors, slow requests) — exactly the ones you'd most want for debugging.
• Con: requires more infrastructure — you have to buffer full trace data for every request temporarily (even ones you'll ultimately throw away), which costs more memory/complexity than head-based sampling's simpler "decide up front" approach.

A strong, senior-level interview answer: "I'd generally prefer tail-based sampling for production systems, because it guarantees the traces you actually need for debugging — the slow ones, the failed ones — are the ones that get kept, rather than gambling on a fixed random percentage catching them. The tradeoff is real infrastructure cost (buffering full trace data temporarily for every request before deciding), so for extremely high-throughput systems, some teams use a hybrid: head-based sampling to cut volume early, combined with rules that always keep anything that looks like an error."

---

OpenTelemetry — The Modern Standard#

For years, tracing was fragmented — different vendors (and open-source projects like OpenTracing and OpenCensus) each had their own instrumentation APIs, meaning your application code had to be tightly coupled to one specific tracing backend. OpenTelemetry (often shortened to "OTel") is the modern, vendor-neutral standard that fixed this, by merging OpenTracing and OpenCensus into one unified project (under the Cloud Native Computing Foundation, the same body that hosts Kubernetes and Prometheus).

Diagram

Why this is such an important, frequently-asked topic: OpenTelemetry doesn't just cover traces — it's a single, unified standard for all three pillars (metrics, logs, and traces), with a shared way of tagging data (resource attributes like service name, and per-signal attributes) that makes cross-pillar correlation (exactly the metric → trace → log workflow from Part 1) much easier and more consistent across an entire organization's tech stack, regardless of what specific backend tools different teams happen to use.

---

The OpenTelemetry Architecture#

Diagram

The Collector — Why It's a Separate Piece#

The OpenTelemetry Collector is a standalone process that sits between your instrumented application and wherever the data ultimately needs to go. Instead of every single application needing to know exactly how to talk to Jaeger, or Prometheus, or a specific vendor's proprietary API, applications just send data to the Collector using one standard protocol (OTLP — OpenTelemetry Protocol), and the Collector is configured (centrally, once) to process that data (batch it, filter it, sample it, add metadata) and forward it to wherever it needs to go.

Diagram

Why this architecture is a good interview talking point: it decouples "how my application generates telemetry" from "where that telemetry ends up and how it's processed" — meaning an org can change observability vendors, add sampling logic, or scrub sensitive data (like credit card numbers accidentally logged) in one central place (the Collector's configuration) instead of needing to change and redeploy every single instrumented application.

---

Instrumentation: Automatic vs Manual#

Diagram

• Automatic instrumentation gets you a baseline "for free" — every incoming HTTP request, every outgoing database call, automatically becomes a span, with zero code changes, using OpenTelemetry's auto-instrumentation libraries for common frameworks.
• Manual instrumentation is what you add on top, for the specific business logic that automatic instrumentation can't know is meaningful — e.g., wrapping a specific pricing-calculation function in its own span so you can see exactly how long that specific business logic took, separate from the generic "HTTP request" span around it.

Practical guidance worth stating in an interview: "I'd always start with automatic instrumentation to get broad coverage cheaply and quickly, then add manual spans selectively around business-critical logic where the generic auto-instrumented spans aren't detailed enough to actually explain what's slow."

---

Where Traces Go: Jaeger, Zipkin, and Backends#

A practical interview note: you don't need deep hands-on experience with every single one of these — knowing the names, roughly what category each falls into (open-source vs. commercial), and that OpenTelemetry lets you swap between them via configuration rather than re-instrumenting your code, covers the vast majority of what gets asked.

---

Trace Gaps — The Silent Blind Spot#

A trace is only as complete as the weakest link in the chain of services it passes through.

Diagram

Why this happens in practice, and it's genuinely common: message queues (Kafka, RabbitMQ, SQS) are a classic source of trace gaps, because the trace context needs to be explicitly carried inside the message payload/headers by the producer, and explicitly read back out by the consumer — this doesn't happen automatically the way it often does for simple HTTP-to-HTTP calls with common auto-instrumentation libraries. Any service that isn't instrumented at all, or any hop where context propagation was implemented incorrectly, breaks the chain at that exact point, and everything downstream of the gap shows up as a disconnected, seemingly-unrelated trace.

Why this matters practically: if you're investigating a slow request and the trace mysteriously "ends" partway through with no visibility into what happened next, a trace gap (not necessarily "nothing happened after that point") is the first, most likely explanation to check.

---

A Full Worked Trace Investigation#

Tying Part 1 and Part 2 together into one realistic story, similar in style to the worked incidents in the Monitoring Methodologies series:

Diagram

This exact investigation shape — metric alert narrows scope, traces pinpoint the slow span, logs confirm the specific detail — is worth having ready to describe fluently, since "walk me through debugging a slow service" is an extremely common SRE interview question, and demonstrating the full three-pillar workflow (from Part 1) in action is a much stronger answer than describing any one pillar in isolation.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A trace waterfall shows a root span of 800ms, with one child span labeled "DatabaseQuery" taking 750ms. A teammate concludes "the database is slow, let's add an index." What follow-up question would you ask before agreeing?

Answer: I'd want to know whether that 750ms is genuinely query execution time, or whether it includes something else entirely — like time spent waiting to acquire a database connection from a pool (a classic case from the Reliability & Architecture Patterns series' worked incident). I'd check if the "DatabaseQuery" span itself has further child spans or attributes distinguishing "connection wait time" from "actual query execution time" — because those two have completely different fixes (adding an index doesn't help at all if the real problem is connection pool exhaustion).

Problem 2: Your team currently uses head-based sampling at a fixed 1%. During a recent incident, almost none of the failed requests were captured in any trace, making root-causing much harder. What would you recommend changing?

Answer: Move to tail-based sampling (or at minimum a hybrid), specifically so that any request which errors or exceeds a latency threshold is always kept, rather than relying on a flat random 1% chance to happen to catch the interesting, rare failures. This does require additional infrastructure to briefly buffer full trace data for every request before the keep/discard decision is made, but it directly addresses exactly the gap the team just experienced.

Problem 3: A trace for a checkout request shows the flow going API Gateway → Cart Service → [nothing else, trace ends] — but you know from application logs that Inventory Service was also called and took a noticeable amount of time. What's the most likely explanation, and how would you confirm it?

Answer: Most likely a trace gap — Cart Service probably calls Inventory Service through a path that isn't properly propagating the traceparent context (a common culprit: an internal message queue, or an un-instrumented internal client library). I'd confirm by checking Inventory Service's own logs/traces around that timestamp for a request that logically corresponds to this checkout flow (e.g., matching order ID or user ID, since the trace ID itself won't connect them), and then fix the actual root cause — usually adding or correcting context propagation on whatever specific call path connects Cart Service to Inventory Service.

---

Summary and What's Next#

• Distributed tracing follows one request across every service it touches, made of a tree of spans (individual units of work) all sharing one trace ID.
• Context propagation (via the traceparent header, part of the W3C Trace Context standard) is the actual mechanism that stitches spans from separate services into one coherent trace — and it's exactly what breaks, causing a trace gap, when a hop (often a message queue) doesn't forward it correctly.
• Reading a trace waterfall means looking past the longest bar to the nested children underneath it — the real bottleneck is often a child span (e.g., a third-party API call), not the parent operation itself.
• You can't trace every request at real scale — sampling is necessary. Head-based sampling decides upfront (cheap, but can miss rare interesting cases); tail-based sampling decides after seeing the outcome (guarantees you catch errors/slow requests, but costs more infrastructure).
• OpenTelemetry is the modern, vendor-neutral standard unifying metrics, logs, and traces under one API and one protocol (OTLP), routed through a central Collector that decouples "how apps generate telemetry" from "where it ends up."
• Automatic instrumentation gives broad coverage for free; manual instrumentation adds detail for the specific business logic that matters most.
• The real power of tracing shows up in a full investigation: a metric alert narrows down that something's wrong, a trace narrows down where in the system, and logs (correlated via the shared trace ID) explain exactly what happened.

Continue to Part 3 (03-alerting-design.md) to see how all of this — metrics, SLIs from the SRE Fundamentals series, and the burn-rate concept — comes together into a well-designed alerting strategy that pages a human at exactly the right moment, and not before.