# Observability — Part 2: Distributed Tracing & OpenTelemetry > **Series:** Observability (2 of 3) > **Part 1:** `01-three-pillars-and-prometheus.md` — Metrics, Logs, Traces & Prometheus > **Part 2:** This file — Distributed Tracing & OpenTelemetry > **Part 3:** `03-alerting-design.md` — Alerting Design & Burn-Rate Alerts > **Questions:** `questions.md` ## Table of Contents 1. [Why Distributed Tracing Exists](#why-distributed-tracing-exists) 2. [The Building Blocks: Traces and Spans](#the-building-blocks-traces-and-spans) 3. [Parent-Child Relationships and the Trace Tree](#parent-child-relationships-and-the-trace-tree) 4. [Context Propagation — How the Trace Follows the Request](#context-propagation--how-the-trace-follows-the-request) 5. [Reading a Real Trace Waterfall](#reading-a-real-trace-waterfall) 6. [Span Attributes, Events, and Tags](#span-attributes-events-and-tags) 7. [Sampling — Why You Can't Trace Everything](#sampling--why-you-cant-trace-everything) 8. [Sampling Strategies, Compared](#sampling-strategies-compared) 9. [OpenTelemetry — The Modern Standard](#opentelemetry--the-modern-standard) 10. [The OpenTelemetry Architecture](#the-opentelemetry-architecture) 11. [Instrumentation: Automatic vs Manual](#instrumentation-automatic-vs-manual) 12. [Where Traces Go: Jaeger, Zipkin, and Backends](#where-traces-go-jaeger-zipkin-and-backends) 13. [Trace Gaps — The Silent Blind Spot](#trace-gaps--the-silent-blind-spot) 14. [A Full Worked Trace Investigation](#a-full-worked-trace-investigation) 15. [Common Mistakes](#common-mistakes) 16. [Worked Practice Problems](#worked-practice-problems) 17. [Summary and What's Next](#summary-and-whats-next) --- ## Why Distributed Tracing Exists Picture a single user clicking "checkout" on a website. Behind that one click, a request might bounce through 15 different microservices — auth, cart, inventory, pricing, payment, notifications, and more — before the user finally sees "Order confirmed!" If that click feels slow, **which of those 15 services is actually the slow one?** ```mermaid graph LR User[User clicks Checkout] --> Gateway[API Gateway] Gateway --> Auth[Auth Service] Gateway --> Cart[Cart Service] Cart --> Inventory[Inventory Service] Cart --> Pricing[Pricing Service] Gateway --> Payment[Payment Service] Payment --> Bank["External Bank API
(third party!)"] Gateway --> Notify[Notification Service] Q["❓ Which ONE of these
8 hops is actually slow?"] -.-> Gateway ``` Without tracing, you'd have to guess, or manually cross-reference timestamps across 8 different services' separate logs — slow, painful, and error-prone. **Distributed tracing solves exactly this problem**: it follows one single request across every service it touches and shows you, visually, exactly where the time went. --- ## The Building Blocks: Traces and Spans ```mermaid graph TD Trace["A TRACE = the entire journey
of ONE request, start to finish"] --> Span1["SPAN 1: API Gateway
(the whole request)"] Span1 --> Span2["SPAN 2: Auth check"] Span1 --> Span3["SPAN 3: Cart lookup"] Span3 --> Span4["SPAN 4: Inventory check
(a 'child' of the Cart span)"] ``` - A **trace** is the complete record of one request's end-to-end journey, identified by a single, unique **trace ID** shared by every piece of it. - A **span** is one individual unit of work within that journey — e.g., "the Auth Service checked the token," or "the database ran this specific query." Each span has its own start time, end time (so, a duration), and a unique **span ID**. **Simple analogy:** think of a trace as an entire relay race, and each span as one runner's individual leg of the race. The trace ID is the race itself; each span ID is one specific runner's timed segment. Looking at all the spans together tells you exactly which runner (which service) was slow. --- ## Parent-Child Relationships and the Trace Tree Spans aren't just a flat list — they form a **tree**, showing which operation triggered which other operation. ```mermaid graph TD Root["Root Span: HTTP request
to /checkout (450ms total)"] --> S1["Span: Auth.verify()
(20ms)"] Root --> S2["Span: Cart.getItems()
(80ms)"] S2 --> S3["Span: Inventory.checkStock()
(45ms) — CHILD of Cart span"] Root --> S4["Span: Payment.charge()
(180ms)"] S4 --> S5["Span: BankAPI.call()
(165ms) — CHILD of Payment span,
the REAL slow part"] Root --> S6["Span: Notification.send()
(15ms)"] ``` **Why the tree structure matters:** it tells you not just "what was slow" but "what caused what." In this example, the Payment span looks slow (180ms) — but the tree reveals that almost *all* of that time (165ms) was actually spent waiting on an external Bank API call, which is a **child** of the Payment span. That's a critical distinction: it tells you the bottleneck isn't your own Payment Service's code — it's a third-party dependency, which completely changes what the fix should be (e.g., add a timeout/circuit breaker around that call, rather than trying to "optimize" Payment Service code that was never the problem). --- ## Context Propagation — How the Trace Follows the Request This is the actual mechanical magic that makes distributed tracing work across separate services (separate processes, possibly separate machines) — and it's a favorite, specific interview question. ```mermaid sequenceDiagram participant Gateway as API Gateway participant Cart as Cart Service participant Inventory as Inventory Service Gateway->>Gateway: Generate trace_id = "abc123"
Start root span Gateway->>Cart: HTTP request
Header: traceparent=abc123-span1 Cart->>Cart: Reads traceparent header,
starts a NEW span,
but tags it with the SAME trace_id Cart->>Inventory: HTTP request
Header: traceparent=abc123-span2 Inventory->>Inventory: Same thing — new span,
same trace_id "abc123" ``` **The core mechanism, in plain terms:** when Service A calls Service B, it includes the current trace ID (and its own current span ID, as the "parent") in the outgoing request — typically as an HTTP header. Service B reads that header, starts its own new span, but tags it as belonging to the *same* trace and as a *child* of Service A's span. This is called **context propagation**, and the specific, now-standardized header format for it is called **`traceparent`** (part of the W3C Trace Context standard — worth knowing this exact name). **Why this matters practically:** if even one service in the chain fails to read and forward this header (e.g., because it wasn't instrumented, or a message queue in between doesn't carry headers through), the trace **breaks** at that point — this is called a **trace gap**, covered later in this tutorial, and it's one of the most common real-world tracing headaches. --- ## Reading a Real Trace Waterfall Trace visualization tools (Jaeger, Zipkin, and most APM products) display a trace as a "waterfall" — a horizontal bar chart where each span is a bar, positioned and sized by its start time and duration. ```mermaid gantt dateFormat X axisFormat %Lms title Trace Waterfall: /checkout request (450ms total) section API Gateway Root span (450ms total) :a1, 0, 450 section Auth Auth.verify (20ms) :a2, 5, 20 section Cart Cart.getItems (80ms) :a3, 25, 80 Inventory.checkStock (45ms) :crit, a4, 60, 45 section Payment Payment.charge (180ms) :a5, 255, 180 BankAPI.call (165ms) :crit, a6, 265, 165 section Notify Notify.send (15ms) :a7, 435, 15 ``` **How to read this, in plain terms:** bars further right started later; longer bars took more time; a bar *nested underneath* another (like BankAPI under Payment) is a child operation happening as part of the parent's work. **The single longest bar isn't necessarily the root cause** — you have to look at nested children to find where the time is *actually* being spent, exactly like the Payment/BankAPI example above. --- ## Span Attributes, Events, and Tags A span isn't just "a start time and a duration" — it can carry rich, structured metadata, which is what makes traces so useful for debugging, not just for measuring speed. ```mermaid graph TD Span["Span: Payment.charge()"] --> Attr["Attributes (key-value pairs):
user_id=12345
order_id=98765
payment_method='visa'
amount_usd=49.99"] Span --> Events["Events (timestamped notes
WITHIN the span):
'retry attempt 1' at +50ms
'retry attempt 2' at +120ms"] Span --> Status["Status: OK or ERROR
(with an error message
if it failed)"] ``` **This is exactly where per-request, high-cardinality detail belongs** — recall from Part 1 that metrics should never carry a `user_id` label (cardinality explosion), but a trace span attribute can absolutely carry `user_id=12345`, because traces are specifically designed to handle unique, per-event detail. This distinction ("cardinality-safe data → metrics; unique, per-request detail → traces/logs") is one of the cleanest ways to demonstrate you understand how the three pillars from Part 1 actually divide responsibility. --- ## Sampling — Why You Can't Trace Everything Here's a real, practical problem: if your service handles 1,000,000 requests per second, generating and storing a full, detailed trace for *every single one* would be enormously expensive — both in overhead added to every request, and in storage cost for all that trace data. ```mermaid graph TD A["Trace 100% of requests"] --> A1["❌ Massive storage cost
❌ Real performance overhead
on every single request
❌ Most of it never gets looked at"] B["Trace a SAMPLE of requests"] --> B1["✅ Manageable cost
✅ Still enough data to spot
patterns and debug real issues"] ``` **The tradeoff to state explicitly:** sampling trades **completeness** for **cost/performance**. The real engineering question is *how* to sample smartly, so you don't accidentally throw away the exact trace you needed. --- ## Sampling Strategies, Compared ```mermaid graph TD Sampling[Sampling Strategies] --> Head[Head-Based Sampling] Sampling --> Tail[Tail-Based Sampling] Head --> H1["Decide whether to trace
THIS request at the very
START, before knowing
how it turns out"] Tail --> T1["Collect the FULL trace for
every request temporarily,
THEN decide afterward
whether to keep it"] ``` ### Head-Based Sampling The decision ("will I record this trace?") is made at the very beginning of the request — usually a simple probability, like "trace 1% of all requests, chosen randomly." - **Pro:** cheap and simple — no need to hold onto trace data for requests you'll ultimately discard. - **Con:** completely blind to outcome — a random 1% sample might easily miss the exact slow or failed request you actually cared about, purely by bad luck. ### Tail-Based Sampling Every request's full trace data is temporarily buffered, and the decision to permanently *keep* it is made **after** the request finishes — based on what actually happened (e.g., "keep it if it was slow, or if it errored, or if it's just a normal fast request, keep only a small random sample of those"). ```mermaid flowchart TD Req[Request happens] --> Buffer["Full trace buffered
temporarily (not yet
permanently stored)"] Buffer --> Decide{"How did it turn out?"} Decide -->|"Slow (e.g. > 1s)"| Keep1["✅ Keep — interesting!"] Decide -->|"Errored"| Keep2["✅ Keep — interesting!"] Decide -->|"Fast and successful"| Keep3["Keep only a small
random sample
(e.g. 1%) — 'boring',
mostly discard"] ``` - **Pro:** guarantees you actually capture the interesting cases (errors, slow requests) — exactly the ones you'd most want for debugging. - **Con:** requires more infrastructure — you have to buffer full trace data for *every* request temporarily (even ones you'll ultimately throw away), which costs more memory/complexity than head-based sampling's simpler "decide up front" approach. **A strong, senior-level interview answer:** "I'd generally prefer tail-based sampling for production systems, because it guarantees the traces you actually need for debugging — the slow ones, the failed ones — are the ones that get kept, rather than gambling on a fixed random percentage catching them. The tradeoff is real infrastructure cost (buffering full trace data temporarily for every request before deciding), so for extremely high-throughput systems, some teams use a hybrid: head-based sampling to cut volume early, combined with rules that always keep anything that looks like an error." --- ## OpenTelemetry — The Modern Standard For years, tracing was fragmented — different vendors (and open-source projects like OpenTracing and OpenCensus) each had their own instrumentation APIs, meaning your application code had to be tightly coupled to one specific tracing backend. **OpenTelemetry (often shortened to "OTel")** is the modern, vendor-neutral standard that fixed this, by merging OpenTracing and OpenCensus into one unified project (under the Cloud Native Computing Foundation, the same body that hosts Kubernetes and Prometheus). ```mermaid graph TD Before["BEFORE OpenTelemetry:
app code tightly coupled
to ONE specific vendor's
tracing library"] --> BeforeProb["❌ Switching tracing
backends meant
re-instrumenting
your whole codebase"] After["WITH OpenTelemetry:
app code uses ONE
vendor-neutral API"] --> AfterGood["✅ Switch backends
(Jaeger, Datadog, Honeycomb,
anything) just by changing
CONFIGURATION — no app
code changes needed"] ``` **Why this is such an important, frequently-asked topic:** OpenTelemetry doesn't just cover traces — it's a single, unified standard for **all three pillars** (metrics, logs, and traces), with a shared way of tagging data (`resource attributes` like service name, and per-signal attributes) that makes cross-pillar correlation (exactly the metric → trace → log workflow from Part 1) much easier and more consistent across an entire organization's tech stack, regardless of what specific backend tools different teams happen to use. --- ## The OpenTelemetry Architecture ```mermaid graph TD App["Your Application Code"] --> SDK["OpenTelemetry SDK
(instruments your code,
generates traces/metrics/logs)"] SDK --> Collector["OpenTelemetry Collector
(a separate process —
receives, processes,
batches, exports)"] Collector --> Backend1["Jaeger
(traces)"] Collector --> Backend2["Prometheus
(metrics)"] Collector --> Backend3["Any vendor's backend
(Datadog, Honeycomb, etc.)"] ``` ### The Collector — Why It's a Separate Piece The **OpenTelemetry Collector** is a standalone process that sits between your instrumented application and wherever the data ultimately needs to go. Instead of every single application needing to know exactly how to talk to Jaeger, or Prometheus, or a specific vendor's proprietary API, applications just send data to the Collector using one standard protocol (OTLP — OpenTelemetry Protocol), and the **Collector** is configured (centrally, once) to process that data (batch it, filter it, sample it, add metadata) and forward it to wherever it needs to go. ```mermaid flowchart LR A1[App 1] --> Collector A2[App 2] --> Collector A3[App 3] --> Collector Collector --> Process["Batching, filtering,
tail-based sampling,
PII scrubbing, etc.
— all handled CENTRALLY,
not per-app"] Process --> Out1[Backend A] Process --> Out2[Backend B] ``` **Why this architecture is a good interview talking point:** it decouples "how my application generates telemetry" from "where that telemetry ends up and how it's processed" — meaning an org can change observability vendors, add sampling logic, or scrub sensitive data (like credit card numbers accidentally logged) in **one central place** (the Collector's configuration) instead of needing to change and redeploy every single instrumented application. --- ## Instrumentation: Automatic vs Manual ```mermaid graph TD Instr[Instrumentation] --> Auto["Automatic Instrumentation
(via agent/library that
hooks into common
frameworks — HTTP servers,
DB clients — with ZERO
code changes)"] Instr --> Manual["Manual Instrumentation
(you explicitly add
spans/attributes around
specific business logic
YOU care about)"] ``` - **Automatic instrumentation** gets you a baseline "for free" — every incoming HTTP request, every outgoing database call, automatically becomes a span, with zero code changes, using OpenTelemetry's auto-instrumentation libraries for common frameworks. - **Manual instrumentation** is what you add on top, for the specific business logic that automatic instrumentation can't know is meaningful — e.g., wrapping a specific pricing-calculation function in its own span so you can see exactly how long *that specific business logic* took, separate from the generic "HTTP request" span around it. **Practical guidance worth stating in an interview:** "I'd always start with automatic instrumentation to get broad coverage cheaply and quickly, then add manual spans selectively around business-critical logic where the generic auto-instrumented spans aren't detailed enough to actually explain what's slow." --- ## Where Traces Go: Jaeger, Zipkin, and Backends | Tool | Origin | Notes | |---|---|---| | **Jaeger** | Originally built at Uber, now a CNCF project | One of the most common open-source tracing backends; widely paired with OpenTelemetry | | **Zipkin** | Originally built at Twitter | An earlier, still-used open-source tracing system; predates a lot of the OpenTelemetry consolidation | | **Commercial APM platforms** | Datadog, Honeycomb, New Relic, etc. | Often ingest OpenTelemetry data directly via OTLP, adding their own analysis/UI on top | **A practical interview note:** you don't need deep hands-on experience with every single one of these — knowing the names, roughly what category each falls into (open-source vs. commercial), and that OpenTelemetry lets you swap between them via configuration rather than re-instrumenting your code, covers the vast majority of what gets asked. --- ## Trace Gaps — The Silent Blind Spot A trace is only as complete as the weakest link in the chain of services it passes through. ```mermaid graph LR A[Service A
instrumented ✅] --> B[Service B
instrumented ✅] B --> C["Message Queue
(does NOT forward
trace headers!) ⚠️"] C --> D[Service D
instrumented ✅] D -.->|"Appears as a totally
SEPARATE, unrelated trace —
the connection to A/B is LOST"| Gap["🕳️ TRACE GAP"] ``` **Why this happens in practice, and it's genuinely common:** message queues (Kafka, RabbitMQ, SQS) are a classic source of trace gaps, because the trace context needs to be explicitly carried inside the message payload/headers by the producer, and explicitly read back out by the consumer — this doesn't happen automatically the way it often does for simple HTTP-to-HTTP calls with common auto-instrumentation libraries. Any service that isn't instrumented at all, or any hop where context propagation was implemented incorrectly, breaks the chain at that exact point, and everything downstream of the gap shows up as a disconnected, seemingly-unrelated trace. **Why this matters practically:** if you're investigating a slow request and the trace mysteriously "ends" partway through with no visibility into what happened next, a trace gap (not necessarily "nothing happened after that point") is the first, most likely explanation to check. --- ## A Full Worked Trace Investigation Tying Part 1 and Part 2 together into one realistic story, similar in style to the worked incidents in the Monitoring Methodologies series: ```mermaid sequenceDiagram participant Alert as Metric Alert participant OnCall as On-Call SRE participant Trace as Tracing Backend (Jaeger) participant Logs as Log Backend Alert->>OnCall: "checkout-service p99
latency elevated" OnCall->>Trace: Search traces for
checkout-service, duration > 1s,
in the last 15 minutes Trace-->>OnCall: Shows 40 slow traces —
ALL have a long span
under "Payment.charge" →
"BankAPI.call" OnCall->>OnCall: Root cause hypothesis:
external Bank API is slow OnCall->>Logs: Filter logs by trace_id
from one of the slow traces Logs-->>OnCall: Payment Service logs show:
"Bank API responded in 4.2s
(normally ~150ms)" OnCall->>OnCall: Confirmed: third-party
dependency is degraded,
NOT our own code ``` **This exact investigation shape — metric alert narrows scope, traces pinpoint the slow span, logs confirm the specific detail — is worth having ready to describe fluently, since "walk me through debugging a slow service" is an extremely common SRE interview question, and demonstrating the full three-pillar workflow (from Part 1) in action is a much stronger answer than describing any one pillar in isolation.** --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Assuming 100% trace sampling is always ideal | Massive storage cost and per-request overhead at real production scale | Use head-based or tail-based sampling deliberately | | Using head-based random sampling and expecting to always catch rare errors | A fixed random % can easily miss the specific rare, interesting failure you needed | Prefer tail-based sampling (or a hybrid) when catching rare errors/slow requests matters | | Assuming a trace that "ends abruptly" means nothing happened after that point | It might just be a trace gap — an un-instrumented hop (often a message queue) that failed to propagate context | Check for trace gaps before concluding a service did nothing | | Coupling application code tightly to one specific vendor's tracing SDK | Makes switching observability vendors extremely costly later | Use OpenTelemetry's vendor-neutral API instead | | Treating the longest span in a waterfall as automatically "the root cause" | The longest span might just be a parent whose time is almost entirely explained by one specific child span (e.g., a slow third-party call) | Drill into nested child spans to find where time is actually spent | | Forgetting that message queues need explicit context propagation | Silently creates trace gaps at every queue hop | Ensure producers write trace context into message headers/payload, and consumers read it back out | --- ## Worked Practice Problems **Problem 1:** A trace waterfall shows a root span of 800ms, with one child span labeled "DatabaseQuery" taking 750ms. A teammate concludes "the database is slow, let's add an index." What follow-up question would you ask before agreeing? *Answer:* I'd want to know whether that 750ms is genuinely query *execution* time, or whether it includes something else entirely — like time spent waiting to acquire a database connection from a pool (a classic case from the Reliability & Architecture Patterns series' worked incident). I'd check if the "DatabaseQuery" span itself has further child spans or attributes distinguishing "connection wait time" from "actual query execution time" — because those two have completely different fixes (adding an index doesn't help at all if the real problem is connection pool exhaustion). **Problem 2:** Your team currently uses head-based sampling at a fixed 1%. During a recent incident, almost none of the failed requests were captured in any trace, making root-causing much harder. What would you recommend changing? *Answer:* Move to tail-based sampling (or at minimum a hybrid), specifically so that any request which errors or exceeds a latency threshold is always kept, rather than relying on a flat random 1% chance to happen to catch the interesting, rare failures. This does require additional infrastructure to briefly buffer full trace data for every request before the keep/discard decision is made, but it directly addresses exactly the gap the team just experienced. **Problem 3:** A trace for a checkout request shows the flow going API Gateway → Cart Service → [nothing else, trace ends] — but you know from application logs that Inventory Service was also called and took a noticeable amount of time. What's the most likely explanation, and how would you confirm it? *Answer:* Most likely a trace gap — Cart Service probably calls Inventory Service through a path that isn't properly propagating the `traceparent` context (a common culprit: an internal message queue, or an un-instrumented internal client library). I'd confirm by checking Inventory Service's own logs/traces around that timestamp for a request that logically corresponds to this checkout flow (e.g., matching order ID or user ID, since the trace ID itself won't connect them), and then fix the actual root cause — usually adding or correcting context propagation on whatever specific call path connects Cart Service to Inventory Service. --- ## Summary and What's Next - **Distributed tracing** follows one request across every service it touches, made of a tree of **spans** (individual units of work) all sharing one **trace ID**. - **Context propagation** (via the `traceparent` header, part of the W3C Trace Context standard) is the actual mechanism that stitches spans from separate services into one coherent trace — and it's exactly what breaks, causing a **trace gap**, when a hop (often a message queue) doesn't forward it correctly. - Reading a trace **waterfall** means looking past the longest bar to the nested children underneath it — the real bottleneck is often a child span (e.g., a third-party API call), not the parent operation itself. - You can't trace every request at real scale — **sampling** is necessary. **Head-based** sampling decides upfront (cheap, but can miss rare interesting cases); **tail-based** sampling decides after seeing the outcome (guarantees you catch errors/slow requests, but costs more infrastructure). - **OpenTelemetry** is the modern, vendor-neutral standard unifying metrics, logs, and traces under one API and one protocol (OTLP), routed through a central **Collector** that decouples "how apps generate telemetry" from "where it ends up." - **Automatic instrumentation** gives broad coverage for free; **manual instrumentation** adds detail for the specific business logic that matters most. - The real power of tracing shows up in a full investigation: a **metric alert** narrows down *that* something's wrong, a **trace** narrows down *where* in the system, and **logs** (correlated via the shared trace ID) explain exactly *what* happened. **Continue to Part 3** (`03-alerting-design.md`) to see how all of this — metrics, SLIs from the SRE Fundamentals series, and the burn-rate concept — comes together into a well-designed alerting strategy that pages a human at exactly the right moment, and not before.