{
  "version": 1,
  "id": "distributed-tracing",
  "title": "Distributed Tracing",
  "summary": "Following one request across services with traces and spans, and where tracing beats metrics and logs.",
  "estimatedMinutes": 20,
  "prerequisites": [],
  "source": "courses-notebook SE303_observability_sre 04_distributed_tracing",
  "quiz": [
    {
      "prompt": "OpenTelemetry is:",
      "options": [
        "A Prometheus alternative only",
        "A vendor-neutral standard for traces, metrics, and logs",
        "A Google-only tracing system",
        "A log aggregator"
      ],
      "answer": 1
    },
    {
      "prompt": "You see one slow database span dominating end-to-end latency. The observability pillar that surfaced this is:",
      "options": [
        "Metrics counters",
        "Distributed tracing (spans)",
        "Unstructured logs",
        "Cron alerts"
      ],
      "answer": 1
    },
    {
      "prompt": "To trace a single user request across 5 services, you should:",
      "options": [
        "Grep each service's logs separately",
        "Propagate a correlation/trace ID and use distributed tracing",
        "Increase log verbosity to DEBUG",
        "Rely on host name only"
      ],
      "answer": 1
    }
  ],
  "lesson": "Topics: trace/span model \u00b7 W3C Trace Context \u00b7 in-process tracer \u00b7 sampling strategies\n\n## 1. The Trace / Span Model\n\nA **trace** represents the end-to-end journey of a request through a distributed system.\n\nA **span** is a single unit of work within a trace:\n- Has a `trace_id` (shared across all spans in one request)\n- Has a `span_id` (unique to this operation)\n- Has a `parent_span_id` (nil for the root span)\n- Records `start_time`, `end_time`, `service`, `operation`, `status`, and optional `attributes`\n\n```\nTrace abc123\n\u251c\u2500\u2500 [span-1] api-gateway  POST /checkout     0ms \u2192 210ms\n\u2502   \u251c\u2500\u2500 [span-2] order-svc  create_order    10ms \u2192 100ms\n\u2502   \u2502   \u2514\u2500\u2500 [span-3] postgres  INSERT orders 15ms \u2192 40ms\n\u2502   \u2514\u2500\u2500 [span-4] payment-svc charge_card    110ms \u2192 200ms\n```\n\n```python\nimport time\nimport uuid\nfrom dataclasses import dataclass, field\nfrom typing import Dict, List, Optional, Any\nfrom contextlib import contextmanager\n\n\n@dataclass\nclass Span:\n    trace_id:      str\n    span_id:       str\n    parent_span_id: Optional[str]\n    service:       str\n    operation:     str\n    start_ns:      int\n    end_ns:        int = 0\n    status:        str = \"ok\"       # ok | error\n    attributes:    Dict[str, Any] = field(default_factory=dict)\n\n    @property\n    def duration_ms(self) -> float:\n        return (self.end_ns - self.start_ns) / 1_000_000\n\n    def finish(self) -> None:\n        self.end_ns = time.monotonic_ns()\n\n    def set_error(self, message: str) -> None:\n        self.status = \"error\"\n        self.attributes[\"error.message\"] = message\n\n\n# Create a standalone span\ns = Span(\n    trace_id=\"abc123\",\n    span_id=str(uuid.uuid4())[:8],\n    parent_span_id=None,\n    service=\"api-gateway\",\n    operation=\"POST /checkout\",\n    start_ns=time.monotonic_ns(),\n)\ntime.sleep(0.01)  # simulate 10ms work\ns.finish()\nprint(f\"Span: {s.operation}  [{s.duration_ms:.2f} ms]  status={s.status}\")\n```\n\n## 2. W3C Trace Context\n\nThe [W3C Trace Context](https://www.w3.org/TR/trace-context/) standard defines HTTP headers for context propagation:\n\n```\ntraceparent: 00-{trace_id_hex_32}-{parent_id_hex_16}-{flags}\ntracestate:  vendor1=value1,vendor2=value2\n```\n\nExample:\n```\ntraceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01\n```\n- Version: `00`\n- Trace ID: `4bf92f3577b34da6a3ce929d0e0e4736` (128-bit)\n- Parent Span ID: `00f067aa0ba902b7` (64-bit)\n- Flags: `01` = sampled\n\n```python\nimport secrets\n\n\ndef new_trace_id() -> str:\n    return secrets.token_hex(16)  # 32 hex chars = 128 bits\n\n\ndef new_span_id() -> str:\n    return secrets.token_hex(8)   # 16 hex chars = 64 bits\n\n\ndef make_traceparent(trace_id: str, span_id: str, sampled: bool = True) -> str:\n    flags = \"01\" if sampled else \"00\"\n    return f\"00-{trace_id}-{span_id}-{flags}\"\n\n\ndef parse_traceparent(header: str) -> Dict[str, str]:\n    parts = header.split(\"-\")\n    if len(parts) != 4 or parts[0] != \"00\":\n        raise ValueError(f\"Invalid traceparent: {header}\")\n    return {\n        \"version\":  parts[0],\n        \"trace_id\": parts[1],\n        \"span_id\":  parts[2],\n        \"flags\":    parts[3],\n        \"sampled\":  parts[3] == \"01\",\n    }\n\n\ntid = new_trace_id()\nsid = new_span_id()\nheader = make_traceparent(tid, sid)\nprint(f\"traceparent header: {header}\")\n\nparsed = parse_traceparent(header)\nfor k, v in parsed.items():\n    print(f\"  {k}: {v}\")\n```\n\n## 3. In-Process Tracer Implementation\n\n```python\nclass Tracer:\n    \"\"\"Minimal in-process tracer \u2014 collects spans in memory.\"\"\"\n\n    def __init__(self, service: str):\n        self.service = service\n        self._spans: List[Span] = []\n        self._active_span: Optional[Span] = None\n\n    @contextmanager\n    def start_span(self, operation: str, attributes: Optional[Dict] = None):\n        parent_id = self._active_span.span_id if self._active_span else None\n        trace_id  = self._active_span.trace_id if self._active_span else new_trace_id()\n\n        span = Span(\n            trace_id=trace_id,\n            span_id=new_span_id(),\n            parent_span_id=parent_id,\n            service=self.service,\n            operation=operation,\n            start_ns=time.monotonic_ns(),\n            attributes=attributes or {},\n        )\n\n        previous = self._active_span\n        self._active_span = span\n        try:\n            yield span\n        except Exception as exc:\n            span.set_error(str(exc))\n            raise\n        finally:\n            span.finish()\n            self._spans.append(span)\n            self._active_span = previous\n\n    def dump(self) -> None:\n        print(f\"\\nTrace dump \u2014 service={self.service} ({len(self._spans)} spans)\")\n        for sp in self._spans:\n            indent = \"  \" if sp.parent_span_id else \"\"\n            print(f\"  {indent}{sp.operation:<35} {sp.duration_ms:6.2f} ms  [{sp.status}]\")\n\n\ntracer = Tracer(\"checkout-service\")\n\nwith tracer.start_span(\"POST /checkout\") as root:\n    with tracer.start_span(\"validate_cart\"):\n        time.sleep(0.005)\n    with tracer.start_span(\"db.query orders\"):\n        time.sleep(0.012)\n    with tracer.start_span(\"payment.charge\"):\n        time.sleep(0.020)\n\ntracer.dump()\n```\n\n## 4. Sampling Strategies\n\nTracing every request at high volume is expensive. Sampling reduces cost while preserving signal.\n\n| Strategy | How | Pros | Cons |\n|----------|-----|------|------|\n| **Head-based** | Decide at trace start (e.g. 1%) | Simple, low overhead | May miss rare errors |\n| **Tail-based** | Decide after trace completes | Can keep all errors | Requires buffering full traces |\n| **Adaptive** | Adjust rate based on traffic | Best coverage | Complex |\n| **Error-always** | Always sample if error occurred | Never miss incidents | Slightly complex |\n\n```python\nimport random as _rand\n\n\nclass HeadBasedSampler:\n    \"\"\"Sample a fixed percentage of traces at ingestion time.\"\"\"\n\n    def __init__(self, rate: float = 0.01):\n        self.rate = rate\n\n    def should_sample(self) -> bool:\n        return _rand.random() < self.rate\n\n\nclass TailBasedSampler:\n    \"\"\"Buffer completed traces; decide based on outcome.\"\"\"\n\n    def __init__(self, base_rate: float = 0.01):\n        self.base_rate = base_rate\n\n    def should_keep(self, spans: List[Span]) -> bool:\n        # Always keep traces with errors\n        if any(s.status == \"error\" for s in spans):\n            return True\n        # Always keep slow traces (>500ms root)\n        root = next((s for s in spans if not s.parent_span_id), None)\n        if root and root.duration_ms > 500:\n            return True\n        # Otherwise probabilistic\n        return _rand.random() < self.base_rate\n\n\n# Simulate sampling decisions\n_rand.seed(42)\nhead = HeadBasedSampler(rate=0.05)\nhead_kept = sum(1 for _ in range(10_000) if head.should_sample())\nprint(f\"Head-based sampler (5%): kept {head_kept} / 10000 traces ({head_kept/100:.1f}%)\")\n```\n\n## Key Takeaways\n\n- A **trace** is a DAG of **spans** sharing one `trace_id`\n- Use **W3C traceparent** headers to propagate context across HTTP calls\n- Implement tracing as a **context manager** to guarantee span completion\n- **Tail-based sampling** is more accurate but requires buffering \u2014 prefer it when errors must never be missed\n- Always record `error` status on spans \u2014 it makes root cause analysis instant\n\n## \ud83e\udde0 Quick Quiz\n\n**Q1.** A distributed trace captures:\n- A) CPU usage across all services\n- B) **The full journey of a request through multiple services as spans** \u2713\n- C) Network packet routing\n- D) Database query plans\n\n**Q2.** A span in distributed tracing represents:\n- A) The entire request lifetime\n- B) A network connection\n- C) **A single unit of work (e.g. one service call, one DB query)** \u2713\n- D) An error event\n\n**Q3.** OpenTelemetry provides:\n- A) A proprietary tracing format\n- B) **A vendor-neutral standard for traces, metrics, and logs** \u2713\n- C) Only distributed tracing\n- D) A hosted observability platform\n\n<details><summary>Answers</summary>1-B, 2-C, 3-B</details>\n"
}