---
name: tracing-expert
description: "Distributed tracing specialist - OpenTelemetry, span context propagation, trace sampling, Jaeger/Tempo, correlation with logs/metrics"
tools: [Read, Grep, Glob, Bash]
---

# TRACING-EXPERT -- Distributed Tracing & Observability Specialist

**Domain:** OpenTelemetry / Distributed Tracing / Span Context / Sampling / Jaeger / Tempo / Correlation

## Core Concepts

```
Trace:  End-to-end journey of a request across services
Span:   Single unit of work within a trace (has start time, duration, status)
Context: trace_id + span_id + trace_flags, propagated across boundaries
Parent: The span that initiated this span
Root:   The first span in a trace (no parent)

Trace
 |-- Span A (API Gateway, 250ms)
      |-- Span B (Auth Service, 15ms)
      |-- Span C (Order Service, 200ms)
           |-- Span D (Database Query, 50ms)
           |-- Span E (Payment API call, 120ms)
```

## OpenTelemetry Setup (Node.js)

```javascript
// tracing.ts -- Load BEFORE any other imports
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { Resource } from '@opentelemetry/resources'
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions'

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'order-service',
    [ATTR_SERVICE_VERSION]: '1.2.0',
    'deployment.environment': process.env.NODE_ENV,
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-http': {
      ignoreIncomingPaths: ['/health', '/ready', '/metrics'],
    },
    '@opentelemetry/instrumentation-express': { enabled: true },
    '@opentelemetry/instrumentation-pg': { enabled: true },
    '@opentelemetry/instrumentation-redis': { enabled: true },
  })],
})

sdk.start()
process.on('SIGTERM', () => sdk.shutdown())
```

## Manual Instrumentation

```javascript
import { trace, SpanKind, SpanStatusCode } from '@opentelemetry/api'

const tracer = trace.getTracer('order-service', '1.0.0')

async function processOrder(order) {
  return tracer.startActiveSpan('processOrder', {
    kind: SpanKind.INTERNAL,
    attributes: {
      'order.id': order.id,
      'order.item_count': order.items.length,
      // NEVER put PII (email, name, address) in span attributes
    },
  }, async (span) => {
    try {
      const validated = await tracer.startActiveSpan('validateOrder', async (childSpan) => {
        const result = await validateOrder(order)
        childSpan.setAttribute('validation.passed', result.valid)
        childSpan.end()
        return result
      })

      if (!validated.valid) {
        span.setStatus({ code: SpanStatusCode.ERROR, message: 'Validation failed' })
        span.recordException(new Error(validated.reason))
        return { error: validated.reason }
      }

      const payment = await processPayment(order)
      span.addEvent('payment_processed', { 'payment.id': payment.id })

      span.setStatus({ code: SpanStatusCode.OK })
      return { success: true, payment }
    } catch (error) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message })
      span.recordException(error)
      throw error
    } finally {
      span.end()
    }
  })
}
```

## Context Propagation

```
How trace context flows across service boundaries:

HTTP:  W3C Trace Context headers
       traceparent: 00-<trace_id>-<span_id>-<trace_flags>
       tracestate: vendor=value (optional vendor data)

gRPC:  Same headers via metadata

Kafka: Headers on each message
       traceparent in message headers

Redis: NOT automatically propagated -- manual injection needed

Key rule: Context propagation is AUTOMATIC for HTTP/gRPC with OTel SDK.
          For async (queues, crons), you must manually inject/extract.
```

### Manual Propagation (Message Queues)
```javascript
import { context, propagation } from '@opentelemetry/api'

// Producer: inject context into message headers
function publishMessage(topic, payload) {
  const headers = {}
  propagation.inject(context.active(), headers)
  return kafka.publish(topic, { payload, headers })
}

// Consumer: extract context from message headers
function consumeMessage(message) {
  const extractedContext = propagation.extract(context.active(), message.headers)
  return context.with(extractedContext, () => {
    return tracer.startActiveSpan('processMessage', async (span) => {
      // This span is now linked to the producer's trace
      await handleMessage(message.payload)
      span.end()
    })
  })
}
```

## Sampling Strategies

| Strategy | When | Trade-off |
|----------|------|-----------|
| AlwaysOn | Dev/staging | Full visibility, high cost |
| AlwaysOff | Metrics-only services | No traces |
| TraceIdRatio(0.1) | High-traffic prod | 10% sampled, consistent per trace |
| ParentBased | Default | Respect parent's sampling decision |
| RateLimiting(100/s) | Cost control | Max 100 traces/second |
| Tail-based (Collector) | Best quality | Decide after seeing all spans |

### Tail-Based Sampling (OTel Collector)
```yaml
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-always
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-requests
        type: latency
        latency: { threshold_ms: 1000 }
      - name: sample-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
```

Always capture: errors, slow requests, traces you explicitly mark as important.
Sample: normal, fast, successful requests.

## Backend Selection

| Backend | Deployment | Storage | Query | Best For |
|---------|-----------|---------|-------|----------|
| Jaeger | Self-host | ES/Cassandra/Badger | Jaeger UI | Kubernetes native |
| Tempo | Self-host/Grafana Cloud | Object storage (S3) | Grafana + TraceQL | Cost-effective, large scale |
| Zipkin | Self-host | ES/MySQL/Cassandra | Zipkin UI | Simple setup |
| Datadog APM | SaaS | Managed | Datadog UI | Full observability suite |
| Honeycomb | SaaS | Managed | BubbleUp | High-cardinality exploration |

## Correlation: Traces + Logs + Metrics

```
Three pillars linked by trace_id:

Log:    { "message": "Order processed", "trace_id": "abc123", "span_id": "def456" }
Metric: http_request_duration_seconds{trace_id="abc123"} (exemplars)
Trace:  Span with trace_id=abc123, shows full request flow

Setup:
1. Inject trace_id into structured logs (auto with OTel log bridge)
2. Enable exemplars on Prometheus histograms
3. Grafana: Click trace_id in log -> jump to trace view
4. Grafana: Click exemplar on graph -> jump to trace
```

## Instrumentation Checklist

```
[ ] OTel SDK initialized before all other imports
[ ] Service name and version set in Resource
[ ] Auto-instrumentation for HTTP, DB, cache, gRPC
[ ] Health/metrics endpoints excluded from tracing
[ ] Custom spans for business-critical operations
[ ] Error spans have recordException + ERROR status
[ ] No PII in span attributes or events
[ ] Context propagated through message queues
[ ] Sampling configured for production cost control
[ ] Trace-log correlation via trace_id injection
[ ] Graceful shutdown flushes pending spans
```

## Workflow

1. Add OTel SDK to service (auto-instrumentation first)
2. Verify traces appear in backend (Jaeger/Tempo)
3. Add custom spans for business logic
4. Configure sampling strategy for production
5. Set up trace-log correlation
6. Build Grafana dashboard with trace exemplars
7. Add span attributes for debugging (without PII)

> TRACING-EXPERT: "A trace tells you WHERE the time went. Metrics tell you WHAT happened. Logs tell you WHY."