---
description: Observability—metrics (Prometheus, RED/USE), structured logs (Loki), traces (OpenTelemetry). SLO-driven; context propagation; actionable alerts.
alwaysApply: false
---

# Observability

Guidelines for metrics, logs, and traces.

## Core Principles

1. **Three Pillars** - Metrics, logs, traces together; correlate via trace_id/span_id.
2. **SLO-Driven** - Define SLOs and SLIs before instrumenting everything.
3. **Context Propagation** - Pass trace context across service boundaries (W3C or OTLP).
4. **Actionable Alerts** - Every alert has a clear response and runbook.

## Metrics (Prometheus)

- **ServiceMonitor**: Selector for app; endpoint port/path/interval; namespaceSelector. Use metricRelabelings to drop high-cardinality or noisy metrics.
- **RED (requests)**: Rate, errors, duration—counter for total requests (by method, path, status); histogram for duration; gauge for in-flight. Normalize path to avoid cardinality explosion.
- **USE (resources)**: Utilization, saturation, errors—e.g. CPU/memory/disk ratios from node exporters. Record rules for reuse in alerts/dashboards.
- **Labels**: Keep cardinality low; avoid user_id or unbounded values in labels.

## Logs

- **Structured**: JSON with level, message, timestamp, trace_id, span_id, and request/service fields. Use a consistent logger (e.g. zap, structlog).
- **Levels**: Debug (dev only), Info (normal ops), Warn (recoverable), Error (needs attention). Avoid Fatal in services; let orchestrator restart.
- **Collection**: Promtail or Fluent Bit; annotate pods for scrape; send to Loki or equivalent. Parse JSON and extract labels for querying.

## Traces (OpenTelemetry)

- **Setup**: OTLP exporter to Tempo/Jaeger; TracerProvider with resource (service name, version, env); ParentBased sampler (e.g. 10% head sampling). Set TraceContext propagator.
- **Spans**: Start span per operation; defer span.End(); record errors; pass context to downstream calls. Use semantic attributes (e.g. http.method, db.statement).
- **Propagation**: Inject trace context into HTTP headers (or gRPC metadata); extract on the other side so traces are end-to-end.

## Alerting

- **Alert on SLO burn rate** or error budget consumption; avoid alerting on raw availability unless necessary. Use multi-window/multi-burn-rate to reduce noise.
- **Runbooks**: Link runbook URL in alert; document steps and ownership. On-call should be able to act without reading code.

## Definition of Done (New Service)

- [ ] Metrics exposed (/metrics); ServiceMonitor or scrape config added.
- [ ] Structured logging with trace_id; log aggregation configured.
- [ ] Tracing instrumented; context propagated to dependencies.
- [ ] SLO/alert and runbook defined.

## Common Pitfalls

- **High-cardinality labels** - Don’t use request_id or user_id as labels; use in logs/traces instead.
- **Missing context** - Always add trace_id to logs so logs and traces correlate.
- **Alert fatigue** - Tune thresholds; use inhibition/silencing; make alerts actionable.