---
name: observability-engineer
description: "Observability with OpenTelemetry: traces, metrics and logs, semantic conventions, structured logging, RED and USE, SLOs and burn-rate alerts, cardinality, sampling. Use when instrumenting a service or designing alerts."
metadata:
  risk: safe
  source: multi-agent-pipeline
  date_added: "2026-09-25"
---

# Observability

Telemetry exists to answer one question during an incident: what changed, for
whom, and where. Collecting more data does not answer it; collecting correlated
data does. This skill covers what to emit, how to keep it affordable, and how to
alert on it without paging people for noise.

## The three signals and how they join

| Signal | Best at | Join key |
|--------|---------|----------|
| Traces | where time went in one request across services | `trace_id`, `span_id` |
| Metrics | how much and how often, cheaply, over long periods | resource attributes, exemplars |
| Logs | the detail of one event | `trace_id` / `span_id` fields |

A metric spike links to example traces through exemplars; a trace links to its
log lines through the trace id. If any of those links is missing, the signals
become three separate searches.

## Structured logging

- One JSON object per line, written to stdout.
- Fixed fields: `timestamp`, `level`, `message`, `service`, `trace_id`,
  `span_id`, plus event fields. Let the logging bridge inject trace context
  instead of passing ids by hand.
- Message is a stable event name ("order payment declined"); variables go in
  fields. Stable messages make grouping and counting possible.
- Levels mean something: `error` is actionable by someone, `warn` is degraded
  but handled, `info` is a business event, `debug` is off in production.
- Redact at the source and again in the Collector: tokens, passwords, card data,
  emails and free-text user input.

```json
{"timestamp":"2026-09-25T10:14:03.112Z","level":"warn","message":"inventory reservation retried","service":"orders","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","order_id":"ord_8812","attempt":2}
```

## Which metrics to emit

For every request-driven service, **RED**:

- **Rate**: requests per second, by route and outcome.
- **Errors**: failed requests, by route and error type.
- **Duration**: a latency histogram, by route.

For every resource (CPU, memory, pool, queue, disk), **USE**:

- **Utilization**: fraction busy.
- **Saturation**: work waiting (queue depth, pool wait time).
- **Errors**: resource-level failures.

Add a few business metrics that show the product working: orders placed,
payments captured, notes saved. They catch failures that return 200.

Use histograms for latency, never averages. OpenTelemetry exponential
histograms avoid hand-tuning bucket boundaries.

## Cardinality

Every unique combination of attribute values is a separate time series, and
cost grows with that count.

- Never use unbounded values as metric attributes: user id, order id, email,
  raw URL, full error message, timestamps.
- Use `http.route` (`/orders/{id}`), not `url.path` (`/orders/8812`).
- Bucket open-ended values (status class `5xx`, size bands).
- High-cardinality detail belongs on spans and logs, which are sampled or
  retained for shorter periods.
- Enforce limits with Collector processors or SDK views that drop or rewrite
  attributes.

## Sampling

- **Head sampling** (decided at the root span, for example 10% with
  `parentbased_traceidratio`) is cheap but blind to outcome.
- **Tail sampling** in the Collector decides after the trace completes: keep all
  errors, all traces slower than a threshold, and a small percentage of the
  rest. It needs every span of a trace to reach the same Collector instance
  (load-balancing exporter keyed by trace id).
- Metrics are never sampled; derive RED metrics from spans before sampling
  (span metrics connector) or emit them directly.

## SLIs and SLOs

An SLI is a ratio of good events to valid events, measured as close to the user
as practical:

- Availability: requests not returning 5xx / all requests.
- Latency: requests faster than 400 ms / all requests.
- Freshness (pipelines): records processed within 5 minutes / all records.

An SLO sets a target over a window, for example 99.9% over 28 days. The error
budget is the remainder (0.1%, about 40 minutes of total outage in 28 days).
Spending policy is agreed in advance: budget exhausted means reliability work
before features.

## Alerting on burn rate

Alert on how fast the budget is being consumed, not on raw thresholds. Burn
rate 1 spends exactly the budget over the window. Common multi-window pairs for
a 28 to 30 day SLO:

| Severity | Long window | Short window | Burn rate | Budget spent if sustained |
|----------|-------------|--------------|-----------|---------------------------|
| page | 1 h | 5 min | 14.4 | 2% in an hour |
| page | 6 h | 30 min | 6 | 5% in six hours |
| ticket | 3 days | 6 h | 1 | 10% in three days |

Both windows must exceed the rate; the short window makes the alert reset soon
after recovery. PromQL shape for the first row, with a 99.9% target:

```promql
(
  sum(rate(http_server_errors_total{service="orders"}[1h]))
  / sum(rate(http_server_requests_total{service="orders"}[1h]))
) > (14.4 * 0.001)
and
(
  sum(rate(http_server_errors_total{service="orders"}[5m]))
  / sum(rate(http_server_requests_total{service="orders"}[5m]))
) > (14.4 * 0.001)
```

Every page links a runbook and a dashboard. Cause-based alerts (CPU high, pod
restarted) become tickets or dashboard panels unless they reliably predict user
impact.

## Dashboards

One service dashboard, top to bottom:

1. SLO status and remaining budget.
2. RED for the service's entry points.
3. RED for each downstream dependency, from the client side.
4. USE for the resources it owns (pools, queues, runtime memory, GC).
5. Deploy and feature-flag annotations on every graph.

## Review checklist

- Every signal carries `service.name`, `service.version` and environment.
- Traces cross every service and queue hop without breaking.
- Logs are JSON, carry trace ids, and contain no secrets or personal data.
- No metric attribute is unbounded.
- Each user-facing service has at least one SLO with burn-rate alerts and a
  runbook link.
- Sampling keeps every error trace.

## References

- `references/patterns.md`: SDK setup, resource attributes, semantic conventions and context propagation, plus a Collector pipeline. Load it when the task reaches those topics.