# Observability > OpenTelemetry tracing in Voltro — the auto-emitted spans for every primitive, span attributes and nesting, the three enabling modes (console / OTLP / buffer), and adding your own spans with Effect.withSpan. --- ## Overview _OpenTelemetry tracing in Voltro — the auto-emitted spans for every primitive, span attributes and nesting, the three enabling modes (console / OTLP / buffer), and adding your own spans with Effect.withSpan._ Every framework primitive emits **OpenTelemetry spans** when tracing is enabled. You don't instrument anything by hand — the handler boundaries, the transactional scope, and the streaming deliveries are all traced automatically, and the spans nest into a single waterfall per request. ## Auto-emitted spans | Span | What it marks | |---|---| | `client.mutation.` / `client.subscription.` | The browser-side span that **roots** the trace and sets the shared `traceId`. | | `mutation.` | The rpc mutation-handler boundary. | | `action.` | The unary action-handler boundary. | | `subscription.` | The streaming-rpc handler **setup** (subscribe → first snapshot). NOT the open-duration. | | `subscription..snapshot` / `.delta` | Each **data delivery** to the subscriber, with its real produce→push latency. | | `store.transactional` | The postgres `BEGIN`/`COMMIT` (or retry) scope. Carries `db.system=postgresql` + `db.operation=transaction`. | | `webhook` | An inbound `*.webhook.tsx` route (an incoming HTTP handler, not a query). Continues the caller's trace when the request carries a `traceparent` header. | ## Span attributes Handler spans carry: - `rpc.tag` — the rpc tag of the call - `subject.type` — the authenticated subject type - `tenant.id` — the active tenant `@effect/rpc`'s own `RpcServer.` transport span is **suppressed** from the trace view — the framework spans carry the real latencies. For streaming subscriptions the transport span would just be a misleading open-duration bar, so the per-delivery `.snapshot` / `.delta` spans surface the real latency instead. ## Nesting Spans nest. A mutation handler span wraps the `store.transactional` span, which wraps any `Effect.withSpan` you add inside the executor: ``` client.mutation.createOrder └─ mutation.createOrder rpc.tag, subject.type, tenant.id └─ store.transactional db.system=postgresql, db.operation=transaction └─ createOrder.charge-card (your custom span) ``` ## Enabling modes Three modes, **env-driven, no code changes**: ```bash # Default — no EXPORTER. `voltro dev` still runs a buffer-only tracer # (powers the Traces dashboards + traceId-in-logs); nothing is shipped # off-box. `VOLTRO_TRACING_BUFFER=off` disables even that. unset FRAMEWORK_TRACING OTEL_EXPORTER_OTLP_ENDPOINT # Console (local dev — spans printed to stdout) FRAMEWORK_TRACING=console voltro dev . # OTLP (production — sends to any OTLP/HTTP collector: Jaeger, Tempo, Honeycomb, etc.) OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 voltro dev . # OTEL_SERVICE_NAME defaults to 'voltro-api'; override per app: OTEL_SERVICE_NAME=my-api OTEL_EXPORTER_OTLP_ENDPOINT=... voltro dev . ``` The framework **auto-detects**: if any standard OpenTelemetry env-var (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`) is set, the mode resolves to `otlp` automatically. Set `FRAMEWORK_TRACING` explicitly (`console` / `otlp` / `off`) to force a mode. Even with no exporter, `voltro dev` installs a **buffer-only** tracer — that's what powers the [DevTools / cloud Traces panel](/docs/observability/distributed-tracing) and the trace id stamped on every log line. Set `VOLTRO_TRACING_BUFFER=off` to disable it entirely. ## Metrics export Separate from tracing, the framework records **metrics** into Effect's global `MetricRegistry` — one source of truth (`voltro_rpc_*`, `voltro_http_*`, `voltro_plugin_hook_*`, `voltro_subscription_*` + `voltro_subscriptions_active`, `voltro_db_*`, plus `effect_fiber_*` and any custom metric). Three ways to get them out: ```bash # OTLP metrics — the SAME OTEL endpoint that enables trace export also enables # metrics. Ships to any OTLP/HTTP collector (Prometheus OTLP, Grafana Agent, # the Datadog Agent's OTLP port, etc.). @effect/opentelemetry auto-bridges the # Effect MetricRegistry into the OTel MeterProvider — no per-metric wiring. OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 voltro start # metrics-only endpoint (no traces): OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://collector:4318/v1/metrics voltro start # console (local dev — periodic metric dump to stdout): FRAMEWORK_METRICS=console voltro dev . # export cadence (ms): OTEL_METRIC_EXPORT_INTERVAL (default 60000). ``` The framework auto-detects: `OTEL_EXPORTER_OTLP_ENDPOINT` (or the metrics-specific endpoint) → metrics export `otlp`; force with `FRAMEWORK_METRICS` (`console` / `otlp` / `off`). Two pull/push consumers read the SAME registry, so they can't disagree: - **[`@voltro/plugin-prometheus`](/docs/plugins/prometheus)** — `GET /metrics` scrape endpoint (Prometheus / Grafana). - **[`@voltro/plugin-datadog`](/docs/plugins/datadog)** — agentless push to Datadog's `/api/v2/series` (for setups without an Agent / OTLP collector). With an Agent, prefer pointing `OTEL_EXPORTER_OTLP_ENDPOINT` at it instead. - The dashboard **Metrics** panel reads the same snapshot via `GET /_voltro/inspect/metrics`. ## Database metrics (`voltro_db_*`) Every dialect store emits the same five series, so a dashboard built against one database keeps working after a migration to another. The labels are `dialect` (`postgres` · `mysql` · `mariadb` · `mssql` · `sqlite` · `turso`) and `op` (`select` · `insert` · `update` · `delete` · `upsert` · `raw` · `transaction` · `ddl`). | Series | Type | What it answers | | --- | --- | --- | | `voltro_db_queries_total` | counter | Query rate, split by operation kind. | | `voltro_db_query_duration_seconds` | histogram | p50/p95/p99 per operation — `histogram_quantile` over the buckets. | | `voltro_db_errors_total` | counter | Statement failure rate. | | `voltro_db_operations_in_flight` | gauge | Concurrency the framework is holding right now. | | `voltro_db_eager_fallback_total` | counter | Eager loads that dropped off the single-roundtrip fast path. | **No table name is ever a label.** Table names grow with your schema, and a label that grows with the schema is how a scrape target falls over. The table appears in the log line instead. **`voltro_db_operations_in_flight` is not the driver's pool queue.** It counts operations the framework currently has in flight, which is an upper bound on the connections it holds — the same number on every dialect. Your driver's own `waiting` count is not exposed. In practice you alert on this gauge sitting near your pool size *together with* the duration histogram's tail growing: a starved pool shows up as acquire time inside the query timing. ### `voltro_db_eager_fallback_total` — the one to alert on An `eager:` query normally compiles to **one** round trip (a JSON aggregate). When it can't, the framework silently uses the portable multi-query walker instead — correct, and one round trip per relation level, on every call. That is a permanent per-query cliff with no error attached to it, which is why it is counted. The `reason` label separates the two very different cases: - `not-compilable` — the query shape can never take the fast path (an unregistered relation, an ambiguous inferred foreign key, or an eager read under physical tenant isolation). Steady state. Worth knowing about, not worth paging on. - `execute-failed` — the fast path compiled, **ran, and threw**, so the query paid for both paths. This is the one to alert on. It usually means a database or driver upgrade changed something under the JSON-aggregate query. Both also log: a `warn` the first time a given table and reason are seen, then again at most every 5 minutes while it persists (`VOLTRO_DB_EAGER_FALLBACK_WARN_INTERVAL_MS`, `0` = once only). The counter is never rate-limited — the log line answers "is this happening now", the counter answers "has this been happening since the deploy three weeks ago". ## Routing traces to a vendor The env-driven OTLP path above ships traces to any OTLP/HTTP collector. For **deep, opt-in vendor integration** — install one plugin, get traces + errors + logs all correlated by the same `traceId`, zero `OTEL_*` env — a plugin can contribute to the framework's tracer directly via `contributeObservability`: ```ts interface ObservabilityContribution { resourceAttributes?: Record spanProcessors?: ReadonlyArray // OTel SpanProcessor — receives every framework span metricReaders?: ReadonlyArray // OTel MetricReader sampler?: unknown // OTel Sampler (use with care — see below) } ``` The CLI gathers every plugin's contribution at boot and passes the merged span-processors / metric-readers / resource-attributes into the framework's NodeSdk tracer. The vendor SDK runs as an OTel **consumer** — it gets a `SpanProcessor` alongside the framework's own buffer sink; it never becomes the global provider. So the in-app **Traces dashboard keeps working** and the vendor receives the identical spans. This is how [`@voltro/plugin-sentry`](/docs/plugins/sentry) (`SentrySpanProcessor`) and [`@voltro/plugin-datadog`](/docs/plugins/datadog) (`OTLPTraceExporter` → DD Agent) route traces with no env fiddling. > **Don't contribute a sampler that drops spans.** The framework's buffer sink (powering the in-app Traces dashboard) is a `SpanProcessor` — it only sees *recorded* spans. A vendor sampler set as the tracer's sampler gates recording for the WHOLE tracer, blinding the dashboard. The framework stays always-on; vendors sample at their own export layer (e.g. Sentry's `tracesSampleRate` is applied by the Sentry client, not a tracer sampler). To attach an error to the exact active span, the interceptor context carries both `ctx.traceId` and `ctx.spanId` — plugins read them explicitly (the active OTel span lives in Effect's fiber context, not the AsyncLocalStorage vendor SDKs read implicitly). ## Adding your own spans Inside a mutation / action / workflow executor, just use Effect: ```typescript import { Effect } from 'effect' export const myAction = defineAction({ name: 'reports.rebuild', guards: [{ scope: 'reports:write' }], /* input, output */ }) export default (input, ctx) => Effect.gen(function* () { yield* someExpensiveWork.pipe( Effect.withSpan('myAction.expensive-work', { attributes: { 'input.id': input.id }, }), ) return { ok: true } }) ``` The span nests under the auto-emitted `action.myAction` span. In synchronous (non-Effect) executors, ad-hoc spans require explicitly adopting the Effect runtime — usually not worth it; rely on the auto spans the framework emits at each handler boundary. Continue to [Distributed tracing](/docs/observability/distributed-tracing) for how one `traceId` flows frontend → api → api, and [Traces & logs from the shell](/docs/observability/cli) for the `voltro traces` / `voltro logs` workflow. --- ## Distributed tracing _One traceId from frontend through every api hop — automatic W3C propagation, ctx.request.traceId, traceId-in-logs, the dev error-surfacing toast, and the DevTools / cloud Traces panel._ Trace continuity is **automatic and on by default** in `voltro dev` — a buffer-only tracer is always installed (set `VOLTRO_TRACING_BUFFER=off` to disable). You don't wire anything. A single request gets **one `traceId`** that flows frontend → api → api, and every log line of that request carries it. ## How propagation works - **Frontend → server is automatic.** `useMutation` / `useSubscription` wrap each rpc call in a client span; `@effect/rpc` propagates that span's W3C trace id to the server, so the handler span — and everything it calls — shares the **same** `traceId`. In any handler, `ctx.request.traceId` is that id. - **api → api is automatic over HTTP.** An action or handler that calls another api via the framework `HttpClient` sends a W3C `traceparent` header; an inbound `*.webhook.tsx` route (an incoming HTTP handler) **continues** the caller's trace. So frontend → api Y → api Z (via webhook / HTTP) is **one** trace. - **Every log line carries it.** Handlers stamp `fields.traceId` onto each log line they emit, so [`voltro logs --trace `](/docs/observability/cli) returns the whole chain across hops, in order. ``` client.mutation.placeOrder traceId=4bf92f35… └─ mutation.placeOrder traceId=4bf92f35… (api Y, ctx.request.traceId) └─ webhook/fulfilment traceId=4bf92f35… (api Z, continued via traceparent) ``` ## Seeing traces - **DevTools "Traces" panel** (dev) — a waterfall per request, failed hops tinted red. Backed by the in-memory ring buffer. - **Cloud dashboard's per-app "Traces" page** — same waterfall, with history kept in `_voltro_traces` (postgres) **when durable persistence is enabled** (dev: on by default, interesting-only; prod: off by default → use OTLP). The in-memory ring is always the live source; see *Durable trace volume* below. ## Durable trace volume (postgres) The in-memory ring always keeps EVERY span for live debugging. SEPARATELY, on postgres, the framework can durably mirror a SUBSET of spans into `_voltro_traces` so the in-app Traces dashboard survives restarts. This is a **dev convenience** — at real volume durable tracing belongs in an **OTLP backend (Tempo / Honeycomb / Datadog), not your OLTP postgres** (set `OTEL_EXPORTER_OTLP_ENDPOINT`). A row-per-span firehose into postgres is what turned this table into ~85% of a production DB. So persistence is **environment-aware + fully configurable**: - **Default OFF in production, `interesting` in dev.** `voltro dev` (not `NODE_ENV=production`) persists only "interesting" spans. `voltro serve` / `voltro start` default `NODE_ENV=production` when it's unset, so they persist NOTHING and **`_voltro_traces` is not even created** — use OTLP in prod. An explicit `NODE_ENV` is never overridden. - **`VOLTRO_TRACING_PERSIST`** = `off` | `errors` | `interesting` | `all` (overrides the env default everywhere). `interesting` = error + slow + per-trace root spans, MINUS the high-volume subscription delivery (snapshot/delta) spans, MINUS the framework's own background-task reads. `errors` = error spans only. `all` = every non-delivery span. `off` = no persistence and no table. **Why background tasks are excluded from "slow".** The framework polls four of its own tables on a timer (`_voltro_schedule_claims`, `_voltro_workflow_pending`, `_voltro_workflow_pauses`, `_voltro_ai_inferences`). Those reads are slow exactly when the database is under pressure — which is when persisting them costs the most. A deployment measured the loop closing on itself: `_voltro_traces` at 476 571 rows / 335 MB, writing ~11 INSERTs/s onto the same 15-slot pooler the app read through, 99 % of it framework poller spans. Pool pressure makes the spans slow, slow spans are "interesting", persisting them costs pool. They are still kept when they ERROR, and `all` mode still keeps everything. A request-path framework table like `_voltro_api_keys` is NOT excluded — a slow lookup there is a real user waiting. - **`VOLTRO_TRACING_SLOW_MS`** (default `500`) — the "slow" threshold used by `interesting`. - **`VOLTRO_TRACING_PERSIST_DELIVERY`** (default off) — also persist the subscription delivery spans (the firehose; rarely wanted). - **`VOLTRO_TRACING_SAMPLE`** (`0`–`1`, default `1`) — per-*trace* sampling (a kept trace keeps all its eligible spans; errors are never sampled out). `=0` with no explicit mode means off. - **`VOLTRO_TRACING_TTL_HOURS`** (default `24`) — the GC prunes rows past the TTL; it loops until the whole over-TTL backlog drains each pass (batched set-based delete), so it always catches up. It WARNs if a pass errors or is shifting very high volume — a silently-failing GC is no longer silent. - **`VOLTRO_TRACING_BUFFER=off`** — disables the durable flusher + the in-memory tracer entirely (and drops the table). `_voltro_traces` has no foreign keys — `TRUNCATE TABLE _voltro_traces` reclaims the space instantly if it ever grew under old defaults. To get the in-app Traces *history* in prod anyway, set `VOLTRO_TRACING_PERSIST=interesting` (keeps it small) — but prefer OTLP for production-grade tracing. ## Dev error-surfacing toast In dev, if any hop of a trace **you** triggered errors — even when the frontend call itself returned OK (a fire-and-forget emit, a detached workflow, a swallowed downstream error) — an auto-toast appears in the browser pointing at the failing hop and at `voltro logs --trace `. This catches the class of failure that's normally invisible: the UI shows success, but a downstream hop quietly failed. The toast surfaces it immediately with the exact `traceId` to chase. ## The debugging loop When you debug a failure: 1. Get the `traceId` — from the dev error toast, an error line's `fields.traceId`, or `/_voltro/inspect/traces?onlyErrors=1`. 2. Run [`voltro logs --trace `](/docs/observability/cli) — that's the end-to-end causal chain, every log line of the request across hops, in order. This is the fastest way to answer "where did this error come from?" — far faster than inferring it from source or from the user's description. ## Shipping traces to a vendor (Sentry / Datadog) The same trace continuity flows to a vendor APM when you install a deep-observability plugin — no `OTEL_*` env. Each contributes an OTel span-processor to the framework's tracer via `contributeObservability`, so the vendor runs as a **consumer** of the existing tracer (the in-app Traces panel keeps working) and receives the identical spans: - [`@voltro/plugin-sentry`](/docs/plugins/sentry) — every mutation/query/action error becomes a Sentry issue correlated to its `trace_id` + `span_id`, with the request's log lines as breadcrumbs; opt-in performance traces (`traces: true`). - [`@voltro/plugin-datadog`](/docs/plugins/datadog) — framework spans → the Datadog Agent's OTLP receiver (`traces: true`), plus log forwarding with `dd.trace_id` correlation. See [Observability › Routing traces to a vendor](/docs/observability/overview#routing-traces-to-a-vendor) for the contribution surface + the sampler caveat. ### "Missing peer" warnings on install (benign) `pnpm install` may print missing-peer warnings for `@opentelemetry/sdk-logs` and `@opentelemetry/sdk-trace-web` — these are **optional peers of `@effect/opentelemetry`** (pulled in transitively), used only if you export logs/browser traces to OTLP. Boot and the in-memory trace ring work **without** them, so the warnings are safe to ignore. To silence them, either add the two packages to your app, or add a pnpm rule: ```json // package.json "pnpm": { "peerDependencyRules": { "ignoreMissing": ["@opentelemetry/sdk-logs", "@opentelemetry/sdk-trace-web"] } } ``` --- ## voltro traces & voltro logs _Inspect traces and logs from the shell — flags, the --trace end-to-end chain workflow, --format json for programmatic analysis, and the guidance for AI coding agents._ `voltro logs` and `voltro traces` give you the running app's log buffer and trace buffer from a shell — no DevTools UI needed. Both discover every running process via `~/.voltro/runtime-registry.json`, fan out to each one's inspect endpoint in parallel, merge by timestamp, and render. ## `voltro logs` Every `voltro dev` / `voltro start` instance buffers the last **2000** server-side `@voltro/logger` records **and** every browser console line that the dev console bridge forwarded. ```sh voltro logs # last 100 from every running process voltro logs --tail 50 --level error # only errors, last 50 voltro logs --since 30s # last 30 seconds voltro logs --scope 'orders' # case-insensitive scope filter voltro logs --filter 'channels.create' # message substring voltro logs --source client # only browser-originated lines voltro logs --trace # the WHOLE causal chain for one request voltro logs --process voltroCloudApi # restrict to one registry name voltro logs --format json | jq '.[]' # machine-parseable ``` Flags: `--tail ` (default 100, cap 2000), `--since ` (`30s` / `5m` / `1h` / unix-ms), `--level ` (`trace|debug|info|warn|error|fatal`, minimum level), `--scope `, `--source `, `--filter `, `--trace `, `--process `, `--format ` (default `pretty`), `--no-color`. `--format json` returns one object per record: `{ ts, level, source: 'server'|'client', scope, message, fields }`. ## `voltro traces` Mirror of `voltro logs` for the trace buffer — a transient ring of 2000 records, no persistence. ```sh voltro traces # 20 newest traces, pretty voltro traces --errors # only traces containing an errored span voltro traces --since 30s --tail 50 # recent voltro traces --process voltroCloudApi # restrict to one process voltro traces --id # one trace, span waterfall (detail mode) voltro traces --errors --format json | jq '.traces[]' ``` List-mode flags: `--tail `, `--since `, `--errors`, `--process `, `--format ` (default `pretty`), `--no-color`. Detail mode (`--id `, alias `--trace`) renders one trace's span waterfall and additionally accepts `--min-duration ` and `--status ` to filter the spans shown. **Subscriptions vs requests in the output**: traces from `useMutation` / `useSubscription` / actions / webhooks show `spans · duration` (the duration *is* the call's latency). Subscriptions show `sub · N deliveries · last Xms` — the open-duration is liveness, not latency, so the command surfaces the latency of the most recent data transfer instead (snapshot + each delta is its own span with its produce→push latency). ## The `--trace ` end-to-end chain The whole point of the trace id is that it's **shared** across the entire frontend → api → api chain (see [Distributed tracing](/docs/observability/distributed-tracing)). Once you have a `traceId`, you can pull the complete causal chain for one request across every hop, in order: ```sh # 1. find a failure voltro traces --errors --format json | jq '.traces[]' # 2. pull the full chain for that one request voltro logs --trace --format json ``` That returns every log line of the request — frontend action → api Y → api Z — in order. It's the fastest answer to "where did this error come from?". ## For AI coding agents **Claude Code, Cursor, GitHub Copilot, and other agents: use these whenever you debug.** Before you grep code or read the user's description of a bug, run `voltro logs --tail 100` (or `--since 30s` after a reproducible action). The buffer carries the actual error message, the actual stack, the actual request/mutation tag — faster and more accurate than inferring what happened from source. Common patterns: - **"Is the change working?"** — make the edit, run the reproducing action, then `voltro logs --since 10s`. New errors show immediately. - **"Why did the mutation fail?"** — `voltro logs --filter '' --level warn`. - **"Is the browser even reaching the server?"** — `voltro logs --source client --tail 20`. - **"Why did this request fail end-to-end?"** — grab the traceId of a failed span (`voltro traces --errors`, or the dev error toast), then `voltro logs --trace --format json` for the whole chain. Discovery uses `~/.voltro/runtime-registry.json` — every `voltro dev` / `voltro start` registers itself with a heartbeat. If nothing is running, both commands print a clear hint instead of failing. The trace buffer is transient; if you need older context than the ring holds, ask the user to reproduce the action. --- ## Time-travel debugger _Record the app's ChangeEvent timeline and scrub a table's rows back and forward through it — read-only point-in-time replay, env-gated + redacted, surfaced in the DevTools / cloud Time-travel panel._ The time-travel debugger records the app's **ChangeEvent stream** into a bounded, redacted ring and lets you **scrub any table's rows back and forward** through it — replaying what the data looked like at any recorded point. Read-only: replay reconstructs a row-set by reversing the recorded changes over the live rows; it never writes. ## Turn it on Recording is **env-gated** and **off by default in production** (volume + PII are the real cost — use OTLP for prod forensics). Set `VOLTRO_TIMELINE`: | value | records | |---|---| | `off` (also `0` / `false`) | nothing | | `interesting` (also `on` / `1` / `true`) | user tables only (skips `_voltro_*` framework noise) | | `all` | every table (except the timeline's own writes) | The default is `interesting` outside production, `off` in production — the same environment-aware default the durable trace persistence uses. Sensitive-looking columns (`password` / `secret` / `token` / `api_key` / `authorization` / `cookie` / `ssn` / `credit_card` / `cvv` / …) are **redacted** before a row enters the ring — PII never sits in the debug buffer. ## Scrub it Open the **Time-travel** panel in the DevTools dashboard (or the cloud dashboard, for a deployed app): pick a table, drag the scrubber across the recorded sequence, and the panel reconstructs that table's rows as of the selected point. The event log shows every recorded change; click one to scrub to just before it. "Jump to now" returns to live. ## The endpoint The panel reads two read-only inspect endpoints: - `GET /_voltro/inspect/timeline` — the recorded events (`?table=&tenantId=&from=&to=&limit=`) plus the current sequence (`currentSeq`). - `GET /_voltro/inspect/timeline/replay?table=&seq=` — ``'s row-set as of sequence ``. ```sh # Gated by the same inspect auth as every /_voltro/inspect/* endpoint. curl "http://localhost:4000/_voltro/inspect/timeline?table=todos&limit=50" curl "http://localhost:4000/_voltro/inspect/timeline/replay?table=todos&seq=42" ``` ## Limits - **Read-only.** Time-travel never writes — restoring a past value is the universal-undo primitive's job (and only for safe inverses). - **In-memory ring** (bounded, per-process) in v1 — it is for scrubbing recent history, not durable forensics: it resets on restart, and a replay is correct only as long as the events after your target sequence are still in the ring. - The inspect surface is a `voltro dev` / DevTools concern; out-of-band DB writes (not in the app's ChangeEvent stream) aren't recorded, and production should use OTLP for forensics.