# Prometheus > Prometheus exporter — scrapes the unified Metrics-API at GET /metrics in text exposition format. --- ## Prometheus _Prometheus exporter — scrapes the unified Metrics-API at GET /metrics in text exposition format._ `@voltro/plugin-prometheus` exposes the framework's metrics to Prometheus / Grafana at a scrape endpoint. It only READS — the framework emits the metrics automatically; the plugin formats them. ## The Metrics-API (single source of truth) The framework records metrics into Effect's global `MetricRegistry`. One snapshot sees everything: - **Core RPC** — `voltro_rpc_requests_total{tag,status}`, `voltro_rpc_errors_total{tag}`, `voltro_rpc_duration_seconds{tag}` (histogram). Emitted at the mutation / action handler boundary. - **HTTP routes** — `voltro_http_requests_total{route,status}`, `voltro_http_duration_seconds{route}`. - **Plugin interceptors** — `voltro_plugin_hook_duration_seconds{hook}`, `voltro_plugin_hook_errors_total{hook}`. - **Subscriptions** — `voltro_subscriptions_active{tag}` (gauge of currently-open subscriptions), `voltro_subscription_deliveries_total{tag,kind}` + `voltro_subscription_delivery_seconds{tag,kind}` (per-delivery produce→push latency; `kind` = `snapshot` | `delta`). - **Schedules (crons)** — `voltro_schedule_runs_total{schedule,status}` (firings by name + outcome — `status` = `succeeded` | `failed`), `voltro_schedule_duration_seconds{schedule}` (histogram), and `voltro_schedule_last_success_timestamp_seconds{schedule}` (a **gauge holding the UNIX time of the last SUCCESS**). Emitted by the framework scheduler, so every cron gets them with no per-handler wiring. A cron fires unattended — the failure mode is silent — so this is the series to alert on: `time() - voltro_schedule_last_success_timestamp_seconds{schedule="…"} > ` fires when a job stops succeeding (a failure counter alone can't catch a job that stopped firing at all, but the last-success gauge going stale does). A failure moves the counter but deliberately NOT the gauge. - **Workflows (durable execution)** — `voltro_workflow_runs_total{workflow,status}` (terminal outcomes — `status` = `succeeded` | `failed`), `voltro_workflow_duration_seconds{workflow}` (histogram), and `voltro_workflow_last_success_timestamp_seconds{workflow}` (last-success gauge). Emitted by the workflow run-recording seam. Because the framework applies no retry of its own, a `failed` run is **terminal** — it is the dead-letter state — so `voltro_workflow_runs_total{status="failed"}` **is** the dead-letter rate, and the last-success gauge going stale is the "this workflow stopped completing" alert (same shape as the schedule alert). A failure moves the counter but not the gauge. - **`@voltro/cache`** counters, Effect's own `effect_fiber_*` runtime metrics, and **any custom metric** you or another plugin defines. Two consumers read the SAME snapshot, so they never disagree: - **`GET /metrics`** (this plugin) → Prometheus text exposition format, consumed by Prometheus / Grafana. - **`GET /_voltro/inspect/metrics`** → JSON `MetricSample[]`, consumed by the dashboard's Metrics panel (quantiles derived from the histogram buckets, `histogram_quantile`-style). ### Custom metrics Define your own counter / gauge / histogram and it shows up in both consumers automatically: ```ts import { counter, gauge, histogramMetric } from '@voltro/runtime' import { Effect, Metric } from 'effect' const signups = counter('app_signups_total', 'User sign-ups.') export default (input, _ctx) => Effect.gen(function* () { // … create the user … yield* Metric.increment(signups) return { ok: true } }) ``` ## Wiring ```ts // app.config.ts import { prometheusPlugin } from '@voltro/plugin-prometheus' export default { type: 'api' as const, name: 'api', plugins: [prometheusPlugin()], } ``` Scrape config — one target **per replica**, never a single load-balanced hostname: ```yaml scrape_configs: - job_name: voltro metrics_path: /metrics static_configs: # one target per replica — NOT one LB-fronted hostname - targets: ['api-0.internal:4000', 'api-1.internal:4000'] ``` ### Multi-replica scraping The registry is **per-process**: each replica serves its own counters, and Prometheus keeps the series apart via the per-target `instance` label — exactly the pull model. But that only works when every replica is scraped **directly**. Behind a load-balancer, successive scrapes hit different processes and the per-process counters interleave into one incoherent series. On Kubernetes, use pod-level discovery so every pod becomes its own target: ```yaml scrape_configs: - job_name: voltro kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: 'true' ``` The plugin's `name` option disambiguates multiple plugin instances **within one process** (it suffixes the plugin name) — it does not identify replicas; replica identity is the scrape target's `instance` label. ## Options ```ts prometheusPlugin({ path: '/metrics', // scrape path (default /metrics) token: process.env.PROMETHEUS_TOKEN, // require `Authorization: Bearer ` processMetrics: true, // include node process_* metrics (default true) version: '1.4.2', // sets voltro_build_info{version} = 1 }) ``` - **`token`** — defaults to the `PROMETHEUS_TOKEN` env. Unset → the endpoint is open; gate it at the network layer (Prometheus uses `bearer_token` to send it). - **`processMetrics`** — emits `process_resident_memory_bytes`, `process_heap_bytes`, `process_cpu_seconds_total`. - The response is `text/plain; version=0.0.4; charset=utf-8` with `cache-control: no-store`. ## Output shape Counters / gauges render as `name{labels} value`. Histograms render as cumulative `name_bucket{labels,le="…"}` series plus `name_sum` + `name_count` + the `+Inf` bucket — exactly what `histogram_quantile()` expects in Grafana. ``` # HELP voltro_rpc_requests_total Total RPC handler invocations. # TYPE voltro_rpc_requests_total counter voltro_rpc_requests_total{status="ok",tag="mutation.todos.create"} 42 # TYPE voltro_rpc_duration_seconds histogram voltro_rpc_duration_seconds_bucket{tag="mutation.todos.create",le="0.001"} 3 voltro_rpc_duration_seconds_bucket{tag="mutation.todos.create",le="+Inf"} 42 voltro_rpc_duration_seconds_sum{tag="mutation.todos.create"} 0.21 voltro_rpc_duration_seconds_count{tag="mutation.todos.create"} 42 ``` ## Scope Mutation / action / HTTP / plugin metrics, subscription metrics (active gauge + per-delivery latency histogram, split snapshot vs delta), schedule/cron metrics AND workflow metrics (run counter + duration + last-success gauge each) are all covered. OTLP metrics export reads the same registry: the `OTEL_EXPORTER_OTLP_ENDPOINT` that enables trace export also exports metrics (`@effect/opentelemetry` bridges the registry into the OTel MeterProvider) — that is the path for the Datadog Agent / any OTLP collector; agentless Datadog is `@voltro/plugin-datadog`. This plugin is specifically the Prometheus **pull** (scrape) surface.