# ObservMe Dashboards, Alerts, and SLOs ## 1. Dashboard Pack ObservMe ships Grafana dashboards as JSON files: ```text dashboards/ ├── observme-overview.json ├── observme-slo-health.json ├── observme-export-health.json ├── observme-trace-journey.json ├── observme-agents.json ├── observme-agent-node-graphs.json ├── observme-cost.json ├── observme-models.json ├── observme-latency.json ├── observme-tools.json ├── observme-errors.json ├── observme-logs-llm.json ├── observme-llm-conversations.json └── observme-branches-compactions.json ``` ## 1.1 Dashboard Map | Dashboard | Primary question answered | Typical next drill-down | |---|---|---| | `ObservMe Overview` | Is ObservMe healthy, is workload normal, are agents/subagents healthy, and are cost or latency elevated? | Open the linked SLO Health, Export Health, Trace Journey, Cost, Models, Latency, Tools, Agents, Errors, or LLM Conversations dashboard with the same time range. | | `ObservMe SLO Health` | Which SLO is burning budget or near an alert threshold? | Export Health for telemetry-path inputs, Agents/Trace Journey for lineage, Errors for failed workflows, or Latency for handler overhead. | | `ObservMe Export Health` | Is the telemetry/export path trustworthy, idle, dropped, redacting, or failing? | Failure-only Loki tables, SLO Health, and `/obs health` for backend reachability. | | `ObservMe Trace Journey` | How did one session/agent/workflow move through turns, LLM calls, tools, bash, subagent spawns, waits, joins, logs, and Tempo traces? | Filtered LLM Conversations, Agents, Errors, Tools, or Tempo trace waterfall. | | `ObservMe Agents and Subagents` | Are subagents spawning, failing, waiting, joining, becoming orphaned, or losing trace context? | Agent Node Graphs for aggregate topology, Trace Journey for per-run logs/traces, and SLO Health for lineage budget. | | `ObservMe Agent Node Graphs` | What aggregate root/role/depth/spawn topology and health shape exists in the selected range? | Agents for ratios/top offenders and Trace Journey for per-execution investigation. | | `ObservMe Cost` | Which providers/models/environments and token/cache patterns are driving spend and burn rate? | Models, Logs and LLM I/O, LLM Conversations, and Trace Journey. | | `ObservMe Models` | How do provider/model traffic, errors, latency, stop reasons, and cost efficiency compare? | Cost, Latency, Logs and LLM I/O, and trace/log rows for representative requests. | | `ObservMe Latency` | Which stage, provider/model, tool, or agent role is slow after accounting for volume? | Trace Journey, Models, Tools, Agents, and Errors. | | `ObservMe Tools` | Which tools are busy, slow, failing, or producing large results? | Errors, Trace Journey, and filtered `tool.call.failed` logs. | | `ObservMe Errors` | Which error family is pressuring reliability and what logs/traces explain it? | Export Health, Tools, Models, Agents, Trace Journey, and Tempo trace rows. | | `ObservMe Logs and LLM I/O` | What LLM lifecycle logs, token trends, and prompt/response size trends exist without duplicating the content timeline? | LLM Conversations for redacted content and Trace Journey/Tempo for execution context. | | `ObservMe LLM Conversations` | What redacted opt-in prompts, responses, thinking, and trace links belong to one session/workflow/agent/run? | Trace Journey and Tempo traces. | | `ObservMe Branches and Compactions` | Did branch, compaction, model, or thinking-level changes explain cost/latency shifts? | Cost, Models, Latency, and Logs and LLM I/O. | ## 1.2 Shared Dashboard UX Conventions New or changed dashboards must use the same variable names, row names, link patterns, and status colors so users can move from overview panels to domain drill-downs without relearning each dashboard. ### Standard variables Use these exact variable names and display labels when a dashboard supports the dimension. Prometheus variables must stay low-cardinality; high-cardinality identifiers are allowed only in Loki/Tempo drill-downs. | Variable | Display label | Datasource scope | Backing label/attribute | Use | |---|---|---|---|---| | `environment` | Environment | Prometheus; mixed only if the log/trace backend exposes the same bounded value | `environment` metric label; `deployment.environment.name`/`observme.environment` attribute | Fleet, stage, or local/CI/prod filtering. | | `provider` | Provider | Prometheus; Loki for LLM lifecycle/content dashboards when promoted | `provider` metric label; `gen_ai_provider_name` Loki label | LLM traffic, latency, error, token, cost, and content-log filtering. | | `model` | Model | Prometheus; Loki for LLM lifecycle/content dashboards when promoted | `model` metric label; `gen_ai_request_model` Loki label | Provider/model comparison; keep values provider-reported and bounded by active use. | | `tool_name` | Tool | Prometheus | `tool_name` metric label | Tool workload, latency, and failure panels. | | `agent_role` | Agent role | Prometheus; Tempo where attributes are searchable | `agent_role` metric label; `pi.agent.role` attribute if emitted | Exact `root`, v2 `lead`/`helper`/`worker`/`validator`, and legacy `subagent`/`orchestrator`/`reviewer`/`unknown` comparison. | | `agent_capability` | Agent capability | Tempo only | `pi.agent.capability` attribute if emitted | Trace search only; launcher-defined capabilities are not Prometheus labels or dashboard metric variables. | | `subagent_depth` | Subagent depth | Prometheus; Tempo where attributes are searchable | `subagent_depth` metric label; `pi.agent.depth` attribute if emitted | Depth/fan-out/orphan/propagation diagnostics. | | `spawn_reason` | Spawn reason | Prometheus | `spawn_reason` metric label | Delegation and fan-out cause analysis. | | `session_id` | Session ID | Loki/Tempo only | `pi_session_id` Loki label; `pi.session.id` Tempo/span attribute | Follow one session across logs and traces. Never use in PromQL. | | `agent_id` | Agent ID | Loki/Tempo only | `pi_agent_id` Loki label; `pi.agent.id` Tempo/span attribute | Follow one root agent or subagent. Never use in PromQL. | | `agent_run_id` | Agent run ID | Loki/Tempo only | `pi_agent_run_id` Loki label; `pi.agent.run.id` Tempo/span attribute | Follow one execution attempt. Never use in PromQL. | | `workflow_id` | Workflow ID | Optional Loki/Tempo only where the backend exposes it | `pi_workflow_id` Loki label; `pi.workflow.id` Tempo/span attribute | Follow a multi-agent workflow. Never use in PromQL. | | `content_kind` | Content kind | Loki only | `pi_llm_content_kind` Loki label | Filter redacted opt-in prompt, response, thinking, or lifecycle content views. Never use raw content values. | Variable scope rules: - Prometheus-primary: `provider`, `model`, `tool_name`, and `spawn_reason`; `provider` and `model` may also be Loki variables on LLM lifecycle/content dashboards when the Collector promotes bounded GenAI labels. - Loki-only: `content_kind` and any other log-only variable whose backing label is promoted by the local Collector and used only by log panels. - Tempo-only: `agent_capability` when a trace-search dashboard explicitly needs it; use trace attributes and row-level links instead of global trace identifiers unless a dashboard is specifically trace-search-only. - Mixed: `environment`, `agent_role`, and `subagent_depth` only when every datasource exposes bounded equivalent values. - Loki/Tempo-only high-cardinality drill-downs: `session_id`, `agent_id`, `agent_run_id`, and `workflow_id`; these identify individual executions and must never be used in PromQL. - `pi.agent.display_name`, `pi.agent.child.display_name`, `pi.agent.capability`, and `pi.agent.child.capability` are never metric labels or Prometheus variables. The local Collector deletes these resource attributes from the metrics pipeline before resource-to-label conversion while retaining them for traces and logs. ### Child-identity observability asset inventory A bounded search for `agent_role`, `agent_capability`, `pi.agent.role`, `pi.agent.capability`, and `pi.agent.display_name` across `dashboards/*` and `observability-stack/config/otel/otel-collector.yaml` identifies exactly five role-aware dashboard JSON files: `observme-agents.json`, `observme-agent-node-graphs.json`, `observme-latency.json`, `observme-overview.json`, and `observme-trace-journey.json`. Their PromQL groups by the emitted `agent_role` without a role matcher or alias, so `lead`, `helper`, `worker`, and `validator` remain separate and `root`, `subagent`, `orchestrator`, `reviewer`, and `unknown` remain visible during migration. The alert and SLO YAML files contain no role, capability, or display-name predicate. Grafana JSON properties named `displayName` are field-rendering metadata, not ObservMe child identity. Because Prometheus resource conversion is enabled in the local Collector, its metrics-only resource processor explicitly deletes the own-agent and child-specific display-name and capability attributes. It intentionally retains the bounded role. Dashboard and cardinality tests lock this inventory so unrelated dashboards and panels do not require edits. ### Standard row names Use these row names consistently and only add domain-specific rows when the dashboard needs additional detail: - `Health`: top-level status, no-data/idle hints, alert state, and failure chips. - `Workload`: sessions, turns, requests, tool calls, spawns, and throughput. - `Cost`: spend, burn rate, tokens, cache/reasoning usage, and efficiency. - `Latency`: p50/p95/p99, stage attribution, and slow-offender tables. - `Agent lineage`: root/subagent health, fan-out, depth, orphan, propagation, wait, and join behavior. - `Logs`: Loki drill-down tables or log panels with parsed labels and empty-state descriptions. - `Traces`: Tempo trace search, trace waterfall entry points, and trace-navigation help. - `SLOs`: SLO attainment, burn rates, error budget, and alert-threshold context. ### Drill-down and data-link patterns Preserve the dashboard time range on all drill-down links using Grafana time macros such as `${__url_time_range}` or explicit `from=${__from}&to=${__to}` parameters. Prometheus panels should link to domain dashboards with low-cardinality variables first, then to Loki/Tempo views for per-execution context. Required link patterns for new or changed panels: - Overview health/SLO panels link to Export Health, Errors, SLO Health when present, and Trace Journey. - LLM, model, token, and cost panels link to Models, Cost, Logs and LLM I/O, LLM Conversations, and Tempo traces when a trace ID or session/run filter is available. - Tool latency/error panels link to Tools, Errors, filtered `tool.call.failed` Loki rows, and Tempo traces. - Agent lineage panels link to Agents, Agent Node Graphs, Trace Journey, filtered `event_category="agent-tree"` Loki rows, and Tempo traces. - Export/redaction/drop panels link to Export Health, failure logs, and the relevant SLO/alert context. - Loki rows should expose a Tempo link from `trace_id`/`span_id` when available; otherwise they should link back to Trace Journey using `session_id`, `agent_id`, and `agent_run_id` filters. Do not place raw prompt, response, command, path, or raw error-message values into dashboard URLs. Prometheus links may pass only low-cardinality labels; high-cardinality values must originate from Loki/Tempo labels or trace/log fields. ### Threshold colors and empty states Use consistent threshold colors: | State | Color | Meaning | |---|---|---| | Healthy | Green | Within objective or zero failures for the selected range. | | Warning | Yellow/Orange | Degraded trend, non-zero recoverable failures, elevated burn, or approaching an alert threshold. | | Critical | Red | Active alert condition, redaction failure, export drop/error, lineage break, or SLO violation. | | Idle/no data | Gray/Blue | No workload or optional capture disabled; panel description must explain why this can be healthy. | Every failure-only log panel must state that an empty table means no matching failures were observed in the selected range. Every zero-safe Prometheus stat that uses `or vector(0)` must explain whether zero means healthy idle, no events yet, or an actual zero failure rate. Content-capture dashboards must state that prompts, responses, and thinking logs are redacted and opt-in. ### Operator drill-down workflows Use the dashboards as a guided investigation path instead of opening raw metric families randomly: 1. **General health:** start in Overview, inspect the `Health` row, then follow the panel link to SLO Health, Export Health, Errors, or Trace Journey. Keep `${__url_time_range}` on all links. 2. **Multi-agent lineage:** start in Overview or Trace Journey, copy/filter by `session_id`, `workflow_id`, `agent_id`, or `agent_run_id` only in Loki/Tempo dashboards, then use Agents and Agent Node Graphs for aggregate ratios and topology. 3. **Cost/model changes:** start in Cost or Models, compare burn rate, token mix, cache/reasoning usage, stop reasons, and model/thinking annotations, then use Logs and LLM I/O or LLM Conversations for per-run context. 4. **Latency/tool/error:** start in Latency, Tools, or Errors, use top-offender tables that pair percentile/rate with volume, and open linked Loki rows or Tempo traces for representative executions. 5. **Export trust:** if panels are blank or suspicious, inspect Export Health before interpreting product dashboards; an idle range can be healthy, but drops, export errors, redaction failures, handler errors, or propagation failures should be visible as non-green chips or failure logs. Zero-state interpretation checklist: - Empty failure-only Loki tables mean no matching failures in the selected range, not that the log pipeline is broken. - `or vector(0)` stats must say whether zero means healthy idle/no activity or a true zero failure ratio. - Optional redacted content panels can be empty when capture is disabled or when the Collector dropped content attributes by policy. - Active-session estimates are approximate across process restarts and counter resets. Raw `observme_active_agents` values are lifecycle claims, not authoritative liveness after an ungraceful exit. - High-cardinality execution identifiers are allowed as Loki/Tempo filter variables only and must never be introduced into Prometheus queries or panel links from aggregate panels. ### 1.3 Active-agent leased-state contract The authoritative metric, duration, export-interval relationship, clock-skew allowance, future-timestamp horizon, convergence target, and failure-mode behavior are defined once in [`04-telemetry-semantic-conventions.md` §12.4.1](04-telemetry-semantic-conventions.md#1241-active-agent-lease-and-clock-contract). Dashboards, alerts, examples, and tests must use that contract without introducing panel-local lease or skew windows. Operational active-agent views combine a positive `observme_active_agents` lifecycle claim with a currently valid `observme_agent_lease_expires_unixtime_seconds` value on the generated `observme_instance_id` resource label. The instance label is an internal join/deduplication key only: it must not be exposed as a dashboard variable, group, legend, URL value, or topology identity. Duplicate scrape/exporter representations of one instance count once. Interpret active-agent states as follows: - positive claim plus a valid current lease: active; - clean-shutdown claim of zero: inactive after normal export/scrape propagation, without waiting for lease expiry; - positive claim plus an expired lease: inactive stale claim, available only as a diagnostic; - missing, malformed, non-finite, or pathologically future lease: inactive (fail closed); - no matching series: zero-safe healthy idle/no activity when the telemetry backend itself is reachable. A crash, `SIGKILL`, cancelled job, or lost runner therefore converges to zero within the semantic-convention bound even if the Collector continues to expose the cached raw claim. A Collector/export outage can intentionally undercount after the last delivered lease expires; use Export Health and backend reachability to distinguish that state from process termination. Clean lifecycle handling and GitHub Actions `if: always()` cleanup remain useful latency optimizations, but neither is the correctness mechanism. ### 1.3.1 Canonical active-agent PromQL Use the following total everywhere current live-agent state is required: ```promql sum( max by (observme_instance_id) ( (observme_active_agents > 0) and on (observme_instance_id) ( (observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305) ) ) ) or vector(0) ``` Both comparisons are required. The lower bound expires stale claims; the upper bound rejects leases beyond the frozen 305-second future horizon. `and on (observme_instance_id)` fails closed when the lease is absent or non-comparable, and `max by (observme_instance_id)` removes duplicate Collector/exporter/scrape replicas before the outer sum. The final `or vector(0)` makes no-series results render as zero; when Prometheus and Export Health are reachable, zero can mean healthy idle, clean shutdown, an expired claim, or a lease that failed closed. For a bounded emitted dimension, retain that dimension in both aggregations. Grouped queries use `or on() vector(0)` so the unlabeled zero appears only when no labeled breakdown exists. Canonical role and environment breakdowns are: ```promql sum by (agent_role) ( max by (observme_instance_id, agent_role) ( (observme_active_agents > 0) and on (observme_instance_id) ( (observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305) ) ) ) or on() vector(0) ``` ```promql sum by (environment) ( max by (observme_instance_id, environment) ( (observme_active_agents > 0) and on (observme_instance_id) ( (observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305) ) ) ) or on() vector(0) ``` The shipped active claim and lease currently emit `agent_role` and `environment`; they do not emit the metric label `subagent_depth`. Therefore current-active depth panels must not group by `subagent_depth`. For a compatible pipeline that actually emits the same bounded `subagent_depth` on the active claim, this is the supported depth shape: ```promql sum by (subagent_depth) ( max by (observme_instance_id, subagent_depth) ( (observme_active_agents > 0) and on (observme_instance_id) ( (observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305) ) ) ) or on() vector(0) ``` Do not substitute resource-derived or high-cardinality execution labels for these dimensions. Use the bounded spawn/tree metrics for depth views in the shipped dashboards until `subagent_depth` is emitted on the active signals themselves. Raw positive lifecycle claims are diagnostic only: ```promql sum(max by (observme_instance_id) (observme_active_agents > 0)) or vector(0) ``` Expired positive claims are diagnostic and never contribute to the live total: ```promql sum( max by (observme_instance_id) ( (observme_active_agents > 0) and on (observme_instance_id) (observme_agent_lease_expires_unixtime_seconds <= time()) ) ) or vector(0) ``` Compare raw claims, expired claims, the canonical live total, and Export Health when diagnosing a producer crash or export gap. A raw-minus-live difference can also include a missing, malformed, or pathologically future lease, so it must not be labeled strictly as “expired.” Never expose `observme_instance_id` as a variable or legend in these diagnostics. ### 1.3.2 Migration from raw active-agent queries Use this rollout order for custom dashboards, recording rules, alerts, and API consumers: 1. Deploy ObservMe producers that emit both `observme_active_agents` and `observme_agent_lease_expires_unixtime_seconds` with the same generated `observme_instance_id` resource label. 2. Confirm the lease metric is present and clocks are synchronized before changing consumers. A missing lease fails closed, so a mixed-version producer without the new metric will not appear in the leased total. 3. Replace raw `sum(observme_active_agents)`, raw role sums, and any equivalent positive-claim aggregation with the canonical total or bounded-dimension form above. Preserve the future-horizon check, instance join, deduplication, and zero-safe fallback together. 4. Keep raw and expired-claim queries only in panels or runbooks explicitly labeled as diagnostics. Do not use them for paging, capacity decisions, or current topology. 5. Keep Collector `metric_expiration` longer than the lease. Do not shorten expiration or schedule Collector restarts to correct liveness; an expired lease stops counting while the cached raw claim remains available for diagnosis. 6. During rollback, old raw consumers continue to see the compatibility lifecycle claim, but they regain the known ghost-agent behavior after ungraceful exits. Treat rollback as a temporary loss of crash-safe liveness semantics. Validate the migrated consumer with live+valid, clean-shutdown, expired, missing, pathologically future, duplicate-replica, and no-series cases before production use. ## 2. Overview Dashboard Purpose: act as the operator landing page. Users should be able to answer whether ObservMe is healthy, workload is normal, agents/subagents are healthy, spend or latency is high, and which focused dashboard to open next without scanning every raw metric series. Rows and panels: - `Health`: SLO/health chips for `Export health`, `Agent lineage health`, `Workflow completion health`, `Error pressure`, `Cost burn / hour`, and `Latency health`. - `Workload`: compact KPI stats with sparklines for `Active sessions estimate`, `Turns/min`, `LLM requests/min`, `Tool calls/min`, `Subagent spawns/min`, and `Active agents`. - `Cost`: selected-range `Total cost`, `Token mix`, and aggregate `Cost trend` without provider/model detail. - `Latency`: compact p95 sparkline stats for turn, LLM, and tool latency. - `Agent lineage`: selected-range `Lineage failures`, p95 tree depth, p95 fan-out, and current active agents by bounded role/depth labels. - `Links`: navigation to Cost, Models, Latency, Tools, Agents, Trace Journey, Errors, Export Health, and LLM Conversations with the current time range preserved. The overview intentionally keeps only landing-page signals. Provider/model, tool, error-family, export, and full trace/log details belong on the domain dashboards linked from the row or panel links. PromQL examples: Export health chip: ```promql 1 - ((sum(rate(observme_telemetry_dropped_total[$__range])) or vector(0)) / clamp_min((sum(rate(observme_events_observed_total[$__range])) or vector(0)), 1e-9)) ``` Agent lineage health chip: ```promql clamp_min(1 - (((sum(rate(observme_subagent_spawn_failures_total[$__range])) or vector(0)) + (sum(rate(observme_orphan_agents_total[$__range])) or vector(0)) + (sum(rate(observme_trace_context_propagation_failures_total[$__range])) or vector(0))) / clamp_min((sum(rate(observme_subagents_spawned_total[$__range])) or vector(0)), 1e-9)), 0) ``` Active sessions estimate: ```promql clamp_min((sum(observme_sessions_started_total) or vector(0)) - (sum(observme_sessions_shutdown_total) or vector(0)), 0) ``` Use the `or vector(0)` fallbacks for session lifecycle panels because fresh active sessions can exist before any shutdown counter series has been exported. The overview describes this panel as an estimate because process restarts and counter resets can affect the value. Compact workload stats: ```promql 60 * (sum(rate(observme_turns_completed_total[$__rate_interval])) or vector(0)) ``` ```promql 60 * (sum(rate(observme_subagents_spawned_total[$__rate_interval])) or vector(0)) ``` Agent fan-out p95: ```promql histogram_quantile(0.95, sum(rate(observme_agent_fanout_count_bucket[$__rate_interval])) by (le)) or vector(0) ``` Dashboard links must preserve the current time range with `${__url_time_range}` and must not include high-cardinality IDs or raw content values. Use Loki/Tempo drill-down dashboards for session, workflow, agent, run, trace, and span identifiers. ## 3. Cost Dashboard Purpose: show selected-range spend, hourly burn, budget pressure, token/cache behavior, and provider/model attribution without misleading denominators. Panels: - Total selected-range cost, cost burn rate per hour, selected-range budget usage, and projected 24h cost. - Cost by provider, model, and environment with percent-of-total context. - Cost over time as a trend panel suitable for model/thinking/compaction annotations. - Token totals by type: input, output, total, reasoning, cache read, cache write, and cache write 1h. - Cache read/write token usage, cache read ratio, and estimated cache savings where supported by metrics. PromQL examples: ```promql sum(increase(observme_llm_cost_usd_total[$__range])) by (provider, model) ``` ```promql sum(increase(observme_llm_cost_usd_total[$__range])) / clamp_min(sum(increase(observme_llm_requests_total[$__range])), 1e-9) * 1000 ``` ```promql sum(increase(observme_llm_input_tokens_total[$__range])) by (provider, model) ``` ```promql sum(increase(observme_llm_cache_read_tokens_total[$__range])) / clamp_min(sum(increase(observme_llm_cache_read_tokens_total[$__range])) + sum(increase(observme_llm_input_tokens_total[$__range])), 1e-9) ``` ## 4. Tool Dashboard Purpose: identify busy, slow, failing, or oversized tools and connect aggregate symptoms to filtered failure logs/traces. Panels: - Tool calls by name, tool failure rate, and tool p95 latency trend panels. - Interactive user-bash (`!`/`!!`) exit codes over the selected range; assistant Bash tool calls remain in the tool-call panels. - Tool result size distribution with character/count semantics when the source metric uses `*_chars`. - Tool failures by severity with failure count and failure-rate sorting. - Tool latency percentiles with volume so sparse p95/p99 series are not overinterpreted. - Captured failed-tool output from `tool.error.captured` logs only when `capture.toolResults` is explicitly enabled; the panel must identify content as opt-in and policy-processed, preserve multiline output, and remain empty when capture is disabled or redaction fails closed. PromQL examples: ```promql sum(rate(observme_tool_calls_total[$__rate_interval])) by (tool_name) ``` ```promql sum(rate(observme_tool_failures_total[$__rate_interval])) by (tool_name) / clamp_min(sum(rate(observme_tool_calls_total[$__rate_interval])) by (tool_name), 1e-9) ``` ```promql histogram_quantile(0.95, sum(rate(observme_tool_duration_ms_bucket[$__rate_interval])) by (tool_name, le)) ``` Operational Loki failure rows use `event_name="tool.call.failed"` and should expose `tool_name`, bounded error/status fields, and a Tempo link when trace metadata is present. Captured output uses the separate `event_name="tool.error.captured", event_category="tool_content"` stream so aggregate failure logs remain content-free and broad session-log views can exclude captured bodies. ## 5. Agent and Subagent Dashboard Purpose: make root/subagent reliability and orchestration behavior obvious, especially spawn failures, fan-out, depth, orphan lineage, trace-context propagation, wait/join latency, and recovered child failures. Panels: - Agent runs by role, subagent spawns by depth/spawn type/reason, and spawn failures. - p95 agent-run and subagent-spawn duration trends. - Active agents by depth, agent-tree depth/width, fan-out per parent operation, and alert-aligned threshold references. - Orphan agents and trace-context propagation failures. - Parent wait/join latency, child-agent failures, and recovered child failures. - Ratio panels for spawn failure rate, orphan pressure vs spawns/runs, propagation failure rate, and child recovery ratio. - Top tables for slow agent roles, failing spawn reasons, orphan status/reason combinations, and high fan-out roles. - Parent/child handoff table with parsed fields and Trace Journey/Tempo links. - Loki-backed identity table showing the running agent display name, exact role, capability, agent/parent/workflow IDs, and spawned child display name/role. Display names are presentation text only; links and joins use agent, workflow, and trace IDs. PromQL aggregates must avoid high-cardinality workflow IDs and agent IDs: ```promql sum(rate(observme_agent_runs_total[5m])) by (agent_role) ``` ```promql sum(rate(observme_subagents_spawned_total[5m])) by (subagent_depth, spawn_type, spawn_reason) ``` ```promql histogram_quantile(0.95, sum(rate(observme_agent_fanout_count_bucket[5m])) by (subagent_depth, le)) ``` ```promql sum(rate(observme_orphan_agents_total[5m])) by (status, reason) ``` ```promql sum(rate(observme_trace_context_propagation_failures_total[5m])) by (agent_role, subagent_depth) ``` ```promql histogram_quantile(0.95, sum(rate(observme_agent_join_duration_ms_bucket[5m])) by (agent_role, le)) ``` ```promql histogram_quantile(0.95, sum(rate(observme_agent_run_duration_ms_bucket[5m])) by (agent_role, le)) ``` The orphan counter emits exactly the bounded `status` and `reason` labels; role, depth, and execution IDs are not available on that metric. Use Tempo or Loki for per-agent/per-workflow drill-down with attributes normalized by the backend, for example `pi_workflow_id`, `pi_agent_id`, `pi_agent_parent_id`, `pi_agent_root_id`, and `pi_agent_spawn_id`. ## 5.1 Agent Node Graphs Dashboard Purpose: visualize aggregate topology and health for root agents, roles, spawn reasons, and subagent depths without implying a single trace tree. Panels: - `Pi root → agents → subagents`: aggregate root-to-role-to-depth node graph with selected-range counts and red health nodes/edges for failed spawns, orphan lineage, and propagation failures. - `Agent role → spawn reason → subagent depth`: aggregate delegation-path node graph using spawn counts as edge weights. Node graphs use only bounded aggregate labels. Per-workflow/session/agent identifiers belong in Trace Journey, Loki handoff rows, or Tempo spans. ## 5.2 Trace Journey Dashboard Panels: - Trace journey map explaining the `session → agent run → turn → LLM → tool/bash → subagent spawn → wait/join → branch/compaction` flow. - Summary stats for active agents, completed workflows, failed workflows, workflow completion ratio, subagent spawns, LLM requests, tool calls, and trace-context propagation failures. - Journey flow rate over time for turns, LLM requests, tool calls, bash executions, subagent spawns, and compactions. - p95 stage-latency bar gauge for agent-run, turn, LLM, tool, bash, wait, and join spans. - Agent-tree shape over time using active agents, p95 depth, p95 width, and p95 fan-out. - Subagent handoffs by depth and spawn reason. - Loki-backed execution journey events and ordered journey log, filterable by `pi_session_id`, `pi_agent_id`, and `pi_agent_run_id`; these use logs panels because Loki log streams do not provide state-timeline frames without extra transforms. - Parent/child handoff table from `event_category="agent-tree"` logs, with `pi.agent.display_name` beside agent ID/role and `pi.agent.child.display_name`/`pi.agent.child.role` on spawn rows. - Tempo TraceQL table for recent `observme-pi-extension` traces so users can open the full waterfall and keep the current session/agent filters beside matching logs. The dashboard uses high-cardinality IDs only for Loki/Tempo drill-down filters. Prometheus panels continue to use low-cardinality aggregate labels only. ## 6. Model Dashboard Purpose: compare provider/model traffic, reliability, latency, stop reasons, and cost efficiency using semantically valid denominators. Panels: - Requests by provider/model with traffic share context. - Error rate by provider/model paired with error counts and request volume. - p95 latency by provider/model, with adjacent volume context in Latency. - Stop reason distribution as selected-range totals or a true time series. - Cost per 1k LLM requests by provider/model. - Overall cost per completed turn when turn attribution is aggregate-only. PromQL examples: ```promql sum(rate(observme_llm_requests_total[$__rate_interval])) by (provider, model) ``` ```promql sum(rate(observme_llm_errors_total[$__rate_interval])) by (provider, model) / clamp_min(sum(rate(observme_llm_requests_total[$__rate_interval])) by (provider, model), 1e-9) ``` ```promql sum(increase(observme_llm_cost_usd_total[$__range])) by (provider, model) / clamp_min(sum(increase(observme_llm_requests_total[$__range])) by (provider, model), 1e-9) * 1000 ``` Do not divide model-level cost by global turn counts and label the result as model cost per turn. Use provider/model LLM-request denominators, or explicitly label aggregate cost per completed turn when the denominator is global turns. ## 7. Branch and Compaction Dashboard Purpose: explain context-branching and compaction pressure, and provide model/thinking change context for cost, latency, and model dashboards. Panels: - Branches per session estimate, using selected-range counts when possible and explaining sparse-session noise. - Compactions over time, with per-session/turn context where available. - Tokens before compaction histogram with p95 and optional p50/p99 companion views. - Branch summaries created and compaction logs as parsed Loki rows. - Model changes and thinking level changes, also suitable as dashboard annotations for Cost, Models, and Latency. PromQL examples: ```promql sum(rate(observme_compactions_total[$__rate_interval])) by (environment) ``` ```promql histogram_quantile(0.95, sum(rate(observme_compaction_tokens_before_bucket[$__rate_interval])) by (le)) ``` ## 8. Export Health Dashboard Purpose: show whether ObservMe is observing Pi handler activity and whether the local telemetry/export path is dropping, redacting, or failing. This dashboard intentionally combines liveness panels with failure-only panels; failure-only panels can be quiet in a healthy range and should not make the whole dashboard appear blank. Healthy local-session behavior: - A trusted local project where `/obs status` and `/obs health` succeed should show recent ObservMe activity from `observme_events_observed_total` after representative Pi events occur. - Failure counters render as `0` when no failure series exists in the selected time range. - Collector/export health renders healthy when events are observed and local drops/export errors remain zero. - Loki failure tables remain empty unless matching failure logs occur; empty `redaction.failed`, `telemetry.dropped`, `export.failed`, and `trace_context.propagation_failed` tables mean no matching failures were observed in the selected range. - The dashboard contract does not require changing project trust, `/obs status`, `/obs health`, local OTLP endpoint selection, Grafana auth/profile, or local debug capture policy. Panels and signals: - Recent observed event rate: `observme_events_observed_total`. - Handler latency and pressure: `observme_handler_duration_ms` and `observme_handler_errors_total`. - Active SDK spans: `observme_active_spans` by bounded `operation`. - Telemetry drops: `observme_telemetry_dropped_total` by bounded `reason`. - Redaction failures: `observme_redaction_failures_total` by bounded `operation`/`error_class` where available. - Export failures: `observme_export_errors_total` by bounded `reason`/`error_class` where available. - Failure logs: Loki `event_name` values `redaction.failed`, `telemetry.dropped`, `export.failed`, and `trace_context.propagation_failed`. Allowed metric labels for Export Health panels are the low-cardinality labels `operation`, `reason`, `error_class`, and `status`. Dashboard PromQL must not group or filter by session IDs, workflow IDs, agent IDs, trace/span IDs, entry IDs, raw prompts, raw commands, raw paths, or raw error messages. Zero-safe PromQL examples for stat panels: ```promql sum(rate(observme_events_observed_total[$__rate_interval])) or vector(0) ``` ```promql sum(rate(observme_telemetry_dropped_total[$__rate_interval])) or vector(0) ``` ```promql sum(rate(observme_redaction_failures_total[$__rate_interval])) or vector(0) ``` ```promql sum(rate(observme_export_errors_total[$__rate_interval])) or vector(0) ``` ```promql sum(rate(observme_handler_errors_total[$__rate_interval])) or vector(0) ``` Breakdowns can keep low-cardinality grouping while still documenting that an unlabeled zero means no series exists yet: ```promql sum(rate(observme_telemetry_dropped_total[$__rate_interval])) by (reason) or vector(0) ``` Observability Export SLO alignment: ```promql 1 - (sum(rate(observme_telemetry_dropped_total[$__range])) / clamp_min(sum(rate(observme_events_observed_total[$__range])), 1e-9)) ``` ## 9. Loki Queries Errors: ```logql {service_name="observme-pi-extension", event_name=~".*[.]failed|.*[.]dropped|agent[.]orphaned"} ``` Tool failures: ```logql {service_name="observme-pi-extension", event_name="tool.call.failed"} ``` LLM failures: ```logql {service_name="observme-pi-extension", event_name="llm.request.failed"} ``` The dedicated `observme-llm-conversations.json` dashboard is the canonical opt-in content view. It shows a redacted conversation timeline, prompt/response/thinking panels, and trace links. Its Loki template filters cover `pi_session_id`, `pi_workflow_id` where promoted, friendly agent name, `pi_agent_id`, `pi_agent_run_id`, provider, model, and content kind. Friendly names are read from `pi_agent_display_name` structured metadata and may be duplicated, so use `pi_agent_id` for exact correlation; the local Collector promotes the other required normalized labels when capture and label policy allow them. Captured prompts, responses, and thinking (redacted, opt-in log bodies): ```logql {service_name="observme-pi-extension", event_name="llm.prompt.captured"} {service_name="observme-pi-extension", event_name="llm.response.captured"} {service_name="observme-pi-extension", event_name="llm.thinking.captured"} ``` Compactions: ```logql {service_name="observme-pi-extension", event_name="compaction.created"} ``` Subagent spawns: ```logql {service_name="observme-pi-extension", event_name="agent.spawn.completed"} ``` Orphan agents: ```logql {service_name="observme-pi-extension", event_name="agent.orphaned"} ``` Trace-context propagation failures: ```logql {service_name="observme-pi-extension", event_name="trace_context.propagation_failed"} ``` These examples assume Loki OTLP ingestion stores ObservMe OTEL attributes as labels or structured metadata. Dotted OTEL attribute names are queried in Loki with underscores, for example `event.name` as `event_name`, `event.category` as `event_category`, and `pi.session.id` as `pi_session_id`. `event.category` is the ObservMe semantic category, so failure dashboards select by `event_name` rather than `event_category="error"`. ## 10. Tempo / TraceQL Concepts Trace search examples depend on Tempo version and configured attributes. Conceptual queries: ```text Find traces where pi.tool.name = bash and status = error Find traces where pi.llm.cost.total_usd > 0.10 Find traces where pi.compaction.tokens_before > 50000 Find traces by pi.session.id Find child traces/spans by pi.agent.parent_id or pi.agent.root_id Find all traces/spans in an orchestrated workflow by pi.workflow.id Find joins where pi.agent.join.status = timeout or failed ``` ## 11. Suggested Alerts The `> 0` detection alerts deliberately have no `for:` pending duration: a single counter increment stays inside a `rate()` window only for that window's length, so a pending duration equal to the window would require sustained failures and could never fire on the isolated events these alerts describe. ### High LLM Error Rate ```promql sum(rate(observme_llm_errors_total[10m])) / clamp_min(sum(rate(observme_llm_requests_total[10m])), 1e-9) > 0.05 ``` Severity: warning ### High Tool Failure Rate ```promql sum(rate(observme_tool_failures_total[10m])) by (tool_name) / clamp_min(sum(rate(observme_tool_calls_total[10m])) by (tool_name), 1e-9) > 0.10 ``` Severity: warning ### Subagent Spawn Failures ```promql sum(rate(observme_subagent_spawn_failures_total[10m])) > 0 ``` Severity: warning ### Export Drops Detected ```promql sum(rate(observme_telemetry_dropped_total[5m])) > 0 ``` Severity: warning ### Cost Spike ```promql sum(increase(observme_llm_cost_usd_total[1h])) > 50 ``` Severity depends on organization budget. ### Redaction Failures ```promql sum(rate(observme_redaction_failures_total[5m])) > 0 ``` Severity: critical if content capture is enabled. ### Runaway Agent Fan-Out ```promql histogram_quantile(0.95, sum(rate(observme_agent_fanout_count_bucket[10m])) by (le)) > 20 ``` Severity: warning; tune the threshold to the organization's normal orchestrator workload. ### Excessive Agent Tree Depth ```promql histogram_quantile(0.95, sum(rate(observme_agent_tree_depth_bucket[10m])) by (le)) > 5 ``` Severity: warning; tune the threshold to expected maximum delegation depth. ### Orphan Agents Detected ```promql sum(rate(observme_orphan_agents_total[10m])) > 0 ``` Severity: warning. ### Trace Context Propagation Failures ```promql sum(rate(observme_trace_context_propagation_failures_total[10m])) > 0 ``` Severity: warning. ### Active Agents Stuck High ```promql ( sum( max by (observme_instance_id) ( (observme_active_agents > 0) and on (observme_instance_id) ( (observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305) ) ) ) or vector(0) ) > 100 ``` Severity depends on normal fleet size; tune per deployment. This alert uses leased live agents, not cached raw claims. ### Expired Active-Agent Claims ```promql ( sum( max by (observme_instance_id) ( (observme_active_agents > 0) and on (observme_instance_id) (observme_agent_lease_expires_unixtime_seconds <= time()) ) ) or vector(0) ) > 5 ``` Severity is deployment-tunable. This is a diagnostic for stale cached claims left behind by ungraceful producer death, not a liveness alert; the expired claims it counts are excluded from leased active totals. ## 12. SLOs The `ObservMe SLO Health` dashboard (`dashboards/observme-slo-health.json`) surfaces runtime SLO scorecards, 1h/30d burn-rate panels, and alert-threshold references. Export Health also links to the SLO dashboard and shows the liveness, drop, export-error, redaction-failure, and handler-overhead inputs separately so healthy idle ranges are distinguishable from unhealthy export behavior. Failure-side terms are guarded with `or vector(0)`: a counter that has never been incremented has no series, and an unguarded empty vector would propagate through the arithmetic and blank the indicator exactly when the system is healthy. Denominator workload absence still reads as no-data, which is the intended "no workload" state. Bounds are clamped because tiny denominators and window edges can otherwise push ratios outside [0, 1]. ### Observability Export SLO 99% of observed telemetry events are not locally dropped over 30 days. Indicator: ```promql 1 - ((sum(rate(observme_telemetry_dropped_total[30d])) or vector(0)) / clamp_min(sum(rate(observme_events_observed_total[30d])), 1e-9)) ``` ### Agent Lineage SLO 99% of known subagent spawns produce either propagated trace context or parent/child lineage attributes. Indicator: ```promql clamp_min(1 - (((sum(rate(observme_subagent_spawn_failures_total[30d])) or vector(0)) + (sum(rate(observme_orphan_agents_total[30d])) or vector(0)) + (sum(rate(observme_trace_context_propagation_failures_total[30d])) or vector(0))) / clamp_min(sum(rate(observme_subagents_spawned_total[30d])), 1e-9)), 0) ``` `observme_subagent_spawn_failures_total` carries only `spawn_type` and `error_class` labels, so all spawn failures count against the SLO; there is no lineage-specific failure reason today. ### Workflow Completion SLO 99% of started workflows complete or explicitly fail with terminal telemetry over 30 days. Indicator: ```promql clamp_max(((sum(rate(observme_workflows_completed_total[30d])) or vector(0)) + (sum(rate(observme_workflow_errors_total[30d])) or vector(0))) / clamp_min(sum(rate(observme_workflows_started_total[30d])), 1e-9), 1) ``` ### Instrumentation Overhead SLO 99% of event handlers complete in under 10ms. Metric: ```text observme_handler_duration_ms ``` ### Redaction SLO 100% of configured secret test patterns are redacted in CI. This is a test SLO, not runtime-only.