---
title: "Instrumentation"
description: "Configure lifecycle instrumentation, handle runtime events, and control the content each destination receives."
url: "/observability/instrumentation"
---

Add lifecycle instrumentation as files under `agent/instrumentation/`. Each
file handles eve runtime events independently. Configure trace exporters and
shared span settings with the [built-in OpenTelemetry APIs](./otel).

The instrumentation directory replaces `agent/instrumentation.ts`. See
[Migrate instrumentation](/docs/observability/instrumentation-migration) when
upgrading from `eve < 0.62.0`.

## Add instrumentation

The filename identifies the instrumentation slot. A lifecycle-event file must
default-export `defineInstrumentation(...)` or `disableInstrumentation()`.
See [OpenTelemetry](./otel) for the supported OTel declarations.

```text
agent/instrumentation/
  audit.ts  lifecycle instrumentation
```

This instrumentation records tool timing and identity without receiving tool
arguments or results:

```ts title="agent/instrumentation/audit.ts"
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation({
  events: {
    "tool.call.started": (event, ctx) => {
      ctx.state.set({ name: event.toolName, startedAt: Date.now() });
    },
    "tool.call.completed": (event, ctx) => {
      const started = ctx.state.get() as { name: string; startedAt: number } | undefined;
      if (started === undefined) return;

      console.log({
        tool: started.name,
        durationMs: Date.now() - started.startedAt,
        outcome: event.outcome,
      });
    },
  },
});
```

`ctx.state` is JSON storage scoped to this file and operation. It survives
durable suspension and is released after the terminal event.

Use `setup` for asynchronous initialization. In `eve dev` and built servers,
eve awaits every provider's setup before loading the server entry. A setup
failure stops startup. Modules imported by an instrumentation file load before
setup, so keep dependencies that need initialization out of those imports.

## Control inputs and outputs

Each lifecycle file created with `defineInstrumentation(...)` has an
independent `tracePolicy`. It decides whether that file receives a trace and
whether its events include input or output content. OpenTelemetry destinations
share the process-wide policy declared by `otel(...)`.

```ts title="agent/instrumentation/audit.ts"
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation({
  tracePolicy: ({ audience, environment }) => ({
    emit: true,
    recordInputs: audience === "public" || environment === "development",
    recordOutputs: audience === "public" || environment === "development",
  }),
  events: {
    "model.call.started": (event) => {
      console.log("input", event.input);
    },
    "model.call.completed": (event) => {
      console.log("output", event.content);
    },
  },
});
```

The function receives `agentName`, `channel`, `audience`, `environment`, and
`principalType`. `audience` is `"public"`, `"private"`, or
`"unknown"`; `environment` is `"development"`, `"preview"`, or
`"production"`.

Without a policy, or when it returns `true`, eve sends metadata for every
trace. It includes input and output content in these cases:

| Environment           | Audience               | Content            |
| --------------------- | ---------------------- | ------------------ |
| Development           | Any                    | Inputs and outputs |
| Preview or production | `public`               | Inputs and outputs |
| Preview or production | `private` or `unknown` | Metadata only      |

Return `{ emit: false }` to not sample the trace for this file. Return
`{ emit: true, recordInputs, recordOutputs }` to choose the two content
directions explicitly. Inputs include model prompts, tool arguments, channel
input, and user responses. Outputs include model responses, tool results,
requests for user input, model provider metadata, and error details. Omitted
content fields are unavailable to the handler.

The channel assigns the audience once when it creates the session. It controls
content capture, not access. See [Audience](../channels/overview#audience) for
how the channel derives `public`, `private`, and `unknown`.

## Redact fields

Lifecycle events are immutable snapshots. Copy the fields you need into a
destination-specific payload and redact that copy before sending it:

```ts title="agent/instrumentation/audit.ts"
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation({
  tracePolicy: () => ({
    emit: true,
    recordInputs: true,
    recordOutputs: false,
  }),
  events: {
    "tool.call.started": async (event) => {
      await sendAuditRecord({
        id: event.idempotencyKey,
        input: redactApiKey(event.input),
        kind: event.kind,
        name: event.toolName,
      });
    },
  },
});

function redactApiKey(value: unknown): unknown {
  if (typeof value !== "object" || value === null || Array.isArray(value)) return value;

  const record = value as Record<string, unknown>;
  return "apiKey" in record ? { ...record, apiKey: "[redacted]" } : record;
}

async function sendAuditRecord(record: unknown): Promise<void> {
  // Send the sanitized record to your destination.
  void record;
}
```

Prefer `recordInputs: false` or `recordOutputs: false` when the destination does not need an entire content direction. Use field-level redaction only when the destination needs part of that content.

## Lifecycle events

`defineInstrumentation({ events })` supports the following event names. A
start and its terminal event share an `idempotencyKey`; use it as a stable
operation key when upserting records across replays.

| Lifecycle        | Events                                                                                                            | Meaning                                                                                                                                                                                |
| ---------------- | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Session          | `session.started`, `session.waiting`, `session.completed`, `session.failed`                                       | Track a session from creation through suspension or termination. `session.waiting` is resumable; `session.completed` and `session.failed` are terminal.                                |
| Turn             | `turn.started`, `turn.completed`, `turn.cancelled`, `turn.failed`                                                 | Track one turn. Cancellation is not a failure and commonly leaves the session waiting for another turn.                                                                                |
| Channel delivery | `channel.delivery.started`, `channel.delivery.completed`, `channel.delivery.cancelled`, `channel.delivery.failed` | Track an inbound channel delivery while eve processes its turn. These events are emitted when the channel provides delivery metadata.                                                  |
| Step attempt     | `step.attempt.started`, `step.attempt.completed`, `step.attempt.failed`, `step.attempt.metadata`                  | Track each model-step attempt; retries have separate attempts. `step.attempt.metadata` is emitted after a step only when the AI SDK supplies provider metadata.                        |
| Model call       | `model.call.started`, `model.call.completed`, `model.call.failed`                                                 | Track an individual call to the model provider. Completion includes usage and finish reason; content fields follow the trace policy.                                                   |
| Tool call        | `tool.call.started`, `tool.call.completed`, `tool.call.failed`                                                    | Track a tool, skill load, local subagent, or remote-agent call through durable settlement. A failed event's `outcome` distinguishes failure, rejection, cancellation, and abandonment. |
| User input       | `input.requested`, `input.resolved`                                                                               | Track a question, tool approval, or session-limit request and its resolution. The resolution includes an outcome such as `answered`, `approved`, `denied`, or `cancelled`.             |
| Memory operation | `memory.operation.started`, `memory.operation.completed`, `memory.operation.failed`                               | Track a memory-store search or write. Events include the operation name, phase, and memory slot; completion may include a record count and records.                                    |

Each call emits one `tool.call.*` lifecycle to authored providers. SDK execution callbacks do not emit a second lifecycle. Tool calls can remain open across approvals, parked turns, and worker replacement.

For local and remote subagents, `tool.call.completed` and `tool.call.failed` include the child turn's `usage` directly. Token counts are always present when the child reports usage; `usage.costUsd` is present only when the model provider reports a price.

When upgrading, replace `action.started`, `action.completed`, and `action.failed` handlers with their `tool.call.*` equivalents. Read `event.toolName` instead of `event.name` on start events. The OpenTelemetry trace schema remains version 4.

Handlers from different files run concurrently and are failure-isolated. Do
not depend on execution order. Use `flush` to drain buffered records and
`shutdown` to release resources.

## What to read next

- [Migrate instrumentation](/docs/observability/instrumentation-migration): replace `agent/instrumentation.ts`.
- [OpenTelemetry](/docs/observability/otel): configure process-wide settings and third-party destinations.
- [Agent Runs](/docs/observability/agent-runs): configure Vercel's built-in trace destination.
- [Local development](../guides/dev-tui): inspect local traces in the TUI
- [Hooks](../guides/hooks): react to runtime events outside the instrumentation lifecycle