---
name: improve-collect
description: "Gather read-only evidence of what a deployed agent (Recipe) did, from test chats, production or a requested sweep: flag failures, sample controls, count prevalence. Use when asked to fix or improve a deployed agent, or handed an issue."
metadata:
  min-cli-version: "0.82.0"
  next: "improve-classify"
---

# Improve: collect

Gather what one deployed version actually did, flag its failures, and hand improve-classify evidence it can diagnose. This step changes nothing.

<!-- rules to carry (improve-collect, from introspection-operator docs/migration-rules.json): all carried. -->

## Before you start

You need the `introspection` CLI 0.82.0 or later, logged in to the project that runs the agent:

```sh
npm i -g @introspection-ai/cli && introspection login
introspection whoami   # the project, its Data Plane and the login's scopes
```

A read the scopes don't cover is a gap to report, not to work around. No deployed version means no traces: go back to build or deploy.

Read a reference when its step links it and the case arises. Production usually needs only [the read set](references/runtime-and-slice.md#the-read-set) and [patterns](references/patterns.md).

## Read-only until the gate

<!-- rule: R-read-only-evidence-before-gate -->

Gather all read-only evidence you can reach without asking; the owner's confirmation comes at improve-classify's gate and covers edits, never reading.

Nothing in this step writes: no branch, case, repository file, annotation or pull request, and no eval runs. Keep exports and notes in the build folder's `runs/`, outside the repository, and record evidence by identifier.

Read [live work](references/live-work.md) before touching a task awaiting input or a live judge, experiment, runtime or binding.

<!-- rule: R-never-simulate-investigation -->

Speak only from evidence you read in this session. When you can't reach it, or for a dry run, say what is unverified and what would settle it. Never claim an inspection you didn't do or invent counts, traces, causes or results.

**Counts come from a command,** named with its view and field; never from `jq`, `grep` or arithmetic over the exports.

## 1. Read the request

A feature, a failure and a hypothesis start differently: read [the request](references/request.md).

## 2. Resolve the version and its commit

<!-- rule: R-confirm-evidence-matches-code-target -->

Resolve the version that served the conversations and the commit it was built from, and confirm the evidence and the code you read describe that version before concluding anything:

```sh
introspection runtimes get <slug|version-id> --resolve -o json > runs/collect/runtime.json
```

Read later runtime facts from this file ([runtime.json](references/runtime-and-slice.md#runtimejson)), and the Recipe at that commit (`git show <sha>:<path>`). Another commit's conversation is another version's evidence.

## 3. Read the baseline at that commit

Read it as [the baseline](references/runtime-and-slice.md#the-baseline) says: a case failing at that commit is evidence before any conversation is read.

## 4. In the loop: the owner's tests of a branch's version

Export and read the owner's tasks on the branch's version as [test chats](references/test-chats.md) says.

## 5. Then production: every evidence family, read the same way

After the tests, or for a production request, count each evidence family, the conversations and the ended tasks, by the version and `production` over one window, never by the runtime group:

```sh
rm -f runs/collect/{feedback,judgement,annotation}.json runs/collect/{named,controls}.ids
V=<version-id> START=<window start> END=<window end>
M='[{"measure":"conversation_id","aggregation":"count_distinct"},{"aggregation":"count"}]'
count() {  # <family> <view> <metrics> <dimensions> [<filter>]
  printf '{"view":"%s","metrics":%s,"dimensions":%s,"filters":[{"field":"runtime_id","operator":"in","value":["%s"]},{"field":"environment","operator":"eq","value":"production"}%s],"from_timestamp":"%s","to_timestamp":"%s","config":{"row_limit":10000}}' \
    "$2" "$3" "$4" "$V" "${5:+,$5}" "$START" "$END" > runs/collect/$1-counts.query.json
  introspection metrics query @runs/collect/$1-counts.query.json -o json > runs/collect/$1-counts.json
  echo "== $1"; jq -c '.meta.truncated, (.data[] | [.dimensions[].value, .metrics[].value])' runs/collect/$1-counts.json
}
count observation observations "$M" '[{"field":"lens"},{"field":"pattern_id"},{"field":"pattern_name"}]'
count observation-conversations observations "$M" '[{"field":"conversation_id"},{"field":"lens"},{"field":"pattern_name"}]' '{"field":"pattern_id","operator":"neq","value":""}'
count feedback events "$M" '[{"field":"event_name"}]' '{"field":"event_name","operator":"eq","value":"introspection.feedback"}'
count judgement judgements "$M" '[{"field":"judge_id"},{"field":"judgement_result"}]'
count annotation events "$M" '[{"field":"event_name"}]' '{"field":"event_name","operator":"in","value":["introspection.annotation.label","introspection.annotation.comment","introspection.annotation.assign"]}'
count conversation conversations '[{"aggregation":"count"},{"measure":"has_error","aggregation":"sum"}]' '[]'
introspection conversations list --filter runtime_id=$V --filter environment=production --start $START --end $END \
  --page-all -o json > runs/collect/conversations.json
introspection tasks list --runtime-id $V --status failed --status cancelled --limit 1000 -o json \
  --query "[?kind=='agent' && metadata.runtime_environment=='production']" > runs/collect/ended-tasks.json
```

A truncated count, a feedback, judgement or annotation row, a redeploy in the window or a `tasks list` `--next` cursor: read [evidence families](references/evidence-families.md) and [metrics query](references/metrics-query.md).

**Then the conversations a signal names:**

```sh
{ jq -r '.data[].dimensions[0].value' runs/collect/observation-conversations-counts.json
  jq -r '.[].conversation_id' runs/collect/{feedback,judgement,annotation}.json 2>/dev/null
  jq -r '.[] | select(.metrics.failed_tool_use_count > 0 or .metrics.has_errors) | .id' runs/collect/conversations.json
  jq -r '.[].id' runs/collect/ended-tasks.json; } | sort -u > runs/collect/named.ids
```

**Then draw the controls** from the rest ([controls](references/controls.md)).

**The first read is one projected stream** over those and the controls, never all of `conversations.json`:

```sh
cat runs/collect/{named,controls}.ids | sort -u > runs/collect/read.ids
introspection conversations export --ids-file runs/collect/read.ids \
  --query "(attributes.gen_ai.operation.name == 'chat' || status.code == 'Error') && {conversation: attributes.gen_ai.conversation.id, turn: attributes.introspection.turn, user: join(' ', attributes.gen_ai.input.messages[?role=='user'] | [-1].parts[?type=='text'].content || to_array('')), agent: join(' ', attributes.gen_ai.output.messages[].parts[?type=='text'].content[] || to_array('')), calls: attributes.gen_ai.output.messages[].parts[?type=='tool_call'].name[], omitted: [attributes.gen_ai.input.messages[?role=='user'] | [-1].parts[?type!='text'].type, attributes.gen_ai.output.messages[].parts[?type!='text' && type!='tool_call' && type!='thinking'].type[]][][], tool: attributes.gen_ai.tool.name, error: status.code == 'Error' && (attributes.error.type || status.code)}" \
  > runs/collect/read.jsonl
```

One line per model call or failed span ([the read set](references/runtime-and-slice.md#the-read-set)).

**Then the version's patterns,** whatever the request, as [patterns](references/patterns.md) says.

## 6. A sweep, only when the owner asks

Improve never sweeps unasked or on a schedule; when the owner asks, run it as [sweep](references/sweep.md) says.

## 7. Flag failures with the four signals

Most trustworthy first:

1. **The owner says so:** a test-chat correction or negative feedback; a user's rating is evidence, not a verdict.
2. **Mechanical signals:** failed tool calls, refused edits, errors, runs cancelled, torn down idle or ending on an unanswered question.
3. **Patterns** and their observations, read under a lens.
4. **A judge model,** only once owner-labelled cases calibrate it; otherwise it gives leads, not flags.

<!-- rule: R-zero-pattern-count-not-proof -->

A zero pattern or observation count is not proof that nothing is wrong: analysis covers conversations partially, and real tool failures can carry no observation. When every signal is zero, check the analysis ran over the window and read the controls ([controls](references/controls.md) sizes them), not the slice, before calling a run clean.

## 8. Read each flagged conversation in full

Read in full only the conversations a signal names, and the controls, from the task row on ([conversation evidence](references/conversation-evidence.md)). Their lines in `read.jsonl` are the full read when they hold all of it ([the read set](references/runtime-and-slice.md#the-read-set) says when); export the rest and [check the export](references/runtime-and-slice.md#a-full-export):

```sh
introspection conversations export --ids-file runs/collect/full.ids --format trajectory --out-dir runs/collect/full
```

MCP servers, experiments, who changed what, retention: [evidence that needs care](references/special-evidence.md).

## 9. Choose the review set

From the flagged conversations and the controls, as [controls](references/controls.md) says.

## 10. Count prevalence on the aggregate surface

<!-- rule: R-prevalence-from-metrics-surface -->

Call something common, rare, rising or regressing only from an aggregate metrics query over the window ([metrics query](references/metrics-query.md)), never from the conversations you read.

<!-- rule: R-no-rates-from-partial-list -->

Take traffic, quality and cost from metrics; never work out a rate for the project from a partial list.

## 11. Review a large body in parallel

More than 30 conversations, or about 50 KB in `read.jsonl`, goes to `evidence-analyst` subagents as [parallel review](references/parallel-review.md) says; only their summaries come back.

## 12. Hand the evidence on

Give improve-classify the version, commit and window; each flagged conversation with its signal, ids, turn and the owner's words; the controls and `read.ids`; every count with its query, and the families' rows; the patterns per conversation, described, those the owner named unwanted, and the newest runs; failing baseline cases with their criteria's reasoning; and every open gap.

## Done when

- The version and commit are resolved and all evidence checked against them; the baseline is read or its absence named.
- Every flagged conversation was read in full with its signal and ids, the owner's corrections in their words, and every count names its query.
- Nothing was written or changed; every gap names the evidence that would close it.

Then go straight on with `improve-classify`.