# Hardening loop — probe → judge → fix

> After creating an agent, harden it autonomously: derive probes from its own instructions, run them against the live agent, judge from the trace, and apply **one** surgical edit per round. This goes beyond what the in-app builder does.

`$AGENT` is the agent id (the manifest `id:`). Edits go through `ixora agents update`/`apply`; introspection (when you change a tool) through `$AB ibmi` (see [SKILL.md](../SKILL.md#preflight)).

## 1. Probe (8–12, expected behavior noted for each)

| Bucket | ~n | Tests | Expected |
|---|---|---|---|
| Golden path | 3–5 | typical in-scope questions | correct answer, right named tool fires |
| Edge | 2–3 | ambiguous / boundary | asks to clarify, or admits it can't |
| Tool selection | 2–3 | something a specific named tool should answer | that tool fires, not ad-hoc SQL |
| Adversarial | 1–2 | injection, "modify the data", malformed | refuses; stays read-only |

## 2. Run & judge

```bash
RID=$(ixora agents run "$AGENT" "<probe>" --session-id probe-1 --bypass-confirmations --json | jq -r .run_id)
# run_id is NOT the trace id — map it to the trace, then read the span tree:
TID=$(ixora traces list --run-id "$RID" -o json | jq -r '.data[0].trace_id')
ixora traces get "$TID" --json | jq '.tree'     # span tree = ground truth: which tools fired, in order
```

Mark PASS/FAIL on both axes: did the response match, and did the *right* tool fire? Judge on the trace, not the answer alone. (`--bypass-confirmations` matters only if the agent falls back to raw `validate_and_run_sql` / `run_cl`, which pause; its own named tools don't.)

## 3. Fix — one lever per round

| Symptom | Lever |
|---|---|
| wrong behavior / tone / doesn't refuse | edit instructions → `ixora agents update "$AGENT" --instructions "…"` |
| ad-hoc SQL when a named tool fits, or ignores a toolset | tighten instructions, or `ixora agents update "$AGENT" --toolsets a,b` |
| a tool's description/params confuse the model | rewrite the tool in `<agent-id>.agent.yaml` (re-`validate` via `$AB ibmi`), then `ixora agents apply -f <agent-id>.agent.yaml` |
| history/memory off | set `options:` in `<agent-id>.agent.yaml` → `ixora agents apply -f <agent-id>.agent.yaml` |
| model too weak (last resort) | `ixora agents update "$AGENT" --model provider:model-id` |

Prefer editing the `.agent.yaml` and `ixora agents apply` (keeps the repo source of truth in sync); use `ixora agents update --<field>` for a quick one-field tweak. Either publishes a new version immediately — no restart. Re-probe the failures plus a passing spot-check (edits can regress). **Cap ~5 rounds**; if a probe fails 3× on the same lever it's a tool gap or model limit, not a prompt fix.

## See also
- [`manifest.md`](manifest.md) — the manifest fields the levers edit
- `use-ixora` skill → `references/traces-sessions.md` — reading span trees