# Strategy notes — 2026-08-14 Captured from a working session investigating why P4-03 (EstimateStudio) kept failing, and what agent-context-card should do next. Not a decision record, just the points worth not losing. ## What we found this session - Original P4-03 run (minimax-m3, routed via `mycoder`): 134 requests, purely `bash`/`read`, zero edits. Session ended mid-"let me start" with no tool call. Not caused by the card — no pinned plan, no `sources`, nothing to pin it on. Looked like plain analysis-paralysis on an oversized task. - Rerun after landing two card fixes (below), same repo state, `mycoder` again: this time it actually edited code (8 successful edits). But the backend model silently changed too (`minimax-m3` → `glm-4.7-flash` — the router isn't a fixed model), so this is not clean evidence the fixes caused the improvement. - Rerun again with a genuinely pinned model (`openrouter-pinned` / `openai/gpt-5-nano`), same starting state: zero edits again. Proximate cause this time was concrete: it read a minified Angular build bundle (`web/dist/.../chunk-*.js`), which alone pushed context usage from ~92% to ~96% of its (128K) window, and its very next response came back empty (`stopReason: "length"`) — it ran out of room before it ever got to decide whether to edit. - Across all three: `retired.findingConsumed` stayed 0. Neither model, in any run, ever populated `findings[].sources` on `update_card`, so the new finding-sourced retirement mechanism has zero live evidence yet, positive or negative. ## Two fixes landed this session (both tested, both live-confirmed for A) - **Task A**: `formatContextCard` no longer lists `filesRead` entries whose state is `"active"` — those are already fully visible in the live transcript, so repeating the path was pure redundant token cost. Confirmed live in both reruns (card correctly omitted the line when all evidence was active). - **Task B**: `update_card` findings can carry an optional `sources: string[]`. A read cited by a later finding's `sources` becomes eligible for retirement after a grace round, mirroring the existing mutation-consumption path. New `findingConsumed` retirement counter. Unit-tested rigorously (including a fault-injection check: disabled the wiring, confirmed the new test failed, restored it). No live evidence yet either way — see above. ## Three next-step ideas I proposed, and why they were rejected 1. **Round-based staleness fallback** ("no mutation after N rounds → stale"). Rejected: this is a timer, the same class of arbitrary cutoff the project's own README explicitly argues against ("fixed limits ignore task complexity"). It would retire evidence that's still relevant just because N rounds passed, and keep irrelevant evidence if they haven't. 2. **Guard against oversized/build-artifact reads** (denylist `dist/`, `.min.js`, size thresholds). Rejected: special-cased to one failure we happened to observe. "Today it's this repo, tomorrow it's something else" — doesn't generalize, and we can't maintain a list of every kind of large-but-unhelpful content a coding agent might read. 3. **Lean harder on the existing eval harness** (`bun run eval:pi:ten-turn`, `evaluation/configs/*`) instead of more one-off manual `pi -p` trials. Directionally right for controlled comparison, but still a curated benchmark — not evidence from real usage. ## The reframe: think bigger The project's own stated design principle is to act only on **observable lifecycle events** (a mutation happened, the agent called `update_card`, a new turn started) — never arbitrary limits, never extra model calls. Both rejected ideas above quietly violated that principle by inventing a new special case (a clock, a denylist) instead of finding the general, already-latent signal. The unifying diagnosis: the unbounded-reads problem and the oversized-single- read problem are the same problem stated twice — **evidence enters context, and nothing observes whether it was ever actually used again.** Mutation- consumption and finding-citation are two specific instances of "this got used." A third, more general instance is already sitting unexploited in the transcript: `hasReferenceOverlap`/`terms()` (in `projection.ts`) already computes whether text overlaps with a piece of evidence — today it's only ever pointed at the _current_ live turn's text, to decide whether to _exempt_ something from retirement. Pointed forward across the whole transcript instead, it becomes a deterministic, zero-extra-calls signal of non-engagement: did any later assistant text, tool call, or finding ever reference this evidence again? If nothing ever came back to it, that's a real fact regardless of how many rounds passed (not a timer) and regardless of what kind of file or output it was (not a denylist). Second thread, orthogonal to the mechanism: the benchmark-vs-real-tests point. What I did by hand this session — pull a session log, work out how much of it was reads nobody ever came back to — is itself a deliverable. Packaging that as a standalone, offline analysis anyone can run against **their own** existing session logs (no live extension required) would let the community see the token-waste problem is real from their own usage, not from a curated benchmark we ran ourselves. DuckDB surfaced a stack of the user's own real EstimateStudio dev sessions (Aug 2, Aug 12) sitting right there as exactly that kind of real material. ## main vs. this branch, checked directly in code (not assumed) `main` already had the context card _and_ the projection engine before this branch existed — `format.ts`/`RuntimeCard` are not new. What main's `RuntimeCard` does NOT have: `findings`, `filesRead`/`hotEvidence`. Main's projection.ts has no `isUpdateCardCall`, no checkpoint mechanism, no `consumedByFinding` (obviously — that's this session's addition). Main's card is built entirely from mechanically observable facts (goal, pinned plan text, an execution journal built by scanning tool calls, capabilities sniffed from package.json, resumed cross-session facts) — nothing in it is the agent's own interpretation. Main decides what raw transcript to forward purely from `consumedReads`/`consumedDiscovery`/turn-boundary collapse — automatic, no agent cooperation needed. This branch (`bc9bbf3` "Add self-maintained context card: agent-declared findings replace guesswork", plus `e7b46ff` forcing it via tool_choice) adds a genuinely different bet on top of the same base engine: let the agent self-report conclusions (`findings`, now `sources`) via `update_card`, and let those self-reports both stand in for retired evidence _and_ (via the checkpoint mechanism) trigger more aggressive collapse of everything before them. This is additive — main's mechanisms are all still running underneath — but the branch's whole thesis rests on the agent actually using `update_card` well. **The forcing mechanism already exists, and we have live evidence on how well it works.** `CARD_ACTIVITY_NUDGE_THRESHOLD = 10` triggers a soft nudge; if unanswered, `tryForceUpdateCardToolCall` forces the tool via `tool_choice`, capped at `CARD_NUDGE_STREAK_CAP = 2` before backing off until the next successful call resets it. Across the two live sessions run _after_ `sources` existed (`b198b128`, `f6e27b58`), `update_card` WAS called (2x, 1x) — the forcing works, in the sense that the tool gets invoked. But the content was thin every time: one call carried `findings: []`, another only `pending`, none carried `sources`. **Forcing compels compliance with the form (call the tool) but not the substance (write a real finding, cite real sources).** That's a specific, now-observed weakness of this branch's central bet, not a hypothetical one. ## Answered: is "point reference-overlap forward" actually proposing main's approach? Yes, in kind. It's an automatic, transcript-only signal that doesn't depend on the agent doing anything right — same category as main's `consumedReads`/`consumedDiscovery`, not the same category as this branch's `findings`/`sources` (which need agent cooperation, and empirically don't reliably get it even when forced). It wouldn't have to replace anything — it could sit alongside `consumedByFinding` as a third trigger in the same family. But the real strategic question it raises: given three live trials never produced a populated `sources` field even with forcing active, should agent self-reporting remain the _primary_ mechanism this branch is betting on, or should it become an opportunistic bonus layered on an automatic safety net that works with or without the agent's cooperation? ## User's next input: caching split + how to actually get summaries **Branch thesis, stated precisely:** send the context card on every LLM call, split into two messages for prompt-caching: 1. Constant: goal, project info — rarely changes, should stay cache-hit across many requests. 2. Variable: findings, reads, sources — changes often, currently forces a full-card re-hash (and cache miss for everything after it) on every change, because `formatContextCard` emits one monolithic string today. This is a real, concrete, currently-unaddressed cost problem, independent of the summarization debate below — worth doing regardless. **The deeper question raised:** the card doesn't have to be pure determinism-or-raw-evidence; there could be a summary in the card in place of raw messages, without depending on the primary agent volunteering it via `update_card`. Three mechanisms proposed: a. **A separate background LLM call** that looks at about-to-retire evidence and writes the summary, mechanically inserted into the card. This is the same tradeoff class as two of the README's own explicitly _rejected_ alternatives, just moved to a different call site: it's "model-maintained memory" (an extra call whose only job is bookkeeping) and it risks "summaries can drop exact implementation details" (the README's first listed reason for not doing this at all). Not free — extra latency, extra cost, and a sync problem (if it's async, is the raw evidence already gone by the time the summary lands?). Worth exploring, but it's a bigger philosophical reversal than it looks like at a glance. b. **Piggyback a synthesis field onto the turn that already consumes the tool output** — no separate call, folded into the primary agent's next response. Structurally the most promising: stays inside "no extra model calls," and can reuse the `tryForceUpdateCardToolCall` infrastructure that already exists and already works _at the form level_ — just retarget the trigger from "10 turns of inactivity" (proven too generic — 3-for-3 empty `sources` even when forced) to "you just read something costly, before you do anything else." A specific, freshly-relevant ask is more likely to get a real answer than a periodic nag that's easy to satisfy with `findings: []`. c. **A small classifier/summarizer model.** User's own read, which matches mine: limited capacity for large/complex content, and it's the exact "summaries can drop exact implementation details" risk the README already named as a reason to avoid this path. Lowest priority. **Done:** hand-ported and extended the split. `formatContextCard` now emits only the stable header (goal, taskId, latestRequest, project, repo, plan, capabilities). New `formatCardStatus` covers everything volatile (execution changes/failures, pending, findings, filesRead, resumed) and renders as a separate `` block, sent as a trailing message after the projected conversation instead of folded into the leading card — same mechanism as `d18c2fd`, extended to cover findings/filesRead/pending which didn't exist when that commit was written. New `STATUS_MESSAGE_TYPE` constant, new `statusChars` audit field. All tests updated and passing (94/94), tsc/eslint/prettier/build clean. **Done: automatic disuse retirement (step 1 of the agreed strategy).** Added `consumedByDisuse` to `src/core/projection.ts`, the third member of the `consumedReads`/`consumedByFinding` family: a read retires once nothing later — no later assistant text, no later tool call's path or arguments — ever references it again, with the same one-successful-round grace requirement as its siblings. Not a timer: there's no fixed N, it's recomputed fresh against the full transcript every request, so a read retired at one point comes back into scope the moment something later genuinely references it again (proven with a dedicated test: same read, same grace point, differs only in whether a later round exists). New `disused` counter on `RetirementCounts`, wired into both the per-turn candidate filter and the global `activeRounds` computation, with its own fault-injection check (disabled the wiring, confirmed multiple tests failed including a pre-existing checkpoint one, restored it). Hit a real, subtle interaction while building it, worth remembering: local recomputation inside `projectTurn` is blind to anything outside whatever slice it's given, so a read in a checkpoint's _prefix_ whose only later reference lives in the checkpoint's _suffix_ looked "never referenced again" to the local computation even though the global one knew better. This is a pre-existing architectural property (the same blindness already affected `consumedReads`/`consumedByFinding`, just usually hidden because mutation/citation triggers are rarer than plain disuse), not something introduced by this change — but disuse is far more sensitive to it, since it requires _positive_ evidence of continued use rather than a specific trigger event. Fixed the immediately-visible test fallout (some fixtures needed a legitimate later reference to isolate what they were actually testing; one test's original assertion turned out to depend on the blindness accidentally working in this project's favor, and got corrected instead of preserved). Did not do a deeper structural fix (threading global active-round truth into every local recomputation) - that's a larger, separate piece of work if it turns out to matter in practice, not something to take on inside this change. **Done: step 2 (tighten the forcing trigger + validate substance).** Two changes to `src/pi/index.ts`: - A read over `COSTLY_READ_CHARS` (4000 chars) now pushes `cardActivitySinceUpdate` straight past the nudge/force threshold instead of waiting for ~10 generic activity units to accumulate — reuses the existing nudge (`turn_end`) and force (`before_provider_request`) machinery unchanged, just retargets when it fires. Guarded against `event.result` being `undefined` (a real crash the first test run caught — `messageText` assumes a defined object). - `update_card`'s `execute()` now checks whether the response has real substance (a non-empty `pending` item, or a finding with non-empty `detail`) whenever it's answering a _forced_ call (tracked via a new `awaitingForcedSubstance` flag set right when `tryForceUpdateCardToolCall` actually forces). A thin forced response no longer resets `cardActivitySinceUpdate`/`cardNudgeStreak`/`forceNudgeStreak` — forcing compels the call, not the content, so only a real answer earns the reset. Voluntary (non-forced) calls are untouched — still always reset, since a modest voluntary update is still a genuine engagement. 5 new tests in `tests/pi-adapter.test.ts`, each fault-injection-checked (disabled the relevant branch, confirmed the test failed, restored it). 102/102 tests passing, tsc/eslint/prettier/build all clean. **Done: the deeper local/global recomputation fix.** Sent to a background agent (isolated worktree, not touching `src/pi/`) with the full root-cause writeup from this file as its brief. It correctly diagnosed the same root cause independently (`projectTurn`'s `current === true` branch ignores its own `activeRounds` parameter and recomputes everything blind to whatever's outside its slice), and landed the minimal fix: local exclusion is now overridden by membership in the passed-down `activeRounds` - `activeRounds.has(round.index) || (...local checks...)`. Since `activeRounds` is always a subset of "not excluded by any global mechanism," this can only add back a round the global pass already vouched for, never suppress a local exclusion the global pass agrees with - so every existing retirement behavior where local and global already agree (the common case) is untouched. Did not merge the agent's diff blindly. Reviewed it line by line, then independently re-ran its fault-injection check myself in its worktree (disabled the override, confirmed exactly its 3 new tests broke, none of the 14 pre-existing ones did, restored it) before touching my own working tree at all. Hand-applied just the two real changes (the `projectTurn` override, 3 new tests) on top of my current files rather than taking its replicated copies of everything else, since its snapshot predated my step-2 work on `src/pi/index.ts`. Ran the same fault-injection check a third time, independently, directly on my real tree, before accepting it. Cleaned up the worktree and its branch afterward. 105/105 tests passing, tsc/eslint/prettier/build all clean, working tree has only the seven intended files changed. All three items from the agreed strategy are now done: automatic disuse retirement, tightened/validated forcing, and the local/global consistency fix underneath both. ## 2026-08-15: bash-based reads were invisible to all of it Fourth live EstimateStudio trial (same pinned `openrouter-pinned/openai/gpt-5-nano`, same clean baseline, after committing and pushing everything above as `eb9b4e7`). Zero edits again, but two real firsts: it reached a natural stop instead of a truncation, and it populated `findings[].sources` for the first time across four trials, with genuine content. Window still climbed to 77% with `findingConsumed`/`disused`/`staleRead` all at 0 nearly the whole session, though - not because retirement failed, but because the session explored via 26 `bash cat`/`grep`-style calls and only 1 `read` call. Every retirement mechanism (`consumedReads`, `consumedByFinding`, `consumedByDisuse`) is gated on `isRead(call)`, which only recognizes the dedicated `read`/`view_file` tool names - a `bash cat file.ts` call is invisible to all three, permanently "active" by construction, no matter how irrelevant it becomes. User's framing, worth keeping verbatim: rejected going further down the "detect every possible read-like tool" path (correctly identified as the same whack-a-mole pattern as the earlier build-artifact-denylist idea I'd already talked myself out of once) in favor of treating a bash call and a dedicated read call as the same _kind_ of event when they produce the same outcome - full file content entering context - regardless of which tool produced it. Floated a further idea (a canonical store the model reads from instead of the raw tool output) but self-identified it as edging into "semantic memory," one of the README's own rejected alternatives, and scoped this pass to the narrower, purely-internal version: widen detection, keep it invisible to the model, no new tools, no behavior change. **Done:** added `bashReadPath`/`readPath`/`isReadLike` to `src/core/projection.ts`. `bashReadPath` recognizes bash calls whose command is a single, unpiped, unchained, unredirected `cat`/`head`/`tail`/ `less`/`more` of one file, and extracts that file's path conservatively (bails to `undefined` rather than guess on multi-file or ambiguous commands; guards against mistaking a flag's numeric value, e.g. the "50" in `head -n 50 file.ts`, for the path). Every consumer of `isRead`/ `filePath` for read-classification purposes (`consumedReads`, `consumedByFinding`, `consumedByDisuse`, `consumedDiscovery`'s listing-then-read check, `isInActiveRounds`, the `projectionDetails` categorization loop and hotEvidence builder, `hasReferenceOverlap`'s path-substring exemption) now goes through `isReadLike`/`readPath` instead, so a bash-based read is treated identically to a dedicated `read` call everywhere in the evidence lifecycle - not a parallel mechanism, the exact same one. 5 new tests in a new `"reads via bash"` describe block: consumed by a later mutation, retired via finding-citation, retired via disuse, a negative case (piped/chained/redirected commands are never attributed to avoid guessing wrong), and the flag-value extraction edge case. Two of the five needed the same fixture fix as earlier sessions today (a second, later `update_card` so the citation lands in the same projection pass instead of the single-round-prefix "keep the final round regardless" fallback swallowing the result) - the same pitfall, recognized faster this time. Fault-injection confirmed: disabled `bashReadPath`, 4 of 5 new tests failed as expected (the negative case correctly still passed either way), restored. 110/110 tests passing, tsc/eslint/prettier/build all clean. ## 2026-08-15: first SWE-bench re-run since the methodology caveat, officially graded Ran `evaluation/benchmarks/swebench-verified-sympy-18211.json` end to end with today's code: `run.mjs` for the baseline/card comparison, then `grade-swebench.mjs` for official Docker-based SWE-bench correctness grading (found the checked-in venv at `.agent-context-card/swebench-venv`, not `.venv`). Three environment problems on the way, none of them bugs in today's changes: the original benchmark model (`llama-cloud/gemma4:31b`, via Ollama Cloud) is gone from this machine entirely, so substituted `google-ai-studio/gemma-4-31b-it` - which promptly hit a 16k-token/minute free-tier quota wall on request 3. Switched to the pinned `openrouter-pinned/openai/gpt-5-nano` already used for today's live trials, which then rejected the config's `thinking: "off"` ("Reasoning is mandatory for this endpoint"). Set `--thinking low` and it ran clean. Official grading itself then failed once on a Docker daemon connection error - Python's `docker` SDK defaults to the legacy `npipe:////./pipe/docker_engine` pipe, but this machine's active context (`desktop-linux`) exposes `npipe:////./pipe/dockerDesktopLinuxEngine` instead. Setting `DOCKER_HOST` explicitly fixed it. **Officially graded result (Docker-verified, not self-reported):** | | Baseline | Card | | --------------------- | -------- | ---------------- | | Resolved | **no** | **yes** | | FAIL_TO_PASS | 0/1 | 1/1 | | PASS_TO_PASS | 54/54 | 54/54 | | Provider input tokens | 883,089 | 209,778 (-76.2%) | | Requests | 27 | 18 (-33.3%) | | Tool calls | 24 | 15 (-37.5%) | | Tool errors | 9 | 3 (-62.5%) | | Duration | 119.7s | 81.1s (-32.2%) | The historical ledger entry for this exact instance (`gemma4:31b`, the now-removed `fresh`-session-bridge architecture) recorded the same qualitative result - baseline unresolved, card resolved - with similar- magnitude efficiency gains (-79% input, -62.5% requests, -66% tool calls, -100% tool errors, -34.5% duration). Different model, different session architecture, none of the mechanisms built this session existed when the original ran. This is the first re-run since the ledger's own methodologyCaveat flagged the old numbers as "pending re-run, not current evidence," and the correctness claim replicated cleanly under completely different conditions - about as strong an independent check as a single instance can give. One caveat worth being honest about: this instance didn't require much back-and-forth (a `-p` single-shot session did produce a working patch here, unlike every EstimateStudio trial), so it isn't direct evidence against the "ends with a plan instead of editing" pattern observed all day - just evidence that when a model _does_ commit to editing, the card measurably helps. ## 2026-08-15: second pilot (sympy-21930) — two real harness bugs found and fixed, and an honest, less flattering result Same setup, same model. Hit two genuine infrastructure bugs in `scripts/evaluation/`, both fixed (not core product code, but real, reproducible, blocking): 1. **`run.mjs` crashed the whole harness on a runaway tool call.** The model issued a malformed bash command (`bash -lc python -V && python - << 'PY' ...` - mixing `-lc` with an unquoted heredoc chain), which streamed continuously for the full 20-minute turn timeout. `run.mjs` unconditionally accumulated all child-process stdout in a single JS string (`stdout += chunk.toString()`); at ~537MB that exceeded V8's max string length and crashed the entire comparison, losing the already- completed baseline result along with it. Fixed by capping the in-memory accumulation at 50MB and killing the child process past that point - the full raw stream still reaches disk via the existing (already correct) `stdoutStream`/`stdoutFile` mechanism regardless, so nothing about normal-sized runs changed, only the pathological case stopped crashing the harness. 2. **`grade-swebench.mjs` crashed on a legitimate empty-patch outcome.** When an agent makes no code changes at all, the official SWE-bench harness correctly skips grading that instance entirely (nothing to test) rather than writing a per-instance report - which looked identical to a genuine grading failure to the wrapper script, which threw `report missing`. An empty patch is a real, common, unambiguous result (definitionally unresolved), not an error. Fixed by short-circuiting before invoking Docker at all when the patch is empty, recording `resolved: false, emptyPatch: true` directly. Also needed the same `DOCKER_HOST` fix as the first pilot, and the same `timeoutMs` bump (300000 -> 1200000) in the checked-in config, since this instance's baseline implement turn hit the original 5-minute cap even before the crash was diagnosed. **Result, honestly - this one does not repeat the clean win:** | | Baseline | Card | | --------------------- | ---------------- | ---------------- | | Patch produced | yes (2754 bytes) | **no (empty)** | | Resolved | no | no | | FAIL_TO_PASS | 0/6 | 0/6 (no attempt) | | PASS_TO_PASS | 45/45 | n/a (not run) | | Provider input tokens | 360,584 | 169,157 (-53.1%) | | Requests | 21 | 9 (-57.1%) | | Tool calls | 18 | 6 (-66.7%) | Baseline's 0/6 matches the historical ledger entry for this exact instance exactly. But the historical _card_ run got 5/6 FAIL_TO_PASS (close, not resolved) - today's card run made no attempt at all, ending with a fully empty patch despite six tool calls across implement and review. That's a real regression relative to the historical run on this specific instance, not something to gloss over. Consistent with the pattern seen all day (EstimateStudio, and to a lesser extent the first pilot): the failure mode on this instance isn't a context-management problem - card used dramatically fewer tokens and requests to get there - it's the same "explores, maybe edits, then stops before finishing" behavior. Efficiency and correctness are not the same axis, and this pilot is a clean example of the card winning heavily on one while not helping (arguably placing exactly where baseline also failed) on the other. ## 2026-08-15: fifth live trial — the biggest improvement yet, mostly for a different reason than expected Same pinned model, same clean baseline, run immediately after the bash-read fix (uncommitted but live, since the extension loads source directly). Zero edits again - fifth trial in a row without one - but context usage stayed dramatically bounded: ~20% of window by request 55, versus ~77% by request 27 last time, despite this run doing _more_ work (25 bash + 15 read + 10 update_card + 4 update_progress vs. last time's 26 bash + 1 read + 1 update_card). `disused` climbed steadily to 5 over the session - the first time automatic disuse retirement has visibly engaged in a live run. Important correction before crediting the wrong fix: this model used the dedicated `read` tool for actual file content (15 calls) and `rg` for searching (already handled by existing discovery-collapse), not `cat`/`head`/`tail` via bash. The bash-read widening from today wasn't really exercised this run - the improvement is attributable to `consumedByDisuse` correctly retiring ordinary `read` calls nothing came back to, not to the new bash detection. Model/run variance chose a different exploration style than the trial that motivated the fix; the fix itself remains unvalidated live, just unit-tested. Also confirmed working end-to-end: forcing fired 10 times (`activity=11; streak=1` every single time) and never needed a second consecutive force - meaning every forced `update_card` call came back with real substance, not a thin no-op. The tightened-trigger-plus-substance- validation mechanism from step 2 is doing exactly what it was built to do. Minor edge case noticed in passing, not yet a problem: this model wrapped bash commands as `bash -lc "actual command"` - a command string starting with "bash", not "cat"/"head"/etc. `bashReadPath` would not recognize a doubly-wrapped `bash -lc "cat file.ts"` as a read (it only matches the command starting directly with a read-like verb). Didn't come up this run since the model used `rg`/`read` instead, but worth knowing about if a future trial uses bash-wrapped cat and doesn't get picked up. **Where this leaves things:** the context-management side of the original thesis now has real, positive live evidence for the first time across five trials - not just unit tests. The remaining, now-clearly-isolated problem is different in kind: every single trial, regardless of model, context usage, or how much of the card mechanism engaged, has ended with a status write-up instead of continuing into `edit`/`write` calls. That's not a context-budget problem anymore - this run had budget to spare. It's a separate question about why these models stop at "here's the plan" instead of executing it in `-p` single-shot sessions, and nothing built this session addresses it. **Proposed synthesis (not yet agreed, still open):** tighten (b)'s trigger to fire immediately after a large/costly read rather than on a generic activity counter, and — this is the part that ties everything today together — _validate substance, not just form_: if a forced `update_card` still comes back with empty `findings`/`sources`, don't reset the nudge streak as if it succeeded; either re-force or fall through to the automatic, non-cooperative signal (main's `consumedReads`-style mechanism, extended per the reference-overlap idea above) as the real backstop. That reconciles both branches' philosophies instead of picking one: try agent self-report, tightly triggered and validated; if it doesn't deliver, the automatic mechanism doesn't care whether the agent cooperated. ## 2026-08-15: root-caused the sympy-21930 empty patch - one real bug found and fixed, one deeper problem left honestly unresolved The second pilot's raw trace was gone (scratch directory, cleaned up before this thread picked back up), so re-ran the card variant alone (`--variant card`, same pinned model/thinking) twice to get real evidence instead of continuing to theorize from the writeup above. **First re-run, trace inspected directly (DuckDB over the JSONL, per [[feedback_duckdb_for_jsonl]]):** found the actual mechanism, not just the symptom. `before_provider_request`'s `tool_choice` forcing pins the model's entire next response to a single `update_card` call - no other tool calls possible in that response. In the captured trace, the model had just read `secondquant.py` and was actively investigating when forcing hit. Its argument-generation partially broke under the constraint (reasoning text leaked into the `pending` array instead of staying in a separate channel - schema-valid because the field is just `string[]`, so it "succeeded" anyway). Worse: its very next (free-choice) response was a complete, _correct_ patch plan - "modify `_print_Pow` in `sympy/printing/latex.py` to brace the base when it's a daggered operator" - written entirely as prose, then `stopReason: "stop"`. Never called `edit`. The forced interruption reads to this model like a wrap-up cue it doesn't recover from. **Fix:** `update_card`'s `execute()` now sends a `steer` message immediately after a _forced_ call resolves (thin or substantive - both branches derailed in testing), explicitly telling the model the interruption wasn't a stopping point and to resume acting - "if you now have a concrete fix in mind, make it with an edit/apply_patch/write call instead of only describing it in text." `pi.sendMessage(..., {deliverAs: "steer"})` from inside a tool's `execute()` queues for delivery before the model's _next_ generation while the agent is still mid-turn (confirmed against `pi-coding-agent`'s own docs, not assumed) - the exact point where the derailment happened. 3 new tests (forced+thin, forced+substantive, voluntary-call-gets-no-extra-nudge), fault-injection confirmed (commented out the new branch, both positive tests failed as expected, the negative one didn't, restored). 113/113 passing, tsc/eslint clean. **Second re-run, fix live, same config:** the targeted mechanism is confirmed fixed. Forcing fired again in the implement turn (same activity=11/streak=1 shape), landed with real content this time, and - this is the change - the model kept working afterward (a further exploratory `bash` call) instead of immediately stopping. That specific derailment point no longer derails. **Honest result: still an empty patch.** A second, different failure surfaced instead, and it is _not_ forcing-related - no forcing fired anywhere near it. In the implement turn (prompt: "Implement ... Inspect current source before editing"), after hitting an unrelated `rg` syntax error, the model's own thinking read "For now, I won't edit any files but will suggest targeted searches using grep instead," and its final text ended with "If you want, I can provide a precise code edit plan (targeted diffs) once you confirm you want me to proceed with editing the relevant latex.py sections." It's asking a user that will never answer in a `-p` batch harness for permission to edit. Same shape recurred in the review turn on the same run (thinking: "I'll focus on recommending changes without implementing them directly"). This reads as a general confirmation-seeking/caution posture in this specific cheap, pinned model (`openrouter-pinned/openai/gpt-5-nano`, `thinking: low`) - triggered here by a tool error, present with or without forcing in play, and consistent with the very first EstimateStudio observation months back ("Looked like plain analysis-paralysis... Not caused by the card"). Baseline resolved this exact instance's implement turn today with the same model, so it isn't unconditional - but nothing in the card's own mechanism explains why baseline pushes through and card sometimes doesn't beyond the one bug just fixed. Didn't chase this further: it looks like an agent-autonomy/system-prompt framing question (does the model believe a user is present to confirm with), not a context-lifecycle one, and doesn't have an obvious fix inside this extension's actual job. Recording it here rather than guessing at a patch for it. Reconsidered that last line immediately: pi's own `before_agent_start` event exists precisely to let extensions append to the system prompt per turn, and `ctx.hasUI` (true only in `tui`/`rpc` modes, per pi's own docs - false in `print`/`json`) is a _fact_, not a guess, about whether anyone could possibly answer a question. Checked pi's actual default system prompt (`system-prompt.js`) directly rather than assuming: it says "You help users by reading files, executing commands, editing code, and writing new files" - interactive-assistant framing, no mention anywhere that a batch/headless run has no one to ask. That gap is real and in-scope: it's exactly what the captured trace showed the model act on ("once you confirm you want me to proceed with editing"). **Fix:** new `before_agent_start` handler in `src/pi/index.ts` appends an explicit note - "no user is available to answer questions or confirm actions... make it directly with the appropriate tool call rather than only describing it or asking whether to proceed" - but only when `!ctx.hasUI`, so an interactive session is never told there's no one listening when there is. Confirmed the gate is correct for how the eval harness actually runs pi (`--mode json --print`) by reading `extensions/runner.js` directly: `hasUI()` returns `uiContext !== noOpUIContext`, and `print-mode.js` never calls `setUIContext` to override the constructor's `noOpUIContext` default - so `hasUI` really is false in exactly the harness's invocation mode, not assumed. 2 new tests (note appended when `hasUI: false`, untouched when `hasUI: true`), fault-injection confirmed. 115/115 passing, tsc/eslint clean. **Third re-run, both fixes live:** first non-empty patch across every card trial today. The model edited `sympy/physics/secondquant.py` directly instead of stopping at a plan - both structural fixes are doing their job. Didn't stop at "a diff exists," though - ran the actual reproduction against the patched tree instead of reading the diff and calling it done: ``` latex(Commutator(Bd(Symbol('0'))**2, B(Symbol('0')))) -> "- \left[b_{0},b^\dagger_{0}^{2}\right]" ``` Still unbraced. The model's fix checked `isinstance(a0, Pow)` assuming the daggered operator stays in `Commutator`'s `args[0]`, but SymPy canonicalizes/reorders commutator arguments (with a sign flip) - for this exact input the Pow lands in `args[1]`, so the one-sided check misses it. Traced why the model's own review-turn self-test didn't catch this: it reproduced with `Bd(0)` (plain int) instead of `Bd(Symbol('0'))` (the actual issue's input), hit an unrelated `AttributeError`, patched around _that_ crash, and declared done without ever re-running the real reproduction case. Not chasing this one: getting the actual fix right, and thoroughly self-verifying against the real repro rather than a nearby one that happens to run clean, is model reasoning quality - not a context-lifecycle mechanism this extension owns. Consistent with baseline's own historical 0/6 on this exact instance: a genuinely hard bug (canonicalized commutator argument order isn't obvious), not evidence the card regressed anything. Both fixes this session are validated at what they were built to fix - the model now attempts real edits instead of stopping short. Whether it gets the answer right is a separate axis this extension doesn't control. ## 2026-08-15: regression-checked sympy-18211 (the one that already worked) - two runs, two different failures, neither traces to today's fixes Both fixes above are committed (`2d015a4`, `204e1b7`). Before calling this session closed, re-ran the previously-resolved instance twice with both live, to check for a regression rather than assume none. **Run 1:** real edit produced (`sympy/solvers/solveset.py`), engagement confirmed - but wrong. Ran the actual reproduction against the patched tree rather than reading the diff and calling it done: still raised `NotImplementedError`, from a code path (`solve_univariate_inequality` via `Relational._eval_as_set`) the patch never touched. Pulled the diff from the original officially-resolved run (`20260806T160120Z.../r2-1/w`) for comparison: the correct fix lives in `sympy/core/relational.py`'s `_eval_as_set`, which this run never even opened. **Run 2:** zero edits. Traced why: the model read `sympy/solveset.py` (wrong path - the real file is `sympy/solvers/solveset.py`) seven times in the review turn, got `File not found` every time, and never corrected to the right path. Separately generated at least one malformed bash command with unescaped nested quotes (`rg -n "A|B|C"` inside an outer `"..."`, closing the outer quote early and running `solveset`/`ConditionSet` as literal shell commands - `command not found`). Neither failure resembles either of today's two fixed bugs (a forced tool_choice interruption, or confirmation-seeking language). Both are plain execution noise from this cheap pinned model - wrong file path never retried correctly, broken shell quoting - the same category of flakiness already surfaced repeatedly all session (the earlier "bash -lc" self- wrapping bug, thin `update_card` responses, etc.), not a new pattern this session's changes introduced. Not enough samples (1-for-1 resolved before, 0-for-2 now) to rule out regression with statistical confidence, but the failure signatures point at pre-existing model unreliability, not at the two mechanisms changed today. Recording as an open question rather than either claiming "no regression" or chasing more repeats to force certainty out of an inherently noisy cheap model. User pushed back on this immediately, and rightly so: "maybe you are not uninstalling and installing plugin in again after fix" - i.e., is the 0-for-2 actually evidence the fixes are live at all, or a stale-extension artifact? Fair question given both fixes had only been checked via self-repro against a hand-run instance, never confirmed inside an actual harness invocation. Didn't answer with reasoning about how `--extension ` loading works - got direct proof instead. First checked whether the `before_agent_start` system-prompt addition left any trace in the existing logs: it didn't, but neither did pi's own default system prompt text, nor any `role: "system"` message at all - the JSON trace format simply never logs the system prompt, so that absence proved nothing either way, for either fix. Added a temporary `taskAudit("session", "info", "before_agent_start fired; ctx.hasUI=...; ctx.mode=...")` line and ran two checks: a fast standalone `pi -p` invocation first (`ctx.hasUI=false; ctx.mode=json`, confirming the handler fires and the gate is correct in under 10 seconds, without paying for a full 3-turn harness run), then the actual harness again to see the same line inside a real run. It fired in all three turns (`before_agent_start fired; ctx.hasUI=false`), proving the fix's `systemPrompt` branch genuinely executes on every real invocation - the extension loads fresh from `index.ts` per spawned process, no stale build or install-once cache in the loop at all. That run (third repeat) also still produced no edit - but for a third, again-different reason: 12 implement-turn tool calls, all redundant/ overlapping `grep`/`rg` searches for the same handful of terms (some of them accidentally scoped to the run's own scratch output directory instead of `sympy/`), never once opening `sympy/core/relational.py`, the file that actually needed the change. Three runs, three distinct failure shapes (wrong file edited; wrong path fixated on plus broken shell quoting; unfocused repetitive search that never converged) - no shared signature with either forced-call derailment or confirmation-seeking language. Question answered: not a stale-plugin artifact, not traceable to today's two fixes. Ordinary variance in a cheap, high-temperature, weak pinned model, now with hard proof behind that conclusion instead of just a plausible-sounding excuse. Kept the `hasUI`/`mode` audit line permanently - cheap, accurate, and exactly the kind of question ("is this extension actually doing anything in this run") worth being able to answer in one grep next time instead of building the proof from scratch again. ## 2026-08-15: repeated-call nudge - fixes the mechanism it targets, but that's not this instance's dominant failure Two of the three sympy-18211 failure shapes above (wrong path fixated on seven times; the redundant grep loop) share a real signature: the model repeating the exact same call after it already failed once. Added a targeted fix for that specific pattern - not the broader "picks a plausible-but-wrong answer" problem, which is a reasoning-depth issue with no clean mechanical fix, and which the user explicitly flagged as outside the actual thesis anyway ("our goal is not to make the harness work better"). New `tool_execution_start` handler records each call's signature (tool + args) by call id; `tool_execution_end` looks it up and tracks a consecutive-identical-failure counter, reset by any success or any differently-signatured failure - strictly back-to-back repetition, not accumulated tolerance across a session. Two consecutive identical failures triggers a steer nudge ("that exact call just failed the same way - try something different"), capped at two nudges per streak (matching the existing `CARD_NUDGE_STREAK_CAP` pattern). 4 new tests (fires on the second identical failure, a different failing call doesn't count, a success in between resets it, caps at two), fault-injection confirmed. 119/119 passing, tsc/eslint clean. Live re-run (4th sympy-18211 repeat, all fixes live): the nudge correctly did not fire - this run had no repeated identical failures to catch. Diff produced, but wrong file again: `sympy/solvers/solveset.py`, not `sympy/core/relational.py` (the actual call path, confirmed the same way as every other check in this session - by running the real reproduction against the patched tree, not reading the diff). Four runs on this instance now: 2 wrong-file, 2 zero-edit-for-other-reasons. The dominant failure is a real one this fix was never meant to touch. Landing it because it's correct and tested for what it targets, not because it resolves this instance - consistent with treating "the harness works better" and "the thesis holds" as separate questions, per the user's own framing.