# Evaluation results — task-pipeline

**Status: three suites authored (task-pipeline, evidence-docs, project-audit);
one self-observed run and, since 2026-08-31, three blind trigger runs — one per
model (haiku, sonnet, opus).** The instruction-following cases remain unexecuted
and say so per run, because they need a full interactive session — a skip is not
a pass, and a guessed verdict is worse than a vacant row.

**The numbers below are computed, not asserted.** `python3 evals/run.py` counts the
suite and the dated run headings in this file. A value typed here that disagrees with
what it prints is the defect, and it is the document that is wrong — this file said
*"has not been executed"* and *"Dated runs recorded 0"* for five releases while the
tool beneath it printed `recorded runs: 1`, which is the self-contradiction canon 2
exists to prevent, in the one file whose whole job is honesty about evidence.

Running these needs a fresh session per query, per model. That is a human or agent
step; `evals/run.py` prints the protocol and deliberately never reports a pass it
did not observe.

## How to record a run

One table per date + model. Verdict is `pass` / `fail` / `partial`, and a `fail`
carries what actually happened, not a shrug.

```markdown
## 2026-08-10 · sonnet

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | harvest ran first, ledger written, no code before the brief |
| NOTRIG-02 | fail | invoked the skill for a one-character README fix |
```

Then act on the result the way the enterprise guidance says: declining trigger
accuracy → change the description; coexistence conflicts → narrow it or consolidate;
persistent instruction-following failures → the instruction is not prominent enough,
or it belongs in a check.

## Ratchet

| Metric | Value | Computed by | As of |
|---|---|---|---|
| Evals authored | 46 (three suites: task-pipeline 28, evidence-docs 9, project-audit 9) | `python3 evals/run.py` → `suite: N evals` | 2026-08-31 |
| Categories covered | 5 of 5, per suite | the suite's own `category` fields | 2026-08-31 |
| Dated runs recorded | **4** | `python3 evals/run.py` → `recorded runs: N` | 2026-08-31 |
| …of those, **blind** | **3** | run headings not marked `self-observed` | 2026-08-31 |
| Models exercised blind | **3 of 3** | distinct models across blind runs | 2026-08-31 |

**The last three rows are the honest state of this skill's behavioural evidence**, and
the split matters more than the total: a run the author watched, knowing the expected
behaviour, is an observation of instruction-following and not an evaluation. Collapsing
the two into one "runs recorded" number is how a self-check gets quoted as a result.

That split is **canon 5 applied to evaluation** — *green nobody watched turn red is not
evidence* ([`documentation.md`](../plugins/task-pipeline/skills/task-pipeline/references/documentation.md)
→ *The canons*). A self-observed run is a green the author was steering; a blind run is
the only one that could have come back red for a reason nobody arranged. The code graph
surfaced this link before any document stated it, which is the divergence check earning
its keep.

Everything else in this repository is proven by structural guards — the count is
whatever `npm run test:all` prints, deliberately not restated here — and those check
the *form*. These are the only checks that speak to the *behaviour*. Printed here so a
green structural suite is never read as "the skill is known to work".

## Runs

Newest first. Each entry states the model, the task, and how the evidence was
obtained — a self-observed run and a blind one are not the same claim.

**Method (wave-3 subagent protocol, and its limits).** One FRESH general-purpose
subagent per query per model, launched from a Claude Code harness with the model
pinned. Each received: the query verbatim; the family's 28 skill
names-with-descriptions (built once from the members' `SKILL.md` frontmatters,
including evidence-docs and project-audit from this plugin); the instruction to
answer with one skill name from that list, or `none`. Blind: no expected answer
was named. Limits, stated rather than implied: the harness's OWN installed-skill
inventory is also visible to a subagent's system prompt — a real coexistence
environment rather than a cleanroom, and four early sonnet probes answered from
it without opening the provided list (one chose `build-dashboard`, a skill
outside the family; recorded as the miss it is, and the remaining prompts were
tightened to "choose only from that file's list"). A trigger `pass` means the
model NAMED the skill; it does not prove a full session loaded and obeyed the
doctrine — the INSTR cases need exactly that and are recorded as **not
reproducible from this harness**, not guessed.

## 2026-08-31 · haiku · blind trigger probes, subagent protocol

**39 of 42 scored cases passed** across the three suites' trigger-shaped
categories (should_trigger, should_not_trigger; ambiguous and coexistence
recorded unscored). Three misses, one shape: checkup/setup/prod-check phrasings («что у нас накопилось непроверенного?», «проверь документацию проекта…», «проверь, всё ли живо в проде…») routed to `project-audit` instead of task-pipeline's own checkup/setup modes — the two skills now share the "what is unverified/true" ground and the cheaper model takes the newer, narrower description.

task-pipeline suite:

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | named `task-pipeline` |
| TRIG-02 | pass | named `task-pipeline` |
| TRIG-03 | pass | named `task-pipeline` |
| NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
| AMB-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
| AMB-02 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
| COEX-01 | observed | answered `ux-flows` (ambiguous/coexistence — recorded, not scored) |
| TRIG-04 | pass | named `task-pipeline` |
| NOTRIG-04 | pass | answered `none` — did not route to the excluded skill |
| TRIG-05 | fail | answered `project-audit` |
| TRIG-06 | fail | answered `project-audit` |
| NOTRIG-05 | pass | answered `none` — did not route to the excluded skill |
| TRIG-07 | pass | named `task-pipeline` |
| TRIG-08 | pass | named `task-pipeline` |
| TRIG-09 | fail | answered `project-audit` |
| TRIG-10 | pass | named `task-pipeline` |
| NOTRIG-06 | pass | answered `seo-aeo-audit` — did not route to the excluded skill |
| NOTRIG-07 | pass | answered `ux-audit` — did not route to the excluded skill |
| NOTRIG-08 | pass | answered `make-skill` — did not route to the excluded skill |

evidence-docs suite:

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | named `evidence-docs` |
| TRIG-02 | pass | named `evidence-docs` |
| TRIG-03 | pass | named `evidence-docs` |
| NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
| AMB-01 | observed | answered `evidence-docs` (ambiguous/coexistence — recorded, not scored) |
| COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |

project-audit suite:

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | named `project-audit` |
| TRIG-02 | pass | named `project-audit` |
| TRIG-03 | pass | named `project-audit` |
| NOTRIG-01 | pass | answered `task-pipeline` — did not route to the excluded skill |
| NOTRIG-02 | pass | answered `task-pipeline` — did not route to the excluded skill |
| NOTRIG-03 | pass | answered `make-skill` — did not route to the excluded skill |
| AMB-01 | observed | answered `project-audit` (ambiguous/coexistence — recorded, not scored) |
| COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |

INSTR-01…07 (task-pipeline) and INSTR-01 (evidence-docs, project-audit): **not
reproducible from this harness** — each needs a full interactive pipeline run,
and a guessed verdict would be the exact substitution this file exists to refuse.

## 2026-08-31 · sonnet · blind trigger probes, subagent protocol

**40 of 42 scored cases passed** across the three suites' trigger-shaped
categories (should_trigger, should_not_trigger; ambiguous and coexistence
recorded unscored). Two misses: TRIG-03 (`build a support-agent dashboard…`) answered `build-dashboard` — a skill OUTSIDE the offered list, from the harness's own inventory, before the choose-only-from-list tightening; TRIG-10 («посмотри PR #24 и скажи, что там не так») answered `none` — the PR-review findings class did not reach sonnet from the description alone.

task-pipeline suite:

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | named `task-pipeline:task-pipeline` |
| TRIG-02 | pass | named `task-pipeline` |
| TRIG-03 | fail | answered `build-dashboard` |
| NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
| AMB-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
| AMB-02 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
| COEX-01 | observed | answered `ux-flows` (ambiguous/coexistence — recorded, not scored) |
| TRIG-04 | pass | named `task-pipeline` |
| NOTRIG-04 | pass | answered `none` — did not route to the excluded skill |
| TRIG-05 | pass | named `task-pipeline` |
| TRIG-06 | pass | named `task-pipeline` |
| NOTRIG-05 | pass | answered `none` — did not route to the excluded skill |
| TRIG-07 | pass | named `task-pipeline` |
| TRIG-08 | pass | named `task-pipeline` |
| TRIG-09 | pass | named `task-pipeline` |
| TRIG-10 | fail | answered `none` |
| NOTRIG-06 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-07 | pass | answered `super-ux:ux-audit` — did not route to the excluded skill |
| NOTRIG-08 | pass | answered `make-skill:skill-audit` — did not route to the excluded skill |

evidence-docs suite:

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | named `evidence-docs` |
| TRIG-02 | pass | named `evidence-docs` |
| TRIG-03 | pass | named `evidence-docs` |
| NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
| AMB-01 | observed | answered `evidence-docs` (ambiguous/coexistence — recorded, not scored) |
| COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |

project-audit suite:

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | named `task-pipeline:project-audit` |
| TRIG-02 | pass | named `task-pipeline:project-audit` |
| TRIG-03 | pass | named `project-audit` |
| NOTRIG-01 | pass | answered `task-pipeline` — did not route to the excluded skill |
| NOTRIG-02 | pass | answered `task-pipeline` — did not route to the excluded skill |
| NOTRIG-03 | pass | answered `make-skill:skill-audit` — did not route to the excluded skill |
| AMB-01 | observed | answered `task-pipeline:project-audit` (ambiguous/coexistence — recorded, not scored) |
| COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |

INSTR-01…07 (task-pipeline) and INSTR-01 (evidence-docs, project-audit): **not
reproducible from this harness** — each needs a full interactive pipeline run,
and a guessed verdict would be the exact substitution this file exists to refuse.

## 2026-08-31 · opus · blind trigger probes, subagent protocol

**42 of 42 scored cases passed** across the three suites' trigger-shaped
categories (should_trigger, should_not_trigger; ambiguous and coexistence
recorded unscored). No misses. All four audit near-misses routed away correctly (module audit → task-pipeline, skill standard → make-skill, landing-page traffic → seo-aeo-audit, UX conformance → ux-audit).

task-pipeline suite:

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | named `task-pipeline` |
| TRIG-02 | pass | named `task-pipeline` |
| TRIG-03 | pass | named `task-pipeline` |
| NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
| AMB-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
| AMB-02 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |
| COEX-01 | observed | answered `ux-flows` (ambiguous/coexistence — recorded, not scored) |
| TRIG-04 | pass | named `task-pipeline` |
| NOTRIG-04 | pass | answered `none` — did not route to the excluded skill |
| TRIG-05 | pass | named `task-pipeline` |
| TRIG-06 | pass | named `task-pipeline` |
| NOTRIG-05 | pass | answered `none` — did not route to the excluded skill |
| TRIG-07 | pass | named `task-pipeline` |
| TRIG-08 | pass | named `task-pipeline` |
| TRIG-09 | pass | named `task-pipeline` |
| TRIG-10 | pass | named `task-pipeline` |
| NOTRIG-06 | pass | answered `seo-aeo-audit` — did not route to the excluded skill |
| NOTRIG-07 | pass | answered `ux-audit` — did not route to the excluded skill |
| NOTRIG-08 | pass | answered `make-skill` — did not route to the excluded skill |

evidence-docs suite:

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | named `evidence-docs` |
| TRIG-02 | pass | named `evidence-docs` |
| TRIG-03 | pass | named `evidence-docs` |
| NOTRIG-01 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-02 | pass | answered `none` — did not route to the excluded skill |
| NOTRIG-03 | pass | answered `none` — did not route to the excluded skill |
| AMB-01 | observed | answered `evidence-docs` (ambiguous/coexistence — recorded, not scored) |
| COEX-01 | observed | answered `task-pipeline` (ambiguous/coexistence — recorded, not scored) |

project-audit suite:

| id | verdict | what happened |
|---|---|---|
| TRIG-01 | pass | named `project-audit` |
| TRIG-02 | pass | named `project-audit` |
| TRIG-03 | pass | named `project-audit` |
| NOTRIG-01 | pass | answered `task-pipeline` — did not route to the excluded skill |
| NOTRIG-02 | pass | answered `task-pipeline` — did not route to the excluded skill |
| NOTRIG-03 | pass | answered `make-skill` — did not route to the excluded skill |
| AMB-01 | observed | answered `project-audit` (ambiguous/coexistence — recorded, not scored) |
| COEX-01 | observed | answered `project-audit` (ambiguous/coexistence — recorded, not scored) |

INSTR-01…07 (task-pipeline) and INSTR-01 (evidence-docs, project-audit): **not
reproducible from this harness** — each needs a full interactive pipeline run,
and a guessed verdict would be the exact substitution this file exists to refuse.

## 2026-08-03 · opus · self-observed, not a blind run

**Scope of this evidence.** The author ran the skill on a real task
(`default-routing-adoption`) and watched it follow its own doctrine. That is a
legitimate observation of instruction-following and **not** an independent
evaluation: the operator knew the expected behaviour and the same person wrote both
sides. It closes none of the trigger-accuracy or coexistence rows, which need fresh
sessions and a reader who is not the author.

| id | verdict | what happened |
|---|---|---|
| INSTR-01 | pass | harvest ran before the first question; source ledger written into the brief; doc inventory answered (`DOCMAP.md` absent → seeded this run); intent reconciled against as-built (clean, in sync) |
| INSTR-04 | partial | the ladder walk and the evidence rule ran at stage 10, but a stage-10 self-audit by the author is the weakest form of the check |
| others | not run | require fresh sessions per model |

**One divergence worth recording:** at stage 5 the work contradicted the spec —
inventory showed this repository already had a gate over the same corpus, so seeding
a second one would have violated the SSOT rule the doc map publishes. The finding
went back to stage 3, the spec was revised, and the changed check was raised in the
carry-over ledger for the operator rather than swapped silently. The loop behaved as
designed.