# Evaluation Design

Read this before choosing or materially changing the behavioral question, evidence class, group shape, pressure, or baseline/candidate boundary.

## Keep Roles Distinct

- **Subject:** performs the declared task under one variant.
- **Evaluator:** isolates subjects and preserves canonical evidence.
- **Deterministic grader:** derives fixed mechanical facts after run persistence.
- **Reviewer:** the active agent or user judges unresolved meaning.
- **Author:** uses Write Skill to revise one measured pressure point.
- **User:** decides whether to revise, use, publish, or reject the skill.

The evaluator does not revise the skill, launch an automatic semantic grader, or own readiness, promotion, or production status.

## Map Claims To Evidence

| Claim | Required evidence |
| --- | --- |
| Description routes the earliest useful prompt | Natural activation |
| Body works with guaranteed context | Explicit body delivery |
| Nearby prompt is not hijacked | Natural activation plus behavioral output |
| Declared dependencies compose | Exact ordered composition |
| Guidance remains useful later | Multi-turn evidence |
| Files or structured state match | Artifact outcome |
| Behavior holds across named hosts | The same group semantics on every named host |

Record separately:

- declaration;
- resource materialization;
- delivery;
- observed reads;
- behavior;
- artifacts;
- derived grades;
- later semantic judgment.

A read does not prove compliance. Correct behavior does not prove which skill caused it.

## Choose The Smallest Group Shape

- **Description prompt:** natural activation and exact read timing.
- **Explicit body task:** first-read body behavior without activation ambiguity.
- **Fixture task:** files or repository state matter.
- **Stateful turns:** ordered conversation and retained use matter.
- **Saved-result review:** canonical evidence already answers the question.
- **Suite:** several independent questions require one ordered serial invocation.

Keep activation and body behavior separate unless their integrated interaction is the exact question and the current runner supports that observing boundary.

## Fix Exact Variants

Every group contains exactly:

- `baseline`;
- `candidate`.

For a new skill, baseline has no target.

For a revision, baseline names an exact immutable previous package. If no exact baseline exists, return to Write Skill before revision.

For a description-only revision, body and resources remain byte-identical.

Freeze prompts or turns, fixture, tools, model, thinking, other skills, context, runtime, and criteria across variants.

## Design Common Questions

### Description activation

Use the earliest natural prompt where the target should become useful.

For a nearby prompt, predeclare whether success means:

- non-trigger; or
- safe behavior after activation.

### First-read body

Explicitly deliver the exact target body with only guaranteed context. Ambient package context cannot repair a hidden dependency.

### Dependency composition

Materialize exact ordered skills and context. Vary only the target snapshot. Do not provide a hidden base stack.

### Retained use

Use ordered turns in one persistent subject process. Repeated body delivery does not prove retained use.

### Artifact outcome

Grade files, changed paths, structured state, and events before relying on the final response. Preserve turn-scoped workspace evidence when later turns may change earlier state.

## Apply Real Pressure

A useful group:

- creates a natural temptation to violate the intended behavior;
- keeps criteria and review questions outside the prompt;
- exposes deterministic evidence where possible;
- distinguishes baseline from candidate behavior;
- remains realistic enough that passing behavior is useful.

If both variants pass, the pressure may be weak or the baseline sufficient.

If both fail, classify skill, fixture, dependency, environment, host, isolation, or criterion before editing.

## Reuse Existing Evidence

When saved canonical evidence already answers the fixed question, inspect it through views and raw artifacts. Do not rerun merely to demonstrate process.

An explicitly delivered body may support first-read behavior while a natural prompt still fails activation. Preserve both conclusions; one does not repair the other.