# Rewrite evaluation

The frozen corpus in `evaluation/frozen.json` contains 56 tools and 168 queries across 9 domains. `evaluation/frozen.sha256` seals the corpus. Aliases are absent.

The production Rewrite comparison is pinned by `evaluation/rewrite.config.json`:

```text
provider: deepseek-official
model: deepseek-v4-flash
promptVersion: query-rewrite-v1
```

Generate the 112 cross-language and same-language requests with:

```sh
pnpm run eval:rewrite:prepare
```

This also builds a one-shot collector and writes the ignored `evaluation/rewrite-runner.patch.yml`. Execute it through an actual DSH runtime and its registered `llm` route:

```sh
npx @deepseek-ai/dsh --profile web \
  --patch evaluation/rewrite-runner.patch.yml \
  --port 0
```

Stop the temporary web process after `evaluation/rewrite-responses.json` appears. The overlay does not edit the selected Profile, refuses to overwrite an existing response artifact, and writes the result atomically only after every call settles. Do not call the provider directly or copy API credentials into this repository. The response file has this shape:

```json
{
  "version": 1,
  "corpusDigest": "<evaluation/frozen.sha256>",
  "provider": "deepseek-official",
  "model": "deepseek-v4-flash",
  "promptVersion": "query-rewrite-v1",
  "maxTokens": 128,
  "responses": [
    {
      "id": "weather_forecast:crossLanguage",
      "query": "查询指定城市的天气预报",
      "status": "applied",
      "reason": "applied",
      "rewrittenQuery": "city weather forecast",
      "latencyMs": 320,
      "usage": {
        "inputTokens": 64,
        "outputTokens": 8,
        "cacheReadTokens": 0,
        "cacheWriteTokens": 0,
        "reasoningTokens": 0
      }
    }
  ]
}
```

Failed calls use `status: "failed"`, retain their real failure reason, and set `rewrittenQuery` to an empty string. The scorer reproduces the plugin's original-query lexical fallback for those rows.

After all 112 responses have been recorded, run:

```sh
pnpm run eval:rewrite:score
```

The ignored `evaluation/rewrite-report.json` contains every raw-BM25 and Rewrite rank, recall metrics, uplift, failures, latency, token usage, route identity, prompt version, and corpus digest. Missing, duplicate, stale, or differently routed responses fail validation. This report is required before a Rewrite recall claim or npm release; the request manifest alone is not evidence.

The separate E5 experiment remains available through `pnpm run eval:dense` and writes `evaluation/report.json`.

Recorded summaries and gate decisions live in [evaluation/RESULTS.md](RESULTS.md).