# Filesystem tools vs Bash-only

This example shows when `pi-coding-agent-eval` is useful: compare multiple Pi setups on the same task while keeping the workspace, prompt, model, validation, and reporting consistent.

It compares:

- `filesystem-tools`: `read`, `write`, and `edit`;
- `bash-only`: only `bash`, so inspection and mutation must be expressed as shell commands.

Both agent profiles solve the same exact-edit task. The report compares reward, elapsed time, tool calls, failures, tokens, cost, and pre-mutation behavior.

## Free deterministic example

`test/examples/tool-profile-comparison/tool-profile-comparison.integration.test.ts` uses scripted assistant responses. Pi still runs as a real process and executes real tools, but it makes no model request and needs no credentials.

```bash
npm run test:integration
```

The test is both executable documentation and a regression test. It proves that the two agent profiles run through the evaluator, produce the expected file, and appear in the pairwise report.

The same suite is also runnable through the public CLI config and direct API example:

```bash
npm run example:cli
npm run example:api
```

`eval.config.mjs` is the complete `PiEvalConfig` for the CLI. `run-api.mjs` is the complete direct `runEvaluation`
example, including a profile override, custom metric, custom renderer, and output check.
The integration test runs all three benchmark presets:

- `easy`: update one file;
- `medium`: update two related files;
- `hard`: update three files in different locations.
Agent profiles can override global `model`, `thinking`, `skills`, `systemPrompt`, and `appendSystemPrompt` settings. An omitted profile field inherits the global run setting; an empty array explicitly clears a list.

To watch the same deterministic scripted run in the native Pi TUI:

```bash
./test/examples/tool-profile-comparison/run-scripted-live.sh
```

Or through npm:

```bash
npm run example:scripted-live
```

This command runs `test/examples/tool-profile-comparison/tool-profile-comparison.integration.test.ts` itself through `pi-test live --once`. There is no duplicate scripted runner: the normal suite and live view execute the same test and assertions. It makes no model requests.

## Real-model experiment

The live launcher uses the same suite, task, agent profiles, validator, and report, but omits scripted conversations. Pi therefore uses the selected real provider and model. It always enables the native Pi TUI, so you can watch each agent profile work while the evaluation is running.

```bash
./test/examples/tool-profile-comparison/run-live.sh \
  --model openai-codex/gpt-5.3-codex \
  --thinking medium \
  --attempts 3
```

The equivalent npm command is:

```bash
npm run example:live -- \
  --model openai-codex/gpt-5.3-codex \
  --thinking medium \
  --attempts 3
```

Options:

- `--model provider/model` selects a model; omit it to use Pi's current model;
- `--thinking level` selects the reasoning level;
- `--attempts N` repeats each agent profile;
- `--agent-profiles ID,...` selects agent profiles, defaulting to both;
- `--benchmark-preset easy|medium|hard` selects the task difficulty;
- `--output PATH` changes the artifact root;
- `--keep-workspace` preserves trial workspaces.

Real-model runs use saved Pi credentials and may incur provider charges. Results are written below `.tmp/tool-profile-comparison/results/` by default.