# pi-tool-stats A [pi](https://pi.dev) extension that tracks how often you actually use your **skills**, **prompt templates**, **extensions**, and **built-in tools** — so you can prune the dead weight and keep pi light. ## Quick Start 1. **Install** — add the extension to `~/.pi/agent/settings.json`: ```json { "extensions": ["/Users//code/pi-tool-stats"] } ``` Then run `/reload` in pi (or restart pi). 2. **Use pi normally** — the extension silently records every skill load, slash command, tool call, and tool result to `~/.pi/agent/tool-stats.jsonl`. 3. **Check your usage** — after a few sessions, run: ``` /tool-stats ``` You'll get a breakdown of everything you've used (and what you haven't) over the last 30 days, including a compact model/tool snapshot. Pass a number to change the window: `/tool-stats 7` for the past week. For the detailed model/tool report, run `/tool-stats model-tools 7`. That's it. No config, no setup wizard, no API keys. ## What it records Every event is appended as one JSON line to `~/.pi/agent/tool-stats.jsonl` (durable, greppable, survives restarts): | Kind | Detected when | | ------------- | ------------------------------------------------------------------- | | `skill` | the agent reads a `SKILL.md`, or you run `/skill:` | | `template` | you invoke a prompt-template slash command | | `ext-cmd` | you invoke an extension-registered slash command | | `tool` | any LLM tool call, with model attribution when available | | `tool-result` | a tool finishes, recording success/failure and duration | ## Report ``` /tool-stats [days] # overview + prune candidates; default 30 days /tool-stats model-tools [days] # detailed model/tool effectiveness /tool-stats models [days] # alias for model-tools /tool-stats tools [days] # alias for model-tools /tool-stats mt [days] # alias for model-tools ``` Use `0` days for all time. The overview shows per-resource **count + last-used date**, sorted by usage, plus a compact **MODEL / TOOL SNAPSHOT** and a **PRUNE CANDIDATES** section listing everything that exists but had **0 hits** in the window — by cross-referencing your live skill dirs, prompt templates, and extensions against the log. The detailed model/tool report shows **BY MODEL**, **BY TOOL**, **FAILURE HOTSPOTS**, and a **MODEL × TOP TOOLS** matrix. `ok%` means tool-execution success, not proof of task success. For skills, the report also calls out **PROMPT-VISIBLE UNUSED SKILLS**: skills present in Pi's `` prompt block but with no observed skill loads (`read` of the skill file or `/skill:`) in the window. This is a prompt-cost signal, not proof that the capability is useless; package skills may also show related extension/tool usage. ### Overview (`/tool-stats`) ``` Usage over last 30d (log: 1281 events total, 1281 in window) --- MODEL / TOOL SNAPSHOT --- model calls ok fail unk ok% top tools weak spot openai-codex/gpt-5.5 28 25 3 0 89% read, bash, grep read 2 fail zai/glm-5.1 24 19 2 3 90% bash, edit, read bash 2 fail anthropic/claude-sonnet-4-6 8 8 0 0 100% agent_browser - details: /tool-stats model-tools [days] --- SKILLS --- name used last land 6 2026-05-30 pi-subagents 1 2026-05-30 --- PROMPT TEMPLATES --- (none) --- EXTENSIONS --- name used last @juicesharp/rpiv-todo 115 2026-05-30 pi-lens 22 2026-05-30 @juicesharp/rpiv-advisor 13 2026-05-30 pi-agent-browser-native 10 2026-05-30 @juicesharp/rpiv-ask-user-question 9 2026-05-30 @aliou/pi-processes 6 2026-05-30 pi-subagents 3 2026-05-30 pi-web-access 1 2026-05-30 --- BUILT-IN TOOLS --- name used last bash 665 2026-05-30 read 258 2026-05-30 edit 82 2026-05-30 grep 16 2026-05-30 write 14 2026-05-30 find 2 2026-05-30 ls 1 2026-05-30 === PRUNE CANDIDATES (0 hits in 30d) === skills : caveman, paseo-loop, ... templates : (none) extensions: (none) === PROMPT-VISIBLE UNUSED SKILLS (0 skill loads in 30d) === These are in and cost prompt tokens, but no read or /skill use was observed. name approx prompt tokens related usage file caveman ~N - /Users/you/.agents/skills/caveman/SKILL.md ``` ### Model / Tool Detail (`/tool-stats model-tools`) ``` Model Tool Effectiveness (last 30d) Attempts: 60 Results: 57 OK: 52 Fail: 5 Unknown: 3 (log: 1281 events total, 1281 in window) Note: ok% is tool-execution success (not proof of task success). --- BY MODEL --- model calls ok fail unk ok% tools top tools openai-codex/gpt-5.5 28 25 3 0 89% 4 read, bash, grep zai/glm-5.1 24 19 2 3 90% 4 bash, edit, read anthropic/claude-sonnet-4-6 8 8 0 0 100% 1 agent_browser --- BY TOOL --- tool calls ok fail unk ok% best model worst model bash 18 15 3 0 83% openai-codex/gpt-5.5 zai/glm-5.1 2 fail read 18 16 2 0 89% zai/glm-5.1 openai-codex/gpt-5.5 2 fail agent_browser 10 10 0 0 100% anthropic/claude-sonnet-4-6 - edit 7 4 0 3 100% zai/glm-5.1 - grep 6 6 0 0 100% openai-codex/gpt-5.5 - find 1 1 0 0 100% openai-codex/gpt-5.5 - --- FAILURE HOTSPOTS --- model tool fail calls ok% openai-codex/gpt-5.5 read 2 12 83% zai/glm-5.1 bash 2 9 78% openai-codex/gpt-5.5 bash 1 9 89% --- MODEL × TOP TOOLS --- model bash read agent_browser edit grep ok/calls ok/calls ok/calls ok/calls ok/calls openai-codex/gpt-5.5 8/9 10/12 - - 6/6 zai/glm-5.1 7/9 6/6 2/2 4/7 - anthropic/claude-sonnet-4-6 - - 8/8 - - ``` The report opens in a scrollable selector and does **not** pollute the LLM context. ## Install ### Local directory (this checkout) ```bash git clone ~/code/pi-tool-stats cd ~/code/pi-tool-stats && bun install ``` Add to `~/.pi/agent/settings.json`: ```json { "extensions": ["/Users//code/pi-tool-stats"] } ``` Then `/reload` (or restart pi). ### As a pi package If published to npm or git, add to the `packages` array in `settings.json`: ```json { "packages": ["npm:pi-tool-stats"] } ``` ## Development ```bash bun install # install dependencies bun run typecheck # tsc type-check bun run build # bun build to /tmp (verification only) ``` ## Tips - **Reset:** `rm ~/.pi/agent/tool-stats.jsonl` - **Watch live:** `tail -f ~/.pi/agent/tool-stats.jsonl` - **Caveat:** an extension only shows usage if it exposes a tool or slash command. Purely passive extensions (event-only hooks) won't register hits — there is no per-event attribution API. ## License MIT