--- title: "Per-Repo Adaptive Scripts: Measured Token Savings vs Per-Session Rediscovery" description: "Every AI coding session rediscovers the same project-specific patterns — branch names, test commands, commit style. We measured the cost and quantified how per-repo adaptive scripts eliminate it." tags: ["ai", "performance", "tokens", "productivity", "opensource"] cover_image: canonical_url: https://github.com/RapierCraftStudios/ForgeDock published: false --- AI coding agents are expensive to run. Every session starts from scratch, re-reading files, running commands, and re-learning the same project-specific patterns that have not changed since last week. This article measures how much that costs, and what per-repo adaptive scripts do about it. --- ## The Problem: Per-Session Rediscovery Every time a ForgeDock agent starts working on an issue, it must answer the same questions about the project: - What is the PR base branch? (staging? milestone/something? main?) - How do I run the tests? (pnpm test? pytest? npm run test:unit?) - What commit format does this project use? (conventional commits? what scope?) - Where do the tests live? (tests/? __tests__/? spec/?) - What workflow labels exist on this repo? None of these change between sessions. The codebase does not reorganize its test directory overnight. The commit convention does not shift on a Tuesday. But the agent re-discovers all of them from scratch on every run — reading files, executing commands, inferring patterns. We measured this cost across five representative completed issues. --- ## Methodology We analyzed five pipeline sessions from the ForgeDock dogfooding repository (issues #674, #669, #671, #679, #676). For each session, we identified every tool call that was a "rediscovery call" — a call whose purpose was to learn something about the project that was already true in the previous session. Token counts were estimated at 1 token per 4 characters (conservative; actual tokenization may differ by model and content type), then adjusted upward ~30% to reflect the claude-sonnet-5 tokenizer, which produces approximately 30% more tokens than claude-sonnet-4.x for the same text. We counted command invocation + output, not context window overhead. --- ## Baseline: Per-Session Rediscovery Cost | Operation | Mechanism (baseline) | Token estimate | |-----------|---------------------|----------------| | forge.yaml full read | Read 111-line, 4,228-byte file | ~1,365 | | Branch name determination | `gh issue view` + LLM milestone-to-slug inference | ~455 | | Commit style detection | `git log --oneline -5` + LLM pattern inference | ~260 | | Test command discovery | Read package.json/pyproject.toml + grep test scripts | ~520 | | Test location discovery | `ls tests/`, `ls __tests__/`, parse package.json | ~325 | | Label scheme discovery | `gh label list` or inferred from pipeline comments | ~390 | | **Total per session** | | **~3,315 tokens** | These numbers are per-session. On a repository with active development (10-30 issues/month), this rediscovery tax runs 25,500–76,500 tokens/month on patterns that have not changed. The branch name example is instructive. Before `classify-lane.sh` was introduced in #669, agents would: 1. Call `gh issue view NUMBER --json milestone` (~200 tokens output) 2. Extract the milestone title string 3. LLM-infer the slug transformation (`Deterministic Pipeline v2` → `milestone/deterministic-pipeline-v2`) 4. Sometimes get it wrong and hallucinate `milestone/project-agnostic` (the bug that motivated #639) After `classify-lane.sh`: one deterministic script execution with a 6-token output. No LLM inference in the critical path. --- ## Target: Per-Repo Adaptive Script Overhead After `/optimize` generates `.forgedock/scripts/` entries from `forge.yaml → learned:`, the same operations become: | Script | Size | Execution cost | |--------|------|----------------| | `branch-targets.sh` | 23 lines, ~470 bytes | ~220 tokens | | `run-tests.sh` | 10 lines, ~200 bytes | ~105 tokens | | `format-commit.sh` | 15 lines, ~300 bytes | ~130 tokens | | `forge.yaml` learned section (4 targeted yq reads) | subset of 4,228-byte file | ~130 tokens | | **Total per session** | | **~585 tokens** | The key insight: a script call costs roughly the script size (to read) plus the output size (to receive). A 23-line branch-targets.sh costs ~153 tokens to read and produces 1 line of output. The equivalent LLM-based rediscovery costs ~455 tokens and occasionally hallucinates. --- ## Savings Calculation ``` savings_per_session = 3,315 - 585 = 2,730 tokens range: low = 1,560 tokens (simple fast-lane issues, no test discovery needed) high = 4,550 tokens (complex multi-file issues with full pattern rediscovery) ``` At claude-sonnet-5 pricing ($3.00 per million input tokens standard; $2.00/M intro through 2026-08-31): ``` 2,730 tokens/session × $3.00 / 1,000,000 = $0.0082/session (standard) 2,730 tokens/session × $2.00 / 1,000,000 = $0.0055/session (intro rate) ``` At observed ForgeDock dogfooding pace (5–15 issues/day, averaging ~10): ``` 300 sessions/month × $0.0082 = $2.46/month per repo (standard rate) 300 sessions/month × $0.0055 = $1.65/month per repo (intro rate through 2026-08-31) ``` For teams running ForgeDock at production scale across multiple repositories, these savings compound. A team with 5 active repos at 300 sessions/month each saves ~$12.30/month from adaptive scripts alone at standard pricing. The cost of running `/optimize` once per repo to generate the scripts: effectively zero (a single session that generates ~5 scripts). **Payback period: 1 session.** --- ## Comparison with DevDocs Per-repo scripts and devdocs solve different problems: | Pattern type | Script approach | DevDocs approach | Winner | |-------------|----------------|-----------------|--------| | Branch target | `branch-targets.sh` — 23 lines, 1 execution call | 1 line in architecture.md | **DevDocs** (simpler for static values already documented) | | Test commands | `run-tests.sh` — executable, deterministic output | Section in devdocs, LLM parses prose | **Scripts** (executable beats interpreted) | | Commit format | `format-commit.sh` — outputs exact string | Convention in devdocs, LLM infers | **Scripts** (deterministic output vs interpretation) | | Architecture decisions | Cannot encode reasoning in scripts | `devdocs/project/architecture.md` | **DevDocs** (irreplaceable for reasoning context) | | Debug runbooks | Script could wrap commands, but context matters | DevDocs with structured steps | **DevDocs** (context-dependent) | The boundary rule: if the knowledge produces a **deterministic output** (a branch name, a command string, a file path), encode it as a script. If it requires **contextual interpretation** (design rationale, tradeoff analysis, conventions with exceptions), encode it in devdocs. Per-repo scripts replace approximately 25–40% of devdocs content for projects with established operational patterns. This reduces per-session devdocs loading by an estimated 500–1,000 tokens, complementing the token savings from selective devdocs loading introduced in #675. **Combined savings per session** (adaptive scripts + devdocs selective loading): ``` Rediscovery elimination (adaptive scripts): ~2,730 tokens Devdocs selective loading (#675): ~1,300 tokens ──────────────────────────────────────────────────────── Total combined: ~4,030 tokens/session ``` --- ## What Gets Generated Running `/optimize` on a ForgeDock repository after the `learned:` section is populated produces: | Script | Trigger | What it replaces | |--------|---------|-----------------| | `branch-targets.sh` | `learned.branch_targets.staging` is set | gh issue milestone lookup + LLM slug inference | | `run-tests.sh` | `learned.test_commands` is non-empty | package.json/pyproject.toml read + LLM test command inference | | `format-commit.sh` | `learned.commit_style` is set + 3+ commits confirm pattern | git log --oneline + LLM conventional commit inference | | `find-tests.sh` | `learned.test_locations` is non-empty | ls tests/, __tests__/ + LLM path inference | | `label-map.sh` | `learned.label_map` is non-empty | gh label list + LLM label mapping | Each script is generated at confidence >= 0.7 (two independent signals) or 1.0 (explicit `learned:` value). The confidence threshold ensures scripts are only generated when there is enough signal to trust the encoded knowledge. --- ## Limitations and Confidence Intervals The estimates in this article carry these caveats: 1. **Token counting method**: We use 1 token per 4 characters. Actual tokenization varies by model and content type (code tokenizes more densely than prose). Error: ±20%. 2. **Baseline measurement**: We analyzed 5 issues from one repository (ForgeDock itself). A ForgeDock-type repository (markdown command specs, minimal code) may have lower rediscovery cost than a large application codebase with many test configurations. Application repos with complex test setups may see 4,000–6,000 tokens of rediscovery per session. 3. **LLM hallucination cost not counted**: Our baseline counts only the token cost of rediscovery, not the token cost of correcting a hallucinated branch name or wrong test command. Including rework, the actual cost of rediscovery-induced errors is higher. 4. **Model pricing changes**: The $3.00/M standard figure is current as of July 2026 for claude-sonnet-5. An introductory rate of $2.00/M input is available through 2026-08-31. Pricing trends downward over time; the absolute dollar savings will decrease, but the proportional savings (rediscovery as a fraction of session cost) remain constant. --- ## Prioritization Conclusion This measurement was created to inform the prioritization decision for #667 (learned section) and #668 (/optimize command). Both are now merged. The data confirms: **adaptive scripts are worth implementing.** The payback period is less than one session. The primary value is not dollar savings (which are modest for individual repos) but **reliability** — replacing LLM inference with deterministic script output eliminates a class of hallucination errors that were causing production-impacting bugs (see #639: agents hallucinating `milestone/project-agnostic` instead of `staging`). Token savings are the measurable proxy for the deeper value: an agent that runs `branch-targets.sh` and gets `milestone/deterministic-pipeline-v2` in 6 tokens cannot hallucinate the branch name. An agent that reads a 350-token prose description and infers the branch name sometimes gets it wrong. Scripts do not just save tokens. They eliminate a category of error.