# pi-ast-grep design notes

This document explains why pi-ast-grep is built the way it is. It lets a
person with no prior context verify behavior or write a patch.

## Goal

Give Pi models the structural capabilities of ast-grep as native tools:
pattern search, YAML rule scanning, code rewriting, and code outlining.
The extension runs the `ast-grep` binary (version 0.44.0 or newer) and
formats its JSON output into bounded text plus structured `details`.

## Architecture

```
src/index.ts      extension factory: registers tools, /ast-grep-check,
                  /ast-grep-rules, footer status, before_agent_start
                  promotion, session lifecycle hooks
src/promotion.ts  bounded before_agent_start guidance block + config
                  (.pi/ast-grep.json promotion.enabled)
src/tools.ts      six ToolDefinitions (TypeBox schemas, execute handlers,
                  prepareArguments, TUI renderers, safety gates)
src/cli.ts        binary resolution, exec runner, output bounding, temp
                  spill, stderr surfacing, target pre-check, version floor
src/formats.ts    JSON parsing (last-complete-element recovery) and text
                  formatting (1-based line/column)
src/languages.ts  generated language catalog (scripts/gen-languages.mjs,
                  stamped with the ast-grep version it was built against)
src/rules.ts      saved-rules library (parse/validate/merge, pure logic)
src/paths.ts      @-stripping and cwd-relative path resolution
scripts/gen-languages.mjs  probes the installed binary and regenerates
                  src/languages.ts with a version stamp
scripts/live-smoke.mjs     live Pi RPC smoke test
test/unit/        pure logic tests (fake runners)
test/contract/    schema + result-shape assertions for all six tools
test/integration/ live tests against the real binary (skip when absent)
test/fixtures/    sgconfig.yml project + sample files for config-mode tests
```

`createAstGrepTools(runner)` creates the tools. `runner` is a `CliRunner`
abstraction. Production binds it to `pi.exec`. Tests bind it to a node
child-process spawner. This keeps the tool logic testable without a live
Pi session.

The package ships one skill at `skills/pi-ast-grep/SKILL.md`. The `pi`
manifest registers it (`pi.skills: ["./skills"]`) and the published
tarball includes it. The skill is a model-facing quick reference: the six
tools, the shared parameters (`context`, `max_results`, `threads`,
`follow`, `no_ignore`, `globs`), and the safety rules (preview rewrites
before you apply them; the extension blocks rewrites in untrusted
projects). Rule-writing guidance also lives in the tools'
`promptGuidelines`.

## Outline is the primary orientation tool

`ast_grep_outline` is the most useful tool for models. It produces a
compact structural map (top-level items plus direct members with line
numbers) for a file or a whole directory. It is cheap enough to run
before any full file read. Every model-facing surface promotes it:

- The tool description carries a "use when" clause (map a file or
  directory before reading it; `items=exports` maps a directory's public
  surface).
- The `before_agent_start` promotion block lists it first.
- The prompts in the extension's own workflow outline before reading
  unfamiliar files.

## Runtime promotion (`before_agent_start`)

`src/promotion.ts` owns a bounded guidance block and its toggle. The hook
in `src/index.ts` adds the block to the chained system prompt only when
ALL of these conditions hold:

1. `.pi/ast-grep.json` `promotion.enabled` is not `false`. The default
   is `true`. Untrusted projects and missing or invalid config files
   keep the default.
2. At least one ast-grep tool is active for the session
   (`systemPromptOptions.selectedTools`).
3. The `ast-grep` binary is reachable (`checkBinary` succeeds). A
   missing binary makes the hook a silent no-op. The block must never
   advertise tools that cannot run. The footer already warns about the
   missing binary.

The gate costs one `ast-grep --version` run per agent start while an
ast-grep tool is active. The cost is accepted because it prevents
advertising tools that cannot run.

The block is a few lines of pointer-only guidance. It stays below 1,200
characters. The toggle config is read from the working directory only.
This mirrors `/ast-grep-rules` config locality.

## The language catalog

`src/languages.ts` is generated by `scripts/gen-languages.mjs`. The
generator probes the installed binary. Nothing is carried over from a
previous generation as fact.

- Aliases: candidate aliases (reference table plus current catalog
  aliases) are probed with `ast-grep run --pattern x --lang <L>
  --json=compact <devNull>`. Exit 0 or 1 means the alias is accepted.
  A `--stdin` match resolves each alias to its canonical name via the
  `language` JSON field.
- Extensions: candidate extensions (reference table plus current catalog
  extensions) are probed by writing a temporary file per extension and
  reading the `language` field of a `run --json=compact` match on it.
  This is extension detection, with no `--lang`. Candidates the binary
  does not detect are dropped.

The header is stamped with the ast-grep version and the generation date.
`--check` mode compares the version stamp AND the full probed content
(aliases and extensions per language) with the catalog file. Any drift
fails the check. The current catalog has 28 languages for 0.45.0,
including Markdown.

## Verified ast-grep behaviors (ast-grep 0.45.0)

The code and tests encode behaviors observed in ast-grep 0.45.0.

1. `ast-grep run --json=compact` prints a JSON array. Exit code 0 means
   matches exist. Exit code 1 means no matches exist. The JSON is still
   printed. The extension accepts both codes and uses the JSON.
2. `ast-grep scan --json=compact` prints a JSON array. Each match has
   `ruleId` and `severity` fields. Exit code 1 means error-severity
   diagnostics were found. Exit code 0 means none were found. Both are
   success for the extension.
3. `--json` suppresses rewrites. When you combine `--json=compact` with
   `--update-all`, the command prints replacement data but does NOT write
   files. The apply path therefore drops `--json`, then verifies each
   file with a separate JSON search. It reports
   `applied_count = before - after`.
4. `ast-grep run --debug-query=cst` prints the debug tree to **stderr**,
   not stdout. `ast_grep_debug_query` concatenates both streams.
5. `ast-grep outline --json=compact` emits one object per file with
   `items` (and nested `members`). Member `signature` can be empty for
   fields. In that case `name` is used. Positions are zero-based.
   The default (auto) view carries member ranges but empty member
   signatures. The expanded view is the only view whose JSON carries
   member signatures; the signatures view omits members entirely. The
   outline tool therefore runs the expanded view once more for the
   default and signatures views and merges the member signatures into
   the result (matched by name and line). This shows member signatures
   with line numbers in the default view and restores the members that
   the signatures view omits.
6. Exit code map:

   | Code | Meaning | Extension action |
   |---|---|---|
   | 0 | success | accept |
   | 1 | no matches (run) or error-severity diagnostics (scan) | accept |
   | 2 | invalid command usage | throw with the stderr text |
   | 3 | no project config, rule not found, or stdin without `--lang` | throw |
   | 4 | test snapshot mismatch | throw |
   | 6 | cannot read file or directory | throw |
   | 8 | rule parse failure (structured `✖ Caused by` chain on stderr; YAML syntax errors include line and column) | throw |

7. Bad paths print an ERROR on stderr with an allowed exit code
   (`outline` and `scan` exit 0, `run` exits 1). The extension
   pre-checks the target with `stat`. As a backstop, it throws when
   results are empty and stderr is non-empty (`surfacedStderr`). There
   is never a silent "0 matches".
8. `--max-results` exists only on `scan`. It caps the TOTAL across
   rules before serialization. `run` and `outline` reject it with exit
   2. `scan` forwards the `max_results` parameter. `run`, `rewrite`,
   and `outline` cap client-side and rely on JSON recovery when output
   is truncated.
9. Project-config mode: `scan` without `--inline-rules` or `--rule`
   loads rules from the nearest `sgconfig.yml`. `ruleDirs` resolve
   relative to the config file's directory. No config means exit 3
   (surfaced as an error). `--inline-rules` REPLACES project rules.
10. Rule files: `scan --rule <file>` runs one rule file (or several,
    separated by `---`). `run` does NOT accept `--rule`. `ast-grep
    test` cannot validate a standalone rule (it needs a project plus
    rule-test YAMLs). Rule validation is therefore implicit: invalid
    YAML fails the scan with exit 8.
11. `--kind` (run) is mutually exclusive with `--pattern` (exit 2). It
    matches whole nodes of that kind with the same JSON shape.
12. `--context N` adds no JSON fields. The match's `lines` becomes
    multi-line spanning the context, and `charCount` re-bases.
    `--heading` requires a value and affects only human output. It is
    not exposed.
13. Knobs: `--threads N`, `--follow`, and `--no-ignore <hidden|dot|
    exclude|global|parent|vcs>` are accepted on run, scan, and outline.
    `--no-ignore` requires a value.
14. `range` is an exact-position matcher. The node start and end must
    EQUAL the given start and end, in line AND column. It is not a
    containment filter. `precedes` and `follows` act on SIBLINGS of the
    matched node. Statement-level patterns with a trailing `;` match;
    bare expression calls usually have no siblings. `nthChild` accepts
    a number or `{position: N}`.
15. Transform, rewriters, and utils pass through `--inline-rules`
    unchanged. 0.45.0 schemas: `transform: { KEY: {convert: {source,
    toCase}} }` (toCase: lowerCase, upperCase, capitalize, camelCase,
    snakeCase, kebabCase, pascalCase), `{substring: {source,
    startChar, endChar}}`, `{replace: {source, replace: <regex>, by:
    <replacement>}}`, `{rewrite: {source, rewriters: [ids], joinBy}}`.
    `rewriters` must be a sequence. `utils:` plus `matches: <id>`
    works. Results appear in JSON as `metaVariables.transformed`.
16. Custom outline rules (`outline --outline-rules <file>`) require
    `role: item|member` in each rule. `--no-default-outline-rules`
    yields `[]` without a custom file. The JSON shape is identical to
    the default.
17. `--stdin` requires `--lang` (exit 3 without it). `pi.exec` cannot
    pipe stdin (ExecOptions = `{signal, timeout, cwd}`). The `code`
    parameter therefore spills the snippet to a temporary file (named
    with the language's first catalog extension) and searches that
    file.

## Output limits and JSON safety

`runAstGrep` bounds stdout at 51,200 bytes and 2,000 lines. These are
Pi's fixed tool limits. The bounds use Pi's official `truncateHead`
utility, imported from `pi-coding-agent` (`DEFAULT_MAX_BYTES` and
`DEFAULT_MAX_LINES`). The extension does not duplicate the values, so a
Pi limit change cannot silently desync it.

When output is truncated, the full stdout goes to a temporary file
(`mkdtemp(join(tmpdir(), "pi-ast-grep-"))` via `withFileMutationQueue`).
The result reports `details.truncation` and `details.fullOutputPath`,
plus a text notice. The `session_shutdown` hook calls
`cleanupSpillDirs`, so temporary files never accumulate across sessions.

Truncated JSON never parses as `[]` by accident. `parseJsonArray` tries
a plain parse first. It then recovers the **last complete top-level
element**: it walks depth boundaries and tries `JSON.parse(prefix +
"]")` longest-first. `parseJsonArrayWithMeta` reports whether recovery
happened. Tools set `details.truncated` accordingly (byte truncation,
JSON recovery, or the client-side `max_results` slice).

## Error handling

- Missing target (run, scan, rewrite, outline): the pre-check `stat`
  throws `"<resolved path>: no such file or directory"` before
  spawning.
- Empty results plus non-empty stderr: `surfacedStderr` throws the
  trimmed stderr (first 500 characters). This catches exit-0-with-ERROR
  cases such as nonexistent nested paths and missing project config.
- Unexpected exit codes (2, 3, 6, 8, ...): `runAstGrep` throws with the
  stderr text.
- Spawn failures (binary missing): the error gives the install hint,
  pointing at ast-grep install options and `AST_GREP_BIN`.
- Empty output with an allowed code: normally treated as a spawn
  failure. The rewrite apply pass is the exception. It sets
  `allowEmptyOutput`. A zero-match `--update-all` prints nothing and
  exits 1. That is an idempotent re-apply, not a broken spawn.

## Rewrite safety

- Default `apply=false` never writes. The dry run returns every match
  with its computed `replacement`.
- The extension gates `apply=true` before any work:
  1. It blocks in untrusted projects (`ctx.isProjectTrusted()`).
  2. With a dialog-capable UI (`ctx.hasUI`), `ctx.ui.confirm` asks
     before writing.
  3. In headless sessions, `apply=true` requires a prior `apply=false`
     preview with the same arguments in the same session. The
     fingerprint excludes `max_results`. The module-level set is
     cleared on `session_shutdown` via `clearPreviewState`.
- `apply=true` resolves the affected files, then runs the rewrite per
  file inside `withFileMutationQueue(file, ...)`. Concurrent built-in
  `edit` and `write` calls on the same file serialize with it.
- A post-apply verification search computes how many matches were
  actually removed per file. This guards against races where the file
  changed between preview and apply.
- Under concurrent identical applies, the post-apply verification can
  double-count `applied_count`. The file-level guarantee still holds:
  exactly-once rewrite, no interleaving.
- `onUpdate` reports coarse progress ("running…", one update per file
  during a multi-file apply).

## Version management

- `checkBinary` parses `ast-grep --version` and computes
  `satisfiesFloor` against `AST_GREP_VERSION_FLOOR` (0.44.0, the
  `outline` dependency). `/ast-grep-check` and the `session_start`
  footer show `ast-grep 0.45.0` when OK, and `ast-grep <v> (older than
  0.44; ast_grep_outline requires 0.44+)` below the floor.
- The binary is resolved from `AST_GREP_BIN` when set, otherwise from
  PATH.
- The runtime floor for Pi is 0.84.0. `peerDependencies` stay `"*"`
  per the Pi packages convention. The floor is documented, not
  package-manager-enforced.

## ExtensionAPI usage

- `prepareArguments` normalizes legacy and alternative argument names
  (`language` to `lang`, `yaml` to `rule_yaml`, `rewrite_mode` to
  `mode`, `pattern2` to `pattern`, `replacement` to `rewrite`, `file`
  to `path`, `maxResults` to `max_results`) before schema validation.
  Schemas stay strict.
- `renderCall` and `renderResult` (TUI-only by the runtime; `Text` from
  `@earendil-works/pi-tui`, declared in peerDependencies) render run
  match lists, rewrite preview and apply summaries with `-` and `+`
  lines, and outline trees. They handle `isPartial`, `expanded`,
  truncation, and undefined details defensively.
- `appendEntry` persistence: `/ast-grep-rules` saves and loads named
  rules. Custom entries never enter model context. The command replies
  via `pi.sendMessage`, so the model sees the result.
- `before_agent_start`: the promotion hook adds the bounded guidance
  block to the chained system prompt. It is conditional on active
  tools, the config toggle, and binary availability. See "Runtime
  promotion" above.
- Safety: `ctx.isProjectTrusted`, `ctx.hasUI` plus `ctx.ui.confirm`
  for interactive applies, `ctx.cwd` for path resolution, and
  `ctx.sessionManager` for the saved-rules branch state.

## Testing

- `test/unit/*` — pure logic with fake runners: path resolution, JSON
  parsing plus last-complete-element recovery, 1-based formatting,
  bounding, spill, exit-code mapping, stderr surfacing, target
  pre-check, version parsing and floor, prepareArguments
  normalization, safety gates, renderers, rules library.
- `test/unit/promotion.test.ts` plus
  `test/integration/promotion.test.ts` — promotion block bounds and
  content, config parse and load defaults and toggle, active-tools
  gate, missing-binary no-op, and the end-to-end hook behavior against
  a fake pi.
- `test/contract/*` — for all six tools: strict schema assertions
  (exact key sets, required sets, bounds, StringEnum values, no legacy
  keys) and result-shape assertions with canned JSON runners.
- `test/integration/live.test.ts`, `features.test.ts`, and
  `correctness.test.ts` — real `ast-grep` binary on fixture files:
  search, relational scan, severity, preview and apply (pattern and
  rule modes) with the safety gates, outline (file, directory, match,
  type, custom outline rules), debug query, unknown-language errors,
  config mode (fixture project), rule-file mode, code mode, kind,
  context, globs, filter, max_results pass-through,
  threads/follow/no_ignore, transform/rewriters/utils,
  nthChild/range/precedes/follows, more than 51 KB truncation with
  spill plus JSON recovery, bad-path errors, AST_GREP_BIN override,
  PATH-stripped spawn failure, multi-rule scan with filter. Skipped
  when the binary is missing.
- `test/integration/live-session.test.ts` — boots pi in RPC mode
  (offline, keyless) with the extension loaded and proves
  `/ast-grep-check` registers and runs. It also drives concurrent
  rewrite applies through Pi's real `withFileMutationQueue` and
  verifies the file ends consistent.
- `scripts/live-smoke.mjs` — loads the extension in real Pi RPC mode
  and runs `/ast-grep-check` (proves the binary plus all six tools
  registered via session name `ast-grep-ok`), once unfiltered and once
  under `--tools read,bash` (a session tool filter must not fail the
  check). With `--model`, it drives `ast_grep_outline` plus
  `ast_grep_run` through DeepSeek V4
  Flash in print mode, requiring the model to return
  `AST_GREP_MODEL_OK`.
- The repository ships no CI workflow. Gates are run locally:
  `npm run check`, `npm run typecheck`, `npm run test:all` (includes
  `check:catalog`), `npm run package:check`, and `npm run smoke`.