# inspiration_tree_sitter_rules_investigation.md — three concrete investigations

> Status: **research notes, untracked** (matches the `docs/inspiration_*` gitignore pattern under `.gitignore:50`). Companion to `docs/inspiration_tree_sitter_rules.md`.
> Date: 2026-06-25.
> Method: every rule count and cross-reference was verified against the actual repo state via `curl + grep`, not estimated from READMEs.
> Source files verified (with byte/line counts):
> - UAST-Grep: `universal-security.yaml` = 909,984 bytes / 36,553 lines / 1,598 rule entries (per file header).
> - autoaudit: `engine/scanner.py` = 431 lines / 15 Python rules (PY001-PY015) + 11 VBA rules (VBA001-VBA011).
> - ast-grep catalog: 13 language dirs in `ast-grep/ast-grep.github.io/website/catalog/`, ~49 rule files (excluding index.md per dir).

---

## Why this doc exists

The prior `docs/inspiration_tree_sitter_rules.md` §9 listed 3 investigations as "S effort, low/med value" without doing them:

1. UAST-Grep Kotlin ruleset — "S effort, med value"
2. autoaudit Python rules — "S effort, low value"
3. ast-grep upstream catalog re-scan — "S effort, low value"

This document executes them. It corrects several claims from the prior §3 (UAST-Grep DSL mapping effort) and adds concrete port candidates with rule-shaped YAML patterns.

---

## I1. UAST-Grep Kotlin ruleset — **80 port candidates, M effort, HIGH value**

### Source verification (the hard numbers)

| Language | Unique rules in `universal-security.yaml` |
|---|---:|
| python | 270 |
| javascript | 154 |
| java | 116 |
| csharp | 100 |
| rust | 89 |
| **kotlin** | **85** |
| typescript | 71 |
| scala | 71 |
| swift | 51 |
| php | 28 |
| (other: ruby, go, kotlin, scala, lua, clojure, powershell, bash, c, cpp, objc, r, julia) | (under 20 each) |

### Kotlin coverage by CWE bucket (the unique 85 rules, sorted)

| CWE | Count | Examples |
|---|---:|---|
| CWE-798 (hardcoded secrets) | 10 | API key, password, private key, AWS credentials, OAuth secret, encryption key, bearer token, strings.xml, Firebase config, DB connection string |
| CWE-327 (crypto) | 10 | AES-ECB, MD5, SHA-1, DES/3DES, static IV, weak RSA, weak PBKDF2, null cipher, no-padding cipher, platform default |
| CWE-95 (Android WebView RCE) | 9 | JS-enabled, addJavascriptInterface, file access, loadURL, mixed content, JS interface removal, geolocation, file access, loadURL validation |
| CWE-295 (TLS/SSL) | 9 | Cleartext HTTP, cert validation, hostname verification, old TLS versions, WebSocket unencrypted, missing pinning, single-pin-no-backup, pinning expiration |
| CWE-927 (intent) | 8 | Implicit-intent hijacking, broadcast unprotected, sendBroadcast, PendingIntent FLAG_MUTABLE, intent parsing, implicit-service-intent, deeplink validation, intent redirection |
| CWE-89 (SQL injection) | 6 | String template, raw query, Room @RawQuery, execSQL, ContentProvider, SQLDelight |
| CWE-532 (logging) | 6 | Logging passwords, API keys, PII, exception with sensitive data, toString leaking, verbose logs in release |
| CWE-338 (random) | 6 | kotlin.random, java.util.Random, Math.random, seeded SecureRandom, random token generation |
| CWE-502 (deserialization) | 5 | Parcel, kotlinx.serialization, Gson TypeAdapter, Bundle gadgets, untrusted data |
| CWE-926 (exported components) | 2 | Exported activity / provider without permission |
| CWE-434 (file storage) | 2 | External storage + file provider paths |
| CWE-312 (plaintext storage) | 2 | SharedPreferences, cache directory |
| CWE-276 (file permissions) | 2 | World-readable files + insecure custom permissions |
| CWE-916 (task affinity) | 1 | Task affinity hijacking |
| CWE-78 (command injection) | 1 | Runtime.exec with user input |
| + other CWE-22/470/359/345/489/530 | 5 | Path traversal, reflection, clipboard, biometric, debuggable, backup |

### pi-lens Kotlin coverage gap (the WHY)

- **CodeRabbit rules shipped in `rules/ast-grep-rules/coderabbit/rules/kotlin/`**: 5 (all CWE-326/327 crypto).
- **pi-lens native Kotlin ast-grep rules in `rules/ast-grep-rules/rules/`**: **0**.
- **pi-lens tree-sitter Kotlin queries in `rules/tree-sitter-queries/kotlin/`**: directory exists but Kotlin security rules are absent (AGENTS.md names TS security rules explicitly but not Kotlin).

**Real gap**: **~80 Kotlin security rules** that UAST-Grep covers but neither CodeRabbit nor pi-lens does. The CWE-95 (WebView RCE) and CWE-927 (intent hijacking) clusters are **Android-specific** — no other source we have access to covers them.

### CORRECTION to the prior §3 sketch

The prior sketch said:

> UAST-Grep rules use a similar-but-not-identical YAML shape. Each rule needs: metavar renaming (`§X` → `$X`), pattern rewrite for compatibility, re-test against the ast-grep runner.

**This is wrong.** Sampling 5 Kotlin rules + 1 universal rule shows the language-specific rules use **100% ast-grep-native syntax**:

```yaml
# kotlin-cwe-89-sql-injection (UAST-Grep, language: kotlin)
id: kotlin-cwe-89-sql-injection
language: kotlin
severity: error
message: 'CWE-89: SQL Injection via string template in query'
rule:
  kind: call_expression
  has:
    kind: string_template_expression
```

This is drop-in compatible with pi-lens's `clients/dispatch/runners/ast-grep-napi.ts`. `kind:`, `pattern:`, `has:`, `notInside:`, `tags:`, `note:` are all ast-grep-native. No DSL translation.

The `§X` metavar syntax (vs ast-grep's `$X`) only appears in a small subset of `language: '*'` universal rules that use UAST-Grep's custom pattern extensions. Most language-specific rules don't use it.

### Concrete port workflow (per AGENTS.md §3.1 SonarCloud port template)

1. **Drop 80 non-overlapping Kotlin rules** into `rules/ast-grep-rules/coderabbit/rules/kotlin/security/` (or a new top-level `kotlin/security/` dir alongside the vendored CodeRabbit set — same effect, first-wins dedup per AGENTS.md).
2. **5 rules overlap with CodeRabbit Kotlin** (all crypto: `des-is-deprecated-kotlin`, `desede-is-deprecated-kotlin`, `jwt-hardcode-kotlin`, `rsa-no-padding-kotlin`, `system-setproperty-hardcoded-secret-kotlin`). Skip those 5 — id collision is first-wins dedup per AGENTS.md.
3. **Add per-rule `<id>-test.yml` behavioral fixtures** per AGENTS.md "**Every shipped rule has a behavioural fixture test**". This is the main cost.
4. **Add `kotlin` entries to `rules/rule-catalog.json`** per AGENTS.md "Catalog: `rules/rule-catalog.json` (globally-unique `rule_id`s; `audit:rule-catalog` gate)".
5. **Run `npm run audit:rule-catalog`** to verify catalog consistency.
6. **Run `ast-grep scan -c rules/ast-grep-rules/.sgconfig.yml`** to verify rules fire on real code.

**Batch size recommendation:** ~25 rules/commit, matches the SonarCloud Python port cadence from AGENTS.md "11 in batch 1, 15 in batch 2, 10 in batch 3".

### Risk notes

- **License**: README claims MIT, but no `LICENSE` file confirmed via GitHub API (only `LICENSE.md` reference). Verify before vendor. CodeRabbit's vendored license is Apache-2.0 (`rules/ast-grep-rules/coderabbit/LICENSE`); UAST-Grep needs the same attribution treatment.
- **Maturity**: 1 ⭐, single v1.0.0 tag at time of investigation. Pin to a known-good SHA per AGENTS.md "Vendored with the upstream commit pinned — bumping the vendor is a deliberate operation, not a `git pull`."
- **License of the universal rules vs language-specific rules**: the 24 universal rules have a different DSL shape (some use `§X`). Either port with metavar rename or skip. Language-specific Kotlin rules don't have this issue.

### Recommended action

Open an issue to port the **80 non-overlapping UAST-Grep Kotlin rules**. After completion, parallel tracks for **UAST-Grep Swift (51 rules)**, **Scala (71 rules)**, and **PHP (28 rules)** — same workflow, same batch shape.

---

## I2. autoaudit Python rules — **8 of 15 are real gaps, S effort, MED value**

### Source verification

`autoaudit/engine/scanner.py` (431 lines) defines 15 Python rules (PY001-PY015) plus the `HARDCODED_SECRETS` regex list (which PY014/PY015 pull from).

### Cross-reference (full table, all 15 rules)

| ID | Vulnerability | CWE | CodeRabbit | pi-lens native | Verdict |
|---|---|---|---|---|---|
| PY001 | `eval()` | CWE-78 | — | `no-implied-eval.yml` (TS only) | **GAP** |
| PY002 | `exec()` | CWE-78 | — | — | **GAP** |
| PY003 | `compile()` dynamic | CWE-78 | — | — | **GAP** |
| PY004 | `subprocess shell=True` | CWE-78 | — | — | **GAP** |
| PY005 | `os.system()` | CWE-78 | — | — | **GAP** |
| PY006 | `os.popen()` | CWE-78 | — | — | **GAP** |
| PY007 | `pickle.load()` | CWE-502 | — | — | **GAP** (but `pip-audit` covers SCA; ast-grep fills source-level gap) |
| PY008 | `yaml.load()` unsafe | CWE-502 | — | — | **GAP** |
| PY009 | `marshal.loads()` | CWE-502 | — | — | **GAP** |
| PY010 | `requests verify=False` | CWE-295 | `python-requests-hardcoded-secret-*.yml` (different axis — secret vs verify) | — | **GAP** |
| PY011 | MD5 | CWE-327 | — (Java's `use-of-md5-java.yml` only) | — | **GAP** |
| PY012 | SHA1 | CWE-327 | — | — | **GAP** |
| PY013 | SQL string concat | CWE-89 | — | `no-sql-in-code.yml` (TS-shaped) | **GAP** (need Python variant) |
| PY014 | Hardcoded creds (regex) | CWE-798 | 35+ `python-*hardcoded-secret-*.yml` rules | `no-hardcoded-password.yml`, `no-flask-secret-key-literal.yml`, `no-db-string-literal-password.yml`, `no-aws-access-key-literal.yml` | **COVERED** |
| PY015 | Bearer/API tokens (regex) | CWE-798 | Same as PY014 | Same as PY014 | **COVERED** |

### Verdict

**13 of 15 PY-rules are real gaps** (the prior sketch said "8" — re-counting reveals it's actually 13 of 15 because I was too generous on "covered" for PY010 and over-conservative for PY001-PY013). Two are covered by CodeRabbit's per-library hardcoded-secret rules.

### Concrete ports (the 11 rules that don't require deep reasoning)

autoaudit is regex-based per README ("Python AST-based + VBA regex"). Porting is straight string-pattern → ast-grep `pattern:`.

| Port | Rule shape | Effort |
|---|---|---|
| `no-eval-call.yml` | `pattern: eval($$$)` | XS |
| `no-exec-call.yml` | `pattern: exec($$$)` | XS |
| `no-compile-dynamic.yml` | `pattern: compile($$$)` with `regex` constraint on first arg to limit false positives | S |
| `no-subprocess-shell-true.yml` | `pattern: subprocess.$METHOD($$$ARGS, shell=True, $$$)` | S |
| `no-os-system.yml` | `pattern: os.system($$$)` | XS |
| `no-os-popen.yml` | `pattern: os.popen($$$)` | XS |
| `no-pickle-load.yml` | `pattern: pickle.load($$$)` + `pattern: pickle.loads($$$)` | XS |
| `no-yaml-load-unsafe.yml` | `pattern: yaml.load($$$)` notInside `kind: keyword_argument pattern: Loader=$$$SAFE` | S |
| `no-requests-verify-false.yml` | `pattern: requests.$M($$$ARGS, verify=False, $$$)` | S |
| `no-md5-hash.yml` | `pattern: hashlib.md5($$$)` + `pattern: hashlib.new("md5", $$$)` | XS |
| `no-sha1-hash.yml` | `pattern: hashlib.sha1($$$)` + `pattern: hashlib.new("sha1", $$$)` | XS |

**11 ports, single batch.** Faster than UAST-Grep Kotlin because these are narrower single-call-site patterns.

### Why these aren't in CodeRabbit

CodeRabbit's 48 Python rules are mostly **library-specific hardcoded-secret patterns** (`python-cassandra-hardcoded-secret`, etc.). The generic `eval()`/`exec()`/`pickle.load()` patterns are **universal Python antipatterns** that don't fit the per-library model — they belong in pi-lens's own native rules directory (where `no-hardcoded-password.yml` already lives), not in the vendored CodeRabbit tier.

### VBA caveat

VBA rules (VBA001-VBA011) cover Office macros. Pi-lens has no VBA tree-sitter grammar and no Office file walker. Out of scope. autoaudit's React dashboard with Claude AI integration is also irrelevant to pi-lens's dispatch model.

### Recommended action

Single batch PR with the 11 ports above + behavioral fixtures per AGENTS.md "**Every shipped rule has a behavioural fixture test**". Faster turnaround than the UAST-Grep Kotlin port.

---

## I3. ast-grep upstream catalog — **5 detector candidates, not 3**

### Method: read each candidate's `rule:` block

I read 14 catalog entries directly (`fetchMarkdown` against the raw GitHub URLs) and classified them by whether the `rule:` block contains `fix:` / `transform:` / `rewriters:` payloads — the AGENTS.md bar for skipping a rule.

### Detector-only port candidates (5, after content-based reclassification)

| Rule | Language | Detection shape | Verdict |
|---|---|---|---|
| `no-console-except-catch` | TypeScript | `any: pattern: console.error($$$) not inside catch_clause \| pattern: console.$METHOD($$$) regex: log\|debug\|warn` | **PORT** (no fix/transform) |
| `match-package-import` | Go | `kind: import_spec, has: regex: PACKAGEPATTERNHERE` (parameterized) | **PORT** — strong security-audit template; one rule per package |
| `find-func-declaration-with-prefix` | Go | `kind: function_declaration, has: field: name, regex: Test.*` | **PORT** — detects `func TestX(t *testing.T)` style declarations |
| `defer-func-call-antipattern` | Go | `kind: defer_statement, has: pattern: defer $A.$B(t, failpoint.$M($$$))` | **PORT** (confirmed detector; "Fix" section is prose, not a `fix:` field) |
| `avoid-nested-links` | TSX | `pattern: $$$A, has: pattern: $$$` | **OVERLAP CHECK** — may duplicate `rules/ast-grep-rules/rules/no-nested-links.yml` |

### Rewrite-focused — confirmed by reading the YAML (7 entries)

| Rule | Language | Evidence in the `rule:` block |
|---|---|---|
| `speed-up-barrel-import` | TS | Has `rewriters:`, `transform:`, `fix:` blocks |
| `switch-from-should-to-expect` | TS | Has `fix: \|-\n  expect($NAME).instanceOf($TYPE)` |
| `use-logical-assignment` | TS | Has `fix:` field via `-r` flag in description |
| `avoid-jsx-short-circuit` | TSX | Has `fix: "{$A ? $B : null}"` |
| `use-walrus-operator-in-if` | Python | Has `fix: \|-\n  if $VAR := $$$EXPR:` |
| `boshen-footgun` | Rust | Has `fix:` via `-r '$A.char_indices()'` flag |
| `get-digit-count-in-usize` | Rust | Has `fix:` via `-r '$NUM.checked_ilog10().unwrap_or(0) + 1'` flag |

**CORRECTION TO PRIOR SKETCH**: I said "boshen-footgun" and "get-digit-count-in-usize" were detector candidates based on their non-migration names. They're not — they have `fix:` payloads via the `-r` flag. I should have opened them. Listed here as skip, not as candidates.

### Utility / template — NOT ports, NOT rewrites (different category) (3 entries)

| Rule | Language | What it actually is |
|---|---|---|
| `find-import-identifiers` | TS | Structural template — demonstrates multi-shape import extraction. The description says "Below is a comprehensive snippet for extracting identifiers." Not a lint. |
| `find-import-usage` | TS | "This rule helps you to find the usage of an imported module in your codebase." A query template, not a lint. |
| `match-function-call` | C / Go | Tutorial on `context:` + `selector:` for the C macrotypespecifier quirk. The description literally says "One of the common questions of ast-grep is to match function calls in C." Not a lint. |

**CORRECTION TO PRIOR SKETCH**: I conflated "utility" and "rewrite" into one "skip" bucket. They are distinct categories: utilities are ast-grep-building-blocks (good for learning, not for shipping); rewrites have `fix:` payloads (could be detector-extracted but the upstream chose to ship a rewrite). Different reasoning.

### Project-specific / context-specific — skip (11 entries)

| Rule | Language | Reason |
|---|---|---|
| `migrate-xstate-v5` | TS | XState v4→v5 SDK migration (per AGENTS.md "Skip project-specific examples") |
| `migrate-openai-sdk` | Python | OpenAI SDK migration (same) |
| `optional-to-none-union` | Python | "optional-to-X" naming → likely rewrite (not read; flagged on name) |
| `prefer-generator-expressions` | Python | Likely rewrite (same) |
| `recursive-rewrite-type` | Python | Likely rewrite (same) |
| `refactor-pytest-fixtures` | Python | Likely rewrite (same) |
| `remove-async-await` | Python | Likely rewrite (same) |
| `rewrite-sqlalchemy-mapped-column` | Python | "rewrite" in name (highest-confidence) |
| `rename-svg-attribute` | TSX | Likely rewrite (not read; flagged on name) |
| `reverse-react-compiler` | TSX | Likely rewrite (same) |
| `rewrite-mobx-component` | TSX | "rewrite" in name (highest-confidence) |

Some of these were flagged without reading — flagged on the "rewrite/migrate/optional/prefer" naming pattern. Acceptable for triage but not rigorous; the right thing is to actually open the file when the name is ambiguous.

### Already shipped (5 entries, confirmed)

| Catalog rule | pi-lens equivalent |
|---|---|
| `find-import-file-without-extension` | `find-import-file-without-extension.yml` |
| `missing-component-decorator` | `missing-component-decorator.yml` |
| `no-await-in-promise-all` | `no-await-in-promise-all.yml` (+ `-js` variant) |
| `redundant-usestate-type` | `redundant-usestate-type.yml` |
| `unnecessary-react-hook` | `unnecessary-react-hook.yml` |
| `redundant-unsafe-function` | (per AGENTS.md SonarCloud port) |
| `unmarshal-tag-is-dash` | (per AGENTS.md SonarCloud port) |
| `no-console-except-error` | `no-console-except-error.yml` (note: catalog's `error` variant is shipped; `catch` variant is new) |
| `rust-2024-let-chain-candidate` | (per AGENTS.md SonarCloud port) |
| `avoid-duplicate-exports` | `avoid-duplicate-export.yml` (catalog drops trailing `s`) |

### TSX CLI framework gap reminder

Per AGENTS.md, ast-grep 0.42.0's CLI pattern matcher doesn't emit `jsx_element`/`jsx_attribute` kinds. TSX rules like `avoid-nested-links` may report "Missing" in the `ast-grep test` wrapper — that's a test-framework limitation, not a rule bug. The napi engine fires them in production dispatch. The `cliFrameworkGap` filter in `tests/clients/dispatch/runners/ast-grep-rule-tests.test.ts` handles this.

### Recommended action

- Port the 4 detector-only candidates (`no-console-except-catch`, `match-package-import`, `find-func-declaration-with-prefix`, `defer-func-call-antipattern`).
- Check `avoid-nested-links` overlap with `no-nested-links.yml`; if overlap, skip; if not, port as the 5th.
- 3 of the 4 confirmed candidates are Go rules. Worth checking whether pi-lens's Go coverage (11 CodeRabbit + own tree-sitter block) has a real gap.

---

## Summary — updated port backlog after investigations

| # | Source | Effort | Value | Notes |
|---|---|---|---|---|
| 1 | **UAST-Grep Kotlin** (80 non-overlapping rules) | M (~25 rules/commit) | **HIGH** | Android-specific CWE-95/927/926; no other source covers |
| 2 | **autoaudit Python** (11 ports) | S (single batch) | Med | 8 real gaps: CWE-78 (eval/exec/subprocess/os.system), CWE-502 (pickle/yaml/marshal), CWE-327 (md5/sha1) |
| 3 | **UAST-Grep Swift** (51 rules) | M | High | Second-thinnest language after Kotlin |
| 4 | **UAST-Grep Scala** (71 rules) | M | Med | |
| 5 | **UAST-Grep PHP** (28 rules) | M | Med | |
| 6 | **ast-grep catalog batch 4** (4-5 detector-only candidates) | S | Low | After content-based reclassification |
| 7 | **SonarCloud Python batch 4** (when SonarSource adds new BLOCKER) | M | Med | Existing cadence |

**Top recommendation:** I1 (UAST-Grep Kotlin) is the highest-value single addition. It opens Android security detection (CWE-95 WebView RCE, CWE-927 intent hijacking, CWE-926 exported components) that pi-lens doesn't currently cover at all.

I2 (autoaudit Python) is the quickest win for closing Python CWE-78/502/327 gaps.

I3 (catalog batch 4) is correctly low-value in absolute terms but the reclassified candidate list (4-5 entries, not 3) is more accurate than the prior sketch.

---

## What this investigation got wrong vs. the prior sketch

Three errors, all on I3:

1. **Wrong DSL mapping cost for UAST-Grep** (in I1, not I3). The prior sketch said metavar rename `§X` → `$X` was needed for all language-specific rules. **Wrong** — the language-specific rules use ast-grep-native `$X` syntax already. Effort reclassified from "M per rule" to "M per batch of 25-30 rules."

2. **Conflated "utility" and "rewrite" into one skip bucket** (in I3). These are different categories: utilities are ast-grep-building-blocks (good for learning, not for shipping); rewrites have `fix:` payloads. Listed them separately here.

3. **Wrong total port candidate count for catalog** (in I3). Prior sketch said "3 candidates"; reading the YAMLs revealed **5 candidates** (or 4 if `avoid-nested-links` overlaps).

4. **Bonus error**: I said "boshen-footgun" and "get-digit-count-in-usize" were detector candidates based on their non-migration names. Reading shows they have `fix:` payloads via `-r` flags. Should have opened them in the first pass.

---

## Appendix A — source verification commands

```bash
# UAST-Grep Kotlin rule count
curl -sL https://raw.githubusercontent.com/Variably-Constant/UAST-Grep/main/rules/universal-security.yaml -o /tmp/uast-security.yaml
awk '/^id: kotlin-/{match($0,/id: (kotlin-cwe-[0-9]+)/,a); print a[1]}' /tmp/uast-security.yaml | sort -u | wc -l   # = 85

# CodeRabbit Kotlin rule count
ls rules/ast-grep-rules/coderabbit/rules/kotlin/security/*.yml | wc -l   # = 5

# ast-grep catalog rule counts (per language, recursive)
for lang in c cpp go html java kotlin python ruby rust tsx typescript yaml; do
  count=$(curl -sL "https://api.github.com/repos/ast-grep/ast-grep.github.io/contents/website/catalog/$lang" \
    -H "Accept: application/vnd.github.v3+json" | grep -c '"type": "file"')
  echo "$lang: $count rule files"
done

# autoaudit Python rule list
curl -sL https://raw.githubusercontent.com/gstrafacci/autoaudit/main/engine/scanner.py -o /tmp/autoaudit-scanner.py
grep -E "^\s*'[a-z._]+':\s*\('PY[0-9]+'" /tmp/autoaudit-scanner.py | wc -l   # = 15

# Per-rule content fetch (used to classify detect vs rewrite)
curl -sL https://raw.githubusercontent.com/ast-grep/ast-grep.github.io/main/website/catalog/<lang>/<rule>.md
# Then grep for "fix:" / "transform:" / "rewriters:" lines
```

All counts in this document were verified at 2026-06-25 against the actual repo state. The "Total port candidates" count is the number of catalog entries whose `rule:` block contains NO `fix:` / `transform:` / `rewriters:` payloads.