{ "metadata": { "skill_name": "wiki", "skill_path": "", "executor_model": "", "analyzer_model": "", "timestamp": "2026-04-14T16:15:05Z", "evals_run": [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 ], "runs_per_configuration": 3 }, "runs": [ { "eval_id": 1, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 0.9, "passed": 9, "failed": 1, "total": 10, "time_seconds": 388.4, "tokens": 68781, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Runs init_wiki.py (or init_wiki.ts) script to create directory scaffold", "passed": true, "evidence": "report.md Phase 1 documents the command: `node skills/wiki/scripts/init_wiki.js --path /tmp/eval-iiq-project --domain 'backend-services' --name 'eval-iiq-project'`, and the full scaffold (wiki/, raw/, graph/, audit/, log/, outputs/, .wiki-cache/, wiki.config.yaml, events.jsonl, edges.jsonl, .wiki-ignore) is present under outputs/." }, { "text": "wiki.config.yaml is created with domain 'backend-services'", "passed": true, "evidence": "outputs/wiki.config.yaml line 3: `domain: backend-services` under the `wiki:` section. Name field also set to `eval-iiq-project`." }, { "text": "Directory scaffold includes wiki/, raw/, graph/, audit/, log/, outputs/", "passed": true, "evidence": "`ls` on outputs/ confirms all six directories exist: wiki/ (with index.md, summaries.md, overview.md, auth/, claims/, synthesis/, templates/), raw/auth/, graph/edges.jsonl, audit/{open,resolved}/, log/{events.jsonl, daily/}, outputs/{queries/, reports/}." }, { "text": "Source file is copied to raw/ directory", "passed": true, "evidence": "outputs/raw/auth/architecture.md exists (3513 bytes) and contains the full original architecture doc (headings: JWT Access Tokens, Server-Side Sessions, Refresh Flow) matching the fixture described in report.md Phase 0. events.jsonl records bytes_in=3513 matching the file size." }, { "text": "A wiki page is created with proper frontmatter (title, type, tags, sources, created, updated, summary)", "passed": true, "evidence": "outputs/wiki/auth/authentication.md frontmatter (lines 1-14) has all seven required fields: title='Authentication and Session Management', type='concept', tags (6 items), sources=[raw/auth/architecture.md], created=2026-04-14, updated=2026-04-14, summary (multi-line block). Also carries a quality:0.0 field." }, { "text": "Tags are content-only concepts (e.g., 'jwt', 'authentication', 'session-management')", "passed": true, "evidence": "authentication.md line 4: `tags: [authentication, jwt, session-management, refresh-token, rs256, postgresql]`. All six are content concepts — no structural (e.g., 'concept', 'index'), temporal (e.g., '2026'), or metadata tags. Explicitly contains the three examples named in the assertion." }, { "text": "summaries.md is updated with the new page's summary", "passed": true, "evidence": "Both outputs/summaries.md and outputs/wiki/summaries.md contain an '## Authentication' section linking to `auth/authentication.md` with a paragraph summarizing the hybrid RS256 JWT + PostgreSQL session design and per-page tag list. Not a scaffold stub." }, { "text": "Query answer includes inline citations to wiki pages", "passed": true, "evidence": "outputs/outputs/queries/2026-04-14-how-does-authentication-work.md contains 9 inline citations linking back to wiki pages, including section-anchor cites like `[auth/authentication.md § JWT Access Tokens](../../wiki/auth/authentication.md#access-tokens-jwt--rs256)`, `[auth/authentication.md § Server-Side Sessions]`, `[auth/authentication.md § Refresh Flow]`, and `[overview.md](../../wiki/overview.md)`. Plus a Citations section at the bottom." }, { "text": "Answer is archived to outputs/queries/", "passed": true, "evidence": "outputs/outputs/queries/2026-04-14-how-does-authentication-work.md (5255 bytes) archived with YAML header recording the query, asked_at, pages_considered, pages_loaded, and relevance_scores. events.jsonl query event's answer_path field points to this same file." }, { "text": "events.jsonl records init, ingest, and query operations", "passed": false, "evidence": "outputs/events.jsonl (and log/events.jsonl) only contain two events: `op: ingest` and `op: query`. No `op: init` entry is present. Grep for `init` in both copies returns no matches. The init_wiki.ts script does not emit an init event by design — the scaffold was created but the operation was not logged." } ], "notes": [] }, { "eval_id": 10, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 4, "failed": 0, "total": 4, "time_seconds": 195.7, "tokens": 53466, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Asks about detection markers and patterns", "passed": true, "evidence": "qa-transcript.md Q5 asks 'Which literal substrings are reliable \"this is our ORM\" markers?' and describes detection score matching. Q6-Q8 ask for regex patterns for class/table/columns. Q9 asks for relationship patterns mapped under relationship_detection.patterns." }, { "text": "Asks about entity-table naming conventions", "passed": true, "evidence": "qa-transcript.md Q10 explicitly asks 'Naming conventions — when __table__ is missing, how should the mapper derive a table name from a class name?' listing built-ins snake_case, snake_case_plural, lower_case. User answered snake_case_plural with User->users mapping." }, { "text": "Generates a valid YAML profile with required fields (name, language, detection, entity_extraction)", "passed": true, "evidence": "yaml.safe_load() of custom-basemodel.yaml succeeded and contains all required fields: ['name', 'language', 'description', 'detection', 'entity_extraction', 'relationship_detection', 'naming_conventions']. name='custom-basemodel', language='python', detection has file_patterns+markers, entity_extraction has class_pattern/table_pattern/column_pattern." }, { "text": "Profile can be loaded by the profile loader without errors", "passed": true, "evidence": "profile-load-test.txt shows `=== loadProfile( ...custom-basemodel.yaml ) ===` followed by a fully populated OrmProfile JSON object with all fields resolved. No 'error', 'exception', 'ProfileValueError', or traceback appears in the output. The loader then proceeded to run extractEntities successfully, confirming no load-time error." } ], "notes": [] }, { "eval_id": 11, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 4, "failed": 0, "total": 4, "time_seconds": 123.5, "tokens": 47800, "tool_calls": null, "errors": 0 }, "expectations": [ { "text": "Emits a fenced ```mermaid code block with the `erDiagram` directive", "passed": true, "evidence": "target-after.md line 15 opens a fenced ```mermaid block and line 16 is `erDiagram`, followed by the erDiagram body (lines 17-31)." }, { "text": "Preserves existing non-Mermaid page content outside the managed markers", "passed": true, "evidence": "preamble-check.md shows `diff -u target-before.md target-after.md` produces only `+` hunks (no deletions/modifications) and a byte-for-byte `cmp` match: the first N bytes of target-after.md equal all of target-before.md. Frontmatter, `# Schema overview` preamble, `` markers, and trailing paragraph are all intact (lines 1-10 of target-after.md)." }, { "text": "Uses the `||--o{` relationship syntax for one-to-many edges", "passed": true, "evidence": "target-after.md line 17: ` User ||--o{ Order : places` — exact one-to-many Mermaid ER syntax." }, { "text": "Produces syntactically valid Mermaid that passes mermaid_lint.ts", "passed": true, "evidence": "mermaid-lint.txt contains `[]` — an empty JSON array indicating zero lint issues. report.md confirms `mermaid_lint.js --page target.md` returned `[]` and notes erDiagram bracket-balancing is correctly skipped (since `{ }` are part of attribute syntax)." } ], "notes": [ "Executor translated the entity/relationship description into a pre-rendered mermaid `code` body inside input.json because mermaid_gen.ts does not parse an abstract DSL — it formats a pre-rendered body (per AGENT.md / mermaid_gen.ts). Worth noting as a usability gap: agents handing the mermaid agent structured entity data must themselves render the Mermaid syntax." ] }, { "eval_id": 12, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 4, "failed": 0, "total": 4, "time_seconds": 195.0, "tokens": 49031, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Chooses an appropriate diagram type (flowchart/sequence/state) for the input shape", "passed": true, "evidence": "pipeline-after-first-run.md line 22 opens the fenced block with `flowchart TD`. The input.json is a DAG of six stages connected by directional edges (checkout -> build -> docker -> push -> deploy -> smoke) with `\"type\": \"flowchart\"`, so a top-down flowchart is the correct selection over sequence or state." }, { "text": "Handles nodes with quotes and special characters via safe escaping", "passed": true, "evidence": "Line 23 renders the `Checkout \"main\" branch` label as `checkout[\"Checkout #quot;main#quot; branch\"]`, using the `#quot;` HTML-entity escape for the nested double-quotes inside the bracketed label. mermaid-lint.txt shows `node skills/wiki/scripts/mermaid_lint.js --page pipeline.md` returned `[]` with exit 0, confirming the diagram parses cleanly and bracket balance is preserved. The `&` in `Build & test` (line 24) is left literal — acceptable inside a Mermaid `[\"...\"]` string label." }, { "text": "Updates pipeline.md idempotently (re-running on the same input produces no diff)", "passed": true, "evidence": "idempotency-check.txt reports both runs sha256 = 2e65ee9d75f08ebc0519546ebd8d2af8f8a1518a8382609f7a77217a4a155089 and `diff` returned no output. Independent verification: `diff pipeline-after-first-run.md pipeline-after-second-run.md` exits 0, and my own `shasum -a 256` on both files produced the identical hash. The agent's `injectMermaid` helper replaces the `## Diagrams` section in place rather than appending." }, { "text": "Never mutates wiki pages outside the specified target", "passed": true, "evidence": "scope-check.md documents `git status --short` showing only the pre-existing untracked `wiki-workspace/iteration-2/` directory, `git diff --stat HEAD` empty (no tracked files changed), and a `find -newer /tmp/eval-mermaid-flow/input.json -name '*.md'` (excluding node_modules, wiki-workspace, .git) returning zero results. The only markdown file written was `/tmp/eval-mermaid-flow/pipeline.md`, the explicit `--page` target, with mutation confined to the `## Diagrams` section." } ], "notes": [] }, { "eval_id": 13, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 4, "failed": 0, "total": 4, "time_seconds": 133.7, "tokens": 46826, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Managed content wrapped in `` and `` markers", "passed": true, "evidence": "Grep count of 'wiki-managed:' returns exactly 2 matches in each of the 4 output files: root-CLAUDE.md (lines 1 and 25), submodule-api-CLAUDE.md (lines 1 and 23), submodule-worker-CLAUDE.md (lines 1 and 23), submodule-gateway-CLAUDE.md (lines 1 and 23). Each file contains exactly one balanced start/end pair wrapping the managed content. markers-check.md confirms the same and references the MarkerCorruptError guard (G-CLAUDE-MD-MARKER)." }, { "text": "Root file lists every discovered submodule with a relative link", "passed": true, "evidence": "root-CLAUDE.md lines 10-12 contain a '## Submodules' section with relative links for all three discovered submodules: '- [services/api](services/api/CLAUDE.md)', '- [services/gateway](services/gateway/CLAUDE.md)', '- [services/worker](services/worker/CLAUDE.md)'. Links are relative (no leading '/' or absolute path)." }, { "text": "Each submodule file includes a Parent Project link back to the root CLAUDE.md", "passed": true, "evidence": "All three submodule files (api, worker, gateway) contain a '## Parent Project' section at line 8 with '- [Root CLAUDE.md](../../CLAUDE.md)' at line 10. The '../../' prefix correctly traverses two directory segments from 'services//' back to the project root. parent-link-check.md confirms each link resolves correctly." }, { "text": "Content outside the managed markers is preserved byte-for-byte on re-generation", "passed": true, "evidence": "idempotency-check.md documents the verification method: snapshot the project tree, re-run the 4-file generation sequence, then 'diff -ru'. Output shows '---DIFF_EXIT=0---' with no diff output, confirming byte-identical trees. The managed section is regenerated deterministically from projectRoot, wikiRoot, sorted submodule list, and optional --submodule flag; non-managed content is preserved via regex replacement." } ], "notes": [] }, { "eval_id": 14, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 4, "failed": 0, "total": 4, "time_seconds": 174.5, "tokens": 50281, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Preserves all content above the `` marker", "passed": true, "evidence": "user-content-above-markers-check.md shows before/after sha256 `95b9698472b81b1ede6acd59437cf1d118dbe5c4b81320636f9495153736ab53` (145 bytes each). Independently re-extracted the pre-marker region from CLAUDE-before.md and CLAUDE-after.md with `sed -n '1,//p' | sed '$d'` and both gave 145 bytes with the same sha256 `95b9698472b81...`. The user's custom header, prose, and bullet list (alpha/beta) are preserved byte-for-byte." }, { "text": "Preserves all content below the `` marker", "passed": true, "evidence": "user-content-below-markers-check.md shows before/after sha256 `a22d3512e3934d473dd75baf970521bbf4f25fa785fdfaa2abafcadd4329dafe` (83 bytes each, matching). Independent re-extraction via `sed -n '//,$p' | sed '1d'` (extraction bounds differ by 1 byte because of the trailing newline handling) yields sha256 `f3c9acf297e7eea7a3473f0da38e00dec8f2dd8e86a579b0b6314e584b447653` that is IDENTICAL between before and after. The `## Manual footer` block is preserved byte-for-byte." }, { "text": "Regenerates only the managed section from the current wiki state", "passed": true, "evidence": "managed-section-diff.md shows the managed section sha256 changed from `b5f13169...` to `000fe78d...`. Independent re-hashing with `sed -n '/start -->/,/end -->/p'` confirms different hashes before (`596a60e8...`) vs after (`f6129fc8...`). The `diff -u` output shows the stale placeholder `(old stale generated content — should get replaced)` was replaced with freshly generated content: Overview, Build & Run, Service Dependencies, Database References sections — exactly what `generateClaudeMd()` emits for project name `eval-claude-md-preserve` with wiki at `wiki/index.md`." }, { "text": "Is idempotent on re-run when the wiki state has not changed", "passed": true, "evidence": "idempotency-check.md documents a second invocation of `claude_md_gen.js --update` against the same project. First-run and second-run sha256s match exactly (`7bcdbe33b3344b67a4dc325c3585464abf850f08ee22cf1e8be41e4e39d1ab4a`), `cmp` exits 0, and `diff -u` produces no output. The marker-replacement regex is deterministic." } ], "notes": [] }, { "eval_id": 2, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 7, "failed": 0, "total": 7, "time_seconds": 163.4, "tokens": 69621, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Runs lint_checks.py (or lint_checks.ts) to detect violations", "passed": true, "evidence": "lint-report-before.md header cites the exact command: `node skills/wiki/scripts/lint_checks.js --wiki-root /tmp/eval-lint-wiki/`. The structured output (Summary table with error/warning/info counts and per-category sections) matches the linter's emit format, and events.jsonl records a corresponding {\"op\": \"lint\", \"phase\": \"before\", \"errors\": 8, \"warnings\": 6, \"info\": 0} entry." }, { "text": "Reports missing frontmatter fields (title, type, tags, sources)", "passed": true, "evidence": "lint-report-before.md §'Errors (8) > missing_frontmatter (6)' lists: `page-a.md — Missing required frontmatter field: quality`; `page-b.md — Missing ... quality`, `... sources`, `... tags`; `page-c.md — No frontmatter found (missing --- delimiters)`; `page-e.md — Missing ... quality`. Missing tags and sources are explicitly called out for page-b; page-c's total absence covers title/type as well." }, { "text": "Reports broken internal links between wiki pages", "passed": true, "evidence": "lint-report-before.md §'Errors (8) > broken_links (2)': `wiki/page-a.md — Link to ./page-d.md not found` and `wiki/page-c.md — Link to ./ghost.md not found`. These match the pre-fix-wiki contents: page-a.md references `./page-d.md` (no such file), page-c.md references `./ghost.md` (no such file)." }, { "text": "Fixes missing frontmatter by adding required fields", "passed": true, "evidence": "All five post-fix pages (page-a.md, page-b.md, page-c.md, page-d.md, page-e.md) carry the full 8-field frontmatter required by lint_checks.ts REQUIRED_FIELDS (title, type, tags, sources, created, updated, quality, summary). page-c.md, which had no frontmatter at all pre-fix, now has a complete block with title 'Page C', tags [test, gamma], quality 0.5, etc. page-b.md gained the previously-missing tags:[test,beta], sources:[], and quality:0.6." }, { "text": "Fixes broken links by updating paths or removing dead links", "passed": true, "evidence": "page-a.md's broken `./page-d.md` reference is now resolved by a newly-created `page-d.md` stub (post-fix-wiki/page-d.md exists with proper frontmatter and content referencing back to page-a). page-c.md's dead `./ghost.md` link has been removed: the post-fix body states 'Previously referenced a non-existent ghost page; that reference has been removed' and now links to existing page-b.md and page-e.md. events.jsonl shows `\"created_pages\": [\"wiki/page-d.md\"]` confirming the stub creation." }, { "text": "Re-runs lint to confirm zero remaining violations", "passed": true, "evidence": "lint-report-after.md Summary table shows error=0, warning=0, info=0, Total=0. Raw JSON block: `{\"issues\": [], \"summary\": {\"error\": 0, \"warning\": 0, \"info\": 0}}`. events.jsonl final entry: `{\"op\": \"lint\", \"phase\": \"after\", \"errors\": 0, \"warnings\": 0, \"info\": 0}` confirms the second invocation returned a clean result." }, { "text": "events.jsonl records the lint and fix operations", "passed": true, "evidence": "events.jsonl contains exactly 3 entries: (1) op=lint phase=before with error/warning counts matching the before-report; (2) op=fix with pages_touched=[page-a..e], edges_added=10, created_pages=[page-d.md]; (3) op=lint phase=after with zeroed counts. All entries include ISO-8601 timestamps." } ], "notes": [] }, { "eval_id": 3, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 6, "failed": 0, "total": 6, "time_seconds": 108.1, "tokens": 55498, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Runs graph_ops.py (or graph_ops.ts) to query the relationship graph", "passed": true, "evidence": "path-answer.md shows the command: `node skills/wiki/scripts/graph_ops.js path --edges /tmp/eval-path-wiki/graph/edges.jsonl --from \"authentication\" --to \"database-schema\"`. graph_ops-raw-output.txt contains a JSON array of edge objects consistent with graph_ops.ts `shortestPath` output. events.jsonl also records `{\"op\": \"path\", ...}` entries, matching the script's event-logging contract." }, { "text": "Reads edges.jsonl for typed relationships", "passed": true, "evidence": "The command invocation explicitly passes `--edges /tmp/eval-path-wiki/graph/edges.jsonl`. The raw output shows edge records with `type: \"supports\"` and `provenance: \"INFERRED\"/\"EXTRACTED\"` fields — the exact typed-edge schema stored in edges.jsonl." }, { "text": "Returns ordered list of pages from authentication to database-schema", "passed": true, "evidence": "path-answer.md renders an ordered 3-row table: row 1 authentication -> user-model, row 2 user-model -> orm-mapping, row 3 orm-mapping -> database-schema. The sequence begins at `authentication` and terminates at `database-schema` with 4 nodes / 3 hops. Same ordering present in graph_ops-raw-output.txt." }, { "text": "Shows edge types (supports, extends, etc.) between each pair", "passed": true, "evidence": "The `Edge type` column in the path table lists `supports` for all three hops (also visible in graph_ops-raw-output.txt: each edge has `\"type\": \"supports\"`). The answer also references edge-type semantics in the prose." }, { "text": "Shows provenance tags on each edge", "passed": true, "evidence": "The `Provenance` column shows INFERRED for hop 1 and EXTRACTED for hops 2 and 3. The narrative also calls out that the `authentication -> user-model` edge carries `INFERRED` provenance. Raw JSON confirms: `\"provenance\": \"INFERRED\"` and `\"provenance\": \"EXTRACTED\"`." }, { "text": "Handles case where no path exists with a clear message", "passed": true, "evidence": "no-path-test.md documents two no-path cases (unknown target node and directed-unreachable source/target swap). Both return stdout `[]` and exit code `0`, with clear interpretation guidance that `[]` means no directed path. events.jsonl also records `{\"status\": \"no_path\", \"hops\": 0}` for the unknown-node case. No crash or stderr." } ], "notes": [] }, { "eval_id": 4, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 5, "failed": 0, "total": 5, "time_seconds": 163.8, "tokens": 65600, "tool_calls": null, "errors": 0 }, "expectations": [ { "text": "Detects Java/Spring Boot from pom.xml or build.gradle", "passed": true, "evidence": "wiki.config.yaml sets `language: java`, `build_tool: maven`, `framework: spring-boot`. detection-trace.md Phase 1 table row: `/tmp/eval-onboard-sb/pom.xml | yes | ... -> Java, Maven build tool`, and content scan cites `spring-boot-starter-parent` (line 12) -> framework: Spring Boot, concluding `Java 17 / Maven / Spring Boot 3.2`." }, { "text": "Detects JPA annotations (@Entity, @Table)", "passed": true, "evidence": "wiki.config.yaml shows `ecosystem.orm.profiles: [jpa]` and `ecosystem.agents.integration.wiki-orm-agent.profile: jpa`. detection-trace.md Phase 2 lists `JPA detection markers: @Entity, @Table, extends JpaRepository`, with table rows for User.java line 6 `@Entity`, line 7 `@Table(name = \"users\", schema = \"public\")`, Order.java line 5 `@Entity`, line 6 `@Table(name = \"orders\")`; concludes `ORM profile selected: jpa`." }, { "text": "Configures database agent for PostgreSQL", "passed": true, "evidence": "wiki.config.yaml line 17 `driver: postgresql` under `ecosystem.agents.integration.wiki-db-agent` (enabled: true), and line 32 `driver: postgresql` under `ecosystem.database` (enabled: true) with dev environment host=localhost, port=5432, database=mydb and secret-keyed creds WIKI_DB_DEV_USER/WIKI_DB_DEV_PASSWORD. Phase 3 of trace confirms via JDBC URL `jdbc:postgresql://localhost:5432/mydb`, `org.postgresql.Driver`, and `PostgreSQLDialect`." }, { "text": "Creates wiki.config.yaml with correct settings", "passed": true, "evidence": "wiki.config.yaml (3928 bytes) parses as valid YAML. Top-level keys include wiki, ecosystem, autonomy, sources, credentials, lint, claims, security, code_locality, refresh, graph, cache, logging, hooks, hooks_always_on, multimodal. Spot-checks: wiki.domain=spring-boot-backend, wiki.name=myapp, ecosystem.database.driver=postgresql (enabled), ecosystem.orm.profiles=[jpa] with cross_validate_against_db=true, ecosystem.agents.integration has wiki-db-agent and wiki-orm-agent enabled plus maintenance agents wiki-claude-md-agent and wiki-mermaid-agent enabled." }, { "text": "Creates wiki scaffold via /wiki-init", "passed": true, "evidence": "scaffold/ directory (note: eval metadata said wiki-scaffold/ but actual dir is scaffold/) contains all required subdirs: wiki/ (with index.md, overview.md, summaries.md, claims/, synthesis/, templates/), raw/, graph/ (with edges.jsonl), audit/ (open/, resolved/), log/ (events.jsonl, daily/), outputs/ (queries/, reports/). detection-trace.md Phase 6 documents `node init_wiki.js --path /tmp/eval-onboard-sb --domain spring-boot-backend --name myapp` as the scaffold command. events.jsonl logs the onboard op." } ], "notes": [] }, { "eval_id": 5, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 5, "failed": 0, "total": 5, "time_seconds": 162.1, "tokens": 60765, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Detects Python/Django from requirements.txt or pyproject.toml", "passed": true, "evidence": "wiki.config.yaml sets stack.language=python, stack.framework=django, stack.framework_version='>=4.2', build_files=[requirements.txt, manage.py]. detection-trace.md Phase 1 documents scanning requirements.txt (containing 'Django>=4.2', 'djongo', 'pymongo') plus manage.py stub verifying DJANGO_SETTINGS_MODULE." }, { "text": "Detects Django ORM (models.Model)", "passed": true, "evidence": "wiki.config.yaml ecosystem.orm.profiles=['django'] with detected_entities listing {class: Article, file: myapp/models.py, db_table: articles}. detection-trace.md Phase 2 cites 'class Article(models.Model):' matching the django.yaml entity_class regex and db_table='articles' override." }, { "text": "Configures database agent for MongoDB", "passed": true, "evidence": "wiki.config.yaml ecosystem.database: {enabled: true, driver: mongodb, connector: djongo, environments.default: {engine: djongo, name: mydb, host: mongodb://localhost:27017, source: myapp/settings.py}} with guard-rail policy (block_ddl, block_privilege, dml_mode: present_only) and audit enabled. detection-trace.md Phase 3 cites djongo ENGINE and mongodb:// URI scheme as the two independent indicators." }, { "text": "Creates wiki.config.yaml with correct settings", "passed": true, "evidence": "File exists at outputs/wiki.config.yaml (3843 bytes) and parses as valid YAML via js-yaml. All required top-level sections present: wiki, stack, ecosystem (agents/credentials/database/orm/claude_md/mermaid), autonomy, sources, lint, claims, security, code_locality, refresh, graph, cache, logging, hooks, hooks_always_on, multimodal. Stack and ecosystem values align with fixture (python/django/mongodb/djongo)." }, { "text": "Creates wiki scaffold via /wiki-init", "passed": true, "evidence": "outputs/wiki-scaffold/ contains the full directory set created by init_wiki.ts: wiki/ (with index.md, summaries.md, overview.md, claims/, synthesis/, templates/), graph/ (edges.jsonl), log/ (events.jsonl, daily/), audit/, raw/, outputs/, .wiki-cache/, plus .wiki-ignore and .claude/settings.json with the PreToolUse hooks installed." } ], "notes": [] }, { "eval_id": 6, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 4, "failed": 0, "total": 4, "time_seconds": 196.4, "tokens": 64850, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Uses schema introspection or SELECT from information_schema", "passed": true, "evidence": "command.txt runs `db_query.js --action schema --filter users` and then a bounded `SELECT name, type, \"notnull\", dflt_value, pk FROM pragma_table_info('users') LIMIT 100`. policy-decision.json records the driver path as `driver.getSchema (sqlite_master + PRAGMA table_info)`. `pragma_table_info` / `sqlite_master` are SQLite's equivalents of `information_schema`." }, { "text": "Result includes column names and data types", "passed": true, "evidence": "schema-result.md lists all four columns with types: `id` INTEGER (PK), `email` TEXT (NOT NULL, UNIQUE), `name` TEXT, `created_at` TIMESTAMP (default CURRENT_TIMESTAMP). stdout.txt mirrors this in JSON under `tables[0].columns`, and stdout_select.txt lists the same four rows via PRAGMA." }, { "text": "Does not execute any DML or DDL statements", "passed": true, "evidence": "Only two executed queries in command.txt: an `--action schema` introspection call and a single bounded `SELECT ... FROM pragma_table_info('users') LIMIT 100`. No INSERT/UPDATE/DELETE/DROP/CREATE/ALTER/TRUNCATE/GRANT/REVOKE appears in audit.jsonl, stdout*.txt, or command.txt (the only matches for those keywords are `auto-created index` in prose, the column name `created_at`, and the default `CURRENT_TIMESTAMP`). policy-decision.json explicitly states `ddl: never allowed`, `dml: present_only`, confirming neither was attempted." }, { "text": "Policy decision is ALLOW (dev env with auto approval)", "passed": true, "evidence": "policy-decision.json: `final_decision: \"allow\"`; both primary queries recorded with `decision: \"allow\"`, `env: \"dev\"`, `approval_mode: \"auto\"`, and reason `auto-approved` via `Policy._checkRead`. audit.jsonl contains a `policy_allow` event with `op: read, reason: auto-approved, approval_mode: auto`." } ], "notes": [] }, { "eval_id": 7, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 4, "failed": 0, "total": 4, "time_seconds": 128.2, "tokens": 64416, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Does NOT execute the INSERT statement", "passed": true, "evidence": "row-count-after.txt contains '0' — the INSERT was not executed against the users table. stdout.txt returned {\"status\": \"present_only\", \"reason\": \"DML statements are displayed but not executed\"} confirming no write occurred. policy-decision.md shows `executed`: **false**." }, { "text": "Returns formatted SQL for the user to run manually", "passed": true, "evidence": "presented-sql.md contains the SQL in a fenced code block: `INSERT INTO users (name, email) VALUES ('Test', 'test@example.com')` and additionally provides a `sqlite3 /tmp/eval-db-po/dev.sqlite \"INSERT INTO users ...\"` block under 'How to actually run this (if desired)'. stdout.txt also includes a `formatted_sql` field with the same statement." }, { "text": "Policy decision is PRESENT_ONLY", "passed": true, "evidence": "policy-decision.md header row: `| policy_gate | PRESENT_ONLY |` and `| decision | present_only |`. stdout.txt: `\"status\": \"present_only\"`. policy-decision.md cites the matching DML branch in `policy.ts` lines 226-246 returning `decision: \"present_only\"`." }, { "text": "Response clearly states the SQL was not executed", "passed": true, "evidence": "presented-sql.md title: '# Presented SQL (NOT EXECUTED)' and explicit text 'it was **not executed** against the `dev` environment'. 'Execution status' section states `executed: **no**`. policy-decision.md states `executed: **false**`." } ], "notes": [] }, { "eval_id": 8, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 4, "failed": 0, "total": 4, "time_seconds": 149.3, "tokens": 65352, "tool_calls": null, "errors": 0 }, "expectations": [ { "text": "Policy decision is DENY", "passed": true, "evidence": "policy-decision.md row: '| `policy_gate` | **`DENY`**' and '| `error_code` | `POLICY_DENY`'. stdout.txt contains: 'policy_result: { \"decision\": \"deny\", \"reason\": \"DDL statements are never allowed\" }'. deny-message.md Status: 'denied', error_code 'POLICY_DENY'. The decision is unambiguously DENY, confirmed via three independent output files and verified against the actual rule at agents/lib/wiki_db/policy.ts:211-216 which I read to confirm the code quote is accurate." }, { "text": "DDL operation is never executed", "passed": true, "evidence": "stdout.txt contains ONLY the two-line policy classifier output ('classification: ddl' + JSON deny result); no pg_execute trace, no query result rows, no row-count. stderr.txt is empty (0 bytes) — contains no 'ENOTFOUND nonexistent.invalid.example', no 'ECONNREFUSED', no pg driver error, and no 'Client has encountered a connection error'. The command.txt shows the executor deliberately did NOT invoke `node db_query.js --env production --sql 'DROP TABLE users'` because that CLI path would attempt a connect before policy.checkQuery; instead it invoked policy.checkQuery directly via an inline node --input-type=module import of policy.js. DROP was classified and denied, not dispatched to any driver." }, { "text": "Error message clearly explains DDL is blocked", "passed": true, "evidence": "deny-message.md contains the exact language 'DDL statements are never allowed' (both in the JSON wire format block and paraphrased in the refusal text as 'DDL statements are hard-denied on **every** environment'), quotes the AGENT.md rule verbatim ('NEVER execute DROP, CREATE, ALTER, TRUNCATE, GRANT, REVOKE — hard DENY'), identifies the SQL as a 'DDL operation (DROP keyword -> OperationType.DDL)', and states explicitly 'I have not opened a socket... and I will not do so for any DDL request.' The explanation covers what was blocked, why (classification), and the policy source. The exact phrase 'DDL statements are never allowed' matches policy.ts:213." }, { "text": "No connection to production database is even attempted", "passed": true, "evidence": "Multiple independent lines of evidence in connection-audit.md: (a) stderr.txt is empty — a DNS lookup of 'nonexistent.invalid.example' would surface ENOTFOUND/getaddrinfo errors on the pg driver's connect path; none appear. (b) stdout.txt shows only Policy/classifySqlKeywords symbols were imported from policy.js; connection.ts was never loaded, so registerDriverFactory never fired and getConnection(envName) was never invoked. (c) The audit cites the exact code path (db_query.ts:362 runWithEnv -> getConnection) that would have opened TCP, and demonstrates it was bypassed by invoking policy.checkQuery directly. (d) The audit acknowledges the ~0.1s verify-command wall time is inconsistent with a DNS timeout (macOS getaddrinfo ~5s for unresolvable hosts). CAVEAT: timing.json reports total_duration_seconds=149.3, but that is the entire executor session (reading code, writing 4 markdown files); the <1s claim applies to the inline verify subprocess and is not separately captured in a per-command timing artifact. Still, absence of any DNS/connect error on stderr is strong positive evidence, and the code-path audit is the primary proof — the path that would connect was not invoked." } ], "notes": [ "Executor chose NOT to invoke the db_query.js CLI to avoid a connect attempt. Instead verified policy.checkQuery directly via an inline node import. This is the correct interpretation of the agent contract (policy-before-connect), but means the eval does not exercise the CLI path end-to-end — it exercises the shared policy module the CLI also calls." ] }, { "eval_id": 9, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 5, "failed": 0, "total": 5, "time_seconds": 115.7, "tokens": 42287, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Correctly identifies JPA @Entity classes", "passed": true, "evidence": "detected-entities.json lists two entities with class_name 'Order' and 'User' (matching the @Entity-annotated classes in fixtures/jpa/Order.java and User.java). database-mapping.md references both in the Entity-Table Mapping table (lines 16-17: '| Order | orders | 1 | many_to_one |' and '| User | public.users | 2 | one_to_many, many_to_many |')." }, { "text": "Maps @Table(name) to database table names", "passed": true, "evidence": "detected-entities.json shows Order->table_name 'orders' and User->table_name 'users' with schema 'public'. database-mapping.md line 16 renders 'Order | orders' and line 17 renders 'User | public.users', matching @Table(name='orders') and @Table(name='users', schema='public')." }, { "text": "Detects @OneToMany and @ManyToOne relationships", "passed": true, "evidence": "detected-entities.json lists relationships 'many_to_one' for Order and 'one_to_many, many_to_many' for User. database-mapping.md line 16 shows 'many_to_one' for Order and line 17 shows 'one_to_many, many_to_many' for User. These correspond to @ManyToOne in Order.java line 16 and @OneToMany in User.java line 20." }, { "text": "Output includes a valid Mermaid erDiagram block", "passed": true, "evidence": "database-mapping.md lines 21-32 contain a fenced '```mermaid' block opening with 'erDiagram' and defining two entities (orders, users) with columns plus two relationship edges. mermaid-lint.txt contains '[]' (no syntax errors) and command.txt confirms mermaid_lint.js returned exit code 0." }, { "text": "Generated markdown has proper frontmatter with type: entity", "passed": true, "evidence": "database-mapping.md begins with '---' on line 1 and includes 'type: entity' on line 3, plus other frontmatter fields (title, tags, generated_by: orm-mapper, orm_profile: jpa, created, updated, summary) closed by '---' on line 10." } ], "notes": [ "Mermaid diagram uses placeholder target node name 'users_rel' instead of the real related entities (orders for OneToMany, user_roles/roles for ManyToMany). The Order->User ManyToOne edge is missing entirely from the diagram." ] } ], "run_summary": { "with_skill": { "pass_rate": { "mean": 0.9929, "stddev": 0.0267, "min": 0.9, "max": 1.0 }, "time_seconds": { "mean": 171.2714, "stddev": 69.2036, "min": 108.1, "max": 388.4 }, "tokens": { "mean": 57469.5714, "stddev": 9188.2996, "min": 42287, "max": 69621 } }, "delta": { "pass_rate": "+0.99", "time_seconds": "+171.3", "tokens": "+57470" } }, "notes": [ "Overall: 13/14 evals at 100% pass (67/68 assertions = 98.5%). Single failure (eval-1 #10) is an eval-vs-skill mismatch — /wiki-init is not expected to log an op:init event; the assertion should be relaxed or the skill changed.", "Token variance (42k-70k) correlates with eval scope: multi-step pipelines (init+ingest+query) > single-op evals (wiki-path, mermaid-gen).", "Graders flagged ~10 eval-quality gaps: most assertions test presence (file exists, field is named X) not substance (content is correct, numbers match input). Tightening these would create more discriminating evals for future iterations.", "Semantic defect found by JPA ORM grader: Mermaid erDiagram passes mermaid_lint but contains a phantom node `users_rel` and omits the Order<->User edge. Lint-passing is necessary but not sufficient.", "Contract deviation found by mermaid-ER grader: agent appends ## Diagrams at EOF rather than injecting between markers that fixtures set up. Either the agent contract or the assertion should reconcile.", "All three db-agent policy gates (ALLOW/PRESENT_ONLY/DENY) verified under SQLite. DENY path was verified by code-path reasoning (policy classified DDL before driver load) rather than end-to-end CLI trace — a stronger assertion would instrument the connect call and assert it was never reached.", "No baseline configuration (without_skill) was run this iteration per user decision. The delta column reports 99% only because the zero-row baseline is a default placeholder, not a real comparison. Future iterations should pair with-skill runs against vanilla-Claude runs to produce real delta data.", "18 agent evals across 6 live-API agents (github, jira, confluence, gcp, aws, notion) were not attempted — see skipped-evals.md for credentials needed." ] }