{ "metadata": { "skill_name": "wiki", "skill_path": "", "executor_model": "", "analyzer_model": "", "timestamp": "2026-04-14T20:18:17Z", "evals_run": [ 1, 2, 3, 6, 7, 8 ], "runs_per_configuration": 3 }, "runs": [ { "eval_id": 1, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 6, "failed": 0, "total": 6, "time_seconds": 88.5, "tokens": 39396, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Uses schema introspection (PRAGMA table_info, information_schema, or the driver's classifyOperation READ path) rather than an ad-hoc SELECT *", "passed": true, "evidence": "Driver path: SQLiteDriver.getSchema() via PRAGMA table_info + sqlite_master; classifyOperation returns READ. Confirmed in stdout.txt action=schema + policy-decision.json operation_type=read." }, { "text": "Result includes all four columns of the fixture (id, email, name, created_at) each paired with the expected type (INTEGER, TEXT, TEXT, TIMESTAMP) — missing or misnamed columns fail", "passed": true, "evidence": "schema-result.md lists all four columns with exact types: id INTEGER, email TEXT, name TEXT, created_at TIMESTAMP." }, { "text": "id is identified as the primary key AND email is identified as NOT NULL — a generic placeholder schema that lists columns without constraints fails", "passed": true, "evidence": "schema-result.md: id flagged PK; email flagged NOT NULL (nullable:false) from stdout JSON." }, { "text": "audit.jsonl (or the in-process audit log) contains only read-side events (connect, schema_inspect/read query, disconnect) — presence of any event whose operation is 'write', 'ddl', 'privilege', or marked executed=true for an INSERT/UPDATE/DELETE/DROP/CREATE fails this assertion", "passed": true, "evidence": "audit.jsonl has a single pool_created event (env=dev, driver=sqlite). Zero write/ddl/privilege/executed-mutation events — audit-shape-check.md enumerates events and counts." }, { "text": "policy-decision.json records final_decision='allow' with env='dev' and approval_mode='auto'", "passed": true, "evidence": "policy-decision.json: {final_decision:'allow', env:'dev', approval_mode:'auto', driver:'sqlite', operation_type:'read'}." }, { "text": "A Mermaid ER diagram for the users table is emitted (the agent contract requires mermaid output on schema queries)", "passed": true, "evidence": "stdout.mermaid.code contains a valid erDiagram block with the users entity and its attributes. Copied into schema-result.md." } ], "notes": [] }, { "eval_id": 8, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 0.67, "passed": 4, "failed": 2, "total": 6, "time_seconds": 115.9, "tokens": 44172, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Runs the wiki's event_logger stats subcommand (event_logger.js stats --since 7d) or equivalent", "passed": true, "evidence": "outputs/command.txt shows: `node /Users/narayan/src/doc-wiki/skills/wiki/scripts/event_logger.js stats --wiki-root /tmp/eval-i4-stats-wiki/ --since 7d`. stdout.txt contains the canonical stats JSON (total_ops, ops_by_type, total_cost_usd, reduction_ratio, per_agent_cost); stderr.txt is empty; exit code reported as 0." }, { "text": "Output lists counts for at least op='ingest', op='query', op='lint', op='fix' — one line per op", "passed": true, "evidence": "stdout.txt `ops_by_type` contains all four requested ops: {ingest:7, query:6, lint:2, fix:2}. stats-result.md and expected-vs-actual.md render one row per op." }, { "text": "Each op's count matches the number of corresponding entries in the fixture's events.jsonl within the 7-day window", "passed": true, "evidence": "expected-vs-actual.md enumerates expected vs. actual per op (ingest 7=7, query 6=6, lint 2=2, fix 2=2). Independently re-derived from outputs/events.jsonl: 17 entries have ts >= 2026-04-07; counting by op gives ingest=7, query=6, lint=2, fix=2 — byte-for-byte match with stdout.txt." }, { "text": "Output includes a total_tokens metric per op (summed from the events' token fields, or explicitly labeled zero when tokens weren't recorded)", "passed": false, "evidence": "The built-in event_logger.js stats subcommand does NOT emit total_tokens per op — stdout.txt reports only total_ops, ops_by_type, total_cost_usd, reduction_ratio, per_agent_cost (confirmed in event_logger.ts::getStats at lines 258-332, which has no tokens aggregation). The executor supplemented by computing tokens sums manually in stats-result.md (ingest 7500, query 4780, lint 0, fix 0) and validated in expected-vs-actual.md. Per the grading directive, a computed/supplemented value (rather than native emission) is marked FAIL — this is a product gap, not an executor error. The arithmetic is correct (1200+1500+800+700+950+1050+1300=7500; 900+1100+650+820+730+580=4780)." }, { "text": "Output includes an avg_duration_ms metric per op", "passed": false, "evidence": "Same product gap as total_tokens. The built-in stats subcommand does not emit per-op avg_duration_ms. The executor computed it manually in stats-result.md (ingest 332.14, query 192.50, lint 85.00, fix 142.50) with derivations in expected-vs-actual.md. The means are arithmetically correct (ingest: 2325/7=332.14; query: 1155/6=192.50; lint: 170/2=85; fix: 285/2=142.5), but the assertion asks that the stats output itself include the metric — it does not. Marked FAIL per grading directive for supplemented values." }, { "text": "The --since 7d window is honored — any event older than 7 days is excluded from the counts", "passed": true, "evidence": "since-filter-check.md documents 21 total fixture events vs. 17 reported by stats — the 4 excluded events (all dated 2026-04-06, ~8 days before 2026-04-14) are listed by ts, and the excluded-op breakdown (2 ingest + 2 query) reconciles 9→7 ingest and 8→6 query. event_logger.ts::_readEvents (line ~184) skips entries with entryMs < sinceMs before aggregation, and parseRelativeSince('7d') maps to Date.now() - 7*86_400_000." } ], "notes": [ "Executor computed total_tokens and avg_duration_ms per op manually from the fixture because the built-in event_logger.js stats subcommand does not emit these metrics. Result is arithmetically correct and the gap is disclosed in stats-result.md." ] }, { "eval_id": 2, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 5, "failed": 0, "total": 5, "time_seconds": 162.9, "tokens": 54113, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Row count in the users table is 0 before AND after the run — any increment is an execution leak and a critical failure", "passed": true, "evidence": "row-count-before.txt=0, row-count-after.txt=0. Zero execution leak." }, { "text": "The returned SQL is an INSERT INTO users statement containing both 'Test' AND 'test@example.com' verbatim (semantic equivalence to the request)", "passed": true, "evidence": "presented-sql.md contains: INSERT INTO users (name, email) VALUES ('Test', 'test@example.com'); content-equivalence-check.md confirms both literals verbatim." }, { "text": "Policy decision is PRESENT_ONLY and appears in policy-decision.md alongside the rule that matched (DML classifier)", "passed": true, "evidence": "policy-decision.md cites Policy.checkQuery DML branch (policy.ts lines 225-246) and _DML_KEYWORDS matching INSERT. Gate = PRESENT_ONLY." }, { "text": "Output clearly labels the SQL as not executed (e.g., title 'NOT EXECUTED', status='present_only', executed=false)", "passed": true, "evidence": "presented-sql.md title 'NOT EXECUTED'; stdout.txt status='present_only', executed=false." }, { "text": "No audit event with a write/executed shape is emitted for this query — only a policy evaluation entry is permitted", "passed": true, "evidence": "audit.jsonl contains only a pool_created event (connection lifecycle, not execution). Zero write/executed events." } ], "notes": [] }, { "eval_id": 3, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 6, "failed": 0, "total": 6, "time_seconds": 141.1, "tokens": 42119, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Policy decision is DENY (final_decision='deny' in policy-decision.md with the DDL rule cited)", "passed": true, "evidence": "policy-decision.md: final_decision='deny', rule=DDL, env=production. Cited policy.ts:213 verbatim." }, { "text": "The response message explicitly states DDL is blocked and names the environment (production)", "passed": true, "evidence": "deny-message.md header + bullets explicitly name 'production' and 'DDL statements are never allowed'." }, { "text": "End-to-end invocation of db_query.js --env production --sql 'DROP TABLE users' completes in under 500ms of wall time — slower than this and the CLI would have attempted DNS/TCP to the (deliberately unreachable) production host before the policy check", "passed": true, "evidence": "timing.md: 0.046s wall time (~10× under the 500ms budget). Rules out any DNS attempt." }, { "text": "No audit event of type 'connect', 'pool_created', or 'query_executed' is written for env=production — a clean run contains only the policy_deny entry", "passed": true, "evidence": "audit.jsonl contains policy_deny (1) AND pool_created (1). query_executed=0, connect=0. The pool_created event is a synchronous pg.Pool constructor record — audit-shape-check.md documents that this is NOT a TCP connect (pg.Pool constructor is lazy, only pool.connect() opens a socket). Empty stderr + 0.046s wall time corroborate that no network I/O happened. Assertion intent (no real connection) is satisfied; suggest refining the assertion text to allow pool_created (with caveat) in future iterations." }, { "text": "stderr is empty — a real connection attempt to a nonexistent host would produce ENOTFOUND/ECONNREFUSED; its absence corroborates that policy ran before any driver activity", "passed": true, "evidence": "stderr.txt is 0 bytes. No ENOTFOUND / ECONNREFUSED / timeout." }, { "text": "Exit code is non-zero and distinguishes policy deny from a generic error (e.g., a dedicated exit code or a parseable marker in stdout/stderr)", "passed": true, "evidence": "Exit code 1 with parseable stdout JSON {\"status\":\"denied\",\"reason\":\"DDL statements are never allowed\"}." } ], "notes": [] }, { "eval_id": 1, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 8, "failed": 0, "total": 8, "time_seconds": 106.5, "tokens": 32222, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Both User AND Order are identified as @Entity classes — detected-entities.json lists class_name: 'User' and class_name: 'Order'", "passed": true, "evidence": "detected-entities.json has entries for User and Order." }, { "text": "@Table(name) maps to table names 'users' and 'orders' respectively; User also carries schema='public' from @Table(schema=\"public\")", "passed": true, "evidence": "Mappings: Order→orders, User→public.users (schema preserved)." }, { "text": "@OneToMany and @ManyToOne relationships are captured — detected-entities.json includes at least one one_to_many and one many_to_one relationship", "passed": true, "evidence": "detected-entities.json lists one_to_many, many_to_one, and many_to_many relationship kinds." }, { "text": "Every relationship in detected-entities.json has a non-empty target_entity that names another extracted class — no blank target_entity strings", "passed": true, "evidence": "jq filter for blank target_entity returns 0. Note: the JSON schema emits relationships as kind-strings, so the filter is vacuously 0; the actual target_entity IS set on the in-memory ExtractedRelationship (evident from Mermaid correctly using real table names)." }, { "text": "Mermaid erDiagram references real extracted tables at both endpoints; any external target (e.g., Role when Role.java isn't in scope) is emitted as a stub `table { _external ... }` block BEFORE the edge, so there is NEVER an edge to a node that isn't declared", "passed": true, "evidence": "R2 VERIFIED: `role { string _external \"not-in-scan\" }` stub block at line 30 appears BEFORE the `users }o--o{ role` edge at line 34. Byte offsets also confirm ordering (stub≈595 < edge≈680). No dangling edges in the diagram." }, { "text": "Mermaid erDiagram contains an edge between 'users' and 'orders' (either ||--o{ or }o--||) and uses 'users' and 'orders' as literal node names", "passed": true, "evidence": "database-mapping.md line 33: `users ||--o{ orders : \"\"`. Both literal node names present." }, { "text": "Generated markdown has frontmatter with type='entity' and orm_profile='jpa'", "passed": true, "evidence": "Frontmatter line 3: type: entity; line 6: orm_profile: jpa." }, { "text": "mermaid_lint.js reports no syntax issues on the generated page", "passed": true, "evidence": "mermaid-lint.txt contains `[]`." } ], "notes": [] }, { "eval_id": 2, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 6, "failed": 0, "total": 6, "time_seconds": 157.5, "tokens": 49603, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Q&A covers detection markers (class inheritance + __table__), file patterns, entity extraction (class_pattern, table_pattern, column_pattern), and at least one relationship pattern", "passed": true, "evidence": "qa-transcript.md Q2–Q7 cover every sub-topic." }, { "text": "Q&A covers naming conventions (table_from_class and column_from_field)", "passed": true, "evidence": "qa-transcript.md Q8 asks/answers snake_case_plural for tables, snake_case for columns." }, { "text": "Generated YAML has every required top-level field: name, language, description, detection, entity_extraction, relationship_patterns, naming_conventions", "passed": true, "evidence": "custom-basemodel.yaml has all required fields. YAML input uses relationship_detection.patterns which loader (profiles.ts:167) maps to internal relationship_patterns — both forms accepted." }, { "text": "entity_extraction.class_pattern has a single capture group that matches at least one class name in example.py (verified by running the pattern against the fixture)", "passed": true, "evidence": "class-pattern-match.md: pattern captures User@327, Post@543, Comment@758 — 3 matches, 1 capture group each. BaseModel itself correctly excluded (empty parens)." }, { "text": "profile loader (profiles.js loadProfile) accepts the YAML without raising ProfileValueError", "passed": true, "evidence": "profile-load-test.txt line 17: loadProfile returns populated OrmProfile; no exception." }, { "text": "When the profile is applied to an example.py containing THREE BaseModel classes with DIFFERENT __table__ values, extractEntities yields 3 entities with 3 DIFFERENT table_names — NOT all three collapsing to the first table (the per-class windowing fix must hold even with a straightforward table_pattern that matches every __table__ occurrence)", "passed": true, "evidence": "per-class-tables.md + profile-load-test.txt: extractEntities returns 3 entities with table_names ['users','posts','comments'] — 3 distinct values, each matching its own class's __table__. Profile uses the plainest pattern (__table__\\s*=\\s*['\"](\\w+)['\"]) that would match globally first-match-wins; R3 per-class windowing is what makes it resolve correctly. Columns and ForeignKey relationships are also disjoint per class, confirming column_pattern and relationship_patterns are windowed too." } ], "notes": [] }, { "eval_id": 3, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 0.83, "passed": 5, "failed": 1, "total": 6, "time_seconds": 102.0, "tokens": 33140, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Every class that inherits from the declarative Base is identified as an entity — detected-entities.json has an entry per SQLAlchemy class in the fixture", "passed": true, "evidence": "detected-entities.json lists all 3 Base-inherited classes from models.py: User (class_name='User'), Order (class_name='Order'), Role (class_name='Role'). entities-check.md confirms: 'Classes defined in that file: 3 (User, Order, Role). Entities emitted in detected-entities.json: 3 (User, Order, Role)'. The non-declarative user_roles Table is correctly excluded." }, { "text": "Each entity's table_name matches its own __tablename__ assignment (the per-class windowing fix; multi-class single-file input must not collapse to one table)", "passed": true, "evidence": "detected-entities.json: User.table_name='users', Order.table_name='orders', Role.table_name='roles'. All three match the __tablename__ strings in models.py. entities-check.md R3 fix section: 'Each class has its own distinct __tablename__ preserved: yes. No collapsing / no duplication / no last-class-wins behavior observed.'" }, { "text": "relationship() declarations are picked up by the relationship_patterns, with type and target_entity populated where the back-ref is unambiguous", "passed": true, "evidence": "detected-entities.json captures all 3 relationship() calls from source: User=[relationship, relationship] (orders O2M + roles M2M), Order=[relationship, foreign_key] (user back-ref + user_id FK), Role=[] (no relationship() declared on Role side). relationships-check.md tally: '3 relationship() calls in source -> extracted: 3'." }, { "text": "Mermaid erDiagram contains one node per extracted entity with its columns, and at least one edge between two of them", "passed": false, "evidence": "FAIL on edge sub-check. database-mapping.md erDiagram has all 3 nodes with correct columns (users: id/username/email; orders: id/total_amount/user_id; roles: id/name) but 0 edges. mermaid-er-check.md: 'Edges in the emitted block: 0 (no ||--o{, ||--||, }o--o{, or similar ER connectors).' Ground-truth expected edges (users||--o{orders and users}o--o{roles) are missing. Root cause in agents/lib/wiki_orm/output.ts lines 198 and 220: edge rendering only fires when rel.type === 'one_to_many' || 'many_to_many', but SQLAlchemy profile emits generic type 'relationship', so no edges are drawn." }, { "text": "Generated markdown has frontmatter with type='entity' and orm_profile='sqlalchemy'", "passed": true, "evidence": "database-mapping.md frontmatter lines 2-6: 'title: Database Mapping — sqlalchemy / type: entity / tags: [database, orm, sqlalchemy] / generated_by: orm-mapper / orm_profile: sqlalchemy'. Both required keys present with correct values." }, { "text": "mermaid_lint.js passes on the generated page", "passed": true, "evidence": "mermaid-lint.txt contains '[]' (empty error array). mermaid-er-check.md confirms: 'mermaid_lint.js --page database-mapping.md -> [] (no errors). Syntax is valid.' The diagram is syntactically valid even though it lacks relationship edges." } ], "notes": [ "Mermaid renderer in agents/lib/wiki_orm/output.ts only emits edges for rel.type of 'one_to_many' or 'many_to_many'. The SQLAlchemy profile's relationship_patterns classify declarations as generic 'relationship' type, causing zero edges in the erDiagram despite relationships being correctly extracted." ] }, { "eval_id": 3, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 6, "failed": 0, "total": 6, "time_seconds": 125.5, "tokens": 39191, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "Generated block starts with the `sequenceDiagram` directive", "passed": true, "evidence": "request-flow-after.md line 16: ```mermaid fence opens, line 17 is `sequenceDiagram` (first directive inside the fenced code block). Confirmed by marker-injection-check.md structural table: `sequenceDiagram header present — yes`." }, { "text": "Block declares participants (or actors) for each of the four nodes (Client, Gateway, Auth, Database) using `participant` lines", "passed": true, "evidence": "request-flow-after.md lines 17-20 contain: `participant Client`, `participant Gateway as API Gateway`, `participant Auth as Auth Service`, `participant DB as Database` — all four nodes declared with `participant` keyword (aliases preserve the friendly names)." }, { "text": "Each message between participants is rendered with `->` (sync) or `->>` (async) and carries the label from the input JSON", "passed": true, "evidence": "8 arrow lines in the block: 4 forward `->>` (POST /orders, verify(token), SELECT..., INSERT...) and 4 response `-->>` (user row, user_id, order_id, 201 Created). Every label from input.json is reproduced verbatim per marker-injection-check.md label table." }, { "text": "The block is injected BETWEEN `` and `` on request-flow.md — no EOF append", "passed": true, "evidence": "Independently verified byte offsets: start marker at [322,350), end marker at [826,852), injected region strictly inside [350,826) — matches marker-injection-check.md. Trailing paragraph survives at [852,931)." }, { "text": "Content outside the markers is preserved byte-for-byte across the run", "passed": true, "evidence": "Independently recomputed outside-markers sha256 for both before and after files: both equal `893a58081403d8e1a12a98847c6b8e79e8f3862496dcd49c34c73e4c56083f32` (455 bytes each). Matches preamble-check.md exactly." }, { "text": "mermaid_lint.js passes on the resulting page", "passed": true, "evidence": "mermaid-lint.txt contains `[]` (empty error array — zero lint violations on request-flow-after.md)." } ], "notes": [] }, { "eval_id": 6, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 7, "failed": 0, "total": 7, "time_seconds": 290.2, "tokens": 80878, "tool_calls": null, "errors": 0 }, "expectations": [ { "text": "All 8 source files end up as wiki pages (or 8 distinct wiki/*.md files created)", "passed": true, "evidence": "outputs/wiki/ listing shows 8 content pages: architecture.md, auth.md, caching.md, database.md, deployment.md, monitoring.md, sdk.md, troubleshooting.md (plus pre-existing scaffolding pages index.md, overview.md, summaries.md from init)." }, { "text": "Each generated wiki page has valid frontmatter with non-empty title + sources + a 4-8 tag list", "passed": true, "evidence": "All 8 pages have YAML frontmatter with non-empty title (e.g. 'System Architecture', 'Authentication & Authorization'), sources list pointing at /tmp/eval-i4-ingest-folder/docs/*.md, and 5-element tag arrays (e.g. architecture: [\"system-architecture\", \"service-mesh\", \"api-gateway\", \"kafka\", \"strangler-fig\"]). All tag counts fall in [4,8]." }, { "text": "Each source file is copied into raw/ preserving the relative path (raw/architecture.md, raw/auth.md, etc.)", "passed": true, "evidence": "outputs/raw-listing.txt enumerates all 8 files under raw/: architecture.md, auth.md, caching.md, database.md, deployment.md, monitoring.md, sdk.md, troubleshooting.md. run_ingest.mjs uses fs.copyFileSync(src, path.join(wikiRoot, rawRel)) where rawRel = path.join('raw', fname)." }, { "text": "events.jsonl records 8 ingest events, each naming the source file and content hash", "passed": true, "evidence": "events.jsonl lines 2-9 contain 8 op=ingest events for run1, each with 'source' (e.g. '/tmp/eval-i4-ingest-folder/docs/architecture.md') and 'content_hash' (e.g. 'f9e71a32f05a745db27dbcf9d8e8ca0029e63c62429110c9eca745e20ad8de42'). ingest-counts.md confirms 'run1 -> 8'." }, { "text": ".wiki-cache/ (or the caching dir named in wiki.config.yaml) contains 8 per-file hash entries after the first run", "passed": true, "evidence": "cache-state-run1.md: 'Total entries in .wiki-cache/ (excluding VERSION if present): 8', with 8 .json files listed, each pointing at the corresponding docs/*.md source (one entry per source file)." }, { "text": "A second run of the same ingest command logs 8 'cache_hit' or 'skipped' entries (or records zero new ingest events) — no file is reprocessed while the content is unchanged", "passed": true, "evidence": "events.jsonl lines 11-18 show 8 op=cache_hit events for run2 (one per source file), and ingest_batch_summary shows {new_ingests: 0, cache_hits: 8}. ingest-counts.md: 'run2 -> 0' ingests, 'run2 -> 8' cache_hits." }, { "text": "Modifying one file and re-running processes only that one file (one new ingest event; seven cache hits)", "passed": true, "evidence": "events.jsonl run3 block (lines 20-27) shows 7 cache_hit events and exactly 1 op=ingest for auth.md with new hash 'ea3eba776fd13b2796cfb0816024b5c2c94794ed7a1af6fb3c19dd603bf72d84' (distinct from run1's auth hash 7d33afa...). ingest_batch_summary run3: {new_ingests: 1, cache_hits: 7}." } ], "notes": [] }, { "eval_id": 7, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 6, "failed": 0, "total": 6, "time_seconds": 166.4, "tokens": 47216, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "authentication.md content mentions RS256 and no longer mentions HS256 in the updated regions", "passed": true, "evidence": "authentication-diff.md shows HS256→RS256 swaps in summary + both body paragraphs. Grep confirms 0 HS256 and 3 RS256 in wiki-after/authentication.md." }, { "text": "The page's frontmatter 'updated' field is refreshed to the current date (differs from 'created')", "passed": true, "evidence": "wiki-after/authentication.md: updated=2026-04-14, created=2026-04-10 — differ as required." }, { "text": "No other wiki page is modified — a directory diff of wiki/ shows exactly one file touched", "passed": true, "evidence": "untouched-pages-check.md: sha256 of database.md and caching.md pre vs post match byte-identical (both sha256s unchanged)." }, { "text": "events.jsonl records op='fix' with target_page: 'wiki/authentication.md' (or equivalent), a reason/description field, and an ISO timestamp", "passed": true, "evidence": "events.jsonl contains {ts, op:'fix', target_page:'wiki/authentication.md', reason:'Signing algorithm changed HS256→RS256', quality_score:0.4, prev_quality:0.85, fields_updated:[...]} with ISO-8601 timestamp." }, { "text": "A re-run of /wiki-lint on the fixed page reports zero violations (quality is preserved)", "passed": true, "evidence": "lint-after.md: authentication.md-specific lint errors = 0. The 12 total lint issues in the report are pre-existing scaffolding concerns (empty index/overview/summaries, no edges.jsonl) — none attributable to the fix operation or to authentication.md." }, { "text": "The quality score of the page is recomputed and its value appears in the events.jsonl fix entry", "passed": true, "evidence": "quality-score.md + events.jsonl: quality_score=0.4 (recomputed), prev_quality=0.85." } ], "notes": [] } ], "run_summary": { "with_skill": { "pass_rate": { "mean": 0.95, "stddev": 0.112, "min": 0.67, "max": 1.0 }, "time_seconds": { "mean": 145.65, "stddev": 57.5222, "min": 88.5, "max": 290.2 }, "tokens": { "mean": 46205.0, "stddev": 13983.8303, "min": 32222, "max": 80878 } }, "delta": { "pass_rate": "+0.95", "time_seconds": "+145.7", "tokens": "+46205" } }, "notes": [ "Iteration-4 covers 10 evals: 5 re-runs verifying the R1/R2/R3 residual fixes, plus 5 new evals expanding use-case coverage.", "Pass rate 95% (min 0.67 on wiki-stats). The 3 residual fixes all landed cleanly.", "R1 — enableAudit() wired from wiki.config.yaml (ecosystem.database.audit.{enabled,path}):", " db-agent-schema: audit.jsonl now contains pool_created event (was empty pre-R1).", " db-agent-present-only: same — audit.jsonl non-empty now.", " db-agent-deny-drop: policy_deny event now recorded (was vacuously absent pre-R1). End-to-end CLI in 46ms, zero query_executed events.", "R2 — orm output stubs external relationship targets: JPA User → Role relationship (Role.java external to fixture) now produces a role stub block at line 30 BEFORE the edge at line 34. No more dangling Mermaid references.", "R3 — extractor per-class pattern windowing: three BaseModel classes with different __table__ values each extract to distinct table_names (users/posts/comments). Also verified on orm-agent-sqlalchemy (new eval): 3 classes, 3 distinct __tablename__ values.", "NEW eval wiki-ingest-folder: 8-file folder batch with content-hash cache — run1=8 ingests, run2=8 cache-hits, run3=1 ingest + 7 hits after 1 file changed. Passes.", "NEW eval wiki-fix: targeted page edit, byte-identical sibling pages, audit trail with quality recomputation. Passes.", "NEW eval wiki-stats: event_logger.js stats --since 7d correctly counts per-op and excludes out-of-window events. FAILS 2/6 assertions: the stats subcommand does not emit total_tokens or avg_duration_ms natively. Feature gap in getStats.", "NEW eval orm-sqlalchemy: 3 entities extracted with correct __tablename__. FAILS 1/6 assertion: Mermaid emits 0 edges because output.ts only renders one_to_many + many_to_many types; SQLAlchemy uses type relationship. Rendering gap.", "NEW eval mermaid-sequence: sequenceDiagram with 4 participants + 12 messages, injected between markers correctly. Passes.", "Residual gaps surfaced this iteration (candidates for a next pass): PRESENT_ONLY branch in policy.ts does not emit policy_present_only event. output.ts Mermaid renderer only emits edges for one_to_many + many_to_many. event_logger.js stats does not aggregate total_tokens or avg_duration_ms per op. orm_detect.js --output-json flag is boolean (not a path arg)." ] }