{ "metadata": { "skill_name": "wiki", "skill_path": "", "executor_model": "", "analyzer_model": "", "timestamp": "2026-04-15T02:14:45Z", "evals_run": [ 4, 5, 12 ], "runs_per_configuration": 3 }, "runs": [ { "eval_id": 4, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 7, "failed": 0, "total": 7, "time_seconds": 104.462, "tokens": 34803, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "All three models (User, Post, Comment) are extracted into detected-entities.json as separate entries", "passed": true, "evidence": "detected-entities.json: 3 entities (User, Post, Comment). Unchanged from iter-7." }, { "text": "Each entity's table_name matches the @@map(...) value in the fixture (e.g., User \u2192 'users', Post \u2192 'posts', Comment \u2192 'comments')", "passed": true, "evidence": "Entity table_names match @@map values: User\u2192users, Post\u2192posts, Comment\u2192comments. Unchanged from iter-7." }, { "text": "Every @relation(fields: [...], references: [...]) field produces a relationship in detected-entities.json with a non-empty target_entity that names another extracted model \u2014 no blank target_entity strings", "passed": true, "evidence": "P1 VERIFIED. All @relation relationships now have populated target_entity: Post.author\u2192User, Comment.post\u2192Post. FLIPPED from iter-7 FAIL (all empty strings). Root: new prisma.yaml patterns with capture group `([A-Z]\\w*)` + extractor preference for relMatch[1] when uppercase." }, { "text": "One-to-many sides (e.g., posts Post[]) produce relationships of type one_to_many; their corresponding many-to-one inverse (e.g., author User @relation(...)) produces type many_to_one, both with the correct target_entity", "passed": true, "evidence": "FLIPPED from iter-7 FAIL. Types now correct: User.posts\u2192one_to_many(Post), Post.comments\u2192one_to_many(Comment), Post.author\u2192many_to_one(User), Comment.post\u2192many_to_one(Post). Previous iter-7 got 'relationship' (generic) on the @relation side; now gets many_to_one because the many_to_one pattern is ordered first." }, { "text": "Generated database-mapping.md frontmatter has type='entity' and orm_profile='prisma'", "passed": true, "evidence": "frontmatter-check.md: type=entity, orm_profile=prisma. Unchanged from iter-7." }, { "text": "Mermaid erDiagram contains nodes named 'users', 'posts', 'comments' (the table names from @@map, not the model names) and at least one edge of cardinality ||--o{ linking users-posts AND another linking posts-comments", "passed": true, "evidence": "FLIPPED from iter-7 FAIL. Mermaid now: `users ||--o{ posts` and `posts ||--o{ comments`. Real table names, no synthetic _rel stubs. Downstream of A3 fix: once target_entity is populated, output.ts resolves to real table_name." }, { "text": "mermaid_lint.js reports no issues on the generated page", "passed": true, "evidence": "mermaid-lint.txt = [] (exit 0). Unchanged from iter-7." } ], "notes": [] }, { "eval_id": 5, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 7, "failed": 0, "total": 7, "time_seconds": 87.092, "tokens": 25624, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "All three entities (Author, Book, Publisher) are extracted into detected-entities.json as separate entries with table_names matching the @Entity('...') first arg (e.g., 'authors', 'books', 'publishers')", "passed": true, "evidence": "detected-entities.json: 3 entities, Author\u2192authors, Book\u2192books, Publisher\u2192publishers. Unchanged from iter-7 (already passing)." }, { "text": "Each @OneToMany(() => Book, ...) and @ManyToOne(() => Author, ...) produces a relationship with target_entity resolved from the arrow-function call argument \u2014 never blank, never the literal string 'Book'/'Author' (the resolver must dereference to the imported class)", "passed": true, "evidence": "T3 VERIFIED. target_entity now resolved: Author.@OneToMany\u2192Book, Book.@ManyToOne\u2192Author, Book.@OneToOne\u2192Publisher. FLIPPED from iter-7 FAIL. Root fix: new regex /^\\s*\\([^)]*\\)\\s*=>\\s*([A-Z][A-Za-z_]\\w*)\\s*[,)]/ ran BEFORE existing call-first-arg branch." }, { "text": "Relationship type matches the decorator name (one_to_many for @OneToMany, many_to_one for @ManyToOne, one_to_one for @OneToOne)", "passed": true, "evidence": "types unchanged from iter-7: Author.@OneToMany\u2192one_to_many, Book.@ManyToOne\u2192many_to_one, Book.@OneToOne\u2192one_to_one." }, { "text": "Generated database-mapping.md frontmatter has type='entity' and orm_profile='typeorm'", "passed": true, "evidence": "frontmatter-check.md: type=entity, orm_profile=typeorm. Unchanged from iter-7." }, { "text": "Mermaid erDiagram contains nodes for all three table names and at least two edges (authors-books and books-publishers, or whatever the fixture declares) using cardinality glyphs that match the decorator (||--o{ for one_to_many, }o--|| for many_to_one)", "passed": true, "evidence": "FLIPPED from iter-7 FAIL. Mermaid now: authors ||--o{ books (one_to_many glyph), books ||--|| publishers (one_to_one glyph). Real entity table names, no _rel stubs. 3 nodes (authors/books/publishers) + edges connecting them directly. A2 fix cascaded here." }, { "text": "Any external target referenced via @ManyToOne whose class file is NOT in the fixture set is emitted as a _external stub before the edge \u2014 never an edge to an undeclared node", "passed": true, "evidence": "FLIPPED from iter-7 FAIL. No edges to undeclared nodes (no _rel fallback fires now). All fixture targets are in-scope so _external path isn't exercised, but the 'never edge to undeclared node' invariant is upheld by A2 fix." }, { "text": "mermaid_lint.js passes on the generated page", "passed": true, "evidence": "mermaid_lint.txt = [] (exit 0). Unchanged from iter-7." } ], "notes": [] }, { "eval_id": 12, "configuration": "with_skill", "run_number": 1, "result": { "pass_rate": 1.0, "passed": 7, "failed": 0, "total": 7, "time_seconds": 194.501, "tokens": 45523, "tool_calls": 0, "errors": 0 }, "expectations": [ { "text": "--since 1h returns total_events=2 (both within the last 30 minutes)", "passed": true, "evidence": "stats-1h.json: total_ops=2 (ingest=1, query=1). Unchanged from iter-7." }, { "text": "--since 30d returns total_events=9 (the 2+3+4 events within 30 days; the 60-day-old ones excluded)", "passed": true, "evidence": "stats-30d.json: total_ops=9 (ingest=4, query=3, lint=1, fix=1). Unchanged from iter-7." }, { "text": "--since 2026-04-01T00:00:00Z returns total_events=5 (only the 30-min and 5-day cohorts survive; the 20-day-old cohort is dated 2026-03-25 \u2014 which is before the 2026-04-01 ISO cutoff \u2014 so those 4 events are correctly excluded, along with the 60-day-old cohort)", "passed": true, "evidence": "E1 FIX VERIFIED. stats-iso.json: total_ops=5 = revised expected value. FLIPPED from iter-7 FAIL (was 5 vs old expected 9 \u2014 arithmetic inconsistency resolved in the assertion text)." }, { "text": "All three calls invoke the same event_logger.js stats subcommand \u2014 the prompt MUST NOT massage results client-side; counts come from the script's stdout", "passed": true, "evidence": "command.txt: all 5 invocations are raw `node event_logger.js stats --wiki-root --since ` calls. JSON captured verbatim from stdout." }, { "text": "Per-op breakdown (by_op) is present in every report \u2014 the count for each individual op (e.g., ingest, query, lint) appears separately and the sum across ops equals total_events", "passed": true, "evidence": "per-op-check.md: ops_by_type present in all 3 valid windows. Sums: 2, 9, 5 \u2014 all match total_ops." }, { "text": "An invalid granularity (e.g., --since 1z) is treated as a literal string by parseRelativeSince (returns the string unchanged), and parsePythonIsoformat then fails it, so the script either errors with a non-zero exit and a human-readable message OR returns total_events=0 with an explicit warning \u2014 silent fallback to 'no filter' is a failure", "passed": true, "evidence": "W1 FIX VERIFIED. stats-invalid.exitcode.txt=2 (non-zero), stats-invalid.stderr.txt=`[event_logger] error: --since value \"1z\" is not a valid relative duration (e.g. 7d, 24h, 15m) or absolute ISO timestamp (YYYY-MM-DD...)`, stdout=empty. FLIPPED from iter-7 FAIL. Fix: CLI-layer regex validation at event_logger.ts main() stats branch rejects inputs that are neither N[dhm] nor start with YYYY-MM-DD." }, { "text": "Absolute ISO timestamp output (with +00:00 UTC offset) is accepted identically to its Z-suffix equivalent (2026-04-01T00:00:00+00:00 and 2026-04-01T00:00:00Z produce the same total_events)", "passed": true, "evidence": "offset-equivalence-check.md: diff stats-iso.json vs stats-utc-offset.json \u2192 no differences. Both produce total_ops=5. Unchanged from iter-7." } ], "notes": [] } ], "run_summary": { "with_skill": { "pass_rate": { "mean": 1.0, "stddev": 0.0, "min": 1.0, "max": 1.0 }, "time_seconds": { "mean": 128.685, "stddev": 57.6562, "min": 87.092, "max": 194.501 }, "tokens": { "mean": 35316.6667, "stddev": 9959.4398, "min": 25624, "max": 45523 } }, "delta": { "pass_rate": "+1.00", "time_seconds": "+128.7", "tokens": "+35317" } }, "notes": [] }