{ "skill_name": "doc-wiki", "evals": [ { "id": 1, "prompt": "I have a new Python project at /tmp/test-project with some markdown docs in a docs/ folder. Initialize a documentation wiki for it with domain 'backend-services', ingest docs/architecture.md (about JWT auth and session management), and then query 'how does authentication work?'.", "expected_output": "Creates wiki scaffold, ingests the architecture doc as a wiki page, and returns a synthesized answer citing the wiki page", "files": [], "expectations": [ "Runs init_wiki.js to create directory scaffold under the project root", "wiki.config.yaml is created with wiki.domain: 'backend-services'", "Directory scaffold includes wiki/, raw/, graph/, audit/, log/, outputs/", "Source file is copied to raw/ (path preserved relative to the ingested root)", "Generated wiki page has frontmatter with non-empty title, type set to 'concept' (or one of concept|index|synthesis|claim|template), sources listing the ingested file, created + updated ISO dates, a non-empty summary, and 4-8 content-only tags", "Tags are content-only concepts (e.g., 'jwt', 'authentication', 'session-management') — no structural/temporal/metadata tags", "summaries.md has a non-stub entry (≥ 20 chars) for the new page", "Query answer includes at least two inline markdown links and each link resolves to a file that actually exists under wiki/", "Query answer mentions at least two concrete facts from the source (e.g., JWT, sessions table, refresh endpoint) — hallucinated answers that cite nonexistent pages must fail", "Answer is archived to outputs/queries/ as a dated .md file", "events.jsonl records separate entries with op='init', op='ingest', and op='query' (one per operation), all with ISO timestamps", "graph/edges.jsonl contains at least one edge with a valid provenance tag (EXTRACTED, INFERRED, or AMBIGUOUS)" ] }, { "id": 2, "prompt": "Run /doc-wiki:lint on my wiki at /tmp/test-wiki/. It has pages with missing frontmatter fields and broken internal links. Fix all issues found.", "expected_output": "Identifies lint violations (missing frontmatter, broken links), reports them, and applies fixes to the affected pages", "files": [], "expectations": [ "lint_checks.js is invoked and its first-pass report lists at least one 'broken_links' violation and at least one 'missing_frontmatter' violation", "Each violation record names the offending file path", "Auto-fix applies the frontmatter fields required by the linter (title, type, tags, sources, created, updated, summary) — running the linter again after fix reports zero errors", "Broken-link fixes either create a stub for the missing target OR remove the link, never rewrite the page body wholesale (original headings and primary prose are preserved byte-identically outside the fix region)", "The fix action went through the wiki skill — events.jsonl records op='lint' AND op='fix' in order, each with target_page paths", "Re-running /doc-wiki:lint after the fix yields zero 'broken_links' and zero 'missing_frontmatter' violations" ] }, { "id": 3, "prompt": "In my wiki at /tmp/test-wiki/, find the shortest path between the 'authentication' concept and the 'database-schema' concept. Show all intermediate pages.", "expected_output": "Returns the shortest path through the relationship graph, listing each intermediate page and edge type", "files": [], "expectations": [ "Runs graph_ops.js with the path subcommand", "Returned path starts at 'authentication' and ends at 'database-schema'", "Every intermediate node appears as either the source or the target of an edge in the fixture's edges.jsonl", "Every consecutive (from, to) pair in the returned path exists as an edge in edges.jsonl (no hallucinated links)", "The returned path is shortest — no alternative chain in edges.jsonl connects the same endpoints with fewer hops", "Edge type (supports|contradicts|extends|supersedes) is reported for each hop", "Provenance tag (EXTRACTED|INFERRED|AMBIGUOUS) is reported for each edge", "A no-path query (unreachable or unknown target) returns an empty result with a human-readable explanation — not a stack trace or bare []" ] }, { "id": 4, "prompt": "Set up wiki for a Spring Boot + PostgreSQL project", "expected_output": "Detects Java/Spring Boot with JPA and PostgreSQL, writes wiki.config.yaml with the stack wired in", "files": [], "expectations": [ "Detects Java/Spring Boot by parsing pom.xml or build.gradle — wiki.config.yaml lists framework='spring-boot' AND a framework version string that matches the fixture (e.g., '3.2.0' when pom.xml pins that version)", "Detects JPA by scanning .java files — ecosystem.orm.profiles includes 'jpa' AND the detection trace lists at least one concrete @Entity class name found (e.g., 'User')", "Detects PostgreSQL from application.properties JDBC URL — wiki.config.yaml's database block sets driver='postgresql' AND host+port+database match the URL (e.g., localhost:5432/mydb when the fixture contains jdbc:postgresql://localhost:5432/mydb)", "Credentials are referenced via secret names (e.g., user_secret: WIKI_DB_DEV_USER) — plaintext passwords anywhere in wiki.config.yaml is a failure", "Policy stanza (block_ddl, block_privilege, dml_mode, audit.enabled) is populated in the database section", "Wiki scaffold (wiki/, raw/, graph/, audit/, log/, outputs/) exists at the configured wiki_root" ] }, { "id": 5, "prompt": "Set up wiki for a Django + MongoDB project", "expected_output": "Detects Python/Django with MongoDB, writes wiki.config.yaml with the stack wired in", "files": [], "expectations": [ "Detects Python/Django — wiki.config.yaml lists language='python' and framework='django' with at least one evidence file cited in detection-trace (requirements.txt, pyproject.toml, or manage.py)", "Detects Django ORM — ecosystem.orm.profiles includes 'django' AND the detection trace names at least one extracted model class with its db_table mapping (e.g., Article → articles)", "Detects MongoDB — wiki.config.yaml's database block sets driver='mongodb' (via djongo or direct) with connection info parsed from settings.py (e.g., mongodb://localhost:27017)", "Policy stanza (block_ddl, block_privilege, dml_mode, audit.enabled) is populated for MongoDB", "Wiki scaffold (wiki/, raw/, graph/, audit/, log/, outputs/) exists at the configured wiki_root", "Seed files (wiki/index.md, graph/edges.jsonl, log/events.jsonl) exist — not just empty directories" ] }, { "id": 6, "prompt": "Ingest all markdown files from the docs/ folder at /tmp/test-ingest-folder/ (8 files covering architecture, auth, database, caching, deployment, monitoring, sdk, and troubleshooting). Use folder batching so I can see progress per-file, and make the second run a no-op for any file whose content hasn't changed.", "expected_output": "All 8 files become wiki pages; per-file events logged; re-running the ingest skips unchanged files via the content-hash cache", "files": [], "expectations": [ "All 8 source files end up as wiki pages (or 8 distinct wiki/*.md files created)", "Each generated wiki page has valid frontmatter with non-empty title + sources + a 4-8 tag list", "Each source file is copied into raw/ preserving the relative path (raw/architecture.md, raw/auth.md, etc.)", "events.jsonl records 8 ingest events, each naming the source file and content hash", ".wiki-cache/ (or the caching dir named in wiki.config.yaml) contains 8 per-file hash entries after the first run", "A second run of the same ingest command logs 8 'cache_hit' or 'skipped' entries (or records zero new ingest events) — no file is reprocessed while the content is unchanged", "Modifying one file and re-running processes only that one file (one new ingest event; seven cache hits)" ] }, { "id": 7, "prompt": "In my wiki at /tmp/test-fix-wiki/, the page authentication.md says the service uses HS256 signing, but the source was recently updated to use RS256. Fix just that page — update the summary and any body mentions of HS256 to RS256.", "expected_output": "Only the authentication.md page is edited; the fix is audited; no other pages change", "files": [], "expectations": [ "authentication.md content mentions RS256 and no longer mentions HS256 in the updated regions", "The page's frontmatter 'updated' field is refreshed to the current date (differs from 'created')", "No other wiki page is modified — a directory diff of wiki/ shows exactly one file touched", "events.jsonl records op='fix' with target_page: 'wiki/authentication.md' (or equivalent), a reason/description field, and an ISO timestamp", "A re-run of /doc-wiki:lint on the fixed page reports zero violations (quality is preserved)", "The quality score of the page is recomputed and its value appears in the events.jsonl fix entry" ] }, { "id": 8, "prompt": "Show me the wiki stats for /tmp/test-stats-wiki/ over the last 7 days. Break it down by operation (ingest, query, lint, fix) and include total-tokens and average-duration per op.", "expected_output": "Runs event_logger stats; produces a summary grouped by op with counts, totals, and averages; respects the 7-day window", "files": [], "expectations": [ "Runs the wiki's event_logger stats subcommand (event_logger.js stats --since 7d) or equivalent", "Output lists counts for at least op='ingest', op='query', op='lint', op='fix' — one line per op", "Each op's count matches the number of corresponding entries in the fixture's events.jsonl within the 7-day window", "Output emits `total_tokens_by_op` NATIVELY from the stats subcommand (no manual post-processing) — with a per-op breakdown summing event.tokens / total_tokens / tokens_in+tokens_out fields", "Output emits `avg_duration_ms_by_op` NATIVELY from the stats subcommand with a per-op average of event.duration_ms / total_duration_ms", "The --since 7d window is honored — any event older than 7 days is excluded from the counts" ] }, { "id": 9, "prompt": "Promote the archived query answer at /tmp/test-promote-wiki/outputs/queries/2026-04-14-how-does-authentication-work.md to a permanent wiki page under wiki/auth/authentication-flow.md. Use topic 'authentication-flow'.", "expected_output": "Copies/transforms the archived query into a permanent wiki page with frontmatter, registers it in the crosslink graph, logs op=promote, and leaves the original archive in place", "files": [], "expectations": [ "A new wiki page exists at wiki/auth/authentication-flow.md (or wiki/authentication-flow.md if no topic dir is configured) with frontmatter type='synthesis' OR type='concept', non-empty title, >=4 content-only tags, a sources array referencing the original query's source URLs/files", "The promoted page's body is derived from (not byte-identical to) the archive — citations to wiki pages in the archive are converted to working relative markdown links, and at least 80% of the archive's prose paragraphs survive into the page", "The original archive at outputs/queries/2026-04-14-how-does-authentication-work.md still exists and is unmodified (mtime and contents preserved); promote MUST NOT move or delete the archive", "wiki/index.md (or summaries.md) has a new entry referencing the promoted page's path", "graph/edges.jsonl gains at least one edge whose source or target is the new page, with a valid provenance tag (EXTRACTED|INFERRED|AMBIGUOUS) and a typed-edge label (supports|extends|supersedes|contradicts)", "events.jsonl records exactly one entry with op='promote' including source_query (path to the archive), target_page (path to the new wiki page), and an ISO timestamp", "Re-running /doc-wiki:promote for the same archive is a no-op or surfaces a 'page already exists' warning — it does not duplicate the page or re-emit the promote event" ] }, { "id": 10, "prompt": "Refresh the previously-ingested source /tmp/test-refresh-wiki/raw/architecture.md against my wiki at /tmp/test-refresh-wiki/. The source has been edited since the last ingest; the wiki page also has a manual frontmatter field reviewer: alice that I added by hand — preserve it.", "expected_output": "Detects the source has changed, re-compiles the wiki page with new content, preserves manual frontmatter additions, updates the raw/ copy, re-runs edge inference, logs op=refresh", "files": [], "expectations": [ "Re-computed content_hash for the source differs from the value stored in .wiki-cache/ from the original ingest — both values appear in the events.jsonl entry as stale_hash and fresh_hash", "The corresponding wiki/architecture.md body is updated to reflect new content from the source (a phrase introduced in the modified source appears in the page body)", "The page's frontmatter still contains reviewer: alice (manual field preserved); standard managed fields (updated, sources, summary) are refreshed; created is unchanged", "raw/architecture.md is overwritten with the new source content (its file hash matches the new source)", "events.jsonl records op='refresh' with target_page, source, stale_hash, fresh_hash, and a delta_summary string of >=20 chars describing what changed", "graph/edges.jsonl is re-evaluated — at least one edge whose source is the refreshed page differs from the pre-refresh edge set (added, removed, or retyped). If no edges change, the events entry must explicitly note edges_unchanged: true", "A second /doc-wiki:refresh run with no further source modification is a no-op: no new events, hash stays identical, page body unchanged byte-for-byte" ] }, { "id": 11, "prompt": "In the wiki at /tmp/test-query-wiki/ (5 wiki pages spanning auth, request-routing, app-server, db-write-path, and audit-log), answer: what's the full request lifecycle from user authentication through DB write? Cite every page you read.", "expected_output": "A synthesized answer grounded in at least three distinct wiki pages, with summary-first search visible in the trace, a knowledge-gaps section if applicable, archive dated and topic-tagged, and reduction_ratio >0.8", "files": [], "expectations": [ "The answer cites at least 3 distinct wiki pages by relative markdown link, each link resolving to a real file under wiki/", "events.jsonl records op='query' with details.search_phase: 'summaries' (or equivalent flag) preceding any per-page reads, evidencing summaries-first search — no op='query' whose first read is a full wiki page is acceptable", "details.reduction_ratio in the events entry is >0.8 (final synthesis token count is <20% of total tokens read)", "The answer ends with a '## Knowledge gaps' (or equivalent labeled) section listing at least one specific gap when the source pages don't cover (a) request-id propagation OR (b) error-path rollback — both of which the fixture intentionally omits", "The archive at outputs/queries/-.md exists with frontmatter containing topic, tags, and created ISO date; filename slug derives from the question", "The synthesized answer mentions concrete artifacts from at least 3 of the 5 pages (e.g., JWT, nginx, pg connection pool, audit_log table) — a generic answer that doesn't surface page-specific terms fails", "The events entry includes tokens_in, tokens_out, and a non-zero pages_read[] array of length >=3" ] }, { "id": 12, "prompt": "Run /doc-wiki:stats on /tmp/test-stats-since-wiki/ three times: first with --since 1h, then with --since 30d, then with --since where is the timestamp 10 days before now rendered as an absolute ISO with a Z suffix (e.g., 2026-04-23T12:00:00Z if now is 2026-05-03T12:00:00Z). Report the per-op counts for each window. The fixture's events.jsonl contains entries timestamped 30 minutes ago (2 events), 5 days ago (3 events), 20 days ago (4 events), and 60 days ago (3 events) — all relative to the moment the eval is run, so the fixture stays time-stable.", "expected_output": "Three distinct stat reports — 1h covers 2 events, 30d covers 9 events, the absolute-ISO cutoff at now-10d covers 5 events (only the 30-min and 5-day cohorts survive; the 20-day and 60-day cohorts fall before the cutoff); each report exposes per-op breakdown", "files": [], "expectations": [ "--since 1h returns total_events=2 (both within the last 30 minutes)", "--since 30d returns total_events=9 (the 2+3+4 events within 30 days; the 60-day-old ones excluded)", "--since returns total_events=5 — the 30-min and 5-day cohorts survive, the 20-day and 60-day cohorts are excluded by the absolute-ISO cutoff. The cutoff is computed as `now - 10 days` rendered with a Z suffix, so it always falls between the 5-day and 20-day cohorts regardless of when the eval is run.", "All three calls invoke the same event_logger.js stats subcommand — the prompt MUST NOT massage results client-side; counts come from the script's stdout", "Per-op breakdown (by_op) is present in every report — the count for each individual op (e.g., ingest, query, lint) appears separately and the sum across ops equals total_events", "An invalid granularity (e.g., --since 1z) is treated as a literal string by parseRelativeSince (returns the string unchanged), and parsePythonIsoformat then fails it, so the script either errors with a non-zero exit and a human-readable message OR returns total_events=0 with an explicit warning — silent fallback to 'no filter' is a failure", "Absolute ISO timestamp with +00:00 UTC offset is accepted identically to its Z-suffix equivalent — re-running the third command with the same wall-clock cutoff but rendered as `+00:00` instead of `Z` produces the same total_events=5" ] }, { "id": 13, "prompt": "Onboard a wiki for the Node project at /tmp/test-onboard-node-pg/. The fixture has package.json with express + pg dependencies; src/db.js opens a new Pool({ connectionString: process.env.DATABASE_URL }) against postgres://wikiuser@db.local:5432/appdb; src/models/user.js exports a plain JS object literal (no ORM library).", "expected_output": "Detects javascript + express + postgres; ORM detection finds no profile match and falls back to raw_sql; wiki.config.yaml populated with all detected fields and a policy stanza", "files": [], "expectations": [ "wiki.config.yaml has language: javascript (or nodejs) inferred from package.json", "framework: express is set when express is in package.json dependencies; alternatively, framework: null (or absent) is acceptable ONLY if a detection-trace entry explicitly notes 'express detected but not configured as framework' — silent omission is a failure", "ecosystem.database.driver: postgresql is set; host: db.local, port: 5432, database: appdb parsed from the connection string in src/db.js", "Credentials are referenced by secret name (user_secret: WIKI_DB_DEV_USER or similar) — no plaintext password OR username appears in wiki.config.yaml", "ecosystem.orm.profiles is either an empty list [] OR contains exactly ['raw_sql'] — none of the shipped ORM profile names (jpa, sqlalchemy, django, prisma, typeorm, activerecord, entity_framework) appear, since the fixture has no ORM library", "Policy stanza (block_ddl, block_privilege, dml_mode, audit.enabled) is populated under the database section with safe defaults (block_ddl: true, block_privilege: true, dml_mode: 'escalate' or 'deny', audit.enabled: true)", "Wiki scaffold (wiki/, raw/, graph/, audit/, log/, outputs/) exists at the configured wiki_root after onboard completes (auto-init triggered)", "events.jsonl contains one op='init' event AND one op='onboard' event (or equivalent), each with ISO timestamps" ] }, { "id": 14, "prompt": "Run /doc-wiki:ingest https://github.com/anthropics/anthropic-sdk-python against my wiki at /tmp/test-gather-wiki/ (initialize the wiki first with /doc-wiki:init). Do NOT set GITHUB_TOKEN in the environment. The point of this eval is to verify the gather() ↔ connector-subprocess ↔ credential-resolution boundary survives end-to-end without needing a real token: gather() should plan the github connector, spawn its subprocess, the subprocess should load narai-primitives/credentials and call resolveSecret() inside its own process, find no credential, and return a structured CONFIG_ERROR envelope to the hub. doc-wiki should record that envelope verbatim in events.jsonl — never a stack trace, never a silent skip.", "expected_output": "Wiki scaffold initialized; gather() dispatches the github connector subprocess; the subprocess calls narai-primitives/credentials.resolveSecret() and finds no GITHUB_TOKEN; the connector emits a structured envelope with status='error' or 'denied' and code='CONFIG_ERROR' (or equivalent) whose reason references GITHUB_TOKEN or the credentials layer; the hub captures the envelope and logs it in events.jsonl; no cleartext token ever appears in any wiki output. Re-running the same ingest produces an identical structured failure (idempotent error path).", "files": [], "expectations": [ "Wiki scaffold exists at /tmp/test-gather-wiki/ with the standard dirs (wiki/, raw/, graph/, audit/, log/, outputs/) — confirms /doc-wiki:init ran", "events.jsonl records at least one op='ingest' entry whose details capture the gather() dispatch — either a top-level `consumer` field set to 'doc-wiki' or an embedded `dispatch_steps[]` array referencing the 'github' connector", "The github dispatch step has a non-success status (e.g., 'error' or 'denied'), with a structured code field — accept any of `code`, `error_code`, or `error.code` set to a CONFIG-class value (e.g., 'CONFIG_ERROR') — proving the github connector's structured-error path fired rather than a thrown exception", "The dispatch step's reason / message text references GITHUB_TOKEN OR narai-primitives/credentials — proves the credential-resolution boundary was reached inside the subprocess (the connector's CONFIG_ERROR text says 'Set GITHUB_TOKEN or register a credential provider via narai-primitives/credentials')", "The github connector ran in a separate Node process, not in-line in the hub — verifiable via either (a) the dispatch envelope matching narai-primitives' DispatchResult contract (envelope/data/error shape produced by createConnector — not raw object literal from in-process code), or (b) the resolved CLI path under narai-primitives/dist/connectors/github/cli.js appearing in the hub log/output", "No stack trace anywhere in the wiki tree (`outputs/`, `raw/`, `wiki/`, `events.jsonl`, `audit/`) — `grep -rE 'at .+\\.js:[0-9]+:|at Object\\.'` returns zero matches; the failure is structured throughout", "The hub process never sees a cleartext token — `grep -rE 'github_pat_[A-Za-z0-9_]{20,}|ghp_[A-Za-z0-9]{36}' /tmp/test-gather-wiki/` returns zero matches (the subprocess-credential-isolation guarantee from architecture invariant #4 still holds)", "Re-running the same /doc-wiki:ingest command produces an identical structured failure (idempotent error path) — does not crash, does not double-create wiki pages, and either (a) emits a second dispatch entry with the same code or (b) skips via cache and records a 'cache_hit' or 'skipped' entry", "resolveAgentCli({name:'github'}) succeeds against the published bundle — the resolved CLI lives at node_modules/narai-primitives/dist/connectors/github/cli.js (with `source: 'bundled-self'` per the 2.1.1+ resolver). A null result here means the bundled-CLI resolver is broken and gather() can't actually dispatch." ] }, { "id": 15, "prompt": "Run /doc-wiki:ingest https://anthropic.atlassian.net/browse/PROJ-1 against my wiki at /tmp/test-gather-jira-wiki/ (initialize the wiki first with /doc-wiki:init). Do NOT set JIRA_API_TOKEN, JIRA_EMAIL, or JIRA_SITE_URL in the environment. The point of this eval is to verify the gather() ↔ jira connector subprocess ↔ credential-resolution boundary survives end-to-end without needing a real token: gather() should plan the jira connector, spawn its subprocess, the subprocess should load narai-primitives/credentials and call resolveSecret() inside its own process, find no credential, and return a structured CONFIG_ERROR envelope to the hub. doc-wiki should record that envelope verbatim in events.jsonl — never a stack trace, never a silent skip.", "expected_output": "Wiki scaffold initialized; gather() dispatches the jira connector subprocess; the subprocess calls narai-primitives/credentials.resolveSecret() and finds no JIRA_API_TOKEN/JIRA_EMAIL/JIRA_SITE_URL; the connector emits a structured envelope with status='error' or 'denied' and code='CONFIG_ERROR' (or AUTH_ERROR) whose reason references JIRA_API_TOKEN or the credentials layer; the hub captures the envelope and logs it in events.jsonl; no cleartext token ever appears in any wiki output. Re-running the same ingest produces an identical structured failure (idempotent error path).", "files": [], "expectations": [ "Wiki scaffold exists at /tmp/test-gather-jira-wiki/ with the standard dirs (wiki/, raw/, graph/, audit/, log/, outputs/) — confirms /doc-wiki:init ran", "events.jsonl records at least one op='ingest' entry whose details capture the gather() dispatch — either a top-level `consumer` field set to 'doc-wiki' or an embedded `dispatch_steps[]` array referencing the 'jira' connector", "The jira dispatch step has a non-success status (e.g., 'error' or 'denied'), with a structured code field — accept any of `code`, `error_code`, or `error.code` set to a CONFIG-class value (e.g., 'CONFIG_ERROR' or 'AUTH_ERROR') — proving the jira connector's structured-error path fired rather than a thrown exception", "The dispatch step's reason / message text references JIRA_API_TOKEN, JIRA_EMAIL, JIRA_SITE_URL, OR narai-primitives/credentials — proves the credential-resolution boundary was reached inside the subprocess", "The jira connector ran in a separate Node process, not in-line in the hub — verifiable via either (a) the dispatch envelope matching narai-primitives' DispatchResult contract (envelope/data/error shape produced by createConnector — not raw object literal from in-process code), or (b) the resolved CLI path under narai-primitives/dist/connectors/jira/cli.js appearing in the hub log/output", "No stack trace anywhere in the wiki tree (`outputs/`, `raw/`, `wiki/`, `events.jsonl`, `audit/`) — `grep -rE 'at .+\\.js:[0-9]+:|at Object\\.'` returns zero matches; the failure is structured throughout", "The hub process never sees a cleartext token — `grep -rE 'ATATT[A-Za-z0-9_-]{20,}' /tmp/test-gather-jira-wiki/` returns zero matches (the subprocess-credential-isolation guarantee from architecture invariant #4 still holds)", "Re-running the same /doc-wiki:ingest command produces an identical structured failure (idempotent error path) — does not crash, does not double-create wiki pages, and either (a) emits a second dispatch entry with the same code or (b) skips via cache and records a 'cache_hit' or 'skipped' entry", "resolveAgentCli({name:'jira'}) succeeds against the published bundle — the resolved CLI lives at node_modules/narai-primitives/dist/connectors/jira/cli.js (with `source: 'bundled-self'` per the 2.1.1+ resolver). A null result here means the bundled-CLI resolver is broken and gather() can't actually dispatch." ] }, { "id": 16, "prompt": "Run /doc-wiki:ingest https://anthropic.atlassian.net/wiki/spaces/PROJ/pages/1 against my wiki at /tmp/test-gather-confluence-wiki/ (initialize the wiki first with /doc-wiki:init). Do NOT set CONFLUENCE_API_TOKEN, CONFLUENCE_EMAIL, or CONFLUENCE_SITE_URL in the environment. The point of this eval is to verify the gather() ↔ confluence connector subprocess ↔ credential-resolution boundary survives end-to-end without needing a real token: gather() should plan the confluence connector, spawn its subprocess, the subprocess should load narai-primitives/credentials and call resolveSecret() inside its own process, find no credential, and return a structured CONFIG_ERROR envelope to the hub. doc-wiki should record that envelope verbatim in events.jsonl — never a stack trace, never a silent skip.", "expected_output": "Wiki scaffold initialized; gather() dispatches the confluence connector subprocess; the subprocess calls narai-primitives/credentials.resolveSecret() and finds no CONFLUENCE_API_TOKEN/CONFLUENCE_EMAIL/CONFLUENCE_SITE_URL; the connector emits a structured envelope with status='error' or 'denied' and code='CONFIG_ERROR' (or AUTH_ERROR) whose reason references CONFLUENCE_API_TOKEN or the credentials layer; the hub captures the envelope and logs it in events.jsonl; no cleartext token ever appears in any wiki output. Re-running the same ingest produces an identical structured failure (idempotent error path).", "files": [], "expectations": [ "Wiki scaffold exists at /tmp/test-gather-confluence-wiki/ with the standard dirs (wiki/, raw/, graph/, audit/, log/, outputs/) — confirms /doc-wiki:init ran", "events.jsonl records at least one op='ingest' entry whose details capture the gather() dispatch — either a top-level `consumer` field set to 'doc-wiki' or an embedded `dispatch_steps[]` array referencing the 'confluence' connector", "The confluence dispatch step has a non-success status (e.g., 'error' or 'denied'), with a structured code field — accept any of `code`, `error_code`, or `error.code` set to a CONFIG-class value (e.g., 'CONFIG_ERROR' or 'AUTH_ERROR') — proving the confluence connector's structured-error path fired rather than a thrown exception", "The dispatch step's reason / message text references CONFLUENCE_API_TOKEN, CONFLUENCE_EMAIL, CONFLUENCE_SITE_URL, OR narai-primitives/credentials — proves the credential-resolution boundary was reached inside the subprocess", "The confluence connector ran in a separate Node process, not in-line in the hub — verifiable via either (a) the dispatch envelope matching narai-primitives' DispatchResult contract (envelope/data/error shape produced by createConnector — not raw object literal from in-process code), or (b) the resolved CLI path under narai-primitives/dist/connectors/confluence/cli.js appearing in the hub log/output", "No stack trace anywhere in the wiki tree (`outputs/`, `raw/`, `wiki/`, `events.jsonl`, `audit/`) — `grep -rE 'at .+\\.js:[0-9]+:|at Object\\.'` returns zero matches; the failure is structured throughout", "The hub process never sees a cleartext token — `grep -rE 'ATATT[A-Za-z0-9_-]{20,}' /tmp/test-gather-confluence-wiki/` returns zero matches (the subprocess-credential-isolation guarantee from architecture invariant #4 still holds)", "Re-running the same /doc-wiki:ingest command produces an identical structured failure (idempotent error path) — does not crash, does not double-create wiki pages, and either (a) emits a second dispatch entry with the same code or (b) skips via cache and records a 'cache_hit' or 'skipped' entry", "resolveAgentCli({name:'confluence'}) succeeds against the published bundle — the resolved CLI lives at node_modules/narai-primitives/dist/connectors/confluence/cli.js (with `source: 'bundled-self'` per the 2.1.1+ resolver). A null result here means the bundled-CLI resolver is broken and gather() can't actually dispatch." ] }, { "id": 17, "prompt": "Run /doc-wiki:ingest https://www.notion.so/Page-abcdef0123456789abcdef0123456789 against my wiki at /tmp/test-gather-notion-wiki/ (initialize the wiki first with /doc-wiki:init). Do NOT set NOTION_TOKEN in the environment. The point of this eval is to verify the gather() ↔ notion connector subprocess ↔ credential-resolution boundary survives end-to-end without needing a real token: gather() should plan the notion connector, spawn its subprocess, the subprocess should load narai-primitives/credentials and call resolveSecret() inside its own process, find no credential, and return a structured CONFIG_ERROR envelope to the hub. doc-wiki should record that envelope verbatim in events.jsonl — never a stack trace, never a silent skip.", "expected_output": "Wiki scaffold initialized; gather() dispatches the notion connector subprocess; the subprocess calls narai-primitives/credentials.resolveSecret() and finds no NOTION_TOKEN; the connector emits a structured envelope with status='error' or 'denied' and code='CONFIG_ERROR' (or AUTH_ERROR) whose reason references NOTION_TOKEN or the credentials layer; the hub captures the envelope and logs it in events.jsonl; no cleartext token ever appears in any wiki output. Re-running the same ingest produces an identical structured failure (idempotent error path).", "files": [], "expectations": [ "Wiki scaffold exists at /tmp/test-gather-notion-wiki/ with the standard dirs (wiki/, raw/, graph/, audit/, log/, outputs/) — confirms /doc-wiki:init ran", "events.jsonl records at least one op='ingest' entry whose details capture the gather() dispatch — either a top-level `consumer` field set to 'doc-wiki' or an embedded `dispatch_steps[]` array referencing the 'notion' connector", "The notion dispatch step has a non-success status (e.g., 'error' or 'denied'), with a structured code field — accept any of `code`, `error_code`, or `error.code` set to a CONFIG-class value (e.g., 'CONFIG_ERROR' or 'AUTH_ERROR') — proving the notion connector's structured-error path fired rather than a thrown exception", "The dispatch step's reason / message text references NOTION_TOKEN OR narai-primitives/credentials — proves the credential-resolution boundary was reached inside the subprocess (the connector's CONFIG_ERROR text says 'Set NOTION_TOKEN or register a credential provider via narai-primitives/credentials')", "The notion connector ran in a separate Node process, not in-line in the hub — verifiable via either (a) the dispatch envelope matching narai-primitives' DispatchResult contract (envelope/data/error shape produced by createConnector — not raw object literal from in-process code), or (b) the resolved CLI path under narai-primitives/dist/connectors/notion/cli.js appearing in the hub log/output", "No stack trace anywhere in the wiki tree (`outputs/`, `raw/`, `wiki/`, `events.jsonl`, `audit/`) — `grep -rE 'at .+\\.js:[0-9]+:|at Object\\.'` returns zero matches; the failure is structured throughout", "The hub process never sees a cleartext token — `grep -rE 'secret_[A-Za-z0-9_]{30,}|ntn_[A-Za-z0-9_]{30,}' /tmp/test-gather-notion-wiki/` returns zero matches (the subprocess-credential-isolation guarantee from architecture invariant #4 still holds)", "Re-running the same /doc-wiki:ingest command produces an identical structured failure (idempotent error path) — does not crash, does not double-create wiki pages, and either (a) emits a second dispatch entry with the same code or (b) skips via cache and records a 'cache_hit' or 'skipped' entry", "resolveAgentCli({name:'notion'}) succeeds against the published bundle — the resolved CLI lives at node_modules/narai-primitives/dist/connectors/notion/cli.js (with `source: 'bundled-self'` per the 2.1.1+ resolver). A null result here means the bundled-CLI resolver is broken and gather() can't actually dispatch." ] }, { "id": 18, "prompt": "Run /doc-wiki:ingest 'arn:aws:s3:::example-bucket' against my wiki at /tmp/test-gather-aws-wiki/ (initialize the wiki first with /doc-wiki:init). Do NOT set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_PROFILE, or AWS_SESSION_TOKEN in the environment. The point of this eval is to verify the gather() ↔ aws connector subprocess ↔ credential-resolution boundary survives end-to-end without needing real credentials: gather() should plan the aws connector, spawn its subprocess, the subprocess should load credentials via narai-primitives/credentials and the AWS SDK default chain inside its own process, find no credentials, and return a structured CONFIG_ERROR (or AUTH_ERROR) envelope to the hub. doc-wiki should record that envelope verbatim in events.jsonl — never a stack trace, never a silent skip.", "expected_output": "Wiki scaffold initialized; gather() dispatches the aws connector subprocess; the subprocess calls the AWS SDK default credential chain (via narai-primitives/credentials) and finds no AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY; the connector emits a structured envelope with status='error' or 'denied' and code='CONFIG_ERROR' (or AUTH_ERROR — AWS surfaces credentials-not-found as AUTH_ERROR via the SDK CredentialsProviderError) whose reason references AWS credentials or the credentials layer; the hub captures the envelope and logs it in events.jsonl; no cleartext key ever appears in any wiki output. Re-running the same ingest produces an identical structured failure (idempotent error path).", "files": [], "expectations": [ "Wiki scaffold exists at /tmp/test-gather-aws-wiki/ with the standard dirs (wiki/, raw/, graph/, audit/, log/, outputs/) — confirms /doc-wiki:init ran", "events.jsonl records at least one op='ingest' entry whose details capture the gather() dispatch — either a top-level `consumer` field set to 'doc-wiki' or an embedded `dispatch_steps[]` array referencing the 'aws' connector", "The aws dispatch step has a non-success status (e.g., 'error' or 'denied'), with a structured code field — accept any of `code`, `error_code`, or `error.code` set to a CONFIG-class value (e.g., 'CONFIG_ERROR' or 'AUTH_ERROR') — proving the aws connector's structured-error path fired rather than a thrown exception", "The dispatch step's reason / message text references AWS credentials OR the credential chain (e.g., 'Could not load credentials from any providers' or AWS_ACCESS_KEY_ID / @aws-sdk) — proves the credential-resolution boundary was reached inside the subprocess", "The aws connector ran in a separate Node process, not in-line in the hub — verifiable via either (a) the dispatch envelope matching narai-primitives' DispatchResult contract (envelope/data/error shape produced by createConnector — not raw object literal from in-process code), or (b) the resolved CLI path under narai-primitives/dist/connectors/aws/cli.js appearing in the hub log/output", "No stack trace anywhere in the wiki tree (`outputs/`, `raw/`, `wiki/`, `events.jsonl`, `audit/`) — `grep -rE 'at .+\\.js:[0-9]+:|at Object\\.'` returns zero matches; the failure is structured throughout", "The hub process never sees a cleartext key — `grep -rE 'AKIA[0-9A-Z]{16}|aws_secret_access_key' /tmp/test-gather-aws-wiki/` returns zero matches (the subprocess-credential-isolation guarantee from architecture invariant #4 still holds)", "Re-running the same /doc-wiki:ingest command produces an identical structured failure (idempotent error path) — does not crash, does not double-create wiki pages, and either (a) emits a second dispatch entry with the same code or (b) skips via cache and records a 'cache_hit' or 'skipped' entry", "resolveAgentCli({name:'aws'}) succeeds against the published bundle — the resolved CLI lives at node_modules/narai-primitives/dist/connectors/aws/cli.js (with `source: 'bundled-self'` per the 2.1.1+ resolver). A null result here means the bundled-CLI resolver is broken and gather() can't actually dispatch." ] }, { "id": 19, "prompt": "Run /doc-wiki:ingest 'gcp://projects/example-project/services' against my wiki at /tmp/test-gather-gcp-wiki/ (initialize the wiki first with /doc-wiki:init). Do NOT set GOOGLE_APPLICATION_CREDENTIALS in the environment, and ensure `gcloud` is not available on PATH. The point of this eval is to verify the gather() ↔ gcp connector subprocess ↔ credential-resolution boundary survives end-to-end without needing real credentials: gather() should plan the gcp connector, spawn its subprocess, the subprocess should detect missing gcloud / Application Default Credentials inside its own process, and return a structured CONFIG_ERROR envelope to the hub. doc-wiki should record that envelope verbatim in events.jsonl — never a stack trace, never a silent skip.", "expected_output": "Wiki scaffold initialized; gather() dispatches the gcp connector subprocess; the subprocess detects missing gcloud or GOOGLE_APPLICATION_CREDENTIALS via narai-primitives/credentials + the gcloud detector; the connector emits a structured envelope with status='error' and code='CONFIG_ERROR' whose reason references gcloud OR GOOGLE_APPLICATION_CREDENTIALS OR Application Default Credentials; the hub captures the envelope and logs it in events.jsonl; no cleartext credential ever appears in any wiki output. Re-running the same ingest produces an identical structured failure (idempotent error path).", "files": [], "expectations": [ "Wiki scaffold exists at /tmp/test-gather-gcp-wiki/ with the standard dirs (wiki/, raw/, graph/, audit/, log/, outputs/) — confirms /doc-wiki:init ran", "events.jsonl records at least one op='ingest' entry whose details capture the gather() dispatch — either a top-level `consumer` field set to 'doc-wiki' or an embedded `dispatch_steps[]` array referencing the 'gcp' connector", "The gcp dispatch step has a non-success status (e.g., 'error' or 'denied'), with a structured code field — accept any of `code`, `error_code`, or `error.code` set to a CONFIG-class value (e.g., 'CONFIG_ERROR' or 'AUTH_ERROR') — proving the gcp connector's structured-error path fired rather than a thrown exception", "The dispatch step's reason / message text references gcloud OR GOOGLE_APPLICATION_CREDENTIALS OR 'Application Default Credentials' OR narai-primitives/credentials — proves the credential-resolution boundary was reached inside the subprocess (the connector's CONFIG_ERROR text says 'gcloud CLI not available on PATH. Install Google Cloud SDK and authenticate with Application Default Credentials')", "The gcp connector ran in a separate Node process, not in-line in the hub — verifiable via either (a) the dispatch envelope matching narai-primitives' DispatchResult contract (envelope/data/error shape produced by createConnector — not raw object literal from in-process code), or (b) the resolved CLI path under narai-primitives/dist/connectors/gcp/cli.js appearing in the hub log/output", "No stack trace anywhere in the wiki tree (`outputs/`, `raw/`, `wiki/`, `events.jsonl`, `audit/`) — `grep -rE 'at .+\\.js:[0-9]+:|at Object\\.'` returns zero matches; the failure is structured throughout", "The hub process never sees a cleartext credential — `grep -rE 'private_key|service_account|client_email' /tmp/test-gather-gcp-wiki/` returns zero matches (the subprocess-credential-isolation guarantee from architecture invariant #4 still holds)", "Re-running the same /doc-wiki:ingest command produces an identical structured failure (idempotent error path) — does not crash, does not double-create wiki pages, and either (a) emits a second dispatch entry with the same code or (b) skips via cache and records a 'cache_hit' or 'skipped' entry", "resolveAgentCli({name:'gcp'}) succeeds against the published bundle — the resolved CLI lives at node_modules/narai-primitives/dist/connectors/gcp/cli.js (with `source: 'bundled-self'` per the 2.1.1+ resolver). A null result here means the bundled-CLI resolver is broken and gather() can't actually dispatch." ] }, { "id": 20, "prompt": "Run /doc-wiki:ingest 'List the tables in the public schema of the evaldb postgres environment via the schema action' against my wiki at /tmp/test-gather-db-wiki/ (initialize the wiki first with /doc-wiki:init). A real PostgreSQL 16 container is already running on localhost:5455 with database 'evaldb', user 'postgres', password 'eval', and a seeded `users` table containing 3 rows. The connector config references the password via env://POSTGRES_PASSWORD which is set in the test runner. The point of this eval is to verify the gather() SUCCESS path end-to-end via the db connector's `schema` action: planner -> spawn db connector subprocess -> resolve credentials via narai-primitives/credentials -> connect via the postgres driver -> execute the schema introspection -> return a structured success envelope (with table_count > 0 and tables[] including 'users') to the hub. doc-wiki should record that envelope in events.jsonl, no cleartext password should leak into the wiki tree, and a re-run should produce an equivalent success without leaking DB connections.", "expected_output": "Wiki scaffold initialized; gather() plans + dispatches a db connector call with action='schema' against the running postgres container; the subprocess loads its config slice, resolves the password via env:// reference, connects via the pg driver, executes the schema introspection, and returns a non-error envelope (status='success' or 'ok') whose data has table_count > 0 and a tables[] array containing 'users'; doc-wiki logs the envelope in events.jsonl. Re-running produces an equivalent success and leaves no leaked pg connections. No cleartext password appears anywhere in the wiki tree.", "files": [], "expectations": [ "Wiki scaffold exists at /tmp/test-gather-db-wiki/ with the standard dirs (wiki/, raw/, graph/, audit/, log/, outputs/) — confirms /doc-wiki:init ran", "events.jsonl records at least one op='ingest' entry whose details capture the gather() dispatch — top-level `consumer: 'doc-wiki'` AND an embedded `dispatch_steps[]` array referencing the 'db' connector with action='schema' (the schema action specifically — the 2.1.3 fix this regression-tests is the schema action returning correct table_count).", "The db dispatch step has a SUCCESS status — either status='success' (createConnector framework envelope) or status='ok' (legacy dispatcher envelope). No error_code on the success path. This proves the full credential-load -> connect -> read path executed end-to-end inside the connector subprocess.", "The schema action returned table_count > 0 (regression check for the 2.1.2 bug where runSchema reported table_count: 0 even with rows present). The seeded 'users' table is one of the listed tables (visible as a row in data.tables[] or equivalent key referencing the public schema), proving the postgres driver actually executed against the live container (not a stub).", "The db connector ran in a separate Node process — verifiable via (a) the dispatch envelope matching narai-primitives' DispatchResult contract (envelope/data shape produced by createConnector), AND (b) the resolved CLI path under narai-primitives/dist/connectors/db/cli.js appearing in the hub log/output (resolveAgentCli({name:'db'}) returns source='bundled-self').", "Credential resolution went through narai-primitives/credentials — the password was referenced via env:POSTGRES_PASSWORD (or env://POSTGRES_PASSWORD) in the connector config, NOT inlined as plaintext in `~/.connectors/config.yaml` or the workspace's `.connectors/config.yaml`. `grep -rE 'password:[ ]*eval[\"]?\\s*$' workspace_config_files` returns zero matches (the literal password 'eval' must not appear as a plaintext password value).", "No cleartext password leaked into the wiki tree — `grep -rE 'password.*eval|POSTGRES_PASSWORD=eval' work/wiki/ work/raw/ work/log/ work/outputs/` returns zero matches (the password lives only in the runner's env, never serialized into wiki/raw/events output).", "Re-running the same /doc-wiki:ingest command produces an equivalent success envelope (idempotent success path) — same status, same table_count > 0, same 'users' reference. Both runs exit cleanly with no leaked postgres connections (the second run completes without timeout, evidencing that the first run's pg pool was shut down properly).", "resolveAgentCli({name:'db'}) succeeds against the published bundle — the resolved CLI lives at node_modules/narai-primitives/dist/connectors/db/cli.js (with `source: 'bundled-self'` per the 2.1.1+ resolver). A null result here means the bundled-CLI resolver is broken and gather() can't actually dispatch.", "Cleanup is idempotent — `docker stop eval-pg-20` succeeds whether the container is running or already stopped (the test runner tolerates the stop-stop case for repeatable runs)." ] }, { "id": 21, "prompt": "Run /doc-wiki:ingest 'https://stub-jira.local/browse/PROJ-1' against my wiki at /tmp/test-gather-jira-success-wiki/ (initialize the wiki first with /doc-wiki:init). The runner stands up a local HTTP stub on a random free port that mimics Atlassian Cloud REST v3 — it answers GET /rest/api/3/issue/PROJ-1 with {\"key\":\"PROJ-1\",\"fields\":{\"summary\":\"Stub issue summary\"}}. JIRA_SITE_URL points at the stub, JIRA_EMAIL=stub@example.com, and JIRA_API_TOKEN=ATATTfaketoken000000000000 are all set in the runner env (fake values — no real Atlassian PAT). Approach A (local HTTP stub): the jira connector takes a configurable site URL via JIRA_SITE_URL, so the stub server can fully impersonate the API for the success path. The point of this eval is to verify the gather() ↔ jira connector subprocess SUCCESS path end-to-end: planner -> spawn jira connector subprocess -> resolveSecret() returns the fake creds -> JiraClient does a GET against the stub -> success envelope returned to the hub -> doc-wiki logs it. A re-run produces an equivalent success.", "expected_output": "Wiki scaffold initialized; gather() (or dispatchPlan() — the test exercises dispatchPlan to keep the result deterministic without an LLM planner) plans + dispatches a jira get_issue call against the stub on JIRA_SITE_URL; the subprocess loads its config slice, resolves credentials via narai-primitives/credentials, builds a Basic-auth GET, hits the stub, and returns a non-error envelope (status='success'); the data carries the stubbed issue summary; doc-wiki logs the envelope in events.jsonl. Re-running produces an equivalent success and leaves no leaked sockets. No cleartext token appears anywhere in the wiki tree (the fake token still must not leak).", "files": [], "expectations": [ "Wiki scaffold exists at the per-eval `work/` dir with the standard dirs (wiki/, raw/, graph/, audit/, log/, outputs/) — confirms /doc-wiki:init ran", "events.jsonl records at least one op='ingest' entry whose details capture the dispatch — top-level `consumer: 'doc-wiki'` AND an embedded `dispatch_steps[]` array referencing the 'jira' connector with action='get_issue'", "The jira dispatch step has a SUCCESS status — either status='success' (createConnector framework envelope) or status='ok' (legacy dispatcher envelope). No error_code on the success path. Proves the full credential-load -> connect -> read path executed end-to-end inside the connector subprocess against the stub", "The dispatch envelope's data references the stubbed issue — the issue key 'PROJ-1' and summary text 'Stub issue summary' (or fields.summary) appear in the data payload, confirming the JiraClient actually executed a GET against the stub (not a stub-of-the-stub)", "The jira connector ran in a separate Node process — verifiable via (a) the dispatch envelope matching narai-primitives' DispatchResult contract (envelope/data shape produced by createConnector), AND (b) the resolved CLI path under narai-primitives/dist/connectors/jira/cli.js appearing in the cli-resolution.json output (resolveAgentCli({name:'jira'}) returns source='bundled-self')", "No cleartext token leaked into the wiki tree — `grep -rE 'ATATT[A-Za-z0-9_-]{20,}' work/` returns zero matches even though the runner sets a fake JIRA_API_TOKEN; the token lives only in the runner's env", "The stub server received exactly one (or two on re-run) authenticated GETs to /rest/api/3/issue/PROJ-1 — the runner's request log records the path AND a non-empty Authorization: Basic <…> header, proving the connector actually built and sent the Basic-auth round trip end-to-end", "Re-running the dispatch produces an equivalent success envelope (idempotent success path) — same status, same data shape, same PROJ-1 reference. Both runs exit cleanly. The stub server tearing down at end-of-test releases its port (no leaked listeners)", "resolveAgentCli({name:'jira'}) succeeds against the published bundle — the resolved CLI lives at node_modules/narai-primitives/dist/connectors/jira/cli.js (with `source: 'bundled-self'` per the 2.1.1+ resolver). A null result here means the bundled-CLI resolver is broken and gather() can't actually dispatch", "Approach: A (local HTTP stub on a random free port pointing JIRA_SITE_URL at it). Picked because the jira connector reads JIRA_SITE_URL directly from process.env and validateUrl accepts http://127.0.0.1:. No real Atlassian credentials required." ] }, { "id": 22, "prompt": "Run /doc-wiki:ingest 'https://stub-confluence.local/wiki/spaces/PROJ/pages/1' against my wiki at /tmp/test-gather-confluence-success-wiki/ (initialize the wiki first with /doc-wiki:init). The runner stands up a local HTTP stub on a random free port that mimics Atlassian Cloud Confluence REST v2 — it answers GET /wiki/api/v2/pages/1 with a stubbed page body referencing 'Stub page title' and a body in storage representation. CONFLUENCE_SITE_URL points at the stub, CONFLUENCE_EMAIL=stub@example.com, and CONFLUENCE_API_TOKEN=ATATTfaketoken000000000000 are all set in the runner env (fake values — no real Atlassian PAT). Approach A (local HTTP stub): the confluence connector takes a configurable site URL via CONFLUENCE_SITE_URL, so the stub server can fully impersonate the API for the success path. The point of this eval is to verify the gather() ↔ confluence connector subprocess SUCCESS path end-to-end. A re-run produces an equivalent success.", "expected_output": "Wiki scaffold initialized; dispatchPlan() plans + dispatches a confluence get_page call against the stub on CONFLUENCE_SITE_URL; the subprocess loads its config slice, resolves credentials via narai-primitives/credentials, builds a Basic-auth GET, hits the stub, and returns a non-error envelope (status='success'); the data carries the stubbed page title; doc-wiki logs the envelope in events.jsonl. Re-running produces an equivalent success.", "files": [], "expectations": [ "Wiki scaffold exists at the per-eval `work/` dir with the standard dirs (wiki/, raw/, graph/, audit/, log/, outputs/) — confirms /doc-wiki:init ran", "events.jsonl records at least one op='ingest' entry whose details capture the dispatch — top-level `consumer: 'doc-wiki'` AND an embedded `dispatch_steps[]` array referencing the 'confluence' connector with action in {get_page, get_content}", "The confluence dispatch step has a SUCCESS status — either status='success' or status='ok'. No error_code on the success path. Proves the full credential-load -> connect -> read path executed end-to-end inside the connector subprocess against the stub", "The dispatch envelope's data references the stubbed page — the page id '1' and stubbed title text appear in the data payload, confirming the ConfluenceClient actually executed a GET against the stub", "The confluence connector ran in a separate Node process — verifiable via (a) the dispatch envelope matching narai-primitives' DispatchResult contract (envelope/data shape produced by createConnector), AND (b) the resolved CLI path under narai-primitives/dist/connectors/confluence/cli.js appearing in cli-resolution.json (resolveAgentCli({name:'confluence'}) returns source='bundled-self')", "No cleartext token leaked into the wiki tree — `grep -rE 'ATATT[A-Za-z0-9_-]{20,}' work/` returns zero matches even though the runner sets a fake CONFLUENCE_API_TOKEN", "The stub server received the expected GET with a non-empty Authorization: Basic <…> header, proving the connector actually built and sent the Basic-auth round trip end-to-end", "Re-running the dispatch produces an equivalent success envelope (idempotent success path) — same status, same data shape, same page-id reference", "resolveAgentCli({name:'confluence'}) succeeds against the published bundle — the resolved CLI lives at node_modules/narai-primitives/dist/connectors/confluence/cli.js (with `source: 'bundled-self'`)", "Approach: A (local HTTP stub on a random free port pointing CONFLUENCE_SITE_URL at it). Picked because the confluence connector reads CONFLUENCE_SITE_URL directly from process.env. No real Atlassian credentials required." ] }, { "id": 23, "prompt": "Run /doc-wiki:ingest 'https://www.notion.so/Page-abcdef0123456789abcdef0123456789' against my wiki at /tmp/test-gather-notion-success-wiki/ (initialize the wiki first with /doc-wiki:init). This eval is gated behind TEST_LIVE_NOTION=1 plus a real NOTION_TOKEN in the env. When TEST_LIVE_NOTION is unset (default) the eval is marked 'skipped' (not failed) with a clear reason. Approach B (TEST_LIVE_ gate) — picked because the Notion API base URL (api.notion.com) is hardcoded inside @notionhq/client and overriding it cleanly requires either a network-level intercept (DNS hosts file or HTTPS proxy) or a JS-runtime fetch monkey-patch inside the connector subprocess; both are higher-friction than a TEST_LIVE_ gate that exercises the real API.", "expected_output": "When TEST_LIVE_NOTION=1 and NOTION_TOKEN is set in env: gather() (or dispatchPlan) plans + dispatches a notion get_page call; the subprocess resolves NOTION_TOKEN via narai-primitives/credentials, hits api.notion.com, returns a success envelope; doc-wiki logs the envelope in events.jsonl. When TEST_LIVE_NOTION is unset: the runner emits a 'skipped' record with reason='TEST_LIVE_NOTION not set; success-path covered only via the live-API gate'. Skip is NOT a failure — it's an explicit gated-test pattern mirroring narai-primitives' live-DB suite.", "files": [], "expectations": [ "Approach: B (TEST_LIVE_NOTION=1 gate). Picked because notion's API base URL is hardcoded inside @notionhq/client; a clean override requires either a DNS/hosts-level intercept or a fetch-monkey-patch in the connector subprocess — both higher-friction than the gate.", "When TEST_LIVE_NOTION is unset (the default in CI today), the runner emits a 'skipped' record with a clear reason — NOT a failed expectation. The harness reports skip distinct from pass/fail", "When TEST_LIVE_NOTION=1 and NOTION_TOKEN is set: wiki scaffold exists at the per-eval `work/` dir; events.jsonl has an op='ingest' entry referencing the notion connector", "When TEST_LIVE_NOTION=1: the dispatch envelope has SUCCESS status (status='success' or 'ok'), no error_code, and data references the requested page id (the 32-char Notion id)", "When TEST_LIVE_NOTION=1: the notion connector ran in a separate Node process — resolveAgentCli({name:'notion'}) returns source='bundled-self' AND resolvedPath under node_modules/narai-primitives/dist/connectors/notion/cli.js", "Whether TEST_LIVE_NOTION is set or not: no cleartext NOTION_TOKEN appears in the wiki tree — `grep -rE 'secret_[A-Za-z0-9_]{30,}|ntn_[A-Za-z0-9_]{30,}' work/` returns zero matches", "The skip path is documented in grading.json — the eval entry records `status='skipped'` AND `skip_reason` referencing TEST_LIVE_NOTION; this mirrors narai-primitives' live-DB integration test gating", "resolveAgentCli({name:'notion'}) succeeds against the published bundle even on the skip path — proves the bundled-CLI resolver is intact regardless of whether the live API is exercised", "Re-running with the same TEST_LIVE_NOTION setting yields the same status (skipped or pass) — deterministic gate behavior" ] }, { "id": 24, "prompt": "Run /doc-wiki:ingest 'arn:aws:s3:::example-bucket' against my wiki at /tmp/test-gather-aws-success-wiki/ (initialize the wiki first with /doc-wiki:init). This eval is gated behind TEST_LIVE_AWS=1 plus real AWS credentials in env (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY or AWS_PROFILE). When TEST_LIVE_AWS is unset (default) the eval is marked 'skipped' with a clear reason. Approach B — picked because while the AWS SDK v3 supports AWS_ENDPOINT_URL for endpoint override (so a localstack/moto stub is conceptually possible), faithfully mocking S3's signed-request protocol for a non-trivial 'list buckets' or 'describe DB' call requires either localstack (~Docker, large dep) or a hand-rolled SigV4-aware HTTP stub that handles the AWS error envelope. Both are far more friction than a TEST_LIVE_ gate that exercises the real account.", "expected_output": "When TEST_LIVE_AWS=1 and AWS credentials are set: gather() plans + dispatches an aws action (list_buckets or describe_db); the subprocess loads credentials via the AWS SDK default chain (which narai-primitives/credentials defers to), hits the real AWS API, returns a success envelope; doc-wiki logs the envelope. When TEST_LIVE_AWS is unset: the runner emits a 'skipped' record with reason='TEST_LIVE_AWS not set; success-path covered only via the live-API gate'.", "files": [], "expectations": [ "Approach: B (TEST_LIVE_AWS=1 gate). Picked because faithfully stubbing S3's signed-request protocol or any other AWS SDK call requires either localstack/moto (big dep) or a SigV4-aware HTTP stub — both higher-friction than the gate. AWS_ENDPOINT_URL would point an SDK at a stub, but the stub itself is the expensive part", "When TEST_LIVE_AWS is unset (the default in CI today), the runner emits a 'skipped' record with a clear reason — NOT a failed expectation", "When TEST_LIVE_AWS=1 and AWS credentials are set: wiki scaffold exists at the per-eval `work/` dir; events.jsonl has an op='ingest' entry referencing the aws connector", "When TEST_LIVE_AWS=1: the dispatch envelope has SUCCESS status (status='success' or 'ok'), no error_code on the success path, and data references the chosen action's payload (e.g., a 'buckets' array for list_buckets)", "When TEST_LIVE_AWS=1: the aws connector ran in a separate Node process — resolveAgentCli({name:'aws'}) returns source='bundled-self' AND resolvedPath under node_modules/narai-primitives/dist/connectors/aws/cli.js", "Whether TEST_LIVE_AWS is set or not: no cleartext AWS key leaks — `grep -rE 'AKIA[0-9A-Z]{16}|aws_secret_access_key' work/` returns zero matches", "The skip path is documented in grading.json — the eval entry records `status='skipped'` AND `skip_reason` referencing TEST_LIVE_AWS", "resolveAgentCli({name:'aws'}) succeeds against the published bundle even on the skip path", "Re-running with the same TEST_LIVE_AWS setting yields the same status (skipped or pass) — deterministic gate behavior" ] }, { "id": 25, "prompt": "Run /doc-wiki:ingest 'gcp://projects/example-project/services' against my wiki at /tmp/test-gather-gcp-success-wiki/ (initialize the wiki first with /doc-wiki:init). This eval is gated behind TEST_LIVE_GCP=1 plus a working gcloud installation (Application Default Credentials configured) and GCP_PROJECT_ID set in env. When TEST_LIVE_GCP is unset (default) the eval is marked 'skipped' with a clear reason. Approach B — picked because the gcp connector probes for the gcloud binary on PATH and relies on Application Default Credentials; faithfully stubbing both (a fake gcloud shim plus signed-token mock) is far more friction than a TEST_LIVE_ gate.", "expected_output": "When TEST_LIVE_GCP=1 and gcloud is installed + ADC configured: gather() plans + dispatches a gcp action (list_projects, describe_service, etc.); the subprocess shells out to gcloud or uses the SDK with ADC, returns a success envelope; doc-wiki logs the envelope. When TEST_LIVE_GCP is unset: the runner emits a 'skipped' record with reason='TEST_LIVE_GCP not set; success-path covered only via the live-API gate'.", "files": [], "expectations": [ "Approach: B (TEST_LIVE_GCP=1 gate). Picked because the gcp connector probes the gcloud binary plus uses Application Default Credentials; faithfully stubbing both is far more friction than a gate that exercises the real account when configured", "When TEST_LIVE_GCP is unset (the default in CI today), the runner emits a 'skipped' record with a clear reason — NOT a failed expectation", "When TEST_LIVE_GCP=1 and gcloud + ADC are configured: wiki scaffold exists at the per-eval `work/` dir; events.jsonl has an op='ingest' entry referencing the gcp connector", "When TEST_LIVE_GCP=1: the dispatch envelope has SUCCESS status (status='success' or 'ok'), no error_code on the success path, and data references the requested resource (project id or service list)", "When TEST_LIVE_GCP=1: the gcp connector ran in a separate Node process — resolveAgentCli({name:'gcp'}) returns source='bundled-self' AND resolvedPath under node_modules/narai-primitives/dist/connectors/gcp/cli.js", "Whether TEST_LIVE_GCP is set or not: no cleartext credential leaks — `grep -rE 'private_key|service_account|client_email' work/` returns zero matches in materialized wiki output (skip path: zero by definition; live path: ADC tokens never serialize)", "The skip path is documented in grading.json — the eval entry records `status='skipped'` AND `skip_reason` referencing TEST_LIVE_GCP", "resolveAgentCli({name:'gcp'}) succeeds against the published bundle even on the skip path", "Re-running with the same TEST_LIVE_GCP setting yields the same status (skipped or pass) — deterministic gate behavior" ] } ] }