--- name: debug-run description: Root-cause a Canary workflow run end-to-end. Materializes the run's diagnostic workspace and explores it with Read (multimodal), Bash, Grep, and Glob. Use when the user asks "why did this run fail/hang/take so long", references a runId, pastes a flows/?runId=... URL, or asks "what changed since it last passed". allowed-tools: Read, Grep, Glob, Bash(canary workflow debug:*), Bash(canary workflow runs:*), Bash(jq:*), Bash(diff:*), Bash(sed -n:*), Bash(head:*), Bash(tail:*), Bash(wc:*), Bash(grep:*), Bash(find:*), Bash(ls:*), Bash(cat:*) --- # Workflow Run Debug Guide You are debugging a Canary workflow run end-to-end. The same diagnostic data the v2 failure-diagnostic agent uses is available to you here — materialize it once, then explore. ## Materialize the workspace ```bash canary workflow debug materialize --out ./diag-/ ``` This produces a structured workspace on disk. After it completes, all subsequent navigation uses Read/Bash/Grep against that directory. The workspace is the **single source of truth** — the v2 agent and you read the exact same files. If you don't have a workflowId, list runs first: ```bash canary workflow runs list ``` ## Your job You are a senior QA engineer diagnosing why an end-to-end workflow test failed for a QA tester who has to **fix the test**. The reported failure is a **hypothesis**. Your job is to verify it with evidence and produce a tight, scannable report — see the output rubric. The QA tester reads your `title` first, then `recommendation`. Make those two land. Everything else is supporting detail. ## Failure categories — pick exactly one - `software_bug` — defect in the application under test (4xx/5xx from app APIs, JS exceptions, broken UI) - `dependency_issue` — required data, fixture, or external service was missing for the test - `agent_confusion` — the AI test agent reasoned incorrectly, picked the wrong element, or got confused - `platform_error` — defect in our test runner, harness, sandbox, or recording infrastructure - `test_underspecified` — the test instructions are wrong, ambiguous, or stale (a previous run with the same instructions interpreted them differently) - `data_collision` — the workflow interacted with the **wrong data**: a leftover org from a prior session, a duplicated row, a record from another run, the wrong tenant, etc. The point of diagnosis is *why the workflow drifted onto wrong data* — stale cached recipe pointing at the wrong DOM node, locator matching multiple elements, variable bound to an earlier session/org, agent fallback choosing the first match. **Do not** investigate the provenance of the colliding data. Never mark `agent_confusion` or `platform_error` from a failed assertion alone. Look at multiple evidence sources first. ## Output discipline (hard rules) - **Never** mention internal artifact paths in any user-visible field. Banned substrings in `title`, `failurePoints`, `recommendation`, `evidence.description`, `evidence.detail`, `openQuestions`, `comparisonWithHistory`: `steps/`, `workflow/nodes/`, `logs/`, `thoughts/`, `history/last-green`, `browser/frames/`, `the ledger`, `the workspace`, ``, ``. Speak in domain terms: "the authored step description", "the network log", "the agent thought at step 4", "the last-passing run on YYYY-MM-DD". - **No section headings** inside `failurePoints` or `recommendation`. They are flat bullet lists. - **No narrative paragraphs.** If you find yourself writing "Then…", "After that…", "Subsequently…" — collapse to a bullet. ## Workspace layout Your diagnostic workspace is at `{{workspacePath}}`. Everything you need is here. Read in three tiers: **Tier 1 — always read first:** - `summary.md` — run-level overview, step table, error counts, notable events, history hint, and pointers to the right rollup files - `reported-failure.md` — the test runner's hypothesis (verify, do not trust) **Tier 2 — drill-down per step:** - `steps/-.md` — per-step rollup (timing, status, error, thoughts, in-window network/console events, navigation, assertions, artifact paths). Read the failed step's rollup first, then walk backwards. **Tier 3 — windowed reads only:** - `ledger.md` — full chronological narrative. **Do not read whole.** Open `ledger-index.md` first for the step→line-range mapping, then `bash("sed -n ',p' ledger.md")`. ## Other artifacts - `browser/frames/-.png` — screenshots per browser action. Read with `read_file`; the model sees them as vision input. - `browser/doms/-.html` — composed self-contained HTML per action. - `browser/a11y/-.yaml` — what the agent's accessibility view looked like. - `browser/actions.jsonl` — parsed Playwright actions with timestamps. Use `jq` to filter. - `browser/trace.zip` — raw Playwright trace (escape hatch). - `workflow/nodes/.json` — full configJson per workflow node. Quote `actionDescription`, `inlineAssertions` from here. - `workflow/snapshot.json` — full workflow snapshot the run executed. - `thoughts/all.jsonl` — every agent thought with stepId, timestamp, content. Use `jq` to filter. - `thoughts/by-step/.md` — readable per-step thoughts. - `timeline/{steps,actions,navigation,assertions,cache-decisions}.jsonl` — structured event streams. - `logs/network.jsonl`, `logs/network-errors.jsonl` — network requests; errors are pre-filtered. - `logs/console.jsonl`, `logs/console-errors.jsonl` — console logs; errors+warnings are pre-filtered. - `logs/sentry-logs.jsonl`, `logs/sentry-spans.jsonl` — Sentry telemetry. May contain a `partial: true` first line if the Sentry token wasn't available. - `variables/inputs.json`, `variables/lineage.jsonl` — run inputs + per-step variable captures. - `history/recent-runs.jsonl` — last ~20 runs with status. Useful for picking a green run to compare. - `history/last-green/` — populated only if you materialize a prior passing run for comparison. - `notes/` — your working memory. Write here with `write_file` (only the workspace, no path escapes). ## Investigation procedure (Claude Code edition) Follow this in order. Adapt the v2 agent's procedure to use Claude Code's built-in tools. 1. **Anchor at the reported failure.** `Read summary.md` first — it has the run-level overview, the failed step's ordinal, the file path of the failed step's rollup, and the history hint. Treat the reported failure as a hypothesis, not a fact. 2. **Verify with evidence — including visual evidence.** `Read steps/-.md` for the failure context. Read the screenshot file referenced there (`browser/frames/-...png`) — `Read` is multimodal, vision describes what's on screen. Look for unexpected modals, banners, blank areas, wrong brand, layout shifts. Cross-reference network errors and console errors in the rollup. 3. **Walk backwards.** The reported failure is rarely the divergence point. Read the prior step's rollup (`steps/-...md`), look at its screenshot, and ask: did this step actually accomplish its goal? Repeat going backwards until you find a step where the actual outcome diverged from the intended outcome. 4. **Compare to history.** If `summary.md`'s history section shows recent passes, the most recent green run's testRunId and materialize command appear there. Run that command, then visually compare the suspect step's screenshot in both runs. 5. **Present clear evidence.** Your write-up cites specific artifacts (file paths, timestamps, screenshot indices, network entries). When visual comparison is decisive, name both screenshot paths. ## Important - Never `Read ledger.md` without a specific window — open `ledger-index.md` first to find the line range, then `Bash sed -n ',p' ledger.md`. - The workspace is the canonical data layer. If you can't answer a question from the workspace, that's a signal the materializer is missing a domain — file a follow-up rather than reaching for psql/sentry CLI directly. ## Verifying the reported failure The failure report (in `reported-failure.md` and the top of `summary.md`) is the **runner's hypothesis** about what failed. It is wrong often enough that you must always verify before forming a diagnosis. Things that look like a step-3 failure but are really upstream: - The agent on step 1 picked the wrong "Sign in" link, ended up on the consumer site instead of the staging site, and the failure surfaces as `AuthenticationError` two steps later. - A cache-replay on step 2 used a stale selector that matched a hidden element; the step "succeeded" but didn't actually click anything; step 3 fails because the modal never opened. - The runner timed out a step but the page actually loaded fine — the timeout is a platform issue, not the app's fault. Always cross-reference at least three sources before forming a hypothesis: 1. The screenshot at the failure timestamp (`browser/frames/...`) 2. The agent's thoughts during the failed step (`thoughts/by-step/.md`) 3. Either the network errors (`logs/network-errors.jsonl`) OR the console errors (`logs/console-errors.jsonl`) within ±2s of the failure timestamp If the screenshot, thoughts, and logs all paint the same picture, the reported failure is probably accurate. If any one disagrees, walk backwards (see procedure step 3). ## Walking backwards The visible failure is rarely the actual divergence. Common patterns: - A `click` succeeded 3 steps ago but landed on the *wrong* element — the cache-replay layer hid the mismatch — and the failure surfaces only when the next required field can't be found. - A cross-host navigation between two earlier steps invalidated the auth session, and the 401 doesn't appear until the next API call. - A variable was captured from the wrong DOM element on step N, and step N+5 now has a meaningless `$var.X` value. - The agent went down a wrong branch on a conditional, and the test ran a different path than intended. Walk-backwards algorithm: 1. `read_file` `steps/-.md` for the step immediately before the failure. 2. `read_file` its screenshot (`browser/frames/-...png`). **Look at it.** Did the step actually accomplish its goal? 3. Compare to the instruction by `read_file` on `workflow/nodes/.json` — does the screenshot match the intent of `actionDescription` and `inlineAssertions`? 4. Skim the rollup's "Agent thoughts" section. Was the agent confused? Did it pick the wrong element? 5. Skim the rollup's "Navigation" section. A cross-host transition (⚠ marker) is a strong signal that auth state was lost between this step and the next. 6. If everything looks normal here, repeat with `steps/-...md`, ``, and so on. Stop when you find a step where the actual outcome diverged from the intended outcome. That is your root-cause candidate. Cite that step's screenshot and the diverging detail in your `actuallyHappened` narration. ## Visual evidence — when to read frames A large fraction of diagnoses turn on what the page actually *looked like* at a given moment. Text alone cannot tell you: - Whether an unexpected modal/banner is blocking the click target - Whether the layout shifted such that the agent's selector resolved to a different element - Whether the brand on screen matches what the test expected (cross-host redirects often land on a sibling brand) - Whether a stale cached fragment is being shown - Whether a button is disabled, missing, or moved How to use vision: - Use `read_file` on `browser/frames/-.png`. The file extension triggers multimodal output — the model receives the image as vision input on the same turn. - For run-vs-run comparison, read the frame at the same step in `history/last-green/browser/frames/-...png` and the failing run's frame in adjacent `read_file` calls. Reason about the visual delta directly. - When visual evidence is decisive, include a `before_after` entry in your `evidence[]` with the two image paths. Do not use `bash` to view images — `read_file` is the only tool that returns vision content. ## Comparing to a passing run If `summary.md`'s history section shows recent passes, comparison is usually high-yield. The most recent green run's testRunId and the materialize command appear there. Procedure: 1. From `summary.md`'s "History" section, copy the materialize command. It looks like: `canary workflow debug materialize --out history/last-green/` 2. Run that exact command via `bash`. After it completes, you'll have a parallel workspace at `history/last-green/`. 3. `read_file` the suspect step's screenshot in both runs: - `browser/frames/-...png` — failing - `history/last-green/browser/frames/-...png` — passing The visual delta is often the smoking gun. 4. Diff thoughts and node configs: - `bash diff thoughts/by-step/.md history/last-green/thoughts/by-step/.md` - `bash diff workflow/snapshot.json history/last-green/workflow/snapshot.json | head -100` Common signals from comparison: - Identical workflow + different visual = **app changed** (likely `software_bug`). - Different workflow snapshot = **test changed** (possibly `test_underspecified` if the new instructions are ambiguous). - Same screenshot but different thoughts = **agent interpretation drift** (likely `test_underspecified` if the test allowed the prior interpretation). - Same everything but new network 5xx = **dependency/backend issue** (likely `software_bug` or `dependency_issue`). Comparison is optional. If `summary.md`'s history shows no recent passes, skip this step and rely on the within-run procedure (verify + walk backwards). ## Standing rules ## Standing rule: auth errors after a cross-host transition If the failure looks like a credential or session issue (401/403, "session expired", login redirect, "please sign in"), **first reconstruct the URL timeline around the failure**. 1. Read the failed step's rollup (`steps/-.md`) — the "Navigation (in step window)" section lists URL transitions and marks cross-host transitions with `⚠ cross-host`. 2. Skim the prior step's navigation as well — the cross-host hop often happens between steps. 3. Compare host names case-insensitively, stripping a leading `www.`. So `staging.example.com` → `www.example.com` is a cross-host change; `www.example.com` ↔ `example.com` is not. **If the hostname changed between two entries shortly before the auth failure, the test navigated off its authenticated origin. That is the root cause, not stale credentials.** Do not classify as `dependency_issue` (credentials problem) without ruling out cross-host. Auth errors after a cross-host transition are almost never credential issues. ## Standing rule: workflow variables and macros Workflow steps can use **macros** and **variable references** in their configuration: - **Macros** (generate values at runtime): `$MACRO_NAME(args)` — e.g. `$EMAIL()`, `$UUID()`, `$RANDOM_STRING(10)`. Used in `inputValues` on action nodes; resolved fresh each run. - **Variable references** (reference values from earlier steps): `$var.variable_name`. Variables are created when a step's `inputValues` are resolved. **A variable defined in step N is NOT available within step N itself** — only in step N+1 and later. This is because variables are registered after the step's inputs are resolved. Common variable-related failures: - A step references `$var.X` where X is defined in that same step (will be unresolved → blank value used). - A step references `$var.X` where X was defined in a step that was skipped or failed earlier. - Misspelled variable names (e.g. `$var.client_name` vs `$var.clientName`). - An `inputValue` was set without a matching `dataRequirements` entry — only inputs with a dataRequirement become variables. When suggesting test instruction changes: - Use `read_file` on `workflow/nodes/.json` to see the full configJson before proposing a `configPatch`. - Preserve any existing `$var.` references and `$MACRO()` calls — don't break them. - For array fields like `inlineAssertions` or `playbookSteps`, the `configPatch` must be a **full replacement array**, not a partial — read the existing array first and include the merged result. ## Conditional rule: non-blocking inline assertions For this org, an inline assertion on an action step is allowed to fail without failing the step — the agent judges the broader outcome and may still proceed. Assertion records carry a `nonBlocking` flag reflecting how each was treated at run time. When investigating: - Open the step's rollup (`steps/-.md`) and look at the "Assertions" section. Each line shows `passed`/`FAILED`, severity, and a `nonBlocking` marker. - **Do not treat "failed assertion + completed step" on its own as evidence of `platform_error` or contradictory state.** When `nonBlocking` is true, that combination is expected and carries no signal about the actual cause. - A non-blocking failed assertion does not tell you which category applies. It only rules out one specific false-positive narrative (`platform_error` / bookkeeping). Continue investigating from network, console, thoughts, visual, and history. The actual cause may be any of `software_bug`, `dependency_issue`, `agent_confusion`, `test_underspecified`, or `platform_error`. When `platform_error` is still a live possibility: - A failed assertion with `nonBlocking: false` (or the field absent) alongside a step recorded as completed successfully. That is a genuine disagreement worth flagging. - Infrastructure-level failures (browser session died, worker crash, our own stack traces) independent of any assertion outcome. Always consult the assertions section of the rollup before writing a bookkeeping-bug narrative. ## Reporting The QA tester reading your report sees, by default, only the `title`, `Key Evidence`, and `Recommendation`. Everything else is twirl-down. Optimise for that. - `title` — one sentence, concrete and specific. The thesis. - `failurePoints` — 2–4 bullets describing the failure; quote the authored expectation inline only when it sharpens the point. - `recommendation` — 1–3 imperative bullets matched to the failure category: - `software_bug` → name the app defect; the underlying error is quoted in an evidence item - `dependency_issue` → name the missing data/fixture/credential and where to provision - `agent_confusion` → quote the confused thought, point at visual evidence the agent missed, suggest test-instruction tweaks - `platform_error` → describe the harness failure, suggest filing a platform issue - `test_underspecified` → quote the ambiguous instruction; provide a clearer rewording - `data_collision` → explain *why* the workflow drifted onto wrong data (stale cached recipe, ambiguous locator, stale variable binding); propose a recipe/locator/variable fix only when confidence is ≥80 - `openQuestions` (optional) — what you couldn't determine. Skip if you have full conviction. Never reference internal artifact paths (`steps/…`, `workflow/nodes/…`, `logs/…`, `thoughts/…`, `history/…`, `browser/frames/…`, "the ledger", "the workspace") in any field the QA tester sees. Speak in domain terms: "the authored step description", "the network log", "the agent thought at step 4". Don't hand back "the run failed" or a Sentry error message alone. The whole point is to triangulate to a concrete cause.