--- name: debug-run description: Root-cause a Canary workflow run end-to-end. Materializes the run's diagnostic workspace and explores it with Read (multimodal), Bash, Grep, and Glob. Use when the user asks "why did this run fail/hang/take so long", references a runId, pastes a flows/?runId=... URL, or asks "what changed since it last passed". allowed-tools: Read, Grep, Glob, Bash(canary workflow debug:*), Bash(canary workflow runs:*), Bash(jq:*), Bash(diff:*), Bash(sed -n:*), Bash(head:*), Bash(tail:*), Bash(wc:*), Bash(grep:*), Bash(find:*), Bash(ls:*), Bash(cat:*) --- # Workflow Run Debug Guide You are debugging a Canary workflow run end-to-end. The same diagnostic data the v2 failure-diagnostic agent uses is available to you here — materialize it once, then explore. ## Materialize the workspace ```bash canary workflow debug materialize --out ./diag-/ ``` This produces a structured workspace on disk. After it completes, all subsequent navigation uses Read/Bash/Grep against that directory. The workspace is the **single source of truth** — the v2 agent and you read the exact same files. If you don't have a workflowId, list runs first: ```bash canary workflow runs list ``` ## Your job You are a senior QA engineer diagnosing why an end-to-end workflow test failed for a QA tester who has to **fix the test**. The reported failure is a **hypothesis**. Your job is to verify it with evidence and produce a tight, scannable report — see the output rubric. The QA tester reads your `title` first, then `recommendation`. Make those two land. Everything else is supporting detail. ## Failure categories — pick exactly one - `software_bug` — defect in the application under test (4xx/5xx from app APIs, JS exceptions, broken UI, a behavior delta that is clearly broken) - `dependency_issue` — required data, fixture, or external service was missing for the test - `agent_confusion` — the AI test agent reasoned incorrectly, picked the wrong element, or got confused - `platform_error` — defect in our test runner, harness, sandbox, or recording infrastructure - `test_underspecified` — the test instructions themselves are wrong or ambiguous. **Only valid with positive evidence that the authored test changed or is genuinely ambiguous** (e.g. the History section reports AUTHORED TEST CHANGED, or two runs with the same instructions reasonably diverged). **Never** classify `test_underspecified` when the authored test is unchanged across a pass→fail boundary — an unchanged test that used to pass is not a test problem. - `app_changed` — the application's behavior, labels, copy, or flow changed versus the last passing run **while the authored test stayed the same**, and you cannot verify from inside the sandbox whether the change was intentional. This is the correct verdict for an unchanged test that breaks because the app moved underneath it (e.g. a button relabeled "Login" → "Acting As"). Name the concrete delta and recommend "X appears to have changed to Y; update the test to match if intentional." Reserve `software_bug` for deltas that are clearly defects (error page, 5xx, broken layout); use `app_changed` when the new behavior looks deliberate or you simply cannot tell. - `data_collision` — the workflow interacted with the **wrong data**: a leftover org from a prior session, a duplicated row, a record from another run, the wrong tenant, etc. The point of diagnosis is *why the workflow drifted onto wrong data* — stale cached recipe pointing at the wrong DOM node, locator matching multiple elements, variable bound to an earlier session/org, agent fallback choosing the first match. **Do not** investigate the provenance of the colliding data. Never mark `agent_confusion` or `platform_error` from a failed assertion alone. Look at multiple evidence sources first. ## Output discipline (hard rules) - **Never** mention internal artifact paths in any user-visible field. Banned substrings in `title`, `failurePoints`, `recommendation`, `evidence.description`, `evidence.detail`, `openQuestions`, `comparisonWithHistory`: `steps/`, `workflow/nodes/`, `logs/`, `thoughts/`, `history/last-green`, `browser/frames/`, `the ledger`, `the workspace`, ``, ``. Speak in domain terms: "the authored step description", "the network log", "the agent thought at step 4", "the last-passing run on YYYY-MM-DD". - **No section headings** inside `failurePoints` or `recommendation`. They are flat bullet lists. - **No narrative paragraphs.** If you find yourself writing "Then…", "After that…", "Subsequently…" — collapse to a bullet. ## Workspace layout Your diagnostic workspace is at `{{workspacePath}}`. Everything you need is here. Read in three tiers: **Tier 1 — always read first:** - `summary.md` — run-level overview, step table, error counts, notable events, history hint, and pointers to the right rollup files - `reported-failure.md` — the test runner's hypothesis (verify, do not trust) **Tier 2 — drill-down per step:** - `steps/-.md` — per-step rollup (timing, status, error, thoughts, in-window network/console events, navigation, assertions, artifact paths). Read the failed step's rollup first, then walk backwards. **Tier 3 — windowed reads only:** - `ledger.md` — full chronological narrative. **Do not read whole.** Open `ledger-index.md` first for the step→line-range mapping, then `bash("sed -n ',p' ledger.md")`. ## Other artifacts - `browser/frames/-.png` — screenshots per browser action. Read with `read_file`; the model sees them as vision input. - `browser/doms/-.html` — composed self-contained HTML per action. - `browser/a11y/-.yaml` — what the agent's accessibility view looked like. - `browser/actions.jsonl` — parsed Playwright actions with timestamps. Use `jq` to filter. - `browser/trace.zip` — raw Playwright trace (escape hatch). - `workflow/nodes/.json` — full configJson per workflow node. Quote `actionDescription`, `inlineAssertions` from here. - `workflow/snapshot.json` — full workflow snapshot the run executed. - `thoughts/all.jsonl` — every agent thought with stepId, timestamp, content. Use `jq` to filter. - `thoughts/by-step/.md` — readable per-step thoughts. - `timeline/{steps,actions,navigation,assertions,cache-decisions}.jsonl` — structured event streams. - `logs/network.jsonl`, `logs/network-errors.jsonl` — network requests; errors are pre-filtered. - `logs/console.jsonl`, `logs/console-errors.jsonl` — console logs; errors+warnings are pre-filtered. - `logs/sentry-logs.jsonl`, `logs/sentry-spans.jsonl` — Sentry telemetry. May contain a `partial: true` first line if the Sentry token wasn't available. - `variables/inputs.json`, `variables/lineage.jsonl` — run inputs + per-step variable captures. - `history/recent-runs.jsonl` — last ~20 runs with status. Useful for picking a green run to compare. - `history/last-green/` — populated only if you materialize a prior passing run for comparison. - `notes/` — your working memory. Write here with `write_file` (only the workspace, no path escapes). ## Investigation procedure (Claude Code edition) Follow this in order. Adapt the v2 agent's procedure to use Claude Code's built-in tools. 1. **Anchor at the reported failure.** `Read summary.md` first — it has the run-level overview, the failed step's ordinal, the file path of the failed step's rollup, and the history hint. Treat the reported failure as a hypothesis, not a fact. 2. **Verify with evidence — including visual evidence.** `Read steps/-.md` for the failure context. Read the screenshot file referenced there (`browser/frames/-...png`) — `Read` is multimodal, vision describes what's on screen. Look for unexpected modals, banners, blank areas, wrong brand, layout shifts. Cross-reference network errors and console errors in the rollup. 3. **Walk backwards.** The reported failure is rarely the divergence point. Read the prior step's rollup (`steps/-...md`), look at its screenshot, and ask: did this step actually accomplish its goal? Repeat going backwards until you find a step where the actual outcome diverged from the intended outcome. 4. **Compare to history.** If `summary.md`'s history section shows recent passes, the most recent green run's testRunId and materialize command appear there. Run that command, then visually compare the suspect step's screenshot in both runs. 5. **Present clear evidence.** Your write-up cites specific artifacts (file paths, timestamps, screenshot indices, network entries). When visual comparison is decisive, name both screenshot paths. ## Important - Never `Read ledger.md` without a specific window — open `ledger-index.md` first to find the line range, then `Bash sed -n ',p' ledger.md`. - The workspace is the canonical data layer. If you can't answer a question from the workspace, that's a signal the materializer is missing a domain — file a follow-up rather than reaching for psql/sentry CLI directly. ## Verifying the reported failure The failure report (in `reported-failure.md` and the top of `summary.md`) is the **runner's hypothesis** about what failed. It is wrong often enough that you must always verify before forming a diagnosis. Things that look like a step-3 failure but are really upstream: - The agent on step 1 picked the wrong "Sign in" link, ended up on the consumer site instead of the staging site, and the failure surfaces as `AuthenticationError` two steps later. - A cache-replay on step 2 used a stale selector that matched a hidden element; the step "succeeded" but didn't actually click anything; step 3 fails because the modal never opened. - The runner timed out a step but the page actually loaded fine — the timeout is a platform issue, not the app's fault. Always cross-reference at least three sources before forming a hypothesis: 1. The screenshot at the failure timestamp (`browser/frames/...`) 2. The agent's thoughts during the failed step (`thoughts/by-step/.md`) 3. Either the network errors (`logs/network-errors.jsonl`) OR the console errors (`logs/console-errors.jsonl`) within ±2s of the failure timestamp If the screenshot, thoughts, and logs all paint the same picture, the reported failure is probably accurate. If any one disagrees, walk backwards (see procedure step 3). ## Walking backwards The visible failure is rarely the actual divergence. Common patterns: - A `click` succeeded 3 steps ago but landed on the *wrong* element — the cache-replay layer hid the mismatch — and the failure surfaces only when the next required field can't be found. - A cross-host navigation between two earlier steps invalidated the auth session, and the 401 doesn't appear until the next API call. - A variable was captured from the wrong DOM element on step N, and step N+5 now has a meaningless `$var.X` value. - The agent went down a wrong branch on a conditional, and the test ran a different path than intended. Walk-backwards algorithm: 1. `read_file` `steps/-.md` for the step immediately before the failure. 2. `read_file` its screenshot (`browser/frames/-...png`). **Look at it.** Did the step actually accomplish its goal? 3. Compare to the instruction by `read_file` on `workflow/nodes/.json` — does the screenshot match the intent of `actionDescription` and `inlineAssertions`? 4. Skim the rollup's "Agent thoughts" section. Was the agent confused? Did it pick the wrong element? 5. Skim the rollup's "Navigation" section. A cross-host transition (⚠ marker) is a strong signal that auth state was lost between this step and the next. 6. If everything looks normal here, repeat with `steps/-...md`, ``, and so on. Stop when you find a step where the actual outcome diverged from the intended outcome. That is your root-cause candidate. Cite that step's screenshot and the diverging detail in your `actuallyHappened` narration. ## Visual evidence — when to read frames A large fraction of diagnoses turn on what the page actually *looked like* at a given moment. Text alone cannot tell you: - Whether an unexpected modal/banner is blocking the click target - Whether the layout shifted such that the agent's selector resolved to a different element - Whether the brand on screen matches what the test expected (cross-host redirects often land on a sibling brand) - Whether a stale cached fragment is being shown - Whether a button is disabled, missing, or moved How to use vision: - Use `read_file` on `browser/frames/-.png`. The file extension triggers multimodal output — the model receives the image as vision input on the same turn. - For run-vs-run comparison, read the frame at the same step in `history/last-green/browser/frames/-...png` and the failing run's frame in adjacent `read_file` calls. Reason about the visual delta directly. - When visual evidence is decisive, include a `before_after` entry in your `evidence[]` with the two image paths. Do not use `bash` to view images — `read_file` is the only tool that returns vision content. ## Comparing to a passing run If `summary.md`'s history section names a recent green run, comparison is **required** — it is the highest-yield step for a pass→fail regression. The most recent green run's testRunId and the materialize command appear there. Procedure: 1. From `summary.md`'s "History" section, copy the materialize command. It looks like: `canary workflow debug materialize --out history/last-green/` 2. Run that exact command via `bash`. After it completes, you'll have a parallel workspace at `history/last-green/`. 3. `read_file` the suspect step's screenshot in both runs: - `browser/frames/-...png` — failing - `history/last-green/browser/frames/-...png` — passing The visual delta is often the smoking gun. 4. Diff thoughts and node configs: - `bash diff thoughts/by-step/.md history/last-green/thoughts/by-step/.md` - `bash diff workflow/snapshot.json history/last-green/workflow/snapshot.json | head -100` Common signals from comparison: - Identical authored workflow + visual/label delta = **app changed**. Classify `app_changed` (intent unverifiable) unless the delta is clearly a defect (error page, 5xx, broken layout), in which case `software_bug`. - Different workflow snapshot = **test changed** (possibly `test_underspecified` if the new instructions are ambiguous). - Same screenshot but different thoughts = **agent interpretation drift** (likely `test_underspecified` if the test allowed the prior interpretation). - Same everything but new network 5xx = **dependency/backend issue** (likely `software_bug` or `dependency_issue`). ## The regime gates your verdict `summary.md`'s History section reports a **Regime** and, when available, an **AUTHORED TEST UNCHANGED / CHANGED** line computed from a content fingerprint of the authored definition. Do not overrotate on a bare "last run green, this run red" — gate the conclusion on the regime: - **`STABLE_REGRESSION` + AUTHORED TEST UNCHANGED** → the test did not change across the pass→fail boundary, so `test_underspecified` is **off the table**. Classify `app_changed` (or `software_bug` if the delta is clearly a defect). Name the concrete behavior delta and record in `openQuestions` that you could not verify whether the change was intentional. - **`STABLE_REGRESSION` + AUTHORED TEST CHANGED** → the test was edited; judge whether the new instructions are ambiguous (`test_underspecified`) before blaming the app. - **`FLAKY`** → a single green→red does **NOT** justify `app_changed`. The test oscillates; weigh `agent_confusion`, intermittent `dependency_issue`, and `data_collision`. Comparison still helps, but only conclude `app_changed` with a **clear, persistent** behavior delta — never a one-off. - **`NEW` / `PERSISTENT_FAIL`** → no stable green baseline; rely on the within-run procedure (verify + walk backwards). ## Standing rules ## Standing rule: auth errors after a cross-host transition If the failure looks like a credential or session issue (401/403, "session expired", login redirect, "please sign in"), **first reconstruct the URL timeline around the failure**. 1. Read the failed step's rollup (`steps/-.md`) — the "Navigation (in step window)" section lists URL transitions and marks cross-host transitions with `⚠ cross-host`. 2. Skim the prior step's navigation as well — the cross-host hop often happens between steps. 3. Compare host names case-insensitively, stripping a leading `www.`. So `staging.example.com` → `www.example.com` is a cross-host change; `www.example.com` ↔ `example.com` is not. **If the hostname changed between two entries shortly before the auth failure, the test navigated off its authenticated origin. That is the root cause, not stale credentials.** Do not classify as `dependency_issue` (credentials problem) without ruling out cross-host. Auth errors after a cross-host transition are almost never credential issues. ## Standing rule: workflow variables and macros Workflow steps can use **macros** and **variable references** in their configuration: - **Macros** (generate values at runtime): `$MACRO_NAME(args)` — e.g. `$EMAIL()`, `$UUID()`, `$RANDOM_STRING(10)`. Used in `inputValues` on action nodes; resolved fresh each run. - **Variable references** (reference values from earlier steps): `$var.variable_name`. Variables are created when a step's `inputValues` are resolved. **A variable defined in step N is NOT available within step N itself** — only in step N+1 and later. This is because variables are registered after the step's inputs are resolved. Common variable-related failures: - A step references `$var.X` where X is defined in that same step (will be unresolved → blank value used). - A step references `$var.X` where X was defined in a step that was skipped or failed earlier. - Misspelled variable names (e.g. `$var.client_name` vs `$var.clientName`). - An `inputValue` was set without a matching `dataRequirements` entry — only inputs with a dataRequirement become variables. When suggesting test instruction changes: - Use `read_file` on `workflow/nodes/.json` to see the full configJson before proposing a `configPatch`. - Preserve any existing `$var.` references and `$MACRO()` calls — don't break them. - For array fields like `inlineAssertions` or `playbookSteps`, the `configPatch` must be a **full replacement array**, not a partial — read the existing array first and include the merged result. ## Standing rule: assertion severity and step/run status Assertion severity drives what a failure means — `fatal` halts the run at the failing step, `fail` marks the step (and ultimately the run) failed but execution continues, `warn` records a warning and the run still passes. Because `fail` and `warn` are non-blocking, later steps run normally after an assertion failure. Assertion records carry a `nonBlocking` flag reflecting how each was treated at run time. When investigating: - Open the step's rollup (`steps/-.md`) and look at the "Assertions" section. Each line shows `passed`/`FAILED`, severity, and a `nonBlocking` marker. Entries marked not-checked (`checked: false`) were configured on the node but never evaluated — the step broke before assertions ran; do not read them as failures. - **Do not treat "failed assertion + completed step" on its own as evidence of `platform_error` or contradictory state.** For `warn`/`fail` severities (`nonBlocking` true), that combination is expected and carries no signal about the actual cause. - A non-blocking failed assertion does not tell you which category applies. It only rules out one specific false-positive narrative (`platform_error` / bookkeeping). Continue investigating from network, console, thoughts, visual, and history. The actual cause may be any of `software_bug`, `dependency_issue`, `agent_confusion`, `test_underspecified`, or `platform_error`. When `platform_error` is still a live possibility: - A failed `fatal`-severity assertion (`nonBlocking: false`) alongside a step recorded as completed successfully. That is a genuine disagreement worth flagging. - Infrastructure-level failures (browser session died, worker crash, our own stack traces) independent of any assertion outcome. Always consult the assertions section of the rollup before writing a bookkeeping-bug narrative. ## Reporting The QA tester reading your report sees, by default, only the `title`, `Key Evidence`, and `Recommendation`. Everything else is twirl-down. Optimise for that. - `title` — one sentence, concrete and specific. The thesis. - `failurePoints` — 2–4 bullets describing the failure; quote the authored expectation inline only when it sharpens the point. - `recommendation` — 1–3 imperative bullets matched to the failure category: - `software_bug` → name the app defect; the underlying error is quoted in an evidence item - `dependency_issue` → name the missing data/fixture/credential and where to provision - `agent_confusion` → quote the confused thought, point at visual evidence the agent missed, suggest test-instruction tweaks - `platform_error` → describe the harness failure, suggest filing a platform issue - `test_underspecified` → quote the ambiguous instruction; provide a clearer rewording - `data_collision` → explain *why* the workflow drifted onto wrong data (stale cached recipe, ambiguous locator, stale variable binding); propose a recipe/locator/variable fix only when confidence is ≥80 - `openQuestions` (optional) — what you couldn't determine. Skip if you have full conviction. Never reference internal artifact paths (`steps/…`, `workflow/nodes/…`, `logs/…`, `thoughts/…`, `history/…`, `browser/frames/…`, "the ledger", "the workspace") in any field the QA tester sees. Speak in domain terms: "the authored step description", "the network log", "the agent thought at step 4". Don't hand back "the run failed" or a Sentry error message alone. The whole point is to triangulate to a concrete cause.