# Tool contract Related docs: - [`../README.md`](../README.md) - [`REQUIREMENTS.md`](REQUIREMENTS.md) - [`ARCHITECTURE.md`](ARCHITECTURE.md) - [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md) - [`ELECTRON.md`](ELECTRON.md) - [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md) ## Browser tools Always-on tools are `agent_browser` (native argv), `agent_browser_code` (fresh JavaScript over a persistent browser), and `agent_browser_tools` (advanced capability discovery/activation). Optional credential-backed `agent_browser_web_search` remains independent. Pi 0.87.0 or newer is required; official and fork qualification is independent. Five specialized tools are registered but initially inactive: `agent_browser_action`, `agent_browser_qa`, `agent_browser_electron`, `agent_browser_source`, and `agent_browser_network_source`. Explicit CLI allowlists, filtered SDK catalogs, and restored Pi tool history are respected. Omit CLI `--tools` for lazy activation, or include every desired advanced tool in its allowlist; the loader reports excluded tools as unavailable rather than bypassing host selection. Pi 0.87 does not distinguish an SDK explicitly selecting the complete browser catalog from its default catalog; SDK hosts choosing activation for that complete catalog can call `session.setActiveToolsByName(...)` after `bindExtensions()`. ## Why this tool shape This keeps the integration: - thin - powerful - low-drift - low-maintenance - close to upstream `agent-browser` It also keeps the main UX where it belongs: the agent invokes the tool directly instead of relying on bash or a large manual command surface. The tool guidance should be written for task discovery first, not wrapper implementation first. That means the description should emphasize browser use cases like reading live pages, clicking, filling, screenshots, extraction, and authenticated/profile-based workflows. Live/current external facts belong on `agent_browser_web_search` when that companion tool is registered. Low-level wrapper details like `stdin` and exact CLI args belong in the schema and guidelines, not the lead description. The tool also needs an operating playbook, not just a capability list. The model should not have to rediscover basics each session, but always-on guidance must stay concise. The canonical agent-facing playbook lives in `extensions/agent-browser/lib/playbook.ts`; it provides compact runtime rules plus absolute installed-package paths to `README.md`, `docs/COMMAND_REFERENCE.md`, and this contract so agents with file tools can read targeted guidance on demand instead of receiving the full docs in prompt context. Generated Markdown fragments are updated by `npm run docs -- playbook write`, and `npm run docs -- playbook check` fails when checked-in documentation drifts. The native command reference in `docs/COMMAND_REFERENCE.md` is driven by the same pattern: `scripts/agent-browser-target.mjs` owns the runtime version and `scripts/agent-browser-capability-baseline.mjs` imports it alongside help/doc inventory; selected regions are generated into the Markdown by `npm run docs -- command-reference write`, and `npm run docs` plus `npm run verify -- command-reference` catch drift (the latter also samples the installed `agent-browser` on `PATH`). Maintainer workflow details live in `AGENTS.md` under upstream capability baseline. ## Host execution hook An SDK host can import the package's compiled `dist/extensions/agent-browser/index.js` default export and register it once through a Pi extension factory: ```ts agentBrowserExtension(pi, { async beforeExecute(toolCallId, ctx) { await saveHostState(toolCallId, ctx.signal); }, }); ``` `beforeExecute` is an optional host callback, not a tool input or package config field. It receives the original outer Pi tool-call ID and the current `ExtensionContext`, with `ctx.signal` set to the dispatched call's abort signal. Hosts must honor that signal when waiting so Stop remains responsive. The extension awaits it after input resolution succeeds and before each ordinary dispatch, including each accepted `browser(...)` call inside code. The code wrapper itself does not call it; inner calls retain the original outer ID and use their own cancellation signals. The existing inner-call queue remains serial even for `Promise.all`, so a completed inner call's files are available to the next callback. Internal helper probes and cleanup do not call it, nor does each row of a native `batch` get a separate callback. Native setup and helper probes do not constitute host callback invocations. Supplying the callback registers `agent_browser` with Pi's native `executionMode: "sequential"`. Pi then finishes earlier sibling tools before entering the callback and dispatching the browser call. Rejection prevents that dispatch: direct calls become Pi tool errors, while code inner calls receive a failed observation. The host owns saving/retry policy; the extension adds no checkpoint store, retry loop, or deadline. Code deadlines and child reaping still apply; finishing code does not close its selected browser. Omitting the callback preserves ordinary scheduling and behavior; the separate web-search tool is unchanged. ## Optional companion web search `agent_browser_web_search` is a separate custom tool, not an `agent_browser` input mode. It is available when the extension can see at least one configured/resolvable Exa or Brave credential source from `~/.pi/config/pi-agent-browser-native/config.json`, `.pi/config/pi-agent-browser-native/config.json`, `PI_AGENT_BROWSER_CONFIG`, or the `EXA_API_KEY` / `BRAVE_API_KEY` environment fallbacks, and runtime execution still checks that the final available merged config has not set `webSearch.enabled` to `false`. Config layers merge global → project → `PI_AGENT_BROWSER_CONFIG` override; under Pi 0.84.0+, globally installed and CLI-loaded copies read `.pi/config/...` when Pi trust allows that project layer, and they skip the project layer when Pi reports the project is untrusted or when launched with `--no-approve`. Disable scope is explicit: a global disable is a normal user default, a project disable applies to one repo, and an override file with `webSearch.enabled: false` is the highest-priority hard disable for that run. Credential sources may be plaintext, `$ENV_VAR` / `${ENV_VAR}` interpolation, escaped literals, or command sources such as `"!op read 'op://Private/Exa/API Key'"` from any loaded config layer; they make the tool available without exposing the value in status text, and command values resolve when the tool executes. Browser profile/executable config uses the same paths and emits prompt guidance from the highest-priority loaded layer, including project config when that layer is loaded. Prefer it for live/current external web facts, current docs/news, and candidate URLs. Prefer it over browser-driving public search-engine forms such as Google: headless typing flows may be redirected to anti-bot or CAPTCHA pages, and agents should use search API results, then `agent_browser` on a target URL, instead of attempting CAPTCHA bypass. Use `agent_browser` when the task needs browser interaction, screenshots, authenticated/profile content, page inspection, or DOM work. The search tool is namespaced to avoid colliding with generic `web_search`, chooses Exa or Brave automatically from available credentials, defaults to Exa when both are available (unless `webSearch.preferredProvider` is set), and must not expose resolved API keys in content, details, errors, status output, docs examples, logs, or PR artifacts. Config shape: ```json { "webSearch": { "enabled": true, "preferredProvider": "exa", "defaultSearchType": "deep-lite", "exaApiKey": "$EXA_API_KEY", "braveApiKey": "$BRAVE_API_KEY" } } ``` Schema: ```json { "query": "search text", "provider": "auto", "searchType": "deep-lite", "includeDomains": ["docs.exa.ai"], "excludeDomains": ["archive.example"], "category": "publication", "additionalQueries": ["Exa search API defaults"], "highlightsDynamic": false, "count": 5, "offset": 0, "country": "US", "searchLang": "en-US", "safesearch": "moderate", "freshness": "pw" } ``` Provider notes: - `provider` is optional; `auto` uses available keys plus `webSearch.preferredProvider`. - `searchType` applies to Exa only and supports `auto`, `fast`, `instant`, `deep-lite`, `deep`, and `deep-reasoning`. Effective precedence is the per-call field, then `webSearch.defaultSearchType`, then `auto`. Use `deep-lite` for implementation research, `deep` for hard multi-source work, and `deep-reasoning` only for the hardest or exhaustive work. - `includeDomains` and `excludeDomains` accept 1–20 Exa hostname, path-prefix, or wildcard-subdomain strings. `category` accepts `company`, `people`, `publication`, `news`, `personal site`, or `financial report`. `company` and `people` reject `freshness` and `excludeDomains` before a request is sent. - `additionalQueries` accepts 1–10 strings only when the effective type is `deep-lite`, `deep`, or `deep-reasoning`. HTTP timeouts are 15 seconds for non-deep types, 45 seconds for `deep-lite`, 60 seconds for `deep`, and 90 seconds for `deep-reasoning`. - Exa requests use `/search` with `contents.highlights: true` for compact excerpts and a fixed `systemPrompt` asking for primary official sources, requested versions/dates, and distinct results. `highlightsDynamic: true` switches to `{ "dynamic": true }` and sends Exa's required beta header. The wrapper intentionally does not expose full page text or structured-output schemas. - Brave-specific `searchLang` is ignored by Exa. Exa maps `country` to `userLocation`, `safesearch` moderate/strict to `moderation: true`, and `freshness` to `startPublishedDate`. Brave keeps ignoring `searchType`; explicitly requested newer Exa-only filters fail before a Brave request. - Requests are serialized. Agents should run one focused query, inspect it, and make at most one follow-up rather than parallel searches. HTTP 429 means stop and report that the provider plan or limit needs time or a change. - After normalization, the wrapper removes later results whose normalized URL is exactly equal to an earlier result, preserves first-result order, does not collapse distinct paths/query URLs by guesswork, and does not overfetch replacements. `details.duplicatesRemoved` is present only when rows were removed, so `details.results.length` can be smaller than the requested `count`. Result details: ```json { "provider": "exa", "query": "search text", "returnedQuery": "search text", "count": 5, "offset": 0, "searchType": "deep-lite", "requestId": "request-id-when-provider-returns-one", "duplicatesRemoved": 1, "fetchedAt": "2026-06-02T00:00:00.000Z", "results": [ { "title": "Result title", "url": "https://example.com/", "description": "Compact summary or first highlight", "highlights": ["Relevant excerpt"], "source": "Example", "pageDate": "2026-06-02", "language": "en" } ] } ``` For Exa, `details.searchType` is the effective requested type even when the provider response does not echo it. `requestId` is included when Exa returns one. Brave omits both fields. `pageDate` comes from Exa `publishedDate` (an estimated page creation date) or Brave `page_age`; Brave may separately return `age`. These provider fields are not crawl/retrieval age or proof that a result matches a requested version. For version-sensitive work, inspect the result, constrain one follow-up to the primary domain (`includeDomains` for Exa or `site:` in a Brave query), and read the primary page. ## Input mode chooser | Need | Tool | Input | | --- | --- | --- | | One native command | `agent_browser` | `{ args, stdin?, outputPath?, timeoutMs?, sessionMode? }` | | Fixed sequence | `agent_browser` | `args: ["batch", "--bail"]` and native JSON-array `stdin` | | Loops, branches, aggregation | `agent_browser_code` | `{ code, session?, namespace?, timeoutMs?, outputPath? }` | | Discover/enable advanced tools | `agent_browser_tools` | `{ enable?: ["action", "qa", "electron", "source", "network"] }` | | Stable locator or native select | `agent_browser_action` | Flat action/locator fields | | Diagnostic verdict | `agent_browser_qa` | Flat URL or attached-check fields | | Desktop app lifecycle | `agent_browser_electron` | Flat action/target fields | | UI/source candidates | `agent_browser_source` | Flat selector/fiber/component fields | | Failed-request/source candidates | `agent_browser_network_source` | Flat request/filter fields | Run `agent_browser_tools {}` for inventory; `enable` adds selected capabilities without removing unrelated active tools. Pi's native selected-tool history handles reload/resume rather than a package-owned activation registry. If an exact recovery action names an inactive advanced tool, enable its capability before using that payload. Use exact labels from the latest snapshot. Return to the model when a fresh observation needs judgment. Neither native batch nor code is a named recipe registry. ## Snapshot and getter batching - **`snapshot -i`**: default for interaction—interactive `@eN` refs, main-content-first trimming, and the usual click/fill workflow. - **`snapshot --compact`**: denser same-page tree when you still need refs but want less output than full interactive snapshot. - **Full `snapshot`** (no `-i`): use only when you need the complete accessibility tree; expect larger output and possible spill files. - Re-run `snapshot -i` after navigation, scrolling, rerendering, or other major DOM changes; refs are page-scoped. - When you need **three or more** `get title` / `get url` / `get text` / similar reads for known refs or selectors on the same page, prefer one `batch` stdin array (for example `[["get","text","@e1"],["get","text","@e2"]]`) instead of serial tool calls. ### Native observation options (0.38+) `args: ["snapshot", "-i", "--delta"]` retains the native `data.snapshot` object with `kind`, `revision`, and either a full `tree`/`refs`, an unchanged `baseRevision`, or `changes` plus `treeChange`. Full states use the normal snapshot presenter; partial states show native revision data. The wrapper obtains complete refs with one ordinary native snapshot for a partial result, without changing the native delta baseline, so direct/batch follow-up refs and transcript restore remain usable. If that read fails, refs are invalidated rather than recorded as an empty page. `--delta --full` forces a native full baseline. Wrapper search/filter/viewport/diff flags use a full tree instead of native delta output. `screenshot --if-changed` and `--threshold <0-1>` preserve native `changed`, `revision`, `pixelChangeRatio`, and `threshold`. An unchanged response has no path, artifact, or image attachment—even if the caller requested an existing file. It succeeds as an observation, not proof that an evidence file was saved. `record start` / `restart` accept `--cursor`, `--contact-sheet`, and `--contact-sheet-threshold `. Reported `contactSheetPath` values become image artifacts: pending during capture, checked on disk after stop, and attached when verified. The contact-sheet destination shares the active video's reservation and transcript lifecycle; same-call/batch artifact and `outputPath` aliases are rejected before dispatch. A successful direct `record restart` retains the prior take's known contact-sheet path from the existing reservation, checks it on disk, and labels it unverified because native restart omits terminal sheet evidence. Use `record stop` then a new start when verified completion of both artifacts is required. Video receipt handling remains separate. Native `--input-mode ` is a sticky, live-changeable session setting, not a launch flag; per-action pointer options pass through. See the [command reference](COMMAND_REFERENCE.md#upstream-0381-rebaseline). ## Wrapper `--json` When explicitly requested, visible JSON remains parseable for early preparation results too: snapshot/network filters, scroll results, and preparation failures use the same JSON envelope as ordinary execution. Diagnostic metadata remains in `details`; `outputPath` notices do not append prose to JSON. Help/version stays native text. The extension always plans normal browser commands with `--json` prepended in `effectiveArgs` so upstream returns structured JSON for presentation and `details`. Omit `--json` in caller `args` for ordinary prose. Include it when you need the visible tool text as a parseable JSON envelope; structured `details` remain available in both modes. Plain-text inspection (`--help`, `--version`) keeps its own output shape. Read-only skills and local/setup commands such as auth/profile/setup, `session list`, and syntactically local state lifecycle operations skip implicit session injection as documented under `sessionMode`. Upstream session/state rows and targets remain visible, and state/config/path operations pass through unchanged. ## Experimental WebMCP Upstream 0.36.0 exposes page-registered tools through ordinary `args`: `webmcp list [tool] [--frame ]`, `webmcp invoke [--params ] [--frame ] [--detach] [--timeout ]`, `webmcp result `, and `webmcp cancel `. Locally managed Chrome enables the experimental CDP feature by default. `--no-webmcp` / `AGENT_BROWSER_NO_WEBMCP` / upstream config `noWebmcp` disables it; attached browsers, providers, Lightpanda, Safari/iOS, and older Chrome builds may return upstream `webmcp_unsupported`. Native 0.37 can include `data.webmcp` on successful navigation. When `available` is true and `toolCount` is a positive integer, the page summary shows the native availability hint and recommends `webmcp list`. Raw metadata remains in `details.data`; absent, unavailable, zero or invalid counts add no hint. The wrapper does not run a discovery probe. Native 0.38 also emits bounded catalog updates on normal responses when page tools change. Their names/descriptions and frame/origin metadata are shown as untrusted page data, including an empty catalog after removal; full schemas still require `webmcp list`. Because native updates are emitted once, helper reads also contribute to the current call's `details.webMcpCatalog`. A helper-only update is shown in prose without changing the main command's `details.data`; caller-requested JSON stays parseable. Collection is call-local, including concurrent sessions and code inner calls, not a persistent catalog cache. The wrapper keeps this as thin CLI pass-through. `webmcp list` is read-only. `invoke`, `result`, and `cancel` may run page code that mutates, rerenders, or navigates, so the wrapper rechecks the live target, emits the normal `pageChangeSummary` and `inspect-after-mutation` follow-up when applicable, and stores `refSnapshotInvalidation.reason: "page-transition"`; old page-scoped refs remain blocked until a fresh `snapshot -i`. A direct or batched call whose result is still `pending`, or a failed `result` / `cancel` attempt made while that target is unknown, does not treat the immediate URL probe or a same-batch snapshot as stable: `details.sessionTabTargetUnknown` stays true until a successful settlement, `get url`, or explicit navigation verifies the page. Its `details.nextActions` replaces the blocked snapshot suggestion with `verify-page-target-after-pending-webmcp` (`get url`); the action warns that URL inspection does not settle the detached page tool. Inside one `batch --bail`, put `get url` after a completed WebMCP mutation and before `snapshot -i`; a snapshot directly against the unknown target remains blocked. Detached invocation ids and page-returned data remain in `details.data`. When top-level `timeoutMs` is omitted, `webmcp invoke` and `webmcp result` extend the wrapper subprocess watchdog to the upstream `--timeout` value plus a small grace window, including effective raw-argument batch rows (which take precedence over stdin exactly as upstream does). `--no-webmcp` is launch-scoped for both bare/`true` and explicit `false` values. Put it on the first call for a session or use `sessionMode: "fresh"` after an implicit managed session exists. The upstream `webmcp-gen` skill is available through stateless `skills get webmcp-gen`; an external MCP server can opt in with `mcp --tools core,webmcp`, but bare long-running `mcp` remains unsuitable for a one-shot Pi tool call. ## Dashboard reverse-proxy origins Upstream 0.35.2 adds `dashboard start --allowed-origins ` and `AGENT_BROWSER_DASHBOARD_ALLOWED_ORIGINS` for comma-separated exact HTTPS reverse-proxy origins. The wrapper treats both explicit `dashboard start` and the no-subcommand `dashboard` equivalent as sessionless local lifecycle commands when they use `--port`, `--allowed-origins`, or `--json`; it does not add a browser session or its own dashboard access layer. ## Headed and local fixture limits Local Chrome uses stock `--args --no-startup-window` at bootstrap and composes it consistently with caller-configured Chrome arguments (CLI overrides environment, which overrides native config). Plain active-session follow-ups do not resend bootstrap settings. A pre-existing custom-argument browser may restart once because stock Chrome launch arguments form part of native session identity; there is no migration or launch-settings journal. Headless remains the default. CDP/auto-connect, providers, Electron, and Lightpanda do not receive this Chrome default. The wrapper does not delete tabs or edit profiles. URL-less `open` is executed as native `get url`, including effective raw/stdin batch rows and code calls. This lazily opens a missing browser without navigating an existing page or issuing a second incomplete launch. `details.args` retains the request; `effectiveArgs`, `command`, native URL data and lifecycle show what ran. No launch receipt is synthesized. Explicit URLs, required-URL errors for `goto`/`navigate`, and top-level help/version argv keep their native meanings. Ignored batch stdin remains unchanged. - Upstream 0.35.0 and newer require separate `args` entries for global flag values. The wrapper rejects `--flag=value` global tokens before normal command dispatch, including trailing tokens; `--restore=` is the explicit upstream-supported exception. Plain help/version inspection preserves exact caller argv, matching upstream. These exceptions are top-level only: global flags for `batch` belong before `batch`, and row-local equals forms fail validation before dispatch. - A bare `--no-sandbox` in the command slot, or an option position after `open` / `goto` / `navigate`, fails validation before dispatch with Chromium launch-argument guidance. The former is an unknown upstream command; navigation ignores the latter. Pass the switch as a separate `--args` value and use `sessionMode: "fresh"` for launch changes. For native batches, `--args` belongs on the outer call before `batch`, not in a row. Other commands keep their native literal text, script, select, path and option-value handling; top-level help/version inspection remains unchanged. - `--headed` is an upstream global flag passed through `args` (for example `{ "args": ["--headed", "open", "https://example.com"], "sessionMode": "fresh" }`). Use it on the first launch for demos, human-observed QA, or a user-completed login. If a managed browser session already exists, use `sessionMode: "fresh"` so the launch-scoped headed/headless choice is not ignored. Wrapper-owned headed launches default `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` to `0` because upstream 0.33.2's multi-origin restore collector opens visible temporary tabs and can delay session policy inspection. The wrapper records the effective launch-time interval and reapplies it to every helper and follow-up subprocess, still-owned off-current session, Electron cleanup close, and transcript reload/resume so daemon configuration remains stable. Native close still saves, but direct window close can lose newer state because upstream exempts headed browsers from idle shutdown; set an explicit interval before launch when periodic preservation matters. Because upstream reads it when the daemon starts, changing the recorded effective interval in either direction on a running wrapper-owned headed session is rejected until close plus a fresh launch. - `--profile ` is upstream Chrome profile selection. `profiles` lists Chrome profile directory names from Chrome's user data directory; `Default` is common but not guaranteed. On profile/user-data-dir failures, use `details.nextActions` or run `profiles` / `doctor`, then tell the user which profile name/path to configure before retrying. - `--executable-path ` selects a custom Chromium-compatible browser executable when upstream can launch it. Use it with `sessionMode: "fresh"` when switching from an already-active implicit session. For non-Chrome Chromium login state, use a full profile/user-data directory path only when upstream accepts it, or attach to a debug-enabled running browser with `--auto-connect` / `connect` when appropriate. - `--ca-cert ` and `--no-ca-cert` are launch-scoped under the wrapper and require a fresh managed session. Upstream 0.35.0 imports PEM/DER CA material into an isolated NSS store for locally launched Linux Chromium; the wrapper disables automatic managed restore when CA trust is enabled and passes caller-selected paths through unchanged. - `--allowed-domains ` is launch-scoped under the wrapper and requires a fresh local Chrome context. As of upstream 0.32.0, it contains workers and popups and disables Chromium `RTCPeerConnection`; upstream rejects combinations with CDP/auto-connect, profiles, restore/state replay, direct-page providers, iOS/Safari, and startup/profile Chrome args because those paths cannot guarantee containment. - On a successful first/fresh local wrapper-managed headed launch whose upstream lifecycle says a browser launched, `details.browserWindow` is `{ mode: "headed", ownership: "wrapper-managed", sessionName, visibility: "unverified" }` and model-visible output adds one headed-login handoff sentence. It proves the requested local headed launch path, not OS desktop visibility; CDP, auto-connect, provider, and Electron attachments do not receive this field or handoff. If the user can see it, they can finish the login and the agent should continue with `sessionMode: "auto"`; otherwise inspect `screenshot`, `tab list`, `get url`, or `snapshot -i` and treat the problem as display/provider/session setup. - `localhost` / `127.0.0.1` URLs are resolved by the browser host, which may differ from the shell or Pi process that started a temporary server. Errors such as `net::ERR_EMPTY_RESPONSE` on local ports are not reliable page-render evidence; they can mean the browser cannot reach the host loopback. Use an environment-specific host-reachable HTTP(S) address. Caller-selected `file://` navigation and follow-up inspection pass through unchanged. - `file://` pages do not provide HTTP headers and can differ from HTTP pages for MIME handling, CORS, storage, and debugger/script behavior. If `eval --stdin` returns `null` or otherwise fails to prove DOM state on a `file://` page, first confirm the script was passed through the native tool `stdin` field (not as a third `args` item after `--stdin`), then treat that verification as inconclusive and use `snapshot -i`, `get text` from current refs, screenshots, or a reachable HTTP fixture instead. - Temporary HTTP servers launched outside the tool are host-owned. The native tool does not allocate ports, track background server PIDs, or clean them up; use a harness or shell cleanup for those processes. - Choose agent_browser for one native command, native batch --bail with JSON-array stdin for a known fixed sequence, or agent_browser_code for loops, branches, and aggregation. Return to the model when a fresh observation needs judgment. Neither tool is a named recipe registry. - agent_browser_code takes { code, session?, namespace?, timeoutMs?, outputPath? }. JavaScript globals are fresh for every call; the selected browser persists and is shared with agent_browser. Use await browser({ args, stdin?, timeoutMs? }), inspect result.success, and emit selected JSON instead of whole envelopes. Use emitImage(result.imageObservations[0]) only when image inspection is needed; image handles belong to the current code call. No imports or host filesystem/network/process APIs are exposed. - Standard workflow: open the page, snapshot -i, interact using current @refs from that snapshot, and re-snapshot after navigation, scrolling, rerendering, or other major DOM changes because refs are page-scoped; the wrapper fails mutation-prone stale/recycled refs before upstream can silently target a different current-page element. On dense pages, use wrapper-side snapshot -i --search or snapshot -i --filter role= to render matching refs while preserving the full ref map in details.refSnapshot, add snapshot --viewport when scroll position or above/below-fold context matters, and add snapshot --diff when a quick before/after ref-map delta would prevent reading a full spill file. - For ordinary forms from one snapshot, batch multiple fill @refs before the submit/click step to avoid serial tool calls; if a fill may autosubmit, navigate, or rerender later fields, split the flow and refresh refs first. - Do not use browser automation to drive public search-engine forms such as Google for discovery; headless jobs that type a query and press Enter can be redirected to anti-bot or CAPTCHA pages. Prefer agent_browser_web_search for live discovery, then agent_browser on a target URL. Do not attempt CAPTCHA bypass. - Snapshot choice: prefer snapshot -i for routine clicks/fills (interactive @refs, main-content-first). Use snapshot --compact when you need a denser same-page tree without full spill; use full snapshot (no -i) only when you need the complete accessibility tree. Re-snapshot after navigation or major DOM changes. When snapshot -i compacts because the tree is oversized, scan visible output for Omitted high-value controls and optional details.data.highValueControlRefIds before opening the spill file: those list bounded searchboxes, textboxes, comboboxes, buttons, named action links, tabs, checkboxes, radios, options, and menuitems that did not fit the key/other ref previews. - When a visible text or accessible-name target should survive ref churn, prefer find locators such as role, text, label, placeholder, alt, title, or testid with the intended action instead of guessing a CSS selector. - For desktop or host-controlled rich inputs, if agent_browser_action fill misses, refresh refs and prefer a current editable @ref from details.richInputRecovery or the latest snapshot; focus or click that ref, then use keyboard type for framework-controlled editors that require real key events. keyboard inserttext is paste-like and can change a DOM value without updating application state, so use it only when later application-state evidence proves the edit was accepted. Do not auto-submit with Enter or a submit button unless the user flow explicitly calls for it. - Do not assume Playwright selector dialects such as text=Close or button:has-text('Close') are supported wrapper syntax unless current upstream agent-browser behavior has been verified. - For authenticated or user-specific content explicitly requested by the user, such as feeds, inboxes, account pages, or private dashboards, use a real profile only when the user/config asks for it or profiles have been inspected; do not assume --profile Default exists on every machine. Do not use a real profile for public pages just because they are dashboards. Treat visible page content from real profiles as model-visible transcript data. On macOS, copied Chrome profiles may omit encrypted cookies, so profile selection alone is not proof of authentication; verify the target page and use a user-approved headed login once when needed. Use --auto-connect only if profile-based reuse is unavailable or the task is specifically about attaching to a running debug-enabled browser. If profile/user-data-dir resolution fails, stop retrying opens, run profiles and/or doctor through agent_browser, then report what the user needs to configure. - Ordinary bare calls share one native browser within your root Pi parent/subagent group; unrelated roots browse independently. Coordinate multi-call navigation inside your group with existing subagent/intercom tools. Explicit native session defaults still win. Pi quit does not close group browsers; never close an unrelated browser. - When using launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --ca-cert, --no-ca-cert, --enable, --executable-path, --webgpu, --no-webmcp, --init-script, --idle-timeout, --args, --user-agent, --headed, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), put them on the first command for that session. If you intentionally use an explicit --session, keep using that same explicit session for follow-ups. - Cooperating Pi processes serialize each complete browser operation and code cell by native socket context, namespace, and session, including target/ref helpers and state updates. Different identities remain concurrent; namespace-wide close waits for matching operations. All participating Pi instances must load this release; humans, third-party clients, and older extensions are outside that coordination. It is not a transaction or rollback. For raw batches whose later content depends on navigation, use exact batch --bail or split the calls. - After a successful `connect`, `--cdp`, or enabled `--auto-connect` call, verify with get url and keep using the resulting session without repeating the attach flag. The wrapper remembers that attachment across active-branch reload/resume and live-checks the URL before later page reads/interactions because an attached browser can drift externally; caller config, file access, launch arguments, and environment pass through unchanged. A successful close clears the marker. When several named sessions share one Chrome, pass --pin-tab once (AGENT_BROWSER_PIN_TAB) so a closed bound tab fails as tab_gone instead of acting on a neighbor; recover with tab new or tab list. --no-pin-tab turns the sticky pin off. tab list includes each tab's CDP targetId, accepted as a tab ref. - If you already used the implicit session and now need launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --ca-cert, --no-ca-cert, --enable, --executable-path, --webgpu, --no-webmcp, --init-script, --idle-timeout, --args, --user-agent, --headed, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), retry with top-level sessionMode set to fresh or pass an explicit --session for the new launch; never pass --session-mode inside args. After a successful unnamed fresh launch, later auto calls follow that new session. - For WebGPU pages, use args ["--webgpu", "open", ""] on a fresh local browser launch; use doctor --webgpu (or --headed on Linux/Windows capture paths) to prove rendering before trusting a non-black screenshot. WebGPU cannot be combined with --cdp, --auto-connect, or provider launches unless --webgpu false overrides an enabled config/environment default. - For experimental WebMCP page tools, use webmcp list, then webmcp invoke with --params and optional --frame/--detach/--timeout; use webmcp result or cancel for detached calls. Locally managed Chrome enables WebMCP by default; a positive navigation hint means the page has tools to list. --no-webmcp is launch-scoped and requires a fresh session; invoke/result/cancel can mutate or navigate, so refresh snapshot refs afterward. - For --allowed-domains, use a fresh local Chrome context. Upstream rejects CDP/auto-connect, profiles, restore/state replay, direct-page providers, iOS/Safari, and startup/profile Chrome args because they cannot guarantee containment; Chromium also disables RTCPeerConnection while the allowlist is active. - For React introspection, launch the page with --enable react-devtools before first navigation, then use react tree, react inspect , agent_browser_source candidates for local UI source hints, react renders start/stop, or react suspense; agent_browser_source is experimental and reports confidence/evidence instead of guaranteed DOM-to-file mappings. For failed fetches and APIs, agent_browser_network_source (experimental) correlates failed network requests with initiator metadata and bounded workspace URL literals—candidates only, not definitive blame. Use vitals [url] for Core Web Vitals and hydration timing, and pushstate for client-side SPA navigation. - For first-navigation setup, use open without a URL plus network route --resource-type , cookies set --curl , or --init-script/--enable before navigate/opening the target page. - For stateful browser context work, use auth save --password-stdin with the tool stdin field for credentials, auth list/show/delete/remove for local auth-profile maintenance, auth login when you need the browser to fill a saved profile, and state save/load/list/show/rename/clear/clean for upstream saved-state lifecycle. State paths, restore identifiers, wrapper-prefixed sessions, and all upstream list rows remain available; credential values inside cookie/storage/auth payloads are still redacted from presentation. - Upstream restore sessions periodically autosave cookies and localStorage while the browser stays open, including page-driven background changes; AGENT_BROWSER_AUTOSAVE_INTERVAL_MS controls the interval (30000 by default; 0 disables periodic saves but keeps save-on-close), while the never value for --restore-save disables automatic saves for that restore session. For wrapper-owned headed launches, the wrapper defaults the interval to 0 because upstream 0.33.2 collects multi-origin storage through visible temporary tabs, then records and reapplies the effective launch-time value to helpers and follow-ups so daemon configuration remains stable. Native close still saves, but direct window close can lose newer state because upstream exempts headed browsers from idle shutdown; set AGENT_BROWSER_AUTOSAVE_INTERVAL_MS before launch when periodic preservation matters, because changing it on a running wrapper-owned headed session requires close plus a fresh launch. - For batch chains that touch cookies, storage, auth, or other secret-bearing commands, use details.batchSteps for per-step artifacts, categories, spill paths, and full structured errors; top-level details.data on batch is only a compact redacted step matrix (success, argv-redacted command, redacted result or scrubbed error text) built from the same presentation rules as standalone calls. - For non-core families, pass current upstream commands through the native tool directly: network requests, network route , network har start/stop [path], diff snapshot, diff screenshot --baseline , diff url , trace start, trace stop [path], profiler start, profiler stop [path], record start , record stop, console/errors [--clear], highlight , inspect, clipboard read, clipboard write , clipboard copy/paste, stream enable/disable/status, dashboard start/stop, device list for iOS simulator inventory, and chat . For compact network requests output, prefer details.nextActions for request detail, route-mock diagnostics, actionable failed-request agent_browser_network_source, filtering, clearing the aggregate buffer before repro, or HAR capture follow-ups instead of guessing request-id syntax. Artifact-producing commands report details.artifacts and verification state; long-running starts such as stream, dashboard, trace/profiler, and record should be paired with the matching stop/disable command when the task is done; stream enable already-enabled outcomes are treated as idempotent success with status/disable follow-ups. - For Electron desktop apps, enable agent_browser_electron through agent_browser_tools for wrapper-owned discovery, isolated launch, status, compact probe, and cleanup: list first, treat likely-sensitive annotations as hints rather than enforcement, launch with the default snapshot handoff unless handoff: "tabs" is the safer diagnostic starting point, use agent_browser_electron probe or snapshot -i/agent_browser_qa attached:true for current-session state, and always cleanup the returned launchId when done. agent_browser_electron launch uses an isolated temporary profile; it does not reuse the app's normal signed-in profile or attach to an already-running authenticated app. For signed-in local app state, host-launch the normal app with --remote-debugging-port when appropriate, then use raw args connect ; after connect, run get url to verify the active target before page-content reads, inspect tab list, select the stable tab id such as tab t2, verify it again with get url, then run a condition wait or snapshot -i before using refs. close commands (`close`, `quit`, or `exit`) only close the browser/CDP session; leave manually launched app shutdown, profile cleanup, and explicit artifacts to the host owner. - For provider or specialized app workflows, load version-matched upstream guidance with skills get agentcore|electron|slack|dogfood|vercel-sandbox|derive-client through the native tool; add --full when you need references/templates, and use skills get --all only for broad skill audits. Use derive-client when recording HAR traffic to generate a standalone API client; prefer network har start (text bodies by default) or network har start --content all|none before multi-step capture. For accessibility audits use a11y or a11y --tags wcag2a,wcag2aa (CDP browsers only). Hosted sandbox workflows should use upstream @agent-browser/sandbox helpers outside this wrapper. Provider launches such as -p ios, --provider browserbase/kernel/browseruse/browserless/agentcore, and iOS --device are upstream-owned setup paths; use sessionMode fresh when switching providers and expect external credentials or local Appium/Xcode setup to be required. - For dialogs and frames, use dialog status/accept/dismiss and frame through native args; dialog commands and eval snippets that look like alert/confirm/prompt/dialog triggers are shorter-bounded than normal browser calls, and timed-out dialog-like interactions may add inspect-dialog-after-timeout, dismiss-dialog-after-timeout, or recover-fresh-session-after-dialog-timeout nextActions. When --confirm-actions produces a pending confirmation, use details.nextActions or exact confirm / deny calls instead of inventing ids. - If a session lands on the wrong page or tab, an interaction changes origin unexpectedly, or an open call returns blocked, blank, or otherwise unexpected results, use tab list / tab / snapshot -i to recover state before retrying different URLs or fallback strategies. For headed demos, put --headed on the first launch with sessionMode=fresh and verify with screenshot/tab/get-url evidence because tool success cannot prove the OS window is visible to the user. For desktop readiness, prefer real conditions first: wait --text, wait --url, wait --fn, wait --load , wait --download, or agent_browser_qa attached:true; for disappearance checks, use wait --fn predicates instead of stale upstream-help examples like wait --state hidden. Use agent_browser_electron probe/status for wrapper-owned launch health or target mismatch. Fixed waits are a last resort: their duration is positional (wait , not wait --time ). Use explicit --timeout or top-level timeoutMs for legitimately slow waits, and treat a successful payload like "waited":"timeout" as elapsed time only—verify completion with an observed condition, fresh snapshot, or screenshot. - For feed, timeline, or inbox reading tasks, focus on the main timeline/list region and read the first item there rather than unrelated composer or sidebar content. - For read-only browsing tasks, use read for documentation or other unstructured text without requiring a Chrome page, or read with no URL for rendered active-tab DOM. Prefer the current snapshot, structured ref labels, getters, or scoped eval --stdin when you need interactive structure or targeted page state. Only click into media viewers, detail routes, or new pages when the current view does not contain the needed information. - For downloads, prefer download when an element click should save a file; native download owns the click and download, including loopback links, generated Blob exports, and redirects. Do not rely on click alone when you need the downloaded file on disk. - On dashboards with nested scroll containers, verify scroll with a screenshot or fresh snapshot -i; if the viewport did not move, details.data.scrolled may be false/noMovement true and you should prefer scrollintoview <@ref> or target the actual scrollable region with scroll [px|percent]. For native selects, use select (or agent_browser_action select) instead of clicking option refs; for custom comboboxes, a click/agent_browser_action may only focus the field, so re-snapshot and use keyboard type for focused input, press ArrowDown or press Enter, or visible option refs. Raw type requires both and . - When using eval --stdin, scope checks and actions to the target element or route whenever possible instead of relying on broad page-wide text heuristics. - When using eval --stdin for extraction, pass the JavaScript through the native tool stdin field, not as an extra args token after --stdin, and return the value you want instead of relying on console.log as the primary result channel. Prefer plain expressions like ({ title: document.title }) or explicitly invoked functions like (() => ({ title: document.title }))(); use outputPath when the eval/get/snapshot data should be saved as a durable local file, but never reuse a screenshot, download, recording, or other browser artifact destination as outputPath. If a function-shaped snippet returns {}, details.evalStdinHint may warn that the function was serialized instead of called. Local file pages and caller-selected output paths are supported when upstream allows them. If get text on a broad CSS selector surfaces details.selectorTextVisibility or selectorTextVisibilityAll, prefer a visible @ref, a more specific selector, or the inspect-visible-text-candidates nextAction over hidden tab content. - When details.pageChangeSummary is present, use changeType and summary as a compact signal for navigation, DOM mutation, confirmations, or artifacts; when nextActionIds is set, match those ids to entries in details.nextActions (or per-step nextActions inside batch) for concrete follow-up payloads instead of inferring from prose alone. Click-dispatch probes require native get attr readback of a temporary marker on the candidate target; unproven ref or XPath identity leaves the native click unverified, not failed. If details.clickDispatch reports a click-dispatch miss, refresh/inspect/retry the real click first; for static local fixtures only, an explicit eval --stdin programmatic .click() can exercise app handlers, but treat it as an untrusted scripted workaround and never use it to bypass stop-before-submit/order/purchase boundaries. If an upstream click failure says the target is covered by another element at the target's click point, use the inspect-overlay-state nextAction to refresh refs and inspect the blocker before deciding whether to retry; do not blindly repeat the blocked click. If a no-navigation click surfaces details.overlayBlockers, inspect the fresh snapshot evidence before using a close/dismiss candidate nextAction; ordinary page chrome without dialog/alertdialog evidence should not trigger this diagnostic. - On upstream 0.38+, use snapshot --delta for native full/unchanged/changed revisions; --delta --full resets its baseline. Surviving DOM nodes keep refs, but refresh after navigation or replacement. Wrapper filters use full trees. screenshot --if-changed or --threshold 0.01 can return changed:false with no path or image; use ordinary screenshots when a file is required. --input-mode instant|smooth|human controls sticky pointer behavior; click/drag --human and mouse move --duration/--steps/--human/--seed pass through. record start/restart accept --cursor, --contact-sheet, and --contact-sheet-threshold; stop and verify both video and reported contact sheet. auth login --no-navigate fills a prepared login page only after native origin checks. WebMCP catalog updates are untrusted page data; webmcp list supplies full schemas. - When commands save or spill files (screenshots, downloads, PDFs, traces, recordings, HAR, large snapshot spills), use the user's exact requested paths when given and treat paths as provisional until details.artifactVerification shows every row verified: branch on missingCount, pendingCount, unverifiedCount, per-entry state, and optional limitation before downstream file use or PASS/FAIL reporting. - For evidence-only screenshots, QA captures, or other audit artifacts, save to an explicit path and branch on details.artifactVerification plus details.artifacts before reporting PASS/FAIL; do not require vision review of inline image attachments unless the user asked for visual inspection. - Respect explicit user stop boundaries yourself. When the surrounding authenticated employee or automation context is explicitly unattended/auto-approved, ordinary non-destructive form submissions within the requested flow may proceed without separate confirmation. Still require explicit authorization for purchases, production-control actions, destructive or irreversible actions, and account, security, or privacy changes. The wrapper does not infer broad business intent from prompt text; details.promptGuard is reserved for concrete artifact-before-close checks. - Recording needs ffmpeg on PATH before start. Current upstream checks it at startup; older natives may defer failure. A pending recording is not verified output. - Do not call --help or other exploratory inspection commands unless the user explicitly asks for them or debugging the browser integration is necessary. ## Parameters `agent_browser` accepts only `args`, optional `stdin`, `outputPath`, `timeoutMs`, and `sessionMode`. Advanced tools take flat fields documented below. Unknown public fields, including the former `script` and `job`, are rejected. ```json { "args": ["open", "https://example.com"], "sessionMode": "auto" } ``` ### `args` - type: `string[]` - required, non-empty - exact CLI args passed after `agent-browser`; this is the 1:1 upstream CLI coverage path for the targeted `agent-browser` version - no shell operators - do not include the binary name - omit `--json` for prose; include it for parseable model-visible JSON (the wrapper injects upstream JSON internally either way) - first-call recipe: `open` → `snapshot -i` → `click` / `fill` with current `@eN` refs from that snapshot → `snapshot -i` again after navigation or DOM changes Examples: ```json { "args": ["open", "https://example.com"] } { "args": ["snapshot", "-i"] } { "args": ["click", "@e2"] } { "args": ["tab", "list"] } { "args": ["network", "unroute"] } { "args": ["quit"] } ``` ### Execution directory Relative file operands, `outputPath`, and source/network-source scans use one `pi-change-working-dir` owner reply captured before browser policy and queues; code calls inherit it. The integration uses the synchronous `pi-change-working-dir:resolve-execution-cwd` event with `{ sessionManager, result?: { cwd, error? } }`. Owner errors or invalid replies fail validation. No owner means native `ctx.cwd`; a known older owner identified through public tool/command package provenance requires an update and Pi restart. No private directory journal is read. Browser/session/config ownership stays separate. Ordinary calls retain the native project or managed launch root, while an unnamed fresh launch and explicit native config resolve against the execution directory. `details.managedSessionCwd`, when present, records the managed launch root for transcript replay, including ordered code transitions. Pi package config/trust and artifact-store identity remain unchanged. A deleted launch directory returns `validation-error` before dispatch; restore it or explicitly choose fresh/config from the new directory. See [working-directory changes](../README.md#working-directory-changes). ### `agent_browser_code` Strict JSON input: `{ code, session?, namespace?, timeoutMs?, outputPath? }`. `code` is a nonempty async JavaScript body, capped at 65,536 characters by schema and 65,536 UTF-8 bytes at execution. No `args`, top-level `stdin`, or `sessionMode` fields are accepted here. - `session` selects a nonempty native session name. Omission follows the same native configured/root/managed default as direct calls; the last explicit direct call does not change that default slot. - `namespace` selects a native namespace; `""` explicitly selects the default namespace. The parent resolves one fixed socket/namespace/session identity before executing the cell. - `timeoutMs` is a positive integer: 120,000 ms default, 300,000 ms maximum. Admission, local queue waits, the shared lock, and inner work use the cell deadline; each inner timeout is clamped to the remaining budget. - `outputPath` saves result data through the ordinary artifact-aware writer and must not alias browser artifacts. The browser persists between code and direct calls. JavaScript variables do not. Native batch, auth, profile, state, connect, local commands, and close remain available through `browser()`, using ordinary parent-side config/environment and ownership rules. Inner commands cannot select another session/namespace or run `close --all`; use `agent_browser` outside the cell for namespace-wide cleanup or a fresh launch. Closing inside a cell closes only its fixed browser identity. A later call may relaunch it according to normal native rules. If it closes a wrapper-managed fresh browser and reopens that fixed name, the reopen is an explicit caller-owned session; the automatic managed slot has already rotated. Close that reopened name explicitly when finished. #### Code globals and output - `await browser({ args, stdin?, timeoutMs? })` returns a JSON-cloned [canonical observation](#canonical-observation): check `success` before consuming `data`. Native batch is permitted, but nested native batch rows remain unsupported by the ordinary executor. - `emit(value)` selects JSON output. One emission becomes `data`; multiple emissions become an ordered array. If there are no emissions, the async body's return value is used; an absent return omits data. - `emitImage(observation.imageObservations[0])` selects a verified real image handle from this cell. Paths, arbitrary objects, and handles from earlier cells are not image inputs. Selection rechecks file size/mtime and ordinary inline-image eligibility. Duplicate selection is coalesced. At most eight selected images and 20 MiB total are allowed, subject to the configurable inline bound (5 MiB per image by default via `PI_AGENT_BROWSER_INLINE_IMAGE_MAX_BYTES`). Selected images are real Pi image content blocks, not JSON strings. ```json { "code": "const page = await browser({ args: ['get', 'title'] }); if (!page.success) throw new Error(page.error); emit({ title: page.data.title ?? page.data.result });" } ``` ```json { "code": "const shot = await browser({ args: ['screenshot', '/tmp/page.png'] }); if (!shot.success) throw new Error(shot.error); emit(shot.artifactVerification); if (shot.imageObservations?.length) emitImage(shot.imageObservations[0]);" } ``` Inner calls are serialized, including calls issued with `Promise.all`. The maximum is 25 attempted calls, 64 KiB emitted JSON, 1 MiB per IPC message, and 8 MiB cumulative IPC. Inner execution keeps full redacted data without full model rendering/spilling/image encoding; only selected outer output is rendered. An observation that cannot fit the bridge limit fails rather than silently discarding recovery or fabricating data. Source runs in a fresh permissioned Node child with an empty child environment, 64 MiB V8 heap ceiling, and a VM context whose string/WebAssembly code generation is disabled. Task globals have null prototypes and JSON-only IPC; no parent promise/object/function is exposed. There are no host filesystem, network, process, import, or timer APIs. This language boundary is separate from native browser capabilities: parent-side browser subprocesses still receive ordinary auth/config/environment. `node:vm` alone is not a security boundary, and this contract is not a universal sandbox certification. #### Failure and persistence `codeRun` reports `callCount`, `emitCount`, `failedCallCount`, `rejectedCallCount`, and timeout/abort flags. `codeSteps` retains bounded audit rows. Handled browser failures may coexist with successful source completion, but remain model-visible in `failures` with exact recovery evidence and in the failure count. A successful code result therefore does not assert that every inner command succeeded. Invalid inner input shapes make the outer result fail even if source handles their observations. Uncaught source exceptions use `script-error`; deadline and cancellation use `timeout` and `aborted`; output/bridge validation retains its specific failure. Pi persistence is required: `--no-session` fails before code execution. Before each accepted browser dispatch, the parent appends an ordered `agent-browser-transition` intent; completion appends reducer-consumed state from the same executor as direct calls. This includes target/ref invalidation, attachment, close/relaunch, and existing ownership evidence. Browser state uses existing replay reducers; there is no persistent JavaScript registry. An interrupted intent leaves the target unknown until inspection. If completion persistence fails after a mutation, the failure says the command may already have run; do not retry blindly. Abort, timeout, branch change, reload, and shutdown stop/reap the child and settle the active inner call before releasing coordination. They do not roll back browser actions or automatically close the selected browser. Existing ownership still controls later shutdown cleanup. The only retained pre-0.7 script machinery reads exact old isolated-session cleanup leases during upgrade; it does not accept old public script execution. ### `agent_browser_action` Enable `action` through `agent_browser_tools`, then call the tool with flat fields. Optional `outputPath` and `timeoutMs` apply to the ordinary executor; this tool has no `sessionMode` field. | Field | Contract | | --- | --- | | `action` | Required: `click`, `check`, `fill`, or `select` | | `selector` | Direct CSS/selector/current-ref target; cannot combine with locator fields for click/check/fill | | `locator` | `role`, `text`, `label`, `placeholder`, `alt`, `title`, or `testid`; select supports only role/label | | `value` | Locator value or one select option; for label select, this is the label | | `values` | One or more select options; required for label select; rejected on other actions | | `text` | Fill text; required for fill, rejected on select | | `role`, `name` | Accessible role and optional name; role may replace locator value and must agree if both are given | | `session` | Explicit native session; preserved in compiled commands and recovery | Locator click/check/fill compiles to native `find`; direct targets compile to their ordinary command. Select uses native `select `. Accessible select resolves exactly one current visible combobox/listbox through a fresh snapshot; missing or ambiguous targets fail before action. Use `role: "combobox"` or `"listbox"` plus `name`, or label plus option `values`. No native `find ... select` capability is assumed. ```json { "action": "fill", "locator": "label", "value": "Email", "text": "user@example.com" } { "action": "select", "locator": "role", "role": "combobox", "name": "Flavor", "value": "chocolate" } { "action": "select", "locator": "label", "value": "Flavor", "values": ["chocolate"] } ``` `details.compiledSemanticAction` retains the compiler's internal name and redacted input/argv; `effectiveArgs` shows actual current-ref resolution. Same-session URL verification, snapshot resolution, and action share the execution lock. Stale locator failures can offer a same-target `find` retry after snapshot refresh; stale direct/select refs get refresh guidance only. Selector misses may add exact visible ref candidates or bounded button/link name candidates. Fill recovery omits submitted text and never submits: focus/click a current editable ref, use native `keyboard type` if needed, and verify application state. `uncheck` remains a direct native command, not a shorthand action. ### `agent_browser_qa` Enable `qa`. Flat input is either `{ url, ...checks }` or `{ attached: true, ...checks }`; these forms are exclusive. Checks are optional `expectedText` (string or string array), `expectedSelector`, `screenshotPath`, `checkNetwork`, `checkConsole`, `checkErrors`, and `loadState` (`domcontentloaded` default, `load`, or `networkidle`). `outputPath`, `timeoutMs`, and `sessionMode` are supported; attached QA only accepts `sessionMode: "auto"`. ```json { "url": "https://example.com", "expectedText": "Example Domain", "screenshotPath": "/tmp/qa.png" } { "attached": true, "expectedText": "Explorer", "checkErrors": true } ``` QA compiles a fixed `batch --bail` diagnostic plan. URL QA clears enabled network/console buffers, captures page-error residue after native clear, opens the requested URL, waits for readiness, allows a bounded 150 ms diagnostic settle when needed, then performs visible-text/selector assertions, enabled diagnostic reads, and optional screenshot. Expected text uses bounded visible-element predicates with a 5-second native timeout. Missing assertions stop before later diagnostics. URL QA does not require the previous page target. Attached QA checks the current browser without navigation or buffer clears. It first requires a live nonempty URL, including file/custom app URLs. Failed target verification returns exact tab/URL recovery instead of running the batch. Diagnostic reads default off because preserved buffers may predate this check; URL diagnostics default on. Explicit `checkErrors: false` skips the error-baseline check. The verdict is `qaPreset: { passed, failedChecks, warnings, summary }`. Console errors, definitely new page errors, ambiguous matching post-clear residue, actionable failed requests, failed batch steps, or failed visible assertions cause `qa-failure`. Matching a nonempty error baseline is an **unverified check**, not proof of a new application error. A clean final buffer can pass. Benign low-impact icon misses may be warnings when the failed row and resource metadata satisfy `classifyNetworkRequestFailure`; ordinary document/script/API failures remain actionable. Native diagnostic-read success alone never constitutes a QA pass. `details.compiledQaPreset` retains the generated plan and checks, including `diagnosticsResetAtStart`; `compiledJob` is internal QA batch-plan metadata, not a public job DSL. Use native batch/code for custom checks. ### `agent_browser_electron` Enable `electron`. Inputs are flat `{ action, ...fields, outputPath? }`; caller `stdin` and `sessionMode` are not accepted. The [Electron guide](ELECTRON.md) covers workflows and ownership. | Action | Fields | Behavior | | --- | --- | --- | | `list` | `query?`, `maxResults?` | Host-only bounded discovery; default 50, cap 200. No timeout field. macOS/Linux scanning only; Windows returns unsupported discovery. | | `launch` | Exactly one of `appPath`, `appName`, `bundleId`, `executablePath`; `appArgs?`, `handoff?`, `targetType?`, `timeoutMs?`, `allow?`, `deny?` | Verify Electron evidence, create a temporary isolated profile/OS-chosen debug port, attach through native connect, hand off, and record ownership. | | `status` | `launchId?` or `all: true`, `timeoutMs?` | Inspect tracked PID/port/profile and targets; explicit cleaned IDs remain inspectable as historical records. | | `probe` | `launchId?`, `timeoutMs?` | Verify URL then read title/focus/tabs/snapshot with owned session settings. Without an ID, inspect the current attached managed session. | | `cleanup` | `launchId?` or `all: true`, `timeoutMs?` | Close the tracked native session, stop only the owned app, verify port shutdown, remove its temporary profile. Partial cleanup fails with exact retry guidance. | ```json { "action": "launch", "appName": "Visual Studio Code", "handoff": "snapshot" } { "action": "cleanup", "launchId": "electron-…" } ``` `handoff` defaults to `snapshot`; `tabs` verifies/discovers targets without an interactive snapshot, and `connect` stops after attach. `targetType` is `page` by default, or `webview`/`any`. Launch cannot reuse the normal signed-in app profile or an existing app; use host launch plus direct native `connect` for that. Windows launch requires a verifiable path rather than scan-based name resolution; this is not Windows qualification evidence. `appArgs` cannot override user-data-dir/debug lifecycle flags or include bare `--`. Caller `allow`/`deny` substring policies match app names/IDs/paths, with deny winning. Likely-sensitive discovery annotations are advisory. Non-Electron executables fail validation. Timeouts are per action: launch readiness defaults to 15 seconds, capped at 120 seconds after discovery; status/probe use the normal 35-second subprocess budget per read (or configured override); cleanup applies its default 35-second managed-close budget separately to close and initial process-exit wait. Local CDP HTTP probes use one second. There is no combined end-to-end teardown deadline. Cancellation before spawn opens nothing; cancellation during launch cleans the new process/profile. `details.electron` retains action-specific `apps`, `launch`, `statuses`, `targets`, `probe`, `cleanup`, and failure diagnostics; `compiledElectron` retains the action plan. `launchId` identifies host lifecycle; `sessionName` identifies browser commands. `userDataDirState` measures the tracked path as present, absent (ENOENT), or unknown; it does not audit all app residue. Failed startup output includes redacted last-4096-byte stdout/stderr tails; private profile-local files follow profile ownership, not a lifetime byte cap. Failed cleanup preserves live profiles. Reload preserves current branch-visible launches and cleans off-branch owned launches; quit cleans owned launches. Restored attachments must match live endpoint/PID/profile evidence before reuse. Post-command process/target loss fails as `tab-drift`; fill verification and broad-shell-text diagnostics remain available. Recovery actions use `agent_browser_electron` with flat params for status/probe/cleanup and `agent_browser` for native tab/URL/ref operations. If inactive, enable `electron` first. Manually launched apps and explicit saved artifacts remain host-owned. ### `agent_browser_source` Enable `source`. Flat fields: at least one of `selector`, `reactFiberId`, or `componentName`; optional `includeDomHints` (default true), `maxWorkspaceFiles` (default 2,000, maximum 5,000), `outputPath`, `timeoutMs`, and `sessionMode`. ```json { "selector": "#save", "componentName": "SaveButton" } ``` Selector inspection runs native `is visible` and optional `get html`; React inspection/tree uses native tooling and requires an exposed DevTools renderer. Component-name search scans bounded local `.ts/.tsx/.js/.jsx` files while skipping build/dependency directories, returning at most ten workspace matches. It does not unpack installed Electron bundles or `app.asar`. `sourceLookup` remains the internal result field: `{ status, candidates, limitations, summary, workspaceRoot?, electronContext? }`. Candidates carry source, confidence, evidence, and optional file/line/column/component. `unsupported` means no candidates and a failed React step; DOM/workspace candidates keep `candidates-found` even if React fails. `no-candidates` alone does not fail a successful batch, but failed native steps still do. Packaged Electron results explain workspace limits and may offer app inspection actions. These are candidates, not authoritative ownership or edit instructions. ### `agent_browser_network_source` Enable `network`. Flat fields: at least one of `requestId`, `filter`, or `url`; optional `session`, `namespace`, `maxWorkspaceFiles` (2,000 default, 5,000 cap), `outputPath`, `timeoutMs`, and `sessionMode`. Explicit `namespace: ""` overrides an ambient namespace. ```json { "requestId": "req-1", "url": "/api/fail" } ``` A request ID runs native `network request`; filter/URL adds `network requests --filter`, with explicit filter winning. Failed HTTP/failed/error rows are correlated with initiator metadata and a bounded URL-literal search under the captured execution directory (up to eight needles and ten workspace matches). `networkSourceLookup` retains `{ status, failedRequests, candidates, limitations, summary }`; statuses are `failed-requests-found`, `no-failed-requests`, or `no-candidates`. URLs and credentials are redacted. Request URLs are diagnostic evidence and never replace the active page target. No candidates alone does not fail a successful batch or assign blame. HAR and full native diagnostics remain available through `args`. ### `stdin` - type: `string` - optional - raw stdin for `eval --stdin`, `batch`, and `auth save --password-stdin` - advanced tools own their generated batch input and do not accept caller `stdin`; code puts stdin inside `browser({ stdin })` - rejected before launch for any other command/stdin combination, including commands such as `click`, `snapshot`, or `open` Examples: ```json { "args": ["eval", "--stdin"], "stdin": "document.title" } ``` For `eval --stdin`, put the script in the top-level `stdin` field. The wrapper normalizes the common mistaken shape `{ "args": ["eval", "--stdin", "document.title"] }` by moving trailing tokens after `--stdin` into stdin before launching upstream, but that recovery is only for simple one-line mistakes; use `stdin` explicitly for multiline or quote-sensitive snippets. ```json { "args": ["batch"], "stdin": "[[\"open\",\"https://example.com\"],[\"snapshot\",\"-i\"]]" } ``` ```json { "args": ["auth", "save", "my-login", "--password-stdin"], "stdin": "password from the user-approved secret source" } ``` ### `outputPath` - type: `string` - optional; can be used with successful browser results, most often `eval --stdin`, `get text`, `get html`, `snapshot`, or diagnostic captures. Recording results also export on failure, pending finalization, timeout and recovery; unrelated failed extractions remain unwritten. - workspace-relative paths resolve against the captured execution cwd; absolute paths are used as-is; a leading `@` is stripped for consistency with Pi file arguments - after the upstream command completes, the wrapper writes `details.data` when present, otherwise the model-facing text content; objects/arrays are written as pretty JSON with a trailing newline and strings are written as-is. If a direct `details.data` value, a `batch` row's `result`, or the whole batch is compacted, the wrapper instead reads and serializes each full command-redacted pre-compaction payload only from the corresponding live `spill` entry in its own `details.artifactManifest` (`persistent-session` or `process-temp`). It never writes a compact metadata object as a substitute; any missing, evicted, malformed, or untrusted required spill makes the result fail and leaves `outputPath` unwritten. - recording exports use `source: "recording-receipt"` and a JSON envelope containing `success`, `error`, `command`/`subcommand`, native session/namespace, `attempt`, `data`, `artifacts`, `artifactVerification`, and optional `recordingRecovery`. A recovered result may have `success: true` while `attempt.success: false` preserves the timeout/error; saving the receipt file never proves the video succeeded. Preflight failures can export an empty receipt with their failed attempt, without inventing an artifact. - `outputPath` must not resolve to the same file as a screenshot, download, recording, or other browser artifact produced by that result, including dangling/existing symlink, hardlink, Unicode-fold, and platform-case aliases. When both destinations are known before launch, preflight rejects the call as `validation-error` without browser activity; if an alias becomes apparent only from the completed result, the writer preserves the browser artifact, rejects the result-data write, and reports `details.outputFile.status: "failed"` - successful writes append `details.outputFile = { status: "saved", path, absolutePath, source, bytes }`; they also append a visible `Output file: …` line except when the caller explicitly passed upstream `--json`, where parseable JSON content is preserved and the saved-file notice lives only in `details.outputFile`. Write failures append `details.outputFile.status: "failed"`, remove success-only category fields, and mark the tool result failed without rolling back browser session state. Example: ```json { "args": ["eval", "--stdin"], "stdin": "({ title: document.title, url: location.href })", "outputPath": "logs/page-state.json" } ``` ### `timeoutMs` - type: positive integer milliseconds - optional per-call wrapper subprocess watchdog for direct/action/QA/source tools; code uses its whole-cell deadline and Electron uses the flat action-specific timeout documented above - managed-session daemon-policy inspection has its own fixed budget of up to 35 seconds before that process and is intentionally not shortened by `timeoutMs`, so a busy valid daemon does not become an unsafe false negative - automatic managed-session cleanup uses `PI_AGENT_BROWSER_IMPLICIT_SESSION_CLOSE_TIMEOUT_MS` (default 35 seconds) separately from the original call or code deadline. The default leaves room for native Chrome shutdown, including its five-second exit grace. Expiry reports the cleanup budget, phase (policy coordination, daemon inspection, or native close), and time spent in that phase—not caller cancellation. Closure remains unconfirmed and cleanup ownership is retained for recovery. The ordinary subprocess watchdog still applies: when extending cleanup beyond it, raise `PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` as well. - use for long opens, large snapshots, paced native batch typing, or captures that legitimately need more than the default watchdog - explicit long `wait` steps are forwarded to upstream; top-level `timeoutMs` only controls the wrapper subprocess watchdog and should be at least the wait duration plus a small grace window when supplied manually - when the watchdog fires, `details.timeoutMs`, `details.timedOut`, and possibly `details.timeoutPartialProgress` explain what was recovered. Explicit URL reads and proven browser-independent read confirmations do not run timeout page probes. A `session info` timeout also leaves browser/page/ref state untouched and offers only `retry-session-info` for the same session/namespace, without claiming liveness. Uncertain recording stops instead use the bounded receipt recovery described below. If the page target is unknown, standalone snapshot suggestions are removed and one session-scoped `verify-page-target-after-timeout` fail-fast batch (`get url`, then `snapshot -i`) appears in visible failure text and `details.nextActions`, so the returned recovery is executable under the same page-target guard. Example: ```json { "args": ["open", "https://slow.example.test/"], "timeoutMs": 45000 } ``` ### `sessionMode` A native `session` default from user/project JSON or `AGENT_BROWSER_SESSION` selects a caller-owned browser for direct, code, action, QA, and source tools. Per-call flags still win. Like literal `--session`, this selection takes precedence over `fresh`; it reports `usedImplicitSession: false` and is not closed when Pi quits. Without a selected native session, attachment, or explicit fresh launch, auto calls select a stable root-Pi named session and root-specific native restore key. `PI_SUBAGENT_ROOT_SESSION_ID` carries the parent identity to descendants; ordinary new/fork/clone roots differ, while resume/cwd changes keep identity. Root sessions report `usedImplicitSession: false` and survive parent/child quit. Global/override package profile names with `policy: "always"` and executable defaults bootstrap only inactive automatic roots. Active roots preserve native launch settings/current restore key, including matching root-name follow-ups; unrelated explicit browsers are unchanged. Profile paths and project-only profile config remain advisory. Native restore covers cookies and web storage, not newly created IndexedDB or memory-only authentication. See the README for communal bootstrap and persistent-profile fallback. Active older/fresh managed sessions retain the lifecycle below. `agent_browser_qa` with `attached: true` can inspect this shared current session using the same live target checks. See [shared browser defaults](../README.md#shared-browser-defaults). - type: `"auto" | "fresh"` - optional - default: `"auto"` Behavior: - if `args` already include `--session` (including argv compiled from `agent_browser_action.session`), upstream session choice wins - `"auto"` prepends the current extension-managed active session when appropriate - after resume, a confirmed inactive wrapper-owned daemon with automatic managed restore enabled reopens its complete recorded URL, including its fragment, before the first current-page operation, including `get url` and `reload`. Non-page calls such as `tab list` or explicit HTTP `read ` can start a daemon without consuming that pending reopen, even across branch/reload replay. The wrapper verifies the observed tab and invalidates old refs; native `open` resets frame scope. This reloads the page with restored cookies/storage, not unsaved forms, JavaScript memory, or history. Live wrong-tab recovery does not navigate. Explicit URL reads, URL `a11y`/`vitals`, `diff url`, `window new`, URL-bearing recording commands, and explicit navigation/context changes do not require the old tab. Caller-owned/attached and restore-disabled sessions are not auto-reopened - `"fresh"` rotates that managed session to a fresh upstream launch so startup-scoped flags like `--profile`, `--executable-path`, `--ca-cert`, `--no-ca-cert`, `--webgpu`, `--no-webmcp`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--headed`, `--enable`, `-p` / `--provider`, or iOS `--device` apply and later default calls follow the new browser; `--idle-timeout` must equal the Pi process's configured managed idle timeout or the wrapper rejects it with restart guidance - upstream `--webgpu` is a launch-scoped optional boolean: both enabled and explicit `false` values require a fresh managed launch once an implicit session exists; enabled WebGPU is local-launch-only and upstream rejects combinations with CDP, auto-connect, or providers - upstream `--no-webmcp` is also launch-scoped for bare/`true` and explicit `false` values because it selects whether locally managed Chrome enables the experimental feature - upstream restore sessions may periodically save cookies/localStorage while open; an explicit `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` passes through unchanged when a daemon launches (`30000` upstream default, `0` disables periodic saves but keeps native close saves), while wrapper-owned headed launches default it to `0` when unset to avoid upstream 0.33.2's visible temporary collector tabs; direct window close can lose newer state because headed browsers are exempt from idle shutdown, the effective interval persists across resume and changing it in either direction on a running wrapper-owned headed daemon requires close plus a fresh launch, and those restore files remain upstream-owned rather than wrapper artifacts - sessionless paths skip that injection even under `"auto"`: plain-text help/version, read-only skills, local auth/profile/setup commands, `session list`, and syntactically local state lifecycle operations keep `effectiveArgs` free of the implicit managed `--session` unless the caller supplied one. All upstream rows and supported targets remain available. Browser-backed or context-dependent commands such as `auth login` and `state save/load` keep normal managed-session injection when the caller did not choose an explicit session (`extensions/agent-browser/lib/command-policy.ts`, `needsManagedSession`; `extensions/agent-browser/lib/runtime.ts`, `buildExecutionPlan`) Recommended use: - use `"auto"` for the common browse/snapshot/click flow inside one `pi` session - use `"fresh"` when switching from an already-active implicit session to domain containment or a new profile/browser executable/debug/auth/provider launch without inventing a fixed explicit session name - when a fresh launch fails or times out before becoming current, check `details.managedSessionOutcome`: it states whether the prior managed session was preserved or whether the attempted fresh session was abandoned because no prior managed session existed; when `sessionMode` is `"fresh"` and the tool ultimately fails, the model-visible result also appends `Managed session outcome: …` (see `#details` below). Failures under `sessionMode: "auto"` still expose the struct on `details` when the extension injects a managed `--session`, but they do not add that extra prose line. ## Wrapper behavior Caller `args` should omit `--json`; the wrapper prepends it for normal execution so `details` and presentation stay structured. See [Wrapper `--json`](#wrapper-json). The extension should: - inject `--json` - invoke `agent-browser` directly, not through a shell - parse JSON output into tool details - handle observed JSON result shapes, including the array returned by `batch --json` - allow plain-text output for native inspection calls and valid sessionless `upgrade` commands; all other commands retain JSON envelope validation - support those inspection calls unconditionally so the tool contract stays local and predictable Native inspection calls use the `agent_browser` tool shape, not shell-like direct-binary commands: - { "args": ["--help"] } - { "args": ["--version"] } These calls return plain text and stay stateless: the extension does not inject its implicit session and does not let inspection consume the managed-session slot needed for later profile, session, CDP, state, auto-connect, or provider-backed launches. - still describe normal browser workflows in guidance so models do not overuse inspection for routine tasks - surface stderr and non-zero exits clearly - attach images when the result points to a screenshot-like artifact ## Result shape ### Canonical observation Direct and advanced results expose the same decision-relevant observation in model-visible JSON or prose; code receives it as data without rendering/reparsing prose. The base is `{ success, resultCategory, data?, error?, summary?, successCategory?, failureCategory? }`. Optional fields include exact `nextActions`, warnings, session/namespace, artifact verification, image observations, page-change evidence, timeout partial progress, and QA/source/diagnostic results. Batch observations retain per-step outcomes. Native help/version remains plain text. `details` retains audit, process, ownership, and replay metadata; it is not the sole channel for actionable failures. JSON remains parseable, including early preparation failures and filters. Prose retains useful command output and appends canonical observation metadata. Final visible text is bounded to 16,000 characters. If necessary, the complete redacted observation is spilled and the result names `observationPath` plus retrieval instructions. Exact recovery objects are either included whole or remain in that complete observation; truncated argv/stdin is never presented as executable recovery. An unavailable spill is reported honestly. Inner code calls use `modelVisible: false` internally: no full model prose rendering, observation spills, or full-image base64 encoding for every result. This is not a public flag or a different executor. Source chooses JSON via `emit` and image handles via `emitImage`; failures and verification evidence remain visible at the outer boundary. ### Image observations `imageObservations[]` carries verified capture metadata: `path`, optional `id` (cell-local code handle), `mimeType`, optional image `pixels` dimensions, `capture` (`viewport`, `full-page`, `element`, or `unknown`), and `geometry`. `geometry.status` is `measured` or `unknown`, with a `reason` and available before/after samples. Samples include CSS viewport/document dimensions, scroll offsets, device pixel ratio, visual viewport, frame evidence, and a uniquely measured CSS element box when available. A measured mapping requires matching pre/post samples and image dimensions that exactly match the CSS crop at the sampled DPR. It includes document-space `crop: { x, y, width, height }` and independent `pixelsPerCssPixel: { x, y }`. This is bracketed measurement, not an atomic capture guarantee. Viewport crop starts at sampled document scroll; full-page crop starts at document origin. Scrolled/ambiguous/non-CSS element captures, child-frame uncertainty, changed geometry, zoom/offset visual viewports, and captures without samples (including native batch captures) remain unknown rather than borrowing final-page geometry. Native mouse coordinates are **current viewport CSS pixels**. For measured geometry, original image pixel `(px, py)` maps to document CSS `(crop.x + px / scale.x, crop.y + py / scale.y)`; subtract current document scroll for viewport coordinates only after rechecking frame/scroll context. Prefer current refs/locators. Pi resize notes map sent-image pixels to original-image pixels only, not CSS coordinates. No provider image-detail override or model-specific coordinate rewrite is applied. Conditional `changed: false` captures have no new image/artifact attachment. A path in JSON is not vision input. Ordinary screenshot attachments use existing verification/size checks; code explicitly selects a handle from its current call. Unknown geometry permits inspection, but cannot justify coordinate input. ### Content Primary content should be: - useful result text for the model, not just a status line - an image attachment when relevant - browser-aware compacting for oversized snapshots so the model gets a concise actionable view before raw page noise - compact snapshots should be main-content-first: prefer the primary content block and nearby sections over top-of-page chrome, ads, or unrelated sidebars when those can be distinguished from the snapshot tree. They are DOM/signal-prioritized, not guaranteed viewport-first after scroll; compact output may include `details.snapshotCompaction.viewportOrdering: "dom-signal-prioritized"` and a visible viewport note when viewport context matters. - when compacting hides actionable controls, snapshot output should add an `Omitted high-value controls` section for bounded editable/searchbox/textbox/combobox controls, named tab/surface controls, primary action buttons, named action links such as row/navigation links and repository-style result links, and other useful controls such as checkboxes, radios, options, and menuitems that were not already shown in key refs - wrapper-side `snapshot -i --search ` and `snapshot -i --filter role=` filters should strip those wrapper-only flags before upstream spawn, preserve the full latest ref map in `details.refSnapshot`, and render matching direct refs plus surrounding snapshot context with `details.snapshotFilter` counts so dense-page agents can find controls without opening raw spill files; search should also run one bounded read-only rendered-DOM probe across the full document so visible below-fold warnings and accessible labels omitted from the accessibility snapshot remain discoverable while hidden nodes stay excluded; the visible summary should distinguish direct ref matches from rendered/contextual matches to avoid apparent count mismatches; wrapper-side `--viewport` should also strip before upstream spawn, run one read-only viewport/scroll probe, and report `details.snapshotViewport`; wrapper-side `--diff` should strip before upstream spawn and report `details.snapshotDiff` against the previous wrapper-tracked ref map for that session Examples: - small `snapshot` results should include the actual snapshot text - oversized `snapshot` results should switch to a compact view that preserves the primary content, nearby sections, a trimmed set of high-value refs, and a separate bounded list of omitted high-value controls when dense pages or desktop host screens would otherwise hide editable inputs, named surfaces/tabs, or primary action buttons, while exposing the full redacted snapshot path directly in the rendered tool text and via `details.fullOutputPath` - successful navigation actions like `click`, `back`, `forward`, and `reload` should include a lightweight post-action title/url summary when the wrapper can address the active session - `tab list` should include a readable tab summary - `screenshot` should include the saved-path summary plus the inline image attachment when available ### Details Recommended details: ```json { "args": ["snapshot", "-i"], "effectiveArgs": ["--json", "--session", "pi-abc123", "snapshot", "-i"], "command": "snapshot", "sessionMode": "auto", "sessionName": "pi-abc123", "usedImplicitSession": true, "resultCategory": "success", "successCategory": "completed", "data": { "origin": "https://example.com/", "refs": { "e1": { "name": "Example Domain", "role": "heading" } }, "snapshot": "- heading \"Example Domain\" [level=1, ref=e1]" }, "summary": "Snapshot: 1 refs on https://example.com/" } ``` Stable category fields are part of the machine-readable contract: - `resultCategory`: always either `"success"` or `"failure"`. - `successCategory`: present on successful results. Current values are `"completed"`, `"artifact-pending"`, `"artifact-saved"`, `"artifact-unverified"`, and `"inspection"`. `artifact-pending` means a recording started but its file is not expected until `record stop`; use the exact `stop-pending-recording` next action and verify the resulting file. Dispatched `record start` and URL-bearing `record restart` attempts append one proactive `Page state:` warning, even on failure, describing conservative ref invalidation, not an observed page change; caller-requested `--json` carries it in `warnings`. Batch warnings require a reached result row. Preflight failures, missing binaries, help, plain restarts and unconfirmed planned rows do not claim a recording page change; the wrapper also invalidates the session’s prior ref snapshot (`refSnapshotInvalidation.reason: "page-transition"`) so old `@e…` refs fail as `stale-ref` until a fresh `snapshot -i`; that invalidation is attempt-scoped to protect older supported natives (0.37 normally keeps the active page and heap) and also covers `record restart` with a URL operand, while a plain `record restart` keeps the page and refs. Failed results also retain that action whenever their artifact rollup still contains a pending recording, except when the live daemon policy permits only cleanup: those results offer `close-pending-recording` instead, explicitly abandoning the unverified recording. `artifact-unverified` means upstream reported success but the merged `artifactVerification` summary still has unverified non-missing rows; inspect its counts and per-entry `state` / optional `limitation` before treating artifacts as durable evidence. - `failureCategory`: present on failed results. Current values are `"aborted"`, `"artifact-missing"`, `"cleanup-failed"`, `"confirmation-required"`, `"download-not-verified"`, `"missing-binary"`, `"parse-failure"`, `"policy-blocked"`, `"qa-failure"`, `"script-error"`, `"selector-not-found"`, `"selector-unsupported"`, `"stale-ref"`, `"tab-drift"`, `"tab-gone"`, `"timeout"`, `"upstream-error"`, and `"validation-error"`. `artifact-missing` means upstream reported a saved/completed artifact path, but the wrapper verified the non-pending file is absent and failed closed. For `agent_browser_code`, the top-level category describes source execution, not an assertion that every browser call succeeded. `codeRun` and visible `failures` retain handled failures; uncaught source exceptions use `script-error`. The selected browser retains ordinary ownership and is not closed at cell completion. See [code failure and persistence](#failure-and-persistence). These categories are intentionally bounded and stable so agents can branch on them instead of parsing prose. They do not replace raw diagnostics: `details.error`, `details.stderr`, `details.parseError`, `details.validationError`, and visible content still preserve the specific upstream or wrapper message after normal redaction. `details.exitCode` preserves the native child close code when available, so it is not a portable timeout indicator: a Windows process stopped with `taskkill` can report `1`. The wrapper uses `124` for its timeout when no native close code is available. Use `failureCategory`, `timedOut`, and `aborted` to distinguish these outcomes. For argv-supplied `--allowed-domains`, the wrapper treats domain containment as launch-scoped. Upstream owns request, worker, popup, and WebRTC containment plus incompatible-mode rejection; the wrapper passes the setting and upstream result through unchanged. Real Pi custom tools only mark a tool result failed when the tool throws during `execute`; returned `isError` fields are not authoritative. The extension therefore also registers a `tool_result` handler for the direct, code, and advanced browser tools that treats a result with `details.resultCategory: "failure"` as a real Pi tool error. For normal prose output, it appends `Result category: failure; failureCategory: …; Pi tool isError: true.` to model-visible text. For caller-requested `--json` output, it only patches `isError` and preserves visible parseable JSON content unchanged. The TUI renderer also repeats that category line at the top of failed rendered results so collapsed failed rows keep the outcome visible. The hook treats `--json` as requested when echoed `details.args` or the original tool `input.args` includes that flag; it skips appending the prose notice when any non-empty text content item is parseable JSON, even if other text items are not parseable. Invalid or non-JSON text still gets the visible prose notice. Implementation: `buildAgentBrowserToolResultPatch` in `extensions/agent-browser/lib/pi-tool-rendering.ts`; `extensions/agent-browser/index.ts` registers the handler. This keeps Pi transcript semantics aligned with the machine-readable result contract, including wrapper-side reclassifications such as `qa-failure` after an upstream-successful batch and `artifact-missing` after an upstream-successful artifact command whose requested file is absent. For `batch`, top-level `details` still carries `resultCategory` plus `successCategory` or `failureCategory` for the **aggregate** tool outcome: if any step fails, the overall result is a failure (`resultCategory: "failure"`) even when later steps succeed—inspect `batchSteps[]` for per-step outcomes. Each `batchSteps[]` entry includes its own `resultCategory` and either `successCategory` or `failureCategory` for that step. When upstream reports it, successful and failed rows also expose a dedicated `lifecycle` field containing only the bounded `effectiveLaunch.browserLaunched` boolean; on failed rows this preserves launch evidence even though the ordinary result payload is omitted. Live state and transcript replay use that evidence to distinguish a terminal nested close from a post-close browser launch. `batchFailure.failedStep` duplicates the first failing step’s details, including its `failureCategory`, bounded lifecycle evidence, and any `nextActions`. Top-level `details.data` on `batch` is a compact per-step roll-up (not a verbatim replay of raw upstream batch JSON): each element is `{ success, command, result? | error? }` where `command` is argv-redacted the same way as echoed invocation args (including `clipboard write` text, `cookies set` cookie values, `storage local|session set` values, and other sensitive flags/positionals), `result` is the presentation-layer data for that step after the same structured redaction as non-batch commands, and `error` is failure text with clipboard-write/cookie/storage/password literals stripped when those values appeared in argv. Prefer `batchSteps[]` for full per-step `details` (artifacts, categories, spill paths); use the roll-up when you only need a redacted matrix of what ran. If a large batch/QA result is compacted and spilled, the inline compacted text still includes bounded failed-step context (first failing step, failure category, failure detail, and any failed-step spill path) before the preview and top-level `Full output path:`. `details.refSnapshot` may appear after successful `snapshot` calls and subsequent same-session calls. It records the latest page-scoped ref ids known to the wrapper, optional per-ref accessible `role`/`name` metadata from the same snapshot, and the page target they came from so mutation-prone `@e…` commands can fail fast instead of silently hitting recycled refs after navigation. For wrapper-tracked Electron sessions, `details.electronRefFreshness` may also appear after a successful `@e…` mutation as a softer same-URL rerender warning: run `snapshot -i` before reusing old refs even if the URL did not change. Ref preflight details (command taxonomy in `extensions/agent-browser/lib/command-taxonomy.ts`, orchestration in `extensions/agent-browser/lib/orchestration/browser-run/session-state.ts`): - **Spelling and operands:** `@eN`, `eN`, and `ref=eN` share the same stale-ref checks, including inside raw and stdin batches. Only upstream ref-resolving selector slots are considered; `get count` and `diff snapshot --selector` keep their CSS/XPath or CSS semantics, while literal text and key/mouse data are not refs. - **URL alignment:** `refSnapshot.target.url` and the session’s current tab URL are compared via `targetsMatch` / `normalizeComparableUrl` in `extensions/agent-browser/index.ts`: values are trimmed, parsed as URLs when possible, compared **after dropping the `#fragment`**, and the query string remains significant. If either side lacks a `url`, `targetsMatch` treats the pair as matching so early-session calls are not blocked. - **Batch stdin ordering:** user `batch` JSON is scanned in order. Any step whose first token satisfies `isRefInvalidatingBatchCommand` sets a latch that blocks later steps whose first token satisfies `isRefGuardedCommand` and that mention `@e…` refs, except for same-snapshot native form-control steps whose current snapshot role metadata identifies all refs as safe controls (`check`/`uncheck` or direct `click`/`tap` on checkbox or radio refs, and `select` on combobox refs). A step whose first token is `snapshot` clears that latch for subsequent steps (pre-spawn intent only; it does not wait for upstream success). These predicates read explicit command capability flags from `command-taxonomy.ts`: navigation/mutation verbs such as `open` / `goto`, `reload`, non-form `click`, and related upstream commands have `invalidatesBatchRefs`, and `record start` steps (any outcome), `record restart` steps with a URL operand, plus WebMCP `invoke` / `result` / `cancel` steps also set the latch because upstream swaps or navigates the active page; same-snapshot `fill` rows and the role-checked native form-control rows stay guarded against missing/stale refs but do not set the latch, allowing ordinary form batches before a final click/submit step. Direct `click`/`tap @e…` is only treated as a safe form-control row when every ref in that step is a latest-snapshot checkbox or radio; other click/tap refs remain invalidating. Ref-guarded commands accept page-scoped refs for interaction (`click`, `fill`, `download`, `scrollintoview` / `scrollinto`, and others centralized in the command taxonomy). Changing either capability requires updating this contract, [`docs/SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md) `RQ-0072`/`RQ-0087` notes, README and command-reference pitfalls, and `test/agent-browser.extension-validation.test.ts`. **Presentation redaction (implementation map):** Successful non-`batch` tool calls and each successful `batchSteps[]` row run upstream `data` through `redactPresentationData` in `extensions/agent-browser/lib/results/presentation/diagnostics.ts`: `cookies` still walk objects/arrays and replace case-insensitive `value` keys with `"[REDACTED]"`; `storage` redacts values when the key or value looks credential-like (token, cookie, auth, secret, JWT, bearer/basic credential, high-entropy token-like string, or nested sensitive JSON) but keeps low-risk primitive QA values such as booleans, numbers, and short strings visible. Redacted storage entries add `valueRedacted` plus `valueRedactionReason` in `details.data`; diagnostic formatters mirror the same decision. Every other command’s payload is recursively scrubbed with the shared `redactSensitiveValue`, which redacts known sensitive key names and applies string-level sensitivity heuristics so network, diff, trace/profiler, stream, dashboard, chat, and other structured results do not echo bearer tokens, proxy credentials, or similar fields verbatim into `details.data`. Echoed `command` arrays in `details` and in batch roll-ups use `redactInvocationArgs` from `extensions/agent-browser/lib/runtime.ts` to mask trailing values for sensitive global flags (including `--body`, `--headers`, `--password`, and `--proxy`), preserve the special positional rules for `cookies set`, `storage local|session set`, and `set credentials`, and scrub other argv tokens for URLs and inline secrets. Raw batch row strings are parsed and command-redacted for presentation and ordered intent/completion journals; execution argv remains unchanged. Failed batch steps additionally run `redactExactValues` on structured step errors so literals taken from that step’s argv (cookie value, storage set value, `--password` / `--password=` tokens) cannot reappear inside formatted error blobs. When the full batch is large enough to need its own aggregate spill, that spill reapplies these per-command data and argv redactors before persistence rather than using generic batch redaction. `nextActions` is an optional list of exact follow-ups. Each entry has a `tool`, `id`, `reason`, optional `safety`, and executable `params` or an `artifactPath`. Native actions use `agent_browser` with argv/stdin; Electron and network-source actions use their actual advanced tool name and flat params. Enable the corresponding capability with `agent_browser_tools` if needed. Both prose and JSON expose recovery; oversized observations retain complete payloads in the named observation spill rather than inventing shortened calls. Browser actions preserve exact session/namespace (including empty namespace) unless fresh-session recovery deliberately omits the old session. Stale refs require a fresh snapshot; unknown targets require `get url` before snapshot; timeouts preserve partial progress and avoid blind mutation retries; covered clicks recommend overlay inspection; recordings retain stop/status/cleanup evidence; confirmations retain exact native IDs. Existing evidence predicates govern locator/ref fallbacks, rich-input recovery, tab correction, selector visibility, and artifact checks. Code-bound actions cannot change its fixed identity; execute cross-identity or namespace-wide recovery outside the cell. **Unknown-command getter hints (failure presentation):** `buildErrorPresentation` in `extensions/agent-browser/lib/results/presentation/errors.ts` only runs this path when upstream error text (after model-facing redaction) matches `unknown command`, `unknown subcommand`, or `unrecognized command` (case-insensitive) **and** the failed invocation’s primary command token is one of `attr`, `count`, `html`, `text`, `title`, `url`, or `value`. Visible text then includes a grouped-`get` hint line plus per-token guidance (`get text `, `get html …`, `get attr …`, `get count …`, `get value …`, `get title`, `get url`). Machine `nextActions` with ids `use-get-title` / `use-get-url` are emitted only for `title` / `url`, with `params.args` optionally prefixed by `--session ` when the failed call targeted a named session. If the error string already contains `Agent-browser hint:` from selector recovery (stale-ref or unsupported selector dialect appendages), the getter block is skipped so two stacked `Agent-browser hint:` headers are not emitted. For `network requests`, `details.nextActions` is bounded to one selected safe request ID, preferring actionable failed rows, then API/fetch-like rows, then benign failed rows, then the first request with a safe ID. Detail/filter/HAR actions use `params.args` and preserve known `--namespace ` / `--session ` prefixes when the current presentation has `details.namespace` / `details.sessionName`; source-candidate actions use flat `agent_browser_network_source` params with the selected `requestId` plus `namespace` / `session` when known and are only emitted for actionable failed rows that the failed-request analyzer can correlate. A `clear-network-requests-before-repro` action can run `network requests --clear` so the next reproduction starts with a current-page-focused diagnostic buffer. Wrapper-side `network requests --current-page` / `--current-origin` renders only rows whose URL matches the active page origin, while `--current-url` renders exact active-document URL matches; those wrapper-only flags are removed before upstream spawn and reported in `details.networkRequestsPageFilter`. Filtered results pass through the same `redactPresentationData` credential redactor as ordinary network diagnostics before entering `details.data` or an `outputPath` export. URLs and query strings are not copied into action params; path filters are skipped when they look sensitive or too large. If the wrapper has observed `network route` in the same session, matching failed, pending, or CORS-looking fetch/XHR rows also add `details.networkRouteDiagnostics[]` with `{ reason, routePattern, mode, requestId?, requestUrl?, summary }` and prepend executable route-mock next actions (`inspect-routed-network-request`, `start-network-har-capture-for-route-mock`) before generic request follow-ups; same-origin/CORS fixture retry guidance stays in visible prose. The route tracker is wrapper-session-local, updated on successful `network route`/`network unroute`, and cleared when that session closes or is replaced. For `batch`, each `batchSteps[]` entry can carry its own `nextActions` for that step’s success or failure. Top-level `details.nextActions` on a failed batch duplicates `batchFailure.failedStep.nextActions` so callers can read one aggregate object. On a fully successful batch, top-level `nextActions` may still list artifact follow-ups derived from the combined step artifacts. `pageChangeSummary` is an optional compact summary for mutation-prone and artifact-producing commands. It includes `changeType` (`"navigation"`, `"mutation"`, `"artifact"`, or `"confirmation"`), `observed`, `command`, a readable `summary`, optional `title`/`url`, optional `artifactCount` or `savedFilePath`, and `nextActionIds`. `observed: false` means upstream dispatched a mutation-capable action but the wrapper did not observe an application change; standalone results also append a visible `Action dispatched; application change unverified` warning. The wrapper maintains explicit command/subcommand capability checks through `isPageChangeSummaryCommand` in `extensions/agent-browser/lib/command-taxonomy.ts`: those commands still emit a `mutation`-typed summary when upstream JSON lacks navigation metadata, as long as no stronger signal (artifact, saved path, navigation fields, or pending confirmation) applies. That capability is independent from `invalidatesBatchRefs` and `triggersPostMutationSnapshot`, so artifact summaries like `download` / `screenshot` and guarded-but-non-invalidating `fill` are documented directly in the capability table instead of implied by broad set spreading. Commands outside that set omit `pageChangeSummary` unless the parsed payload shows navigation, a confirmation prompt, saved files, or artifacts—including read-only inspection commands, which normally have no summary unless one of those signals appears. For `batch`, the top-level summary favors artifact rollups when any step produced artifacts; otherwise it synthesizes an observed-or-unverified summary from step evidence. Visible batch output promotes dispatch-only mutation evidence before step details and states that fixed waits are not postconditions. Agents should verify URL/text/state or an external receipt for important mutations before continuing. `clickDispatch` may appear after a **top-level non-Electron** direct `click` when the wrapper installed a target-specific DOM-event probe, upstream reported success, and the post-click probe found no trusted DOM event reached the resolved target. Target-specific probes cover `xpath=` targets and role-gated `@e…` refs when the latest wrapper-tracked snapshot has role/name metadata; eligible ref roles are `button`, `checkbox`, `menuitem`, `radio`, `switch`, and `tab`, and the role/name must be unique in both the saved snapshot and the live candidates. Before trusting either a ref or XPath candidate, the wrapper sets a per-probe temporary DOM attribute and requires native `get attr ` readback to match it. Missing or mismatched identity, including XPath/frame scope mismatches, cleans up the probe and leaves the native click to run without dispatch verification. This does not implement a new accessible-name algorithm or promise that every ref is observable. Duplicate-name refs pass through without a probe because snapshot order can change and is not proof of target identity. Raw `find … click` locator calls, including compiled `agent_browser_action` clicks that still execute as upstream `find`, are not probed because the wrapper has no concrete element before upstream resolves the locator, and document-level probes can falsely fail frame-scoped clicks. It does **not** take a fresh pre-click snapshot because that could recycle upstream refs before the intended click. The wrapper does **not** replay clicks in-page. On a miss it marks the tool failed, appends `Click dispatch diagnostic: …`, and sets `clickDispatch.status` to `"no-native-event-observed"` with `reason: "native-click-produced-no-target-dom-event"`, `nativeEventCount`, and a redacted `target` descriptor (`kind: "xpath"` plus `selector`, or `kind: "accessible"` plus `refId`, `role`, and redacted `name`). `details.nextActions` gains `inspect-click-dispatch-miss` (`snapshot -i`) and `retry-click-after-dispatch-miss` (same upstream click argv, session-prefixed when applicable). If a local static fixture must be exercised despite this diagnostic, a caller may explicitly run a programmatic activation via `eval --stdin` such as `document.querySelector(...).click()`, but that emits an untrusted scripted event and is only a debugging/workaround path; it must not be used as proof that real user-like clicking works or to bypass prompt stop boundaries. This diagnostic is only for standalone top-level direct click calls; `find` locator clicks and native batch/QA click steps remain upstream-owned behavior. `promptGuard` may appear on wrapper-blocked calls only for concrete, machine-checkable prompt requirements. `reason: "requested-artifacts-missing-before-close"` blocks `close` / `quit` / `exit` when the prompt used a direct screenshot/recording creation phrase with a destination such as `here`, `at`, `as`, or `to` and the session artifact manifest has not verified that exact path; a destination heading can carry that intent across contiguous plain or Markdown-bulleted path-only list lines. Bare, review-only, fenced-reference, conditional, permissive/uncertain, directly negated, and Pi clipboard/attachment image/video paths are treated as input, not output requirements, while subordinate requirements such as “do not close until you save” remain output intent. A recording-availability qualifier may precede, appear within, or follow its path/list and is scoped through the next path boundary; if the same path appears more than once, any required occurrence takes precedence. Markdown-link destinations resolve to the destination path. Explicitly optional artifacts are not close requirements. The classifier is deliberately conservative, so use wording such as `Save a screenshot here: ` when machine-enforced close blocking matters. Optional recording paths are only required when recording appears available. The wrapper does **not** parse broad user/business intent such as “do not place the order” or “do not post anything” into click/key blocks; agents must follow those instructions themselves. Prompt guards return `failureCategory: "policy-blocked"` and `validationError` text instead of invoking upstream. `overlayBlockers` may appear on a successful `snapshot` whose own refs show strong modal context, or after a successful **top-level non-Electron** `click` (the unified `details.command` is `click`, not native batch/QA flows that compile to `batch`) only when upstream JSON includes a string `data.clicked` ref (`@e…` / `ref=`), no `clickDispatch` diagnostic fired for the same result, the session’s prior pinned tab URL (`priorSessionTabTarget.url`) and `details.navigationSummary.url` both exist and stay equal after the same URL normalization used for ref preflight (trimmed hosts/paths; **`#fragment` dropped** while the query string stays significant), and the wrapper did not apply session tab correction or an about-blank mismatch recovery in the same result. Wrapper-tracked Electron clicks prefer lifecycle health and ref-freshness diagnostics because desktop app chrome produced too many false overlay candidates in dogfood. For post-click diagnostics, the wrapper uses existing `details.navigationSummary.url` as same-URL evidence, then issues **one** extra session-scoped `snapshot -i`; CSS selector clicks do not run this overlay probe. For snapshot diagnostics, it scans that snapshot result directly. It only emits diagnostics when **both** are true: at least one ref has a strong modal role (`dialog` or `alertdialog`), and there are up to **three** separate `button`/`link`/`menuitem` refs whose names match close/dismiss-style patterns (for example “Close”, “Dismiss”, “No thanks”, or a lone `×`). Page-wide text such as “privacy”, “sign in”, or “banner” without a dialog role is not enough, which avoids warning on ordinary same-page menu opens or app button mutations. Each candidate carries `ref` (`@eN`), optional `role`/`name`, exact `click` argv in `args`, and a short evidence `reason`. The struct also includes a `summary` string (one sentence describing the snapshot/click evidence and likely dismiss controls) plus a `snapshot` object (same shape as `details.refSnapshot` after a normal snapshot): on success the wrapper may treat that snapshot as the session’s latest ref map for subsequent calls, so agents should assume refs can move to match this post-diagnostic tree. Visible text appends the same bullets under `Possible overlay blockers`, and `details.nextActions` gains `inspect-overlay-state` plus `try-overlay-blocker-candidate-1`…`3` after any presentation `nextActions` (for example `inspect-after-mutation`); when `details.sessionName` is set, those appended actions preserve session context; namespaced sessions use `--namespace --session ` and non-namespaced sessions use `--session ` unless argv already carries that context. This is conservative evidence, not proof the candidate should be clicked; prefer `inspect-overlay-state` first unless the dismiss control is clearly safe. Example shape (fields vary by scenario): ```json "nextActions": [ { "tool": "agent_browser", "id": "inspect-after-mutation", "reason": "Refresh interactive refs after a browser mutation, navigation, scroll, or rerender.", "safety": "Do not reuse prior @refs until a fresh snapshot confirms they still exist.", "params": { "args": ["snapshot", "-i"], "sessionMode": "auto" } } ] ``` When `agent_browser_action` produced compiled `find` argv and the unified result is `failureCategory: "stale-ref"` with `details.compiledSemanticAction` still present, `nextActions` chains snapshot refresh then the compiled `find` retry; `select` shorthands with stale `@refs` stop at refresh guidance. `reason` / `safety` strings match `buildAgentBrowserNextActions` in `extensions/agent-browser/lib/results/action-recommendations.ts` and the append in `extensions/agent-browser/index.ts`: ```json "nextActions": [ { "tool": "agent_browser", "id": "refresh-interactive-refs", "reason": "Get current interactive refs before retrying the element action.", "safety": "Prefer a current @ref or a stable find locator; do not retry stale refs blindly.", "params": { "args": ["snapshot", "-i"] } }, { "tool": "agent_browser", "id": "retry-semantic-action-after-stale-ref", "reason": "Retry the same semantic target via its compiled find command after the upstream stale-ref failure proves the prior action did not execute.", "safety": "Use only for the same intended target; direct stale @refs still require a fresh snapshot or stable locator before retrying.", "params": { "args": ["find", "text", "Submit", "click"] } } ] ``` ```json "pageChangeSummary": { "changeType": "navigation", "command": "open", "observed": true, "summary": "Opened Example Domain", "title": "Example Domain", "url": "https://example.com/", "nextActionIds": ["inspect-opened-page"] } ``` Implementation and precedence: - Shared machine-readable types are centralized in `extensions/agent-browser/lib/results/contracts.ts` (including re-exports such as `AgentBrowserNextAction` from `next-actions.ts`). Classifiers live in `categories.ts` (`classifyAgentBrowserSuccessCategory`, `classifyAgentBrowserFailureCategory`, `buildAgentBrowserResultCategoryDetails`—the last prefers an explicit `failureCategory` when the caller already knows the bucket, otherwise it runs the classifier). Generic follow-up assembly lives in `action-recommendations.ts` (`buildAgentBrowserNextActions`). Tab/session recovery ids live in `recovery-actions.ts` (`AGENT_BROWSER_RECOVERY_NEXT_ACTION_IDS`, `AGENT_BROWSER_RICH_INPUT_RECOVERY_NEXT_ACTION_IDS`, `getAgentBrowserRichInputRecoveryNextActionId`, `getAgentBrowserRichInputRecoveryNextActionIds`, `buildRecoveryNextActions`) and session-aware wrappers live in `recovery-next-actions.ts`. Selector miss and rich-input diagnostic shapes/actions live in `selector-recovery.ts`. Failed upstream `network requests` rows flow through `classifyNetworkRequestFailure` / `summarizeNetworkFailures` in `network.ts` for QA analysis (`analyzeQaPresetResults` in `extensions/agent-browser/index.ts`) and for actionable-vs-benign lines plus request-specific nextActions in `network requests` presentation (`extensions/agent-browser/lib/results/presentation/diagnostics.ts`). - Artifact verification: `ArtifactVerificationSummary` / `ArtifactVerificationEntry` types live in `contracts.ts`. `buildArtifactVerificationSummary`, `getArtifactVerificationEntry`, and `getManifestVerificationEntry` in `presentation/artifacts.ts` merge each resolved file artifact with manifest rows whose `storageScope` is not `explicit-path` (those rows duplicate file artifacts) and whose `path` is in the current result’s spill path set. Presentation fails closed with `failureCategory: "artifact-missing"` when a non-pending artifact (including a previous recording finalized by `record restart`) is absent or when its `mtimeMs` falls outside the command's bounded start/end window with two seconds of filesystem precision tolerance (`status: "stale"`, `state: "unverified"`, and `updatedAtMs` expose that evidence). Batch preflight canonicalizes existing path ancestry, normalizes Unicode and folds case on macOS/Windows, and rejects duplicate explicit artifact destinations, including recording start/restart paths, so filesystem aliases or an earlier step cannot satisfy a later step's verification. Pending video entries from `record start` / `record restart` remain successful as `artifact-pending` until `record stop`. - Inner success categories (`classifyAgentBrowserSuccessCategory` in `categories.ts`, after verification counts are clear): if `inspection` is true → `"inspection"`; else if any pending recording artifact exists → `"artifact-pending"`; else if any artifact lacks confirmed on-disk presence (`exists !== true`) and was not upgraded to an `artifact-missing` failure → `"artifact-unverified"`; else if there is a `savedFile` or any `artifacts` → `"artifact-saved"`; else → `"completed"`. - Failure: the classifier walks a single ordered chain (first match wins): explicit `options.confirmationRequired` → `tab-gone` (`tab_gone:` signature, before lastUrl can match aborted/policy/about:blank heuristics) → upstream locator-detail misses (`selector-not-found`, including 0.32.4+ `Names seen:` / `No element found: getByRole(...)` / `Element not found: … Verify the selector, role, or name`) → text-derived `confirmation-required` → `timeout` → `missing-binary` → `parse-failure` → `aborted` → `policy-blocked` → `cleanup-failed` → explicit `options.validationError` → `tab-drift` → `stale-ref` (including “unknown ref” text and a narrow `@eN` plus “element not found” heuristic) → `selector-unsupported` → other `selector-not-found` shapes → `download-not-verified` (download / wait-download style failures) → default `upstream-error`. Locator-detail misses are classified before text-derived confirmation/timeout so an accessible name containing those phrases cannot suppress selector recovery. Wrapper-known missing artifact checks pass an explicit `artifact-missing` category rather than relying on this text classifier. - The main tool implementation merges these fields into Pi-facing `details` from `extensions/agent-browser/index.ts` and from `extensions/agent-browser/lib/results/presentation.ts` for presentation-time failures. Additional structured fields can appear when relevant: - `sessionTabReopenPending: boolean` persists a confirmed-cold managed session's outstanding URL reopen through non-page commands and branch/reload replay. `true` means a daemon may have started but the remembered page has not been reopened; `false` means a reopen attempt or an executed explicit context/navigation command consumed the obligation, not that navigation succeeded. Cancellation after the reopen CLI starts returns `failureCategory: "aborted"`, the exact `sessionName` / `namespace`, the consumed `false` marker and ref invalidation through the normal result path, so replay cannot repeat the navigation. Cancellation before the CLI starts leaves the obligation pending. Successful close clears it with the rest of that session's page state. Internal remembered URLs retain their complete fragment; comparison remains fragment-insensitive and presentation redaction is unchanged. Old persisted targets without a fragment cannot reconstruct it. - `closeAllApplied: true` when a successful direct or nested `close` / `quit` / `exit --all` reached upstream. The marker makes live state and transcript replay clear every managed/attached/page/ref/route/trace/recording identity in the effective canonical namespace; a later batch row that proves browser reactivation may rebuild only the effective session. - `attachedBrowserSession: true` on successful calls that establish or reuse a wrapper-tracked CDP/auto-connect/Electron attachment, and on a failed fresh attachment only when `managedSessionOutcome.activeAfter` proves its daemon remained active for cleanup. The marker restores attachment continuity from the active transcript branch, including that active-after-failure case; live state and transcript replay remove it after a terminal successful close/cleanup even when aggregate verification failed; a close followed by a later step whose lifecycle reports a browser launch preserves it, while a successful non-launching diagnostic leaves the close terminal. Caller config, environment, paths, and file-access settings remain upstream-owned; the marker only adds live-URL verification and lifecycle continuity. - `lifecycle: { effectiveLaunch: { browserLaunched } }` when upstream returned that boolean. It separates starting the requested `agent-browser` CLI process from the effective Chrome session context. `readSource` exposes upstream's string `data.source` for direct `read` calls and identifies the raw HTTP fetch path; its lifecycle boolean can be `false` before a browser launch or `true` when the same managed session already has an active browser. Direct reads also append one visible `Read execution` line with the source, CLI-start result, managed browser lifecycle, and managed-session outcome so Pi models do not have to infer model-invisible details. - `browserWindow: { mode: "headed", ownership: "wrapper-managed", sessionName, visibility: "unverified" }` only after a successful first/fresh local wrapper-managed headed call (including `batch`) that is not an attachment and whose lifecycle proves a browser launched and whose managed-session outcome is `created` or `replaced`. One visible handoff sentence tells the user to complete the login in that window if they can see it, then continue with `sessionMode: "auto"`; the field never claims OS desktop visibility. - `sessionTabTargetUnknown: true` after a spawned `connect`, `state load`, history navigation, tab-selection/close, `window new`, or `diff url` call changes the active page without a trustworthy observed target. Direct `window new` / `diff url` and reached native batch rows retire the old target and refs; an intentional new blank window or an observed blank URL-diff destination never triggers old-tab recovery. URL diff inputs do not prove the final URL after redirects. Successful standalone tab selection/close now live-probes URL and a fresh non-blank title before state is committed, even when the new tab shares the prior URL; explicit selection of an existing `about:blank` tab and a post-close blank target are retained, so this marker remains only when the probe cannot verify the target. It is persisted and restored across branch/reload replay, clears stale refs and tab pinning, and blocks page inspection until `get url` or explicit navigation observes a target; `tab list`, tab selection, close, and blocking-dialog `status` / `accept` / `dismiss` remain available. A timeout against an unknown target removes standalone snapshot actions and returns `verify-page-target-after-timeout`, a session-scoped `batch --bail` whose stdin runs `get url` before `snapshot -i`. - `compiledSemanticAction` when the call used `agent_browser_action` and the result includes the unified `details` merge: `{ action, locator, args }` for `find` actions or `{ action: "select", selector?, locator?, values, args }` for `select`, with the same redaction rules as `args` / `effectiveArgs`; omitted for plain direct native calls and omitted on some early error returns that omit this field (see the `agent_browser_action` section above) - `compiledJob` is internal QA batch-plan metadata: `{ args: ["batch", "--bail"], failFast: true, stdin, steps }`, with redacted step argv. It does not expose a public job input or compiler. - `compiledQaPreset` when the call used `agent_browser_qa`: the compiled native batch fields plus the QA `checks` object. `args` is `batch --bail` and `failFast` is `true` for QA presets. `checks.attached` is `true` for current-session QA, `checks.url` is present only for URL-opening QA, and `checks.diagnosticsResetAtStart` is `true` only for URL-opening QA because `agent_browser_qa` with `attached: true` preserves existing session diagnostics. - `compiledSourceLookup` when the call used `agent_browser_source`: `{ args: ["batch"], stdin, steps, query }` with the generated local-evidence plan and original query fields (`selector?`, `reactFiberId?`, `componentName?`, `includeDomHints?`, `maxWorkspaceFiles?`). - `sourceLookup` when the call used `agent_browser_source`: `{ status, candidates, limitations, summary, workspaceRoot?, electronContext? }`; wrapper-tracked packaged Electron no-candidate diagnostics may carry `workspaceRoot` plus `electronContext` and live Electron nextActions without marking the successful batch as a tool failure. - `compiledNetworkSourceLookup` / `networkSourceLookup` when the call used `agent_browser_network_source`: the generated batch plan plus bounded failed-request/candidate evidence as described above. - `qaPreset` when the call used `agent_browser_qa`: `{ passed, failedChecks, warnings, summary }`. `failedChecks` includes “page-error check could not be verified” when final errors match a nonempty post-clear baseline; this sets `passed: false` without claiming those matched rows are new application errors. Definitely new rows are counted separately as page errors. Network rows inside the `network requests` batch step use `summarizeNetworkFailures` / `classifyNetworkRequestFailure` in `network.ts`: actionable failures appear in `failedChecks` (and fail the tool when the upstream batch still succeeded); benign icon-classified failures appear only in `warnings` and in `summary` as `QA preset passed with warnings: …` when nothing else failed. - `networkRouteDiagnostics` after successful `network requests` when the wrapper has observed active `network route` patterns for that session and a matching request row is pending/no-status or carries CORS/preflight-looking error text. Each row includes `reason` (`"pending-routed-request"` or `"cors-likely-routed-request"`), `routePattern`, `mode`, optional `requestId`, optional `requestUrl`, and `summary`; visible text starts with `Network route diagnostics`, and `details.nextActions` prepends executable route-mock inspection/HAR follow-ups before generic request follow-ups. - `compiledElectron` when the call used `agent_browser_electron`: redacted action plan for `list`, `launch`, `status`, `cleanup`, or `probe`. - `electron` when the call used `agent_browser_electron`: action-specific lifecycle, discovery, probe, and cleanup data; see the `agent_browser_electron` contract above. - `batchFailure` and `batchSteps` for `batch` rendering, including mixed-success runs - `navigationSummary` for navigation-style commands like `click`, `back`, `forward`, `reload`, `window new`, `diff url`, and successful standalone tab selection/close; reached window-new/URL-diff rows also probe the final page after native batches, including failed batches that may already have changed the page (not aborted or timed-out calls). Unreached rows do not change target or ref state; `urlChanged` records whether the live URL differs from a known pinned pre-command URL, so same-URL clicks and clicks without a comparison baseline remain dispatch-only rather than being mislabeled as observed navigation. Helper probes run `get url` first and run `get title` for any verified non-`about:blank` URL. The title read is skipped when the probed URL already carries a wrapper-observed title for this session, except after a tab selection/close, which always refreshes a non-blank title even when the URL is unchanged: titles are last-observed labels for that URL, while the URL itself is live-probed on every call. Href-less CSS selector clicks use this same post-command helper so `sessionTabTarget` cannot stay on the pre-click page; any click-dispatch check still runs first. A failed non-batch `eval`, `back`, `forward`, `reload`, `connect`, `state load`, or `tab` selection also runs this helper (browser started, not aborted, not watchdog-timed-out), so an observed page stays verified instead of forcing a manual `get url` round trip; a failed or empty probe keeps the prior unverified-page behavior. Because a failed transition can still have mutated or replaced the document, a successful probe also invalidates the prior page-scoped ref snapshot (matching the previous unknown-target behavior, which dropped refs), so the next `@e…` use requires a fresh `snapshot -i`. - `pageChangeSummary` for compact mutation/artifact/navigation summaries on commands that can change browser state - `clickDispatch` when a top-level non-Electron direct `click` reported upstream success but the target-specific probe found no trusted event reached the resolved XPath or accessible `@ref` target; shape follows `ClickDispatchDiagnostic` in `extensions/agent-browser/lib/orchestration/browser-run/types.ts` - `promptGuard` when the requested-artifact-before-close guard blocks browser close before required prompt artifact paths are verified; implementation lives in `extensions/agent-browser/lib/orchestration/browser-run/prompt-guards.ts` - `overlayBlockers` for conservative overlay/banner/dialog blocker candidates when a successful snapshot itself contains strong modal evidence, or after a qualifying top-level `@e…` / `ref=` click stays on the same URL, no `clickDispatch` diagnostic fired, and a fresh snapshot provides evidence (`candidates`, `summary`, and `snapshot` per `OverlayBlockerDiagnostic` in `extensions/agent-browser/index.ts`). CSS selector clicks do not run this overlay probe. - `visibleRefFallback` after a raw `find` or compiled `agent_browser_action` fails with `selector-not-found` and a fresh snapshot finds exact role/name `@ref` matches. Shape follows `VisibleRefFallbackDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, snapshot, summary, target }`, where each candidate has `ref`, `role`, `name`, optional direct ref `args`, and `reason`; visible text appends `Current snapshot ref fallback`. Non-fill candidates with direct args add `try-current-visible-ref` or numbered `try-current-visible-ref-N` actions. Fill candidates omit direct args and target text so recovery details do not repeat potentially sensitive fill text. - `refSnapshotInvalidation` after a confirmed cold managed-session shutdown (`reason: "page-transition"`, including when reopening fails), a session `snapshot` fails with `No active page`, any upstream-executed `record start` attempt or URL-bearing `record restart` conservatively invalidates refs (including failures, for older-native protection rather than proof of a page change), a direct or reached batch `window new` / `diff url` attempt changes the page, or a failed non-batch transition command (`eval`, `back`, `forward`, `reload`, `connect`, `state load`, `tab` selection) whose live URL re-verification probe observed the page (a failed transition can still have mutated the document, so the verified URL is kept but the prior refs are not). Shape follows `SessionRefSnapshotInvalidation` in `extensions/agent-browser/lib/session-page-state.ts`: `{ reason: "no-active-page" | "page-transition", summary }`; replay preserves the persisted summary. The wrapper deletes prior refs for that session, persists the invalidation for resume, and blocks mutation-prone `@e…` preflight with `failureCategory: "stale-ref"` until a successful fresh `snapshot -i` records refs again. - `snapshotFilter` after wrapper-side `snapshot -i --search ` or `snapshot -i --filter role=`. Shape: `{ cleanArgs, search?, role?, matchedRefs, totalRefs, visibleLines, totalLines, renderedTextMatches?, renderedTextTotalMatches?, renderedTextTruncated? }`. Search runs one bounded read-only rendered-DOM probe across the full document; each visible match carries bounded `text`, `tagName`, `kind` (`text` or prioritized `validation`), `offscreen`, optional `role`/accessible `name`, and a unique mapped `ref` when the full snapshot supports it. Hidden elements are excluded. The filtered accessibility snapshot remains separate, while `details.refSnapshot` still records the full upstream ref map for later stale-ref checks. - `snapshotViewport` after wrapper-side `snapshot --viewport` (with or without `-i`, `--search`, or `--filter`). Shape matches the scroll-position probe: viewport scroll offsets, inner/document dimensions, sampled scrollable-container count, and bounded container offsets. The wrapper strips `--viewport` before upstream spawn and gathers this with a read-only `eval --stdin` call. - `snapshotDiff` after wrapper-side `snapshot --diff` (with or without `-i`, `--search`, `--filter`, or `--viewport`). Shape: `{ addedRefs, removedRefs, changedRefs, unchangedRefs, summary }`, comparing ref ids plus role/name metadata from the previous wrapper-tracked snapshot for the session with the newly returned full ref map. It is a quick ref-map delta, not a visual diff. - `networkRequestsPageFilter` after wrapper-side `network requests --current-page`, `--current-origin`, or `--current-url`. Shape: `{ cleanArgs, currentUrl, mode, matchedRows, totalRows }`; the visible rows and `details.data.requests` / `items` / `entries` are filtered while the active session page target is read with `get url`. - `richInputRecovery` after a raw `find` or compiled `agent_browser_action` `fill` fails with `selector-not-found` and the same current-ref diagnostic finds exact editable `searchbox` / `textbox` candidates. Shape follows `RichInputRecoveryDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, inputMethodHint, nextActionIds, summary, target }`, where each candidate has `ref`, `role`, `name`, `focusArgs`, `clickArgs`, and `reason`. Visible text appends `Rich input recovery`, and `details.nextActions` gains ids from `getAgentBrowserRichInputRecoveryNextActionIds`: `focus-current-editable-ref` / `click-current-editable-ref` (or numbered variants). These actions are bounded to focus/click/inspect-style recovery: they do not include the fill text, do not press `Enter`, and do not submit. After the right current editable ref is focused, use `keyboard type` for framework-controlled editors that require real key events. Use paste-like `keyboard inserttext` only with separate application-state verification, and submit only when explicitly required by the flow. - unsupported `scrollintoview text=