# pi-control-chrome [English](./README.md) · [简体中文](./README-zh-CN.md) Codex-aligned Chrome and Edge browser control for Pi, Codex and DSH. It reuses the user's existing Chromium profile through a local WebSocket Bridge and a Manifest V3 extension. > The Stage 1/2 core implementation is complete. Later-stage capabilities remain tracked separately in [`FEATURES.md`](./FEATURES.md). ## What it provides - Reuses the current Chrome or Edge profile, login state, cookies, extensions, and tabs. - Lets Pi inspect and claim explicitly selected existing tabs without moving them by default. - Places Agent-created tabs in a dedicated blue `Pi` tab group. - Tracks Agent ownership, sessions, `handoff`, and `deliverable` lifecycle states. - Closes unmarked Agent temporary tabs at turn end, releases claimed user tabs without closing them, and preserves only handoff or deliverable tabs marked for the current turn. - Provides DOM, accessibility, locator, coordinate, and native CDP controls. - Returns bounded semantic page state by default: visible/actionable snapshot nodes, 20,000-character/200-node snapshot and DOM-CUA budgets, and a shared 12,000-character extract budget. Same-origin iframe text is included by default with bounded `frames` diagnostics; pass `includeFrames: false` for shell-only inspection. `browser_snapshot` keeps refs and `snapshotId` while Pi/DSH projections omit duplicate raw accessibility and frameTree data; the Bridge advertises `capabilities.compactResponses` and negotiates compact page-read responses, while unspecified mode remains raw-compatible for older consumers; when tab metadata already carries title and URL, projections do not repeat them inside page results, and full page text is reserved for `browser_extract` when no semantic state is available. - Accessibility reads prefer the real Chromium AX tree and support full, incremental-diff, and unchanged states; actionable/focusable AX nodes may carry document-scoped `aN` refs, which require the matching `snapshotId` for operations. Sensitive values stay redacted in AX and locator results. If the Accessibility domain is unavailable, reads safely fall back to the DOM semantic tree. Pass `disableDiffing: true` only when a complete AX state is required. Snapshot, accessibility, extract, and DOM-CUA reads accept optional `selector`, `includeFrames`, and output budgets. Pi and DSH adapters omit blank optional `selector`, `snapshotId`, `incarnation`, and screenshot `path` fields before dispatch. - Ordinary `role`/`name`, label, and accessible-text locator, wait, and interaction targets now try Chromium AX first. AX resolves the computed role/name/state, then maps the current backend DOM node under the existing document/frame fence; successful actions expose internal provenance as `resolvedBy: "chromium_ax"`. AX ambiguity, incomplete trees, and unsafe DOM mapping fail closed. Only an unavailable Accessibility domain, or a text-only target whose custom clickable element is absent from AX, uses the bounded DOM fallback; selectors, test IDs, and placeholders remain DOM-driven. - Bounds `browser_evaluate` results to depth 8, 2,000 array items, 200 object fields, and 200,000 characters before returning them through the Bridge. - Supports screenshots, page extraction, Console, Network, JavaScript dialogs, file upload, downloads, and clipboard text. Console and Network listings retain at most 200 entries and 20,000 serialized characters per read; Console reads support bounded `only: "errors"` filtering and `since`/`nextSince` cursors, including `pageerror` events. - `browser_probe_interaction` performs exactly one explicit page action and returns before/after document identity, action confirmation, post-action Console errors, settle status, and post-action target state for UI regression diagnosis. It never replays an uncertain side effect. - Captures ordinary active-tab viewport screenshots without opening a DevTools debugger session; full-page and background-tab captures use a short session-owned debugger lease. - Persists debugger lease identity across an MV3 worker restart. Ordinary cleanup reports an unverified old lease instead of detaching an untracked target; explicit stale recovery verifies the current tab fence and CDP target identity before detaching it. - Fences tab handles with a tab fence and document identity. Snapshot refs and DOM-CUA node ids are live observations within their originating document: title/focus changes, user tab switching, later observations, and unrelated DOM churn do not invalidate them; an original detached node can be rebound once only when a strongly equivalent replacement is unique. Navigation, reload, and document replacement remain hard boundaries and reject old observations with `BROWSER_DOCUMENT_CHANGED`; tab closure and tab-fence changes reject them with `BROWSER_TAB_CLOSED` and `BROWSER_TAB_FENCE_CHANGED`; `navigate(wait: false)`, `back`, `forward`, and `reload` return a tab marked `transitionPending` whose handle omits unstable URL/title and document-incarnation fields, so wait or re-observe before document-bound work. Read-only page reads re-observe a user tab and retry once when its document changes, while side-effecting operations that lose document identity return an uncertain result and are not replayed automatically. - Keeps a Manifest V3 Bridge socket active with application heartbeats, preventing idle worker suspension from creating avoidable connection-generation churn during long browser waits. - Includes a reusable `pi-control-chrome` Skill. Browser tool schemas are hidden until that Skill is explicitly loaded for the current session; the bundled CLI remains available for explicit human/developer workflows and tests. ## Semantic page interaction The common click and form tools accept a semantic `target` such as `{ "role": "button", "name": "Submit" }`, `{ "label": "Email" }`, `{ "placeholder": "Search" }`, `{ "text": "Next" }`, or `{ "testId": "submit-button" }`. Semantic interactions wait for one visible match and fail closed when a target is missing, hidden, disabled, or ambiguous; use an explicit zero-based `index` only when multiple matches are intentional. Actions apply that index after visibility filtering, so hidden duplicate controls do not consume it. Role/name, label, and accessible-text targets prefer Chromium's computed AX semantics and return `resolvedBy: "chromium_ax"` on successful mapped actions; CSS selectors, test IDs, and placeholders continue through DOM resolution. Same-origin iframe AX nodes participate in the same bounded semantic read, while cross-origin frame content remains a diagnostic boundary. `browser_snapshot` returns document-scoped live `eN` refs and a `snapshotId`; pass that matching `snapshotId` with every ref-based interaction or element-state wait. The resolver uses the original connected element first and allows one unique, strongly equivalent same-document rebind; CSS selectors remain supported. Text targets used by action and element-state locators project a matching text leaf to its nearest actionable ancestor, so `isEnabled` and `browser_wait` report the state of the control that receives the action. `browser_wait` supports `load`, `url`, `text`, `text_gone`, `visible`, `hidden`, and `enabled`. Text conditions use `text`; element conditions use the same semantic `target` accepted by interactions. `url` and `urlIncludes` can constrain every wait condition. `hidden` succeeds when the target has no visible matches, including an absent target or multiple hidden matches; `visible` and `enabled` require one visible match, and multiple visible matches fail closed. For `visible` and `enabled`, an explicit index is applied after visibility filtering. Browser page matching runs inside the selected tab, while the Bridge continues to enforce the existing browser target and connection-generation fence. If a side-effecting interaction loses its injected result after navigation, it reports an uncertain outcome and is never replayed automatically. ## Architecture ```text Pi Extension ↕ WebSocket 127.0.0.1 Local Bridge ↕ WebSocket Chrome / Edge Manifest V3 Extension ↕ Current Chromium Profile ``` The Bridge binds to loopback and requires a local pairing token for WebSocket operations. Its `/pair` bootstrap response is intentionally available to any caller on that loopback port so an unpacked extension can pair without reading the host token file; any local process or installed extension that can reach the port is therefore trusted with browser control. Keep the Bridge port off network proxies. Host-launched instances expose a non-secret instance id, launcher label and capability list; any paired DSH or Pi Host in the same local-user control domain may request a cooperative restart when the Bridge exposes `capabilities.localUserRestart: true` and has no pending browser request. The instance id prevents stale restart races, and a restart lock serializes concurrent requests. Unknown legacy Bridges remain untouched when they do not expose the local-user capability. Installing the extension and completing local pairing are the trust steps; normal browser operations do not request repeated per-action authorization. ## Installation ### From npm (recommended) ```powershell pi install npm:pi-control-chrome ``` ### From GitHub ```powershell pi install git:github.com/lyd123qw2008/pi-control-chrome ``` ### From a local checkout ```powershell pi install /pi-control-chrome ``` The package registers the Pi extension and the bundled Skill through its `package.json` `pi` manifest. It requires Node.js 22 or newer. ### DSH integration This repository also contains the standalone [`@lyd123qw2008/dsh-tool-control-chrome`](./dsh-tool-control-chrome/README.md) package. It registers the same browser-control surface as model-facing DeepSeek Harness tools and routes calls through the local Bridge. Install the DSH package in the active DSH Profile: ```powershell dsh plugin --profile web add @lyd123qw2008/dsh-tool-control-chrome ``` Then merge the `insert` entry from its `cordis.patch.yml` into the existing `cordis.patch.yml` when installing by hand (`dsh plugin --profile web add` applies that bundle patch automatically), and keep Bridge settings in `/settings.yaml`. The DSH package reuses this project's Bridge and Manifest V3 extension. Its default `lazyTools: true` mode exposes only the `pi-control-chrome` Skill metadata initially; after a successful Skill load, all 43 `browser_*` tools are registered in that Agent and remain active across turns in the current Agent session. At turn end, the host closes unmarked Agent temporary tabs, releases claimed user tabs without closing them, and detaches the session debugger lease. Bridge, browser tools, and Browser binding remain available for later turns. A model must call `browser_mark_handoff` or `browser_mark_deliverable` to preserve an Agent tab through the current turn cleanup, and repeat the mark in a later turn when needed. Only an explicit user request to close temporary tabs, release claims, or clean the browser task may trigger `browser_cleanup`; it performs immediate task cleanup while retaining the lazy tools and healthy Bridge. `browser_context_reset` is the separate explicit user-requested operation that finalizes resources and deactivates lazy tools. Agent and plugin disposal retry final cleanup; failed recovery blocks a replacement Agent that reuses the same session ID until cleanup succeeds. Set `lazyTools: false` for eager visibility. Pi registers the same native tools once but hides them with its active-tool set until the explicit `/skill:pi-control-chrome` expansion or another successful Skill activation. Ordinary web search does not activate browser control. The DSH package also provides human-only `/chrome status`, `/chrome targets`, `/chrome profile [browserId]`, `/chrome connect`, `/chrome disconnect`, `/chrome doctor`, `/chrome restart`, and `/chrome tabs` commands. It does not install browser extensions automatically, read Chrome Profile files, or expose the Bridge beyond loopback. ### Codex CLI and Desktop The repository includes a Codex plugin manifest and a local MCP `stdio` adapter. After installing a package version containing this adapter, register it with `codex mcp add pi-control-chrome -- pi-control-chrome-codex`, or point `codex mcp add` at `codex/mcp-server.mjs` in this checkout. The adapter reuses the existing Bridge and exposes the bounded Codex browser tool set, including `browser_restart` after explicit user confirmation; it does not open an additional MCP port. See [`codex/README.md`](./codex/README.md) for setup and safety rules. ## Load the browser extension Pi cannot install an unpacked browser extension automatically. Load the shared `extension/` directory once in Chrome or Edge: 1. Open `chrome://extensions` or `edge://extensions`. 2. Enable **Developer mode**. 3. Select **Load unpacked**. 4. Choose the repository or installed package's `extension/` directory. 5. Start or reload Pi. Check the connection in Pi: ```text /chrome status /chrome targets /chrome profile /chrome tabs ``` The extension is shared by Chrome and Edge and uses Chromium Manifest V3 capability detection rather than separate browser-specific implementations. One Bridge can now keep multiple identified browser targets connected at once. Each target is keyed by its stable `browserId`; requests use target-qualified routing and connection-generation fencing, so a newer connection for Profile A cannot replace Profile B or satisfy Profile A's stale requests. When multiple targets are available, select one explicitly with `/chrome profile ` or `browser_status`/the Skill CLI `--browser-id`; the runtime never guesses from the newest connection or active window. ## Skill CLI The bundled scripts are for explicit human/developer workflows and automated tests. They connect to the Bridge directly and must not be invoked by a model shell as a substitute for the Skill-gated native tools: ```powershell node skills/pi-control-chrome/scripts/browser.mjs status --browser-id node skills/pi-control-chrome/scripts/browser.mjs tabs --browser-id --json node skills/pi-control-chrome/scripts/browser.mjs group --browser-id --json node skills/pi-control-chrome/scripts/browser.mjs view https://example.com --browser-id --session example-session --turn 1 --screenshot "$env:TEMP\example.png" node skills/pi-control-chrome/scripts/browser.mjs cleanup --browser-id --session [--recover-stale] ``` Managed CLI commands use explicit lifecycle identity: `open` and `cleanup` require `--session `, and `view` requires both `--session ` and `--turn ` unless it is explicitly temporary. The CLI does not derive ownership from a process id, so a later invocation can address the same tabs safely. Use `cleanup --recover-stale` only for an explicit recovery decision after an extension runtime change; it forgets matching unknown-incarnation ownership records without closing those tabs and reports their ids in `recovered` for manual inspection. The bundled scripts support these environment variables. The Pi extension also accepts `PI_CONTROL_CHROME_BRIDGE_PORT` when the local Bridge uses another loopback port; its host remains `127.0.0.1`: ```text PI_CONTROL_CHROME_BRIDGE_HOST # bundled CLI only PI_CONTROL_CHROME_BRIDGE_PORT # bundled CLI and Pi extension ``` The checked-in browser extension still connects to `127.0.0.1:17318` and its manifest CSP allowlist is fixed to that endpoint. Use a matching rebuilt extension and CSP before pointing the Pi extension or DSH client at another endpoint. Use the native `browser_*` Pi or DSH tools for model browser work. They preserve the current Agent session, target identity, and tab ownership protections. User tabs remain valid browser targets, but their documents can be replaced while the user or DSH UI is active; read-only page reads re-observe a changing document once, while side-effecting operations retain strict document fencing. Automated browser verification should use an Agent-owned test tab or an isolated browser profile instead of the active DSH GUI tab. ## Development and tests ```powershell cd /pi-control-chrome npm install npm run check npm test npm run test:pi:lifecycle npm run test:skill npm run pack:check npm run test:package-install ``` `npm run test:skill` requires a connected Chrome or Edge profile and the local Bridge. It creates temporary Agent tabs, so when multiple ready targets exist it skips rather than choosing one; set `PI_CONTROL_CHROME_TEST_BROWSER_ID` only for an explicitly authorized target. The high-coverage browser smoke test is: ```powershell npm run smoke:e2e ``` The multi-target acceptance test launches two isolated temporary profiles of the configured browser (Edge by default, or Chrome for Testing when selected) and verifies explicit routing, target disconnect isolation, and same-Profile reconnection fencing: ```powershell npm run smoke:e2e:multi-profile ``` The smoke test defaults to Edge. The current worktree has passed the same isolated coverage on Edge and Chrome for Testing 149.0.7827.55. Run it against Chrome for Testing with: ```powershell $env:PI_CONTROL_CHROME_BROWSER = "\chrome-for-testing\chrome.exe" npm run smoke:e2e ``` This requests a unique temporary `--user-data-dir` rather than the normal user profile. The harness fails if the browser process exits before the extension handshake, which detects common Windows singleton delegation; Chrome for Testing provides the most reliable isolated executable. The installed Google Chrome may reject command-line unpacked-extension flags; load `extension/` manually from `chrome://extensions` for a normal-profile check. ## Design principles 1. **Install once, operate with low friction.** Extension installation and local pairing are the user-facing trust steps. 2. **The extension is the browser capability boundary.** Pi communicates with it only through the local Bridge. 3. **User tabs are preserved by default.** Claiming a tab establishes control ownership; it does not move the tab into the Agent group. 4. **Agent tabs are reclaimable.** Agent-created tabs carry ownership and session metadata and are cleaned up according to lifecycle policy. 5. **Chrome and Edge share one implementation.** The extension uses Manifest V3 and capability detection. 6. **Observable behavior is aligned without copying private Codex runtime code.** Pi uses its own Bridge protocol and Extension API. 7. **Capability layer, not a site adapter.** The plugin understands page structure, identity, safety boundaries, and generic primitives; a product's DOM, workflow, and field names belong to the calling Skill. See [Capability layer and personalization layer](./ARCHITECTURE.md#capability-layer-and-personalization-layer). ## Documentation - [`ARCHITECTURE.md`](./ARCHITECTURE.md) — English architecture documentation. - [`README-zh-CN.md`](./README-zh-CN.md) — Simplified Chinese guide. - [`FEATURES.md`](./FEATURES.md) — feature scope and future work. - [`CODEX-ALIGNMENT.zh-CN.md`](./CODEX-ALIGNMENT.zh-CN.md) — Codex behavior alignment notes. - [`DECISIONS.zh-CN.md`](./DECISIONS.zh-CN.md) — project decisions. - [`docs/RELEASE-CHECKLIST.zh-CN.md`](./docs/RELEASE-CHECKLIST.zh-CN.md) — package/version, dependency, PR, npm publication and active DSH Profile release checklist. - [`skills/pi-control-chrome-release/SKILL.md`](./skills/pi-control-chrome-release/SKILL.md) — executable release procedure for Pi and DSH packages. - [`skills/pi-control-chrome/SKILL.md`](./skills/pi-control-chrome/SKILL.md) — bundled Pi Skill. - [`CHANGELOG.md`](./CHANGELOG.md) — release history. - [`BROWSER-ACTIVATION-DESIGN.zh-CN.md`](./BROWSER-ACTIVATION-DESIGN.zh-CN.md) — browser capability activation and on-demand Skill design. - [`docs/BROWSER-LIFECYCLE-CODEX-ALIGNED.zh-CN.md`](./docs/BROWSER-LIFECYCLE-CODEX-ALIGNED.zh-CN.md) — implemented Codex-aligned browser lifecycle. - [`docs/BROWSER-OUTPUT-COMPACTION-DESIGN.zh-CN.md`](./docs/BROWSER-OUTPUT-COMPACTION-DESIGN.zh-CN.md) — implemented bounded model-facing browser output and AX revision design. - [`docs/AGENT-BROWSER-RUNTIME-DESIGN.zh-CN.md`](./docs/AGENT-BROWSER-RUNTIME-DESIGN.zh-CN.md) — Agent-first live-ref runtime design, implementation status, and AX runtime status. ## Current future scope The following items are planned for later stages and do not block the current browser-control loop: - WebMCP, GSuite export, and browsing history APIs. - Dedicated media-download interfaces. - Capability discovery beyond the current Bridge target and extension capability handshake. - Chrome Web Store and Edge Add-ons release packages. - Dedicated Brave and Chromium acceptance coverage. - Cross-Bridge orchestration and simultaneous multi-target control within one session. ## License MIT. See [`LICENSE`](./LICENSE).