# cua-driver: macOS specifics This file is the macOS-specific extension to `SKILL.md`. The cross-platform core (snapshot invariant, CLI/MCP defaults, behavior matrix, canonical loop, pixel-click contract, common error patterns) is in `SKILL.md`. Read this in addition to `SKILL.md` when you're driving an app on macOS. ## The no-foreground contract **Background window actions must preserve the user's frontmost app.** Foreground and desktop actions follow the authorization boundary in [RUNTIME.md](RUNTIME.md#foreground-boundary). This is the whole reason cua-driver exists. Users pay for the right to keep typing in their editor while an agent drives another app in the background. Violate this rule and every other nice property the driver gives you (no cursor warp, no Space switch, no window raise) stops mattering: you just shipped the Accessibility Inspector with extra steps. Before running any shell command, ask: **"does this raise, activate, foreground, or make-key any app?"** If yes, don't run it. Every one of the commands below activates the target on macOS and is therefore forbidden unless the user **explicitly** asked for frontmost state: - **Every form of the `open` CLI: `open -a `, `open -b `, `open `, `open `, `open `: always activates.** macOS routes all forms through LaunchServices, which unhides and foregrounds the target regardless of whether you passed an app name, a bundle id, a document, a URL, or the bundle path itself. The activation happens even when the only intent was "start the process." **Never use `open` for any app launch.** This includes launching a just-built .app from a local build dir (e.g. `open build/Build/Products/Debug/MyApp.app`): resolve the `CFBundleIdentifier` from `Info.plist` and use `launch_app` with that id. See "The narrow carve-out" below for why `launch_app` is safe even when the app internally calls `NSApp.activate`. - `osascript -e 'tell application "X" to activate'`: activates by design. Same for `... to open `, `... to launch`, and anything with `activate` in the tell block. - `osascript -e 'tell application "System Events" to ... frontmost'` in a mutating form (setting `frontmost` rather than reading it). - AppleScript files that invoke `activate`, `launch`, or `open` against the target app. - `cliclick` (moves the user's real cursor to the target coords before clicking: a focus-steal-equivalent even if the app's window state is unchanged). - `CGEventPost` with `cghidEventTap` targeting a coordinate over a different app's window (warps the cursor, possibly activates on hit). - `AppleScriptTask`, `NSAppleScript`, `Process` wrapping `osascript` that contains any of the above. - `NSRunningApplication.activate(options:)` called from your own helper binary: same class. - Dock clicks and any `open` invocation (see the first bullet; every form of `open` goes through LaunchServices which activates, full stop). - **Keyboard shortcuts that semantically mean "focus here": most notably Chrome / Safari / Arc's `⌘L` (focus omnibox) and Finder's `⌘⇧G` (Go to Folder).** These aren't pure key events: the receiving app interprets "user wants to type here" as activation intent and raises its window to be key. Even when delivered to a backgrounded pid via `hotkey`, the downstream app pulls focus. For an exactly bound Chromium page, use `browser_navigate`; otherwise use `launch_app({bundle_id, urls})` to create a separately addressable window. Do not emulate navigation by writing the omnibox and pressing Return in a background window. - **Tab-switching shortcuts in browsers (`⌘1..⌘9`, `⌘]`, `⌘[`, `⌘⇧[`, `⌘⇧]`) are visibly disruptive even when delivered to a backgrounded pid.** The app's key handler processes the shortcut, the window re-renders the new tab's content, and the user sees their tabs flipping. The typed browser route can inspect and mutate a returned `tab_id` without driving this native shortcut. For unsupported browsers, prefer separately addressable windows over visible tab switching. Reading frontmost state is fine (`osascript -e 'tell application "System Events" to get name of first application process whose frontmost is true'`). Mutating it is not. **Corollary: the AXMenuBar rule.** Do not manually drive a background application's `AXMenuBarItem`: the visible macOS menu bar belongs to the frontmost app, and command items may be disabled otherwise. Use `invoke_menu` for a known application-menu path. With the app in the background it resolves the whole path without opening a menu and never activates the app; if macOS keeps the item unavailable to a background app it refuses with `background_unavailable` (then, if the user allows it, `bring_to_front` first). One exception: a disabled Save (the ⌘S item) of a document window is saved through the app's scripting interface instead, still in the background, and the result says so; TextEdit disables Save whenever it is behind the user's window. A Save is verified on disk. A background `hotkey` whose combo is a menu item's key equivalent (⌘S, ⌘W, ⌘C, ...) in a native app takes the same route, because background key events never reach an app's menus; no keys are sent. macOS asks the user once to allow Cua Driver to control the app (Automation) the first time a scripting route runs; when that is off, the call refuses and nothing is saved. With the app already in front it resolves every AX level live. Prefer an in-window element action when the same command has an ordinary control, and prefer `set_window_frame` for exact geometry. Full rationale is in “Navigating native menu bars” below. **"Open \" in user speech means launch, not activate.** `cua-driver launch_app` is the one correct path for process startup: it's idempotent (no-op on a running app), returns the pid, and has an internal `FocusRestoreGuard` that catches `NSApp.activate(ignoringOtherApps:)` calls the target makes during `application(_:open:)` and clobbers the frontmost back to what it was before the launch. That guard is why `launch_app` with `urls` (e.g. `{"bundle_id": "com.colliderli.iina", "urls": ["~/video.mp4"]}`) is safe even for apps that normally foreground on media-load (Chrome, Electron, media players). ## Intent → tool mapping (macOS-specific) | Intent | Use | Don't use | | ------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | Open / launch an app | `launch_app({bundle_id})` or `launch_app({bundle_id, urls:[...]})` | `open -a`, `osascript 'tell app … to launch/activate/open'` | | Find a pid | `list_apps` or `launch_app`'s return | `pgrep`, `ps`, `osascript frontmost` | | Enumerate an app's windows | `list_windows({pid})`: or read the `windows` array `launch_app` already returns | `osascript 'every window of app …'` | | Move or resize one exact window | `set_window_frame({pid, window_id, x, y, width, height})` | `osascript` position/size writes or title-bar dragging | | Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open ` | | Loop, branch or retry over app actions | `run_script({script})` (sandboxed JavaScript over the `cua` API, one call); `run_steps` for a fixed list | an AppleScript or shell loop around `osascript` or `cua-driver` calls | | Drag / drag-and-drop / marquee select | `drag({pid, window_id, from_x, from_y, to_x, to_y})` in the background (pixel-only: macOS AX has no semantic drag). Verify with a screenshot; `delivery_mode:"foreground"` only if nothing changed. A bound browser tab: `browser_pointer` `action:"drag"` | `cliclick dd:`, `osascript drag`, refusing a drag-only step as "needs the window in front" | | Hover (reveal a hover-only control) | Window-targeted `move_cursor`; its result says whether it hovered (AX alternate UI or an Electron page). A web page in Chrome, Edge or Brave: `browser_pointer` hover after `get_browser_state`. If not, look for the same command elsewhere (menu item, context menu, shortcut, help text in `get_window_state`); only `move_cursor({scope:"desktop", delivery_mode:"foreground"})` (moves the user's pointer) or `bring_to_front` reveals it, and only if the user allows it | assuming a window-targeted `move_cursor` hovered when its result says the app sees no hover; a desktop `move_cursor` over a background browser page | | Screenshot | `get_window_state` (window) or authorized `get_desktop_state` (desktop) | `screencapture` | | Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` | | Hand a file/URL to an app | `launch_app({bundle_id, urls:[]})` | `open -a `, `open ` | | View a local file in the GUI | `launch_app({bundle_id, urls:["file:///…"]})` | typing `file://` into the omnibox | ### The narrow carve-out For authorized foreground input, use the Cua action's `delivery_mode:"foreground"`. For requested persistent foreground state, use `bring_to_front`. Neither requires a shell activation workaround. When a cua-driver call surprises you, diagnose cua-driver first: - **Large tree or repeated reads?** `get_window_state` returns one compact markdown tree capped at 250 nodes; pass `since:"latest"` for a diff (`no change since …; focused element is …` names the focused row on macOS) and `tree_format:"elements"` for structured rows. `full_output:true` restores the old payload (both forms, `_note`, `background_input`, 2 000 nodes). - **Empty `tree_markdown`?** `get_window_state` returns **both** the AX tree and a screenshot by default: there's nothing to configure and no capture mode to pick. An empty tree means the surface isn't AX (a non-AX surface: Electron/Chromium/canvas), and the response carries `degraded: true`, so act by **`px`** off the screenshot that's already in the same response. `capture_mode` is **deprecated and ignored** (still accepted so old callers don't error, but it has no effect; tree + screenshot come back regardless); don't reach for `get_config` to "switch modes," there is no mode to switch. - **`has_screenshot: false`?** The window capture failed (transient race against a close, or the window has no backing store yet). Re-snapshot; if persistent, pick a different `window_id` via `list_windows`. - **`stale_element_token` / no cached AX state?** Re-snapshot the exact window and use the new `element_token`. A new snapshot of that window invalidates older targets immediately; the refusal names the current snapshots so you can tell whether you already hold one. - **Sparse Chromium AX tree?** Retry `get_window_state` once: the tree populates on second call. Only after those are ruled out, and only if the user's action genuinely needs frontmost state, fall through to the activate fallback. Always name the focus steal in your response ("I'll briefly bring Chrome to the front because …"). ### Verifying actions: cross-check the tree against the pixels you already have There is no `ax`/`vision` capture toggle. **Every `get_window_state` returns both the AX tree and a screenshot** (default), so verifying that an action **landed** never means "go grab a screenshot": it means cross-check the tree diff against the pixels you already have in the same response, and only switch _dispatch rung_ on a real signal: 1. **Re-snapshot and read the tree diff**: a changed `AXValue`, a new element, a collapsed menu, a disabled button. If the tree shows the change, you're done. When you only need the tree diff and don't need fresh pixels, pass `since:"latest"` and `include_screenshot:false` to skip the grab: a **perf** knob, not a mode flip. The window keeps its last screenshot's pixel frame while its size is unchanged, so `x,y` from that screenshot still work afterwards. 2. **Trust the screenshot and do an element px action** when the tree **lies**: the action response carried `effect:"suspected_noop"`, the re-snapshot came back `degraded` (empty tree), or the tree looks unchanged/unreadable / disagrees with the pixels on a surface where it's known to lie: - **Canvas-backed editors**: Monaco (VSCode, Cursor), xterm, Figma, WebGL. The AX tree shows the chrome but nothing for the canvas content; a snapshot's tree can look unchanged after a successful edit while the pixels show it landed. - **Catalyst / iOS-on-Mac text views**: see "Known text-input limits" above. `AXValue` can lag the rendered pixels or report the placeholder while the field is actually populated. On these surfaces you read the result off the screenshot already in the response, then address the target by `x,y`: an **element px action**. `px` is your **conscious switch to the pixel addressing path**, not a different capture: the screenshot was always there, you just change _how you address_ the target. The point is to catch the "type → AX-check succeeds → believe the lie → find out three calls later" trap on exactly the surfaces that warrant it. Rule of thumb: - **element ax action** (default): the element lookup before a click AND the first verify after it; you address by the `[N]` row's `element_token` and read the tree diff. - **element px action**: when the tree is unreadable / `suspected_noop` / `degraded` / disagrees with the pixels, or for pure visual inspection (reading a chart). You address by `x,y` off the screenshot that's already in the snapshot response. ### Self-check pattern Before every `Bash` call whose command line touches any macOS app (launching, opening, clicking, typing, scripting, screenshotting), run the self-check: 1. **Does this command foreground the target?** If yes: stop and translate to the cua-driver equivalent from the mapping table. 2. **Does this command move the user's real cursor?** (`cliclick`, any `CGEventPost` at `cghidEventTap` over another app's window). If yes: stop; use `click({pid, x, y})` which routes per-pid via SkyLight and never warps the cursor. 3. **Does this command bypass cua-driver entirely?** (`osascript` mutating GUI state, AppleScript files, external helpers.) If yes: stop; find the cua-driver tool that does the intent. If all three are "no," the command is safe. If you can't answer, default to stop and ask rather than proceed. A single `open -a` run by accident kills the demo, the trust, and the user's in-flight editor state. ## Prerequisites: macOS 1. `cua-driver` is on `$PATH` (`which cua-driver`). If not, point the user at `scripts/install-local.sh` and stop. 2. Start the daemon with `open -n -g -a CuaDriver --args serve` (the recommended form: goes through LaunchServices so TCC attributes the process to CuaDriver.app). `cua-driver serve &` also works; the CLI auto-relaunches through `open -n -g -a CuaDriver` when it detects a wrong-TCC context (any IDE-spawned shell: Claude Code, Cursor, VS Code, Conductor). Verify with `cua-driver status`. 3. Run `cua-driver permissions status --json`. This path is read-only: it checks Accessibility and Screen Recording but deliberately does not run Tahoe's prompt-capable direct-capture probe. Therefore `screen_recording_capturable` is `null` and `direct_capture_status` is `"not_checked"` until the explicit grant flow. - If Accessibility is `false`, stop. AX reads and actions cannot work; tell the user to run `cua-driver permissions grant` and approve it. - If Screen Recording is `false`, continue only when the task can be completed and verified from the AX tree. Call `get_window_state` with `include_screenshot:false` and use `element_token` AX actions. Do not use screenshots, pixel coordinates, or pixel-based verification. - If the task materially needs pixels, stop and ask the user to run `cua-driver permissions grant`. That command explains and deliberately triggers the additional private-window-picker bypass dialog before verifying live capture. macOS mentions screen and audio in the combined consent, although Cua Driver's current recorder does not enable audio. If the installed app is absent from **Screen & System Audio Recording**, the user should click **+**, add `/Applications/CuaDriver.app` (or `/Applications/CuaDriverLocal.app`), enable it, and rerun the command. ## Resolve the requested application Reuse a discovered live target when available. Otherwise use `launch_app` when launch is requested or implied, then select the returned window. If the window has not appeared yet, bound retries of `list_windows`. - `launch_app({bundle_id: "com.apple.finder"})`, preferred, unambiguous. - `launch_app({name: "Calculator"})`, when bundle_id isn't known. `launch_app` is a **hidden-launch primitive by design**: that's the entire point of cua-driver: agents drive apps in the background while the user keeps typing in their real foreground app. The target's window is initialized (AX tree fully populated, clickable via `element_token`, the pid appears in `list_apps`) but not drawn on screen. The driver never activates or unhides apps on its own; that would violate the no-foreground contract the whole driver exists to protect. If the user explicitly wants the window visible (usually for a demo or recording), they unhide it themselves: Dock click, Cmd-Tab, or Spotlight. Do not reach for `open` / `osascript activate` as a shortcut to make the window visible; those paths break the backgrounded invariant on every call, not just the call that "needed" the foreground. Say out loud what the user needs to do ("click the Todo app in your Dock to bring it forward") and let them do it. Never shell out to **any** form of `open` (including `open ` for a just-built binary: resolve the bundle id from `Info.plist` and use `launch_app` with that), `osascript 'tell app … to launch/open'`, or similar. Those paths activate the target, bypass the driver's focus-restore guard, and require a Bash permission prompt the agent loop shouldn't be burning on app launch. ## Pixel-click dispatch (macOS) The pixel click is routed through SkyLight's per-pid event path (`SLEventPostToPid`), not the system HID stream. The dispatch recipe is the backgrounded "noraise" sequence: a focus-without-raise SLPS event record to the target window, followed by an off-screen user-activation primer and the real click. The target window becomes AppKit-key for event routing, but it does **not** rise to the front of the z-stack, macOS's "switch to Space with windows for app" follow is suppressed, and the user's front window keeps key status, so their typing keeps landing there. ### Menus opened in the background keep the user's keyboard A popup button, a `` in Chrome does not follow a background drag (pointer-event widgets do). Use `set_value` on its slider element, or `browser_pointer` `action:"drag"` on a bound tab, which sends the drag through CDP `Input.dispatchMouseEvent` with the button held. A drag that starts a macOS drag-and-drop session (reordering a list by drag and drop, moving a file, HTML5 `draggable` content) cannot drop in the background: macOS tracks those sessions with the real pointer. The driver sees the session start, releases at once, and returns `background_unavailable` with `reason: "drag_and_drop_session"`. Look for a non-drag route first (Move Up/Down, a menu or context-menu command, a shortcut, cut and paste), then foreground if the user allows it. Toolkits that read the hardware pointer (Tk) or drop routed mouse events (LibreOffice) refuse the background drag with `background_unavailable` before anything is sent. A foreground window-scoped **pixel** `click`, `double_click`, or `right_click` (`x`/`y` with `window_id`) is delivered like a desktop-scope click, not through the per-pid path: Cua Driver activates the exact window (refusing with `foreground_unavailable` when that window never becomes focused), moves the hardware pointer to the mapped screen point, and posts through the HID event tap. The pointer stays at the target afterward, matching Windows `SendInput` and X11 XTest foreground clicks, and the prior frontmost app is restored. The result reports `route: "global_input"` and says the pointer moved. Modified clicks are the deliberate exception: pass `delivery_mode:"foreground"` and a concrete `window_id`. macOS applications can discard PID-routed modifier state after initially publishing a transient selection, so Cua Driver refuses that background combination. The foreground rung holds physical HID modifier keys around the click, restores the prior foreground app, and confirms list-like selection changes with a stable AX readback. Element-addressed modified clicks also restore the hardware cursor; pixel modified clicks leave it at the target like other foreground pixel clicks. macOS-specific residuals worth knowing (the rest of the capture/dispatch/ addressing params are a shared cross-platform contract: see `SKILL.md` → _Cross-platform parameter contract_): - **`check_permissions.prompt` is macOS-only and public calls are status-only.** Omitted `prompt` defaults to `false`; explicit `true` is refused before platform dispatch in every mode. For a signed standalone install, the human-run `cua-driver permissions grant` command launches a short-lived CuaDriver app instance through LaunchServices so macOS owns the approval UI and direct ScreenCaptureKit probe. In an in-process SDK runtime, private embedded host, or `cua-driver mcp --direct`, the embedding host owns permission UX. There is no Windows/Linux equivalent. - **`session` always worked on macOS;** the cross-platform change is that Windows/Linux stopped _rejecting_ it. No macOS-side change to how you pass it. - **`target`** selects the action coordinate space uniformly on all platforms. Use `target:{"kind":"desktop","display_id":"primary"}` for screen-absolute pointer actions or foreground keyboard actions, and `target:{"kind":"window","pid":PID,"window_id":WINDOW_ID}` for exact window coordinates. Legacy flat `scope`, `pid`, and `window_id` fields remain compatibility inputs, but do not combine them with `target`. ### Canvases, viewports, games (Blender, Unity, GHOST, Qt, wxWidgets) Apps whose main surface is an OpenGL / Metal / Qt / wxWidgets viewport expose **no useful AX tree**: the whole surface is one opaque `AXGroup` or `AXWindow` from AX's perspective. Per-pid event paths (`SLEventPostToPid`, `CGEvent.postToPid`) are filtered by the viewport's own event-source check and silently dropped: the event loop wants "real HID origin". The working pattern: 1. Observe the target and the background route's refusal or lack of effect. 2. Obtain authorization for visible foreground control if not already given. 3. Retry only the necessary Cua action with `delivery_mode:"foreground"`, then verify from fresh state. Do not inject raw CGEvents or use shell activation. There is no backgrounded path that reaches these apps today. ### Known pixel-click limits - **Toolkits that read the hardware pointer (Tk, some Java and game toolkits)** derive a click's location from the real pointer position, not from the delivered event. Background delivery never moves the pointer, so a background pixel click lands wherever the pointer happens to be. When the target maps a Tk library or Python's `_tkinter`, a background window pixel click returns `background_unavailable` with `reason: "pointer_reading_toolkit"`. Other pointer-reading apps cannot be detected cheaply; their background result stays "not driver-verified" and says the pointer was not moved. Retry with `delivery_mode:"foreground"`, which moves the pointer first. - **Chromium `