# Windows Computer Use

Use this workflow only when `computer.getPlatform()` returned `windows`.
The shared entrypoint has already initialized `computer`.

## API surface

```ts
type WindowTarget = { pid: number; windowId: number };
type ElementTarget = {
  pid: number;
  windowId?: number;
  elementToken: string;
};
type CoordinateTarget = WindowTarget & { x: number; y: number };
type PointOrElementTarget = CoordinateTarget | ElementTarget;
type ExactActionTarget = WindowTarget | ElementTarget;
type DeliveryOptions = { deliveryMode?: DeliveryMode };

type ComputerUse = {
  click: (
    args: PointOrElementTarget &
      DeliveryOptions & { button?: MouseButton; count?: number },
  ) => Promise<object>;
  doubleClick: (
    args: PointOrElementTarget & DeliveryOptions,
  ) => Promise<object>;
  rightClick: (
    args: PointOrElementTarget & DeliveryOptions & { modifier?: string[] },
  ) => Promise<object>;
  drag: (
    args: WindowTarget &
      DeliveryOptions & {
        fromX: number;
        fromY: number;
        toX: number;
        toY: number;
      },
  ) => Promise<object>;
  observeWindow: (
    args: WindowTarget & {
      disableDiff?: boolean;
      includeScreenshot?: boolean;
      maxTextChars?: number;
    },
  ) => Promise<WindowObservation>;
  listApps: () => Promise<Array<App>>;
  listWindows: (args: {
    pid: number;
    onScreenOnly?: boolean;
  }) => Promise<Array<Window>>;
  performSecondaryAction: (
    args: ElementTarget & { action: string },
  ) => Promise<object>;
  pressKey: (
    args: ExactActionTarget &
      DeliveryOptions & { key: string; modifiers?: string[] },
  ) => Promise<object>;
  hotkey: (
    args: ExactActionTarget & DeliveryOptions & { keys: string[] },
  ) => Promise<object>;
  scroll: (
    args: PointOrElementTarget &
      DeliveryOptions & { direction: Direction; amount?: number },
  ) => Promise<object>;
  setValue: (args: ElementTarget & { value: string }) => Promise<object>;
  typeText: (
    args: ExactActionTarget & DeliveryOptions & { text: string },
  ) => Promise<object>;
  close: () => Promise<void>;
};

type App = {
  name?: string;
  bundle_id?: string;
  pid?: number;
  running?: boolean;
  launch_path?: string;
};

type Window = {
  window_id: number;
  title?: string;
  is_on_screen?: boolean;
  on_current_space?: boolean;
};

type WindowObservation = {
  pid: number;
  windowId: number;
  mode: 'full' | 'diff' | 'no_change';
  resyncReason?: string;
  text: string;
  elements: Array<Element>;
  screenshot?: Screenshot;
};

type Element = {
  element_token?: string;
  role?: string;
  label?: string;
  automation_id?: string;
  value?: unknown;
  actions?: string[];
};

type Screenshot = {
  images: Array<{ mimeType: string; dataBase64: string }>;
};

type Direction = 'up' | 'down' | 'left' | 'right';
type MouseButton = 'left' | 'right' | 'middle';
type DeliveryMode = 'background' | 'foreground';
```

## Workflow

### 1. Initialize

Start by getting the state for the app and window you want to use. When the task
names an app, filter for that name directly:

The snippets below show individual steps. When the app and visible window
unambiguously match the task, combine discovery, selection, and observation in
one `node_repl` call. Print candidate lists for selection when they are ambiguous.

```js
var apps = await computer.listApps();
var matches = apps.filter(
  (app) => app.name === 'Target App' || app.bundle_id === 'com.example.target',
);
nodeRepl.write(JSON.stringify(matches));
```

Use the matching app's observed PID as `targetPid`, then list its visible windows:

```js
var windows = await computer.listWindows({
  pid: targetPid,
  onScreenOnly: true,
});
nodeRepl.write(JSON.stringify(windows));
```

Select the intended document or dialog from the returned titles and window IDs;
the first entry is not necessarily the task window. Set `target` to
`{ pid: targetPid, windowId: selectedWindowId }` using that observed ID, then read
its state:

```js
var state = await computer.observeWindow(target);
nodeRepl.write(state.text); // This will return the accessibility tree
```

If you cannot identify an app from the task, prior context, or builtin apps,
start by discovering the available apps:

```js
var apps = await computer.listApps();
nodeRepl.write(JSON.stringify(apps));
```

After performing one or more UI actions, call `observeWindow(...)` before
deciding what to do next. This keeps you in the current UI state. Read the
current actionable `element_token` values from `state.elements`; tokens for
unchanged elements remain valid across `diff` and `no_change` observations.

For token efficiency, when appropriate, the accessibility tree will be returned
as a diff from the most previous accessibility tree, listing only the elements
that were removed, added, or changed. Prefer this default diff output; pass
`disableDiff: true` only when you need a fresh full accessibility tree. If you
disregard the text from a previous call to `observeWindow`, such as when you only
emit the screenshot, get the full tree next time you inspect AX text.
`state.elements` remains the current full actionable element list when
`state.text` is a diff or reports no change.

Returned text defaults to at most 12,000 characters. Set `maxTextChars` (minimum 512) to adjust the limit. When text is truncated, filter `state.elements` for
the controls you need or request `disableDiff: true` with a larger
`maxTextChars`. Do not repeatedly print the whole array. An omitted row does not
prove absence. Incomplete captures retain only currently issued action tokens;
after a read failure, use only tokens from the latest observation.

### 2. Actions using the exact window

Choose the delivery mode and current window before batching actions:

- End the input batch when opening a dialog or menu. For a new dialog, call `listWindows({ pid, onScreenOnly: true })` and observe the matching returned window ID before typing. Do not guess an ID if no matching window is returned. For ordinary menus that remain in the current window, refresh that window's observation and use its current menu tokens.
- An AX action can return an error after the app has already opened or closed a window. Check the current windows and state before retrying; do not assume the error means nothing happened or repeat the old token immediately.

Batch actions whose target remains the same, then fetch the latest state:

```js
await computer.click({ pid: target.pid, elementToken });
await computer.setValue({ pid: target.pid, elementToken, value: 'openai.com' });
await computer.pressKey({ ...target, key: 'Enter' });
await computer.typeText({ ...target, text: 'hello' });
await computer.scroll({
  pid: target.pid,
  elementToken,
  direction: 'down',
  amount: 1,
});
await computer.performSecondaryAction({
  pid: target.pid,
  elementToken,
  action: 'Show Menu',
});
nodeRepl.write((await computer.observeWindow(target)).text);
```

Notes:

- Prefer `element_token`-based actions over coordinate actions. If AX actions or AX text are unavailable or behave unexpectedly, switch to screenshots, coordinate clicks, and key presses.
- For window-targeted coordinate actions, use pixels in the PNG returned by `observeWindow` for that exact window, with its top-left corner as `(0, 0)`. AX `frame` values are screen-space logical points, not PNG pixels; do not pass an AX frame's center directly as `x`/`y`. Use a current element token or locate the control in the screenshot. The driver already accounts for display scale and image downscaling.
- `doubleClick` and `rightClick` invoke their dedicated SDK actions; `rightClick` also accepts optional modifiers.
- If the UI is not behaving as expected, try fetching the latest `observeWindow(...)` state to make sure you have the latest context.
- Prefer using accessibility text over screenshots for efficiency, but if the interface is not fully working or not providing enough context, make sure to fetch a screenshot to get more context. The accessibility interface may be incomplete in some applications, so a screenshot helps fully understand what is going on.
- `performSecondaryAction` invokes an accessibility action that an element exposes besides a normal click, such as expanding a disclosure row, showing a menu, incrementing a control, or cancelling something. It requires an action actually exposed for that element in the accessibility text. Do not guess action names.
- `pressKey` presses one key and accepts optional modifiers; `hotkey` sends a key combination such as `{ keys: ['ctrl', 'c'] }`. Single-key examples include `"a"`, `"Enter"`, `"Tab"`, and `"Up"`.
- Take care when passing strings containing `\n` or `\r` to `typeText`, as it simulates pressing the return key. Many apps with message composers or forms will respond by sending the message or submitting the form rather than inserting a newline.

## Reading screenshots

Request a screenshot with the observation, read its accessibility text, and
emit each returned image:

`includeScreenshot: true` is the parameter that requests a screenshot.

```js
var state = await computer.observeWindow({
  ...target,
  includeScreenshot: true,
});
nodeRepl.write(state.text);
for (const image of state.screenshot?.images ?? []) {
  await nodeRepl.emitImage(`data:${image.mimeType};base64,${image.dataBase64}`);
}
```

When all Computer Use work is complete:

```js
await computer.close();
globalThis.computer = undefined;
```

Reset the Node REPL only when no other persistent state is needed.