# dsh-mac-vision

[简体中文](./README.md)

A **native macOS vision plugin for DeepSeek Harness**. It lets text-only models inspect local images, clipboard images, screens, and application windows on demand, using Apple Vision for fully on-device OCR and detection.

## One plugin, not two products

Users install `dsh-mac-vision` once.

The plugin contains two cooperating parts:

- **Native tool layer:** performs capture, OCR, and Vision analysis and registers `mac_vision_inspect` and `mac_vision_list_windows` for the model.
- **Bundled usage policy:** automatically registers as a Harness Skill and teaches the model when to call the tools and how to separate direct observations, semantic inferences, and uncertain OCR.

The bundled Skill is not another product to install, enable, or choose. It cannot perform OCR by itself and does not compete with the plugin; it is the model-facing operating policy shipped inside the plugin.

```text
dsh-mac-vision native plugin (install once)
├── Native Tools: provide macOS Vision capabilities
└── Bundled Skill: teaches the model to use those tools reliably
```

## Capabilities

- Inspect local files, clipboard images, the full screen, the frontmost window, or a selected window.
- Run local OCR through macOS Vision.
- Return structured evidence containing text, confidence, coordinates, alternatives, and verification state.
- Automatically crop and upscale suspicious small text or notation for a second OCR pass.
- Optionally run image classification, barcode, saliency, people, and cat/dog detection.
- Distinguish `not_requested`, completed detection with no results, and conflicting OCR.
- Respect Harness cancellation signals, timeouts, and output-size limits.
- Keep image analysis on the Mac without uploading images to a vision-model service.

Model-facing tools:

- `mac_vision_inspect`: inspect an image, clipboard, screen, or window.
- `mac_vision_list_windows`: list visible windows before selecting one.

## Requirements

- macOS
- Node.js 22 or later
- DeepSeek Harness
- Xcode Command Line Tools or Xcode with a Swift compiler

On the first call, the plugin compiles its bundled Swift helper into the macOS user cache. Screen and window capture may require granting Screen Recording permission to the DeepSeek Harness host in System Settings.

## Installation

Build a tarball from the project directory:

```sh
pnpm install
pnpm pack
```

Install it into a DeepSeek Harness profile:

```sh
dsh plugin --profile default add ./dsh-mac-vision-0.1.0.tgz
dsh --profile default --dump-config
dsh --profile default
```

The tarball contains prebuilt JavaScript, so installation does not require permission to run a git dependency's `prepare` script. The native tools and bundled policy load together; do not separately copy or install `SKILL.md`.

## Usage examples

Ask for tasks that depend on visual information:

```text
Read the title and main content of /absolute/path/to/slide.png.
```

```text
Inspect the frontmost window and separate what is directly observed from your interpretation.
```

```text
List the visible windows, then inspect the error shown in the browser window.
```

The bundled policy tells the model to choose the narrowest source and report:

- **Observed:** evidence directly returned by OCR or completed detectors.
- **Inferred:** interpretations derived from layout, text placement, or multiple observations.
- **Uncertain:** conflicting OCR, unverified notation, and detectors that were not run.

## Configuration

Defaults are defined in [`cordis.patch.yml`](./cordis.patch.yml):

```yaml
config:
  timeoutMs: 45000
  maxOutputBytes: 8388608
  defaultMode: fast
  defaultLanguages: []
  defaultRefineText: true
  defaultRefineLimit: 12
  defaultRefineScale: 3
  allowedSources: [file, clipboard, screen, front-window, window]
```

Override them in the profile's own `cordis.patch.yml`. Harness patch layers replace the whole `config` value instead of deep-merging it, so repeat the complete config block when overriding the row.

Use `allowedSources` to disable capture sources that a deployment should not expose. For example, allow local files only:

```yaml
allowedSources: [file]
```

## Local development

```sh
pnpm install
pnpm check
```

When developing from a DeepSeek Harness source checkout, create an overlay with absolute paths:

```yaml
- insert:
    - id: mac-vision-tools
      name: '/absolute/path/to/ds-mac-vision/src/index.ts'
    - id: mac-vision-skill
      name: '/absolute/path/to/ds-mac-vision/src/skill.ts'
```

Start Harness from its repository:

```sh
pnpm dsh web --patch /absolute/path/to/overlay.yml
```

These are two Cordis entry points of the same native plugin: the first registers tools and the second registers the bundled policy. A normal bundle installation contributes both automatically, so users do not configure them separately.

## Verification

```sh
pnpm typecheck
pnpm test
pnpm pack
```

Tests cover argument mapping, the native helper, the real Cordis tool execution pipeline, and the Harness Skill registry.