# pi-vision-agents 🧩🎬

Vision agents for [pi](https://github.com/earendil-works/pi). If your pi model
doesn't support images or videos, this package gives you two tools that hand
the work to dedicated vision/video models:

- **`forward_to_pimage`** — describe, OCR, or analyze an image with a vision model
- **`forward_to_pideo`** — summarize/transcribe a YouTube or TikTok video

Both are file-bridge daemons: the extension writes a request, a small Python
daemon picks it up, calls the provider API, and writes back the result. No
terminal multiplexer required — works in any terminal where pi runs.

> **cmux note:** you can run these in a plain pi session. The `examples/`
> directory contains optional pane-launch scripts for
> [cmux](https://github.com/manaflow-ai/cmux) users who want a dedicated
> vision pane, but they are **not** required.

---

## Install

```bash
pi install npm:pi-vision-agents
```

Then **add your API keys** to your shell environment (or pi's `.env`):

```bash
# pimage (image analysis) — OpenAI-compatible vision API
export OPENCODE_API_KEY="your-key-here"

# pideo (video analysis) — OpenRouter
export OPENROUTER_API_KEY="your-key-here"
```

### Dependencies

| Requirement | Needed for | Install |
|---|---|---|
| `python3` (≥3.10) | both daemons | system Python |
| `yt-dlp` | pideo (video metadata + subtitles) | `pip install yt-dlp` or `brew install yt-dlp` |

Both daemons start **lazily** on first tool call and stop on session end — no
manual daemon management needed.

---

## Usage

In pi, just ask:

```
forward_to_pimage: /path/to/screenshot.png — "Extract all text from this image"
forward_to_pideo:  https://www.youtube.com/watch?v=... — "Summarize this video"
```

The extension also auto-detects when you attach an image in a non-vision
model session and tells the model to use `forward_to_pimage`.

---

## Configuration

Everything is env-var driven. No hardcoded paths, no baked-in secrets.

| Env var | Default | Purpose |
|---|---|---|
| `PIMAGE_DIR` | `~/.pimage` | pimage state dir (inbox/outbox/signal) |
| `PIMAGE_MODEL` | `opencode-go/mimo-v2.5` | Vision model |
| `PIMAGE_API_BASE` | `https://opencode.ai/zen/go/v1` | OpenAI-compatible API base |
| `PIMAGE_DAEMON` | package `daemons/` | Explicit daemon path override |
| `PIMAGE_TIMEOUT_MS` | `120000` | Response timeout |
| `OPENCODE_API_KEY` | — | **Required** for pimage |
| `PIDEO_DIR` | `~/.pideo` | pideo state dir |
| `PIDEO_MODEL` | `google/gemini-2.5-flash` | Video analysis model |
| `PIDEO_API_BASE` | OpenRouter chat completions | API base URL |
| `PIDEO_DAEMON` | package `daemons/` | Explicit daemon path override |
| `PIDEO_TIMEOUT_MS` | `300000` | Response timeout |
| `OPENROUTER_API_KEY` | — | **Required** for pideo |

---

## How it works

```
pi session                          python daemon
┌──────────────────┐   file write   ┌──────────────────┐   HTTPS   ┌─────────────┐
│ forward_to_pimage│ ─────────────▶ │ pimage-daemon.py │ ────────▶ │ vision API  │
│ extension        │ ◀───────────── │ polls ~/.pimage  │ ◀──────── │ (OpenAI-    │
│ polls result     │   result.json  │                  │  result   │  compatible)│
└──────────────────┘                └──────────────────┘           └─────────────┘
```

1. Tool call writes `{id, image_path/url, instruction}` to `$DIR/in/<uuid>/`
2. Signal file `$DIR/current` wakes the daemon
3. Daemon calls the provider (base64 image / yt-dlp transcript)
4. Result written to `$DIR/out/<uuid>/result.json`, extension returns it

---

## Security & privacy

- **No secrets in the package.** All API keys come from your environment.
- **No PII.** The package collects nothing — requests are only sent to the
  provider API you configure.
- **Files stay local.** Images/transcripts pass through your local state dir
  and are deleted after processing (`$DIR/in/<uuid>/` is cleaned up on
  success/timeout).
- Your data goes to whichever provider you configure (`OPENCODE_API_KEY`
  provider for pimage, OpenRouter for pideo). Review their policies.

---

## Development

```bash
git clone https://github.com/ezadEzanee/pi-vision-agents
# try locally without installing:
pi -e ./path/to/pi-vision-agents
```

Run a daemon manually for debugging:

```bash
PIMAGE_DIR=/tmp/pimage-test OPENCODE_API_KEY=... python3 daemons/pimage-daemon.py
PIDEO_DIR=/tmp/pideo-test OPENROUTER_API_KEY=... python3 daemons/pideo-daemon.py
```

---

## License

MIT