# pi-vision-fallback

![pi-vision-fallback](https://github.com/mforce/pi-vision-fallback/raw/main/screenshot.png)

Pi extension that keeps your text-only coding model in the driver's seat while still letting it "see".

When a prompt or tool result contains images but the active model has no `image` input modality, this extension asks a configured vision model to describe each image and hands the text model a description (while keeping the image block so the TUI still shows it). The session model, transcript, and thinking level are never changed.

## Install

```bash
pi install npm:pi-vision-fallback
# or from a git checkout:
pi install git:github.com/mforce/pi-vision-fallback
```

## Configure

One-time, permanent — set it from the pi TUI:

```
/vision-model                       # opens a picker of all image-capable models
/vision-model openai-codex/gpt-5    # or pass "provider/modelId" directly
/vision-model clear                 # unset
```

This writes to the **project** config `<cwd>/.pi/vision-fallback.json` (per-repo), falling back to the **global** `~/.pi/agent/vision-fallback.json` when no project dir applies. On read, the project value wins over the global, which wins over env vars — so you can set a global default and override it per repository.

The picker lists the **image-capable models from your configured providers** (the same set `/model` shows), sorted by provider. It's paged (`PgUp`/`PgDn`, `Home`/`End`), type-to-filter (e.g. start typing `gpt`), and marks `★ configured` (the current setting, pre-selected) and `✓ active` (your session model). Descriptions show the model name and context window.

Environment variables also work and are checked after the config file:

```bash
export PI_VISION_FALLBACK_MODEL="openai/gpt-4o"     # "provider/modelId" (or just "modelId" for the active model's provider)
export PI_VISION_FALLBACK_PROMPT="..."              # optional custom description prompt
```

(Legacy names `VISION_FALLBACK_MODEL` / `VISION_FALLBACK_PROMPT` still work.)

Without a configured model — or when the active model already supports images — the extension is inert.

The vision model's provider must be authenticated (API key or OAuth) like any other pi model.

## Behavior

- Active model supports images → no-op.
- Active model is text-only and images are present → each image is sent alone to the vision model with a description prompt. The image block is **kept** (the TUI still renders it; pi's provider layer strips image blocks for text-only models) and a text description is added right before it for the parent model.
- **Prompt-injection guard** — each description is framed as *UNTRUSTED DATA* (the image itself may contain embedded instructions), so the text model treats it as content to analyze, never as commands to follow.
- **Caching** — a description is keyed by model + prompt + image bytes (in-memory LRU + `~/.pi/agent/vision-fallback-cache.json`, 24h TTL). The same image isn't re-described across turns or restarts.
- **Graceful failure** — if the vision call fails, the image is swapped for a placeholder so the parent model moves on instead of retry-looping a dead API.
- A one-time notice is shown if the extension is needed but not configured, or if the vision provider has no credentials.

## Limitations

- The main model receives a text description, not the raw image; fine-grained visual details can be lost.
- Uses pi's own stream dispatch (`streamSimple`), so any builtin provider API works.
- Keeping the original image block (so the TUI still shows it) relies on pi's provider layer stripping image blocks for text-only models. This extension only runs when the active model has no `image` modality.
- The cache lives under `~/.pi/agent/`; delete `vision-fallback-cache.json` to force re-description.

## Development

```bash
npm run check    # typecheck
npm test         # unit tests
```

## Related

Filed against pi core as an enhancement: https://github.com/earendil-works/pi/issues/8815 (request-level routing to a `visionModel` setting would make this extension unnecessary).