<div align="center">

# Pi Vision Bridge — Vision for Text-Only Models

**Give text-only models (DeepSeek, Llama, Qwen, local Ollama models) the ability to see images — without switching models mid-task.**

[![License](https://img.shields.io/badge/License-MIT-green?logo=opensourceinitiative&logoColor=white)](./LICENSE)
[![Pi](https://img.shields.io/badge/Pi-0.83+-6B5B95?logo=pi&logoColor=white)](https://pi.dev)
[![Zero Deps](https://img.shields.io/badge/Zero-Dependencies-2E8B57?logo=npm&logoColor=white)](./package.json)
[![PRs Welcome](https://img.shields.io/badge/PRs-Welcome-brightgreen?logo=github)](https://github.com/wuxiangru915/pi-vision-bridge/pulls)

[English](README.md) | [中文](README_CN.md)

</div>

---

Your coding agent runs on a text-only model and executes the whole task. When it needs to understand an image — a page screenshot, an error message, a UI mockup — it delegates to a vision model, gets a text description back, and keeps going. The active model never changes.

## Features

- **`describe_image` tool** — the agent calls it at any point during a task: pass an image path and an optional question, get a text description back. Works with any vision model pi can talk to.
- **Automatic image-to-text fallback** — paste or attach an image while your model has no vision; it is described before reaching the model, so the conversation keeps working with no API errors.
- **Automatic model discovery + fallback** — finds a vision-capable model in your pi registry; if a candidate fails, it transparently tries the next one.
- **Dual API support** — OpenAI-compatible endpoints (`/chat/completions`) and Google Generative AI (`generateContent`) out of the box.
- **Proxy-aware** — honors the standard `HTTPS_PROXY` / `HTTP_PROXY` / `PI_VISION_PROXY` environment variables via a zero-dependency CONNECT tunnel.
- **Zero runtime dependencies** — no npm packages, no binary downloads.

## How it works

```
Your agent (text-only, e.g. DeepSeek)
    │  executing a long task
    │  ── needs to see an image ──▶ calls describe_image(path, question)
    │                                    │
    │                                    ▼
    │                        vision model (Gemini / Qwen-VL / GLM / ...)
    │                                    │
    │  ◀── gets text description ────────┘
    │  continues the task, model never switched
```

## Installation

```bash
pi install npm:@wuxiangru/pi-vision-bridge
```

Or install from git:

```bash
pi install git:github.com/wuxiangru915/pi-vision-bridge
```

Or try it without installing:

```bash
pi -e git:github.com/wuxiangru915/pi-vision-bridge
```

> **Note:** pi extensions run with full system access. Review the source before installing.

## Configuration

The vision model is resolved from pi's model registry (`~/.pi/agent/models.json`), so any provider pi can authenticate with works out of the box.

### Option A: Automatic discovery (default)

With no explicit config, the extension picks the first available model that accepts images, has configured auth, and scores best on common chat formats — preferring your current provider, then the top candidates with automatic fallback on failure.

### Option B: Explicit configuration (recommended)

Pin a specific provider/model with environment variables:

```bash
export PI_VISION_PROVIDER=google
export PI_VISION_MODEL=gemini-3-flash-preview
```

Both must be set; this takes priority over auto-discovery. The provider and model must be defined in `~/.pi/agent/models.json` with `"input": ["text", "image"]`:

```json
{
  "providers": {
    "google": {
      "baseUrl": "https://generativelanguage.googleapis.com/v1beta",
      "api": "google-generative-ai",
      "apiKey": "$GEMINI_API_KEY",
      "models": [
        { "id": "gemini-3-flash-preview", "input": ["text", "image"], "contextWindow": 1000000 }
      ]
    }
  }
}
```

### Supported vision models

Any model pi can authenticate and call:

| Provider | Example models | API format |
|----------|---------------|------------|
| Google Gemini | `gemini-3-flash-preview`, `gemini-2.5-pro` | `google-generative-ai` |
| Alibaba Qwen | `qwen-vl-max`, `qwen2.5-vl` | OpenAI-compatible |
| Zhipu GLM | `glm-4v`, `glm-4v-plus` | OpenAI-compatible |
| OpenAI | `gpt-4o-mini`, `gpt-4o` | OpenAI-compatible |
| Local | Ollama vision models (`llama3.2-vision`) | OpenAI-compatible |

### Proxy (optional)

If your vision model API requires a proxy, set the standard variables (the extension also honors `PI_VISION_PROXY`):

```bash
export HTTPS_PROXY=http://your-proxy:port
```

## Usage

- **Agent-driven** — when a task involves images, the agent calls `describe_image` on its own. You can direct it: *"Screenshot the page and check whether the design matches the requirements."*
- **User-driven** — paste or attach an image directly in the chat. If your active model is text-only, it is described automatically.

## Requirements

- pi v0.83+ (uses `ctx.modelRegistry.getApiKeyAndHeaders`)
- A vision-capable model in `~/.pi/agent/models.json` (or set `PI_VISION_PROVIDER` / `PI_VISION_MODEL`)
- Network access to your vision model's API

## License

MIT