# PRD: pi-vision-proxy 1.4.0 **Status:** Final **Target version:** pi-vision-proxy 1.4.0 **Date:** 2026-05-03 --- ## Changes since v1.3.0 1. **`analyze_image` tool** — agent-facing tool for targeted re-querying of images with multi-form crop support and optional grounding. 2. **Multi-image batched comparison** — adaptive joint vision calls when ≥2 images arrive together. 3. **`/vision-proxy describe` slash command** — user-facing re-query and re-describe with extended crop syntax. 4. **Optional grounded-coordinate output** — per-model native-format grounding with `grounding_format` metadata. 5. **Spatial-awareness fix** — image dimensions, filenames, and `crop_origin` in all fence types so agents can reason about coordinates without prior knowledge of image geometry. --- ## Background pi-vision-proxy currently provides automatic, transparent image description for non-vision models in Pi. It runs in `before_agent_start`, sends each attached image to a configured vision model once, persists the description in the session keyed by image hash, and re-injects the description on every subsequent LLM call so descriptions survive across turns. Three structural limitations remain: 1. **Generic descriptions, no question context.** Detail-level questions ("what error code?", "what is the y-axis maximum?") often require information the generic pass omitted. 2. **Per-image isolation.** Comparison questions cannot be answered from independently generated per-image descriptions. 3. **No user-side override.** No user-facing way to refresh or re-query a wrong description. --- ## Summary of features | # | Feature | Type | Default | |---|---|---|---| | 1 | `analyze_image` tool with multi-form crop & question support | Agent-facing tool | enabled when proxy enabled | | 2 | Multi-image batched comparison with adaptive prompting | Behaviour change | enabled when ≥ 2 images in one turn | | 3 | `/vision-proxy describe` slash command | User-facing command | always available | | 4 | Optional grounded-coordinate output with format metadata | Capability flag per vision model | curated Tier 1 list pre-populated | --- ## Image metadata and dimensions All three fence tags carry image dimensions and filename. Dimensions are stored in an in-memory map populated on first ingestion — no session-entry persistence needed (images are always re-decoded each session). ```typescript interface ImageMeta { width: number; height: number; filename?: string; // basename only } const _imageMeta = new Map(); ``` When `readImageFileWithReason` returns an image, it also returns the basename. The caller stores it in `_imageMeta` alongside dimensions extracted via `image-size` (header-only, no full decode). **Open question:** Whether `sharp` (for cropping and pHash) should be a hard or optional dependency, and what the degradation behavior is when it's absent. `image-size` is always a hard dependency. --- ## Fence tag reference Three distinct tags — each has a clear semantic role so the agent can distinguish generic descriptions from targeted analyses from joint comparisons. | Tag | Producer | Semantics | |---|---|---| | `` | Auto-proxy (`before_agent_start`) | Generic, per-image, always produced for attached images | | `` | `analyze_image` tool | Targeted, question-driven, possibly cropped/grounded | | `` | Auto-proxy or tool (≥2 images) | Multi-image comparison | Closing-tag neutralisation is applied to all three fence types. The `fenceUntrusted` function is updated to handle all three tags. **Precedence rule:** `analyze_image` results (``) are authoritative for the specific question asked. The cached generic description (``) remains the default for everything else. The agent receives both in context and resolves contradictions using question specificity and recency. No correction entry is created. --- ## Feature 1 — Targeted analysis via `analyze_image` tool ### Problem The automatic proxy generates one generic description per image. When the agent later needs a specific detail, it has no recourse — and even with a vision-capable active model, images attached many turns ago tend to fall out of effective attention. ### Functional requirements **FR-1.1** When the proxy is enabled (`mode != off`), pi-vision-proxy registers a tool named `analyze_image` and exposes it to the active model alongside Pi's other tools, including when the active model supports images natively. **FR-1.2** Tool schema: ```typescript analyze_image({ images: string[], // 1..maxImagesPerCall; path or "sha256:" question: string, // required, non-empty, max 4000 chars model?: string, // optional; provider/model-id crop?: CropEntry[], // optional; per-image crop reason?: string // optional; logged for analytics only }) -> string type CropEntry = { image_index: number; } & ( | { region: NamedRegion } | { normalized: { x: number; y: number; width: number; height: number } } | { pixels: { x: number; y: number; width: number; height: number } } ); type NamedRegion = | "top-left" | "top-right" | "bottom-left" | "bottom-right" | "top" | "bottom" | "left" | "right" | "center" | "top-half" | "bottom-half" | "left-half" | "right-half"; ``` Each `CropEntry` must include exactly one of `region`, `normalized`, or `pixels`. Specifying more than one is a tool error. **FR-1.2.1** *Crop semantics.* - **`region`** — proxy resolves to a normalized rectangle internally. Quadrants (`top-left` etc.) cover 50% × 50%. `top` / `bottom` / `left` / `right` cover the indicated 50% × 100% strip. `center` is the centre 50% × 50%. `*-half` aliases are explicit names for the 50%-strip forms. Always valid; never errors. - **`normalized`** — `x`, `y`, `width`, `height` ∈ [0.0, 1.0]. Resolved to pixels by multiplying by image dimensions. Out-of-bounds values are clamped; if the resulting rectangle has zero area, tool error. - **`pixels`** — absolute pixel coordinates. Out-of-bounds values are clamped to image dimensions. Zero-area after clamping → tool error. Use this form only when the agent has authoritative pixel coordinates from a prior fence or from a previous grounded response. The proxy converts all three forms to pixels internally before cropping locally and transmitting the cropped region to the vision model. **FR-1.2.2** *`reason` field.* Optional, logged when supplied. No semantic role — purely for analytics. **FR-1.2.3** *Tool description text.* The tool description registered with the active model must include all three crop forms with concrete examples. Required text: > Use `analyze_image` when (a) the cached description of an image lacks a detail you need, (b) you need to compare or cross-reference multiple images, or (c) you need to focus on a specific region. > > **Cropping.** Three forms, in order of preference: > > - **`region`** — coarse cut by name. Use when you don't have exact dimensions: `{ image_index: 0, region: "bottom-right" }`. > - **`normalized`** — fractional coordinates 0.0–1.0. Default choice for precise crops without knowing image dimensions: `{ image_index: 0, normalized: { x: 0.5, y: 0.5, width: 0.4, height: 0.4 } }`. > - **`pixels`** — absolute pixels. Use only when you have authoritative coordinates from a prior `` or `` (which carry `width` and `height` attributes) or from a previous grounded response. Example: `{ image_index: 0, pixels: { x: 1840, y: 120, width: 840, height: 360 } }`. > > Image dimensions and filenames are available in the `width`, `height`, and `filename` attributes of ``, ``, and `` blocks in your context. > > When a crop is applied, the response fence carries a `crop_origin` attribute (e.g. `crop_origin="1840,120"`). Add the origin's x to any returned x-coordinate and the origin's y to any returned y-coordinate to map coordinates back to the original full image. > > The tool result is authoritative for the specific question asked; the cached generic description remains the default for everything else. **FR-1.3** Path resolution and security. Delegates path policy to Pi's `read` tool. Adds: `..` segments rejected, symlink-escape rejected, `images.length ≤ maxImagesPerCall`. **FR-1.4** Model override. Must be registered, must support image input. Agent-initiated overrides honoured even when `PI_VISION_PROXY_MODEL` is env-locked. **FR-1.5** Per-provider consent. First call to a not-yet-consented provider returns a tool error pointing to `/vision-proxy consent yes`. Consent for the auto-proxy's provider carries over to `analyze_image` calls using the same provider. **FR-1.6** Persistence. Image bytes by sha256. Image dimensions are computed on first ingestion and stored in the in-memory `_imageMeta` map (see "Image metadata and dimensions" above). Tool results are not persisted as canonical descriptions. Result LRU keyed by `(sorted_image_hashes, crop_signature, question_hash, model_id)` — `crop_signature` is a stable hash over the resolved-pixels rectangle, so semantically equivalent crops in different forms hit the same cache entry. **FR-1.7** Configuration: - `/vision-proxy tool on|off` — default `on` at GA, `off` during beta. - `/vision-proxy max-images-per-call ` — default 10, range 1–20. - `/vision-proxy cache-size ` — default 50, range 0–500. **FR-1.8** *Result fencing with image metadata.* The tool result is wrapped in a `` fence that carries the dimensions, filename, and (when applicable) grounding format of the source image: ``` The error dialog [1840, 120, 2680, 480] shows "Connection timed out"... ``` Required attributes: `image` (always), `width`, `height` (always, in pixels). Optional attributes: `filename` (basename only — full paths are never exposed in fence attributes, per security note), `grounding_format` (only present when grounding is enabled and the model is in `groundingModels`; see FR-4.2 and FR-4.7). For cropped calls, `width` and `height` are the dimensions of the **cropped region** sent to the vision model, not the original image, and `image` is suffixed with the crop signature: `image="sha256:abc...#crop:1840,120,840,360"`. A `crop_origin` attribute is added: ``` The error dialog [1840, 120, 2680, 480] shows "Connection timed out"... ``` **`crop_origin`** — `","` comma-separated pixel offset of the crop's top-left corner within the original image. Present **only** on cropped calls. Absent on uncropped calls. The agent adds `crop_origin.x` to any returned x-coordinate and `crop_origin.y` to any returned y-coordinate to map coordinates back to full-image space. Closing-tag neutralisation is applied to the body. **FR-1.9** Telemetry. `vision_proxy.tool_call` entries with timestamp, image hashes, crop form and resolved pixels, question, supplied `reason`, model id, latency_ms, provider-reported token usage, `cache_hit` flag. **FR-1.10** *Precedence between tool results and cached descriptions.* `analyze_image` results (``) are authoritative for the specific question asked. The cached generic description (``) remains the default for everything else. The agent receives both in context and resolves contradictions using question specificity and recency. No correction entry is created. --- ## Feature 2 — Multi-image batched comparison ### Problem VLMs natively struggle to link visual cues across images even when both images are in the same prompt. But forcing a contrastive structure when the user just said "describe both" over-constrains. ### Approach The vision model itself routes between contrastive and co-presented output structure — it's already multilingual and is already reading the user's question. The proxy supplies language-independent structural hints (filenames, perceptual similarity) as additional context. ### Functional requirements **FR-2.1** When the automatic proxy processes a turn containing `N` ≥ 2 images: - N independent vision calls produce per-image canonical descriptions (unchanged). - One additional vision call produces a "joint description" cached by `(sorted_hashes, turn_index)`. Injected for *this turn only*. **FR-2.2** When `analyze_image` is invoked with `2 ≤ N ≤ maxImagesPerCall` images, one vision call is made with all N images. **FR-2.3** Joint calls obey the existing consent flow, prompt-injection fence, and `include_context` setting. **FR-2.4** `maxBatch` config (slash: `/vision-proxy max-batch `; range 1–10; default 4). **FR-2.5** *Adaptive joint-call system prompt.* ``` You are analysing N images that the user has provided together. Refer to them as Image 1 (filename), Image 2 (filename), ... . Each image's dimensions will be visible to your reasoning as "Image N: WxH pixels". Read the user's question carefully. If the user is asking about comparison, difference, change, or relationship between the images, structure your response as: (1) similarities across the images, (2) specific differences, (3) a direct, step-by-step answer to the user's question. Otherwise, describe each image in turn and note any obvious relationships between them. [Structural hints, if available: ...] [User's prompt: ...] ``` This works in any language the underlying VLM supports — comparison cues in German, Italian, Japanese, Chinese, etc. are handled natively. No per-language code in the proxy. **FR-2.5.1** *Filename hints.* See Appendix D. Basename-only. **FR-2.5.2** *Perceptual similarity hint.* pHash computed after resizing both images to a fixed square with letterboxing (e.g. 64×64 with gray padding) to normalise aspect ratio before hashing. Threshold default 0.80 (lowered to account for residual aspect-ratio distortion in content-heavy crops). Configurable via `PI_VISION_PROXY_PHASH_THRESHOLD`. *Known limitation:* Heavy crops (e.g. a small modal window cropped from a full-screen screenshot) may still fall below threshold. This is acceptable — the hint is advisory (FR-2.5.3) and the VLM can still compare the images without it. **FR-2.5.3** Hints are advisory; the user's question wins. **FR-2.5.4** Hints suppressed for `analyze_image` tool path — the agent's `question` is already explicit. **FR-2.6** *Joint description fencing with per-image metadata.* ``` Image 1 and Image 2 differ primarily in the navigation sidebar... ``` `dimensions` is a JSON-encoded array, one entry per image, in the order presented to the vision model. Closing-tag neutralisation is applied. **FR-2.7** Cost telemetry. Provider-reported tokens only. ### Edge cases | Case | Behaviour | |---|---| | 1 image in a turn | No joint call | | `maxBatch` exceeded | Joint call skipped; per-image descriptions still produced | | One image fails to decode | Joint call skipped | | `maxBatch=1` | Joint calls disabled | | Filenames contain no recognised pattern | No filename hint; pHash hint may still apply | | pHash computation fails | Skip pHash hint; proceed | | Cropped images in joint call | Each cropped image's dimensions in the `dimensions` array reflect the cropped region | --- ## Feature 3 — `/vision-proxy describe` slash command ### Functional requirements **FR-3.1** New slash subcommands. Crop syntax supports all three coordinate forms: ``` /vision-proxy describe [ ...] [--question ""] [--crop :
] [--model ] [--save] /vision-proxy redescribe [--model ] ``` `` syntax for `--crop`: | Form | Syntax | Example | |---|---|---| | Named region | `r=` | `--crop 0:r=top-right` | | Normalized | `n=,,,` | `--crop 0:n=0.5,0.5,0.4,0.4` | | Pixels | `p=,,,` | `--crop 0:p=1840,120,840,360` | Specifying more than one form per crop entry is rejected with a clear error message. **FR-3.2** `describe` semantics: - Resolves inputs identical to FR-1.3. - Omitting `--question` triggers default generic system prompt. - Multiple inputs trigger a joint call (FR-2.5 adaptive prompt + filename/pHash hints). - Result printed to TUI only by default; `--save` overwrites canonical session description (single image, no `--question`, no `--crop`). **FR-3.3** `redescribe` is sugar for `describe --save` with no question and no crop. **FR-3.4** Inline consent prompts. **FR-3.5** TUI output uses distinct visual style with `[Vision Proxy]` prefix. **FR-3.6** Logged as `vision_proxy.command` entries. --- ## Feature 4 — Optional grounded-coordinate output ### Functional requirements **FR-4.1** Per-model capability flag in the model registry: `supportsGrounding: boolean`. **FR-4.1.1** *Curated default list* (1.4.0 ships with these marked `supportsGrounding: true`): **Tier 1 — designed-in grounding:** - `Qwen/Qwen2.5-VL-3B-Instruct`, `-7B-Instruct`, `-32B-Instruct`, `-72B-Instruct` - `Qwen/Qwen3-VL` family - `allenai/Molmo2-8B`, `allenai/Molmo2-72B` - `deepseek-ai/deepseek-vl2-tiny`, `-small`, `-base` - `OpenGVLab/InternVL3` family **Tier 2 — opt-in with quality caveat:** - `google/gemini-2.5-pro`, `google/gemini-3-pro` **Excluded (warned on `add` attempt):** - `anthropic/claude-*`, `openai/gpt-4o`, `gpt-5`, `meta/llama-*-vision` **FR-4.1.2** Slash commands: ``` /vision-proxy grounding-models add [--format ] /vision-proxy grounding-models remove /vision-proxy grounding-models list /vision-proxy grounding-models reset ``` `add` for an excluded model triggers a confirmation prompt explaining the unreliability. **FR-4.2** *Native-format grounding output.* When `supportsGrounding: true`, the system prompt for both per-image and joint calls is appended with a model-specific instruction matching the model's training format. The proxy does **not** force a uniform output format across models. Instead, the proxy: 1. Looks up the model's `grounding_format` in the registry (FR-4.7). 2. Appends an instruction phrased in the model's native convention. 3. Records the format in the response fence's `grounding_format` attribute so the agent knows what convention to interpret. Example instruction for `qwen_pixels` models: > When you describe a spatial element, follow the description with bounding-box coordinates as `[x1, y1, x2, y2]` in absolute pixels relative to the image. Use `Image-N:` prefix for multi-image inputs. Example instruction for `molmo_points` models: > When you describe a spatial element, follow the description with point coordinates as `` using your standard percentage-based convention. **FR-4.3** When `supportsGrounding: false`, system prompt unchanged; `grounding_format` attribute on the fence is `"none"`. **FR-4.4** The proxy does not parse, validate, or rewrite returned coordinates. Downstream agents and tooling can extract them using the `grounding_format` attribute as a key. **FR-4.5** Purely additive — a model that ignores the grounding instruction simply produces a description without coordinates; `grounding_format` will still be set per the registry but the response body will contain no coordinates. **FR-4.6** *Crop-and-grounding interaction.* When grounding is enabled and the call includes a `crop`, returned coordinates are relative to the **cropped region** sent to the vision model. The fence includes a `crop_origin` attribute (e.g. `crop_origin="1840,120"`) giving the pixel offset of the crop's top-left corner in the original image. Agents can recover full-image coordinates by adding `crop_origin` to any returned coordinate. The fence's `width` and `height` reflect the cropped dimensions, and the `image` attribute is suffixed with the crop signature (FR-1.8). Documented in the tool description (FR-1.2.3). **FR-4.7** *Per-model `grounding_format` registry.* New required field on every entry in `groundingModels`. The registry maps each grounding-capable model to one of the following format identifiers: | Format identifier | Convention | Example output | |---|---|---| | `qwen_pixels` | absolute pixels, two-corner `[x1, y1, x2, y2]` | `[1840, 120, 2680, 480]` | | `molmo_points` | percentage-based points (0–100), Molmo's native `` element | `` | | `deepseek_bbox` | DeepSeek's `<\|ref\|>desc<\|/ref\|><\|det\|>[[x1,y1,x2,y2]]<\|/det\|>` format | (as documented by DeepSeek) | | `internvl_pixels` | InternVL's native bbox format, absolute pixels | `[100, 200, 300, 400]` | | `gemini_normalized_1000` | normalized 0–1000 coordinates per Gemini API convention | `[420, 124, 670, 248]` | | `none` | no grounding | — | **Default mappings shipped in 1.4.0:** ```json { "Qwen/Qwen2.5-VL-7B-Instruct": "qwen_pixels", "Qwen/Qwen2.5-VL-72B-Instruct": "qwen_pixels", "Qwen/Qwen3-VL-7B": "qwen_pixels", "allenai/Molmo2-8B": "molmo_points", "allenai/Molmo2-72B": "molmo_points", "deepseek-ai/deepseek-vl2": "deepseek_bbox", "deepseek-ai/deepseek-vl2-small": "deepseek_bbox", "OpenGVLab/InternVL3-8B": "internvl_pixels", "google/gemini-2.5-pro": "gemini_normalized_1000", "google/gemini-3-pro": "gemini_normalized_1000" } ``` When users add a model not in the default mapping, they must specify its format: ``` /vision-proxy grounding-models add --format ``` If `--format` is omitted, defaults to `qwen_pixels` (the most common convention) with a TUI warning. --- ## Cross-feature interaction | Combination | Behaviour | |---|---| | `analyze_image` with N ≥ 2 images | Single joint vision call using FR-2.5 adaptive prompt; filename/pHash hints suppressed (FR-2.5.4) | | `analyze_image` with `crop` and N ≥ 2 | Crops applied per-image before transmission, then batched. Joint description's `dimensions` array reflects cropped sizes | | `analyze_image` with `pixels` crop on an image not previously described | Proxy clamps using actual dimensions (computed at ingestion); proceeds without warning | | Automatic proxy + `analyze_image` follow-up in same turn | Both happen; tool result is authoritative for its specific question (FR-1.10) | | `/vision-proxy describe` during automatic proxy turn | Serialised | | Mode `off` | Tool unregistered; joint calls disabled; slash commands refuse | | Grounding enabled + cropped call | Coordinates relative to cropped image; `crop_origin` provided for mapping back (FR-4.6) | | User in non-English locale | FR-2.5 routing happens inside the VLM in user's language; hints language-independent | --- ## Configuration summary | Setting | Slash | Env | Default | Range | |---|---|---|---|---| | Tool exposure | `/vision-proxy tool on\|off` | `PI_VISION_PROXY_TOOL` | `on` (GA) / `off` (beta) | — | | Max images per call | `/vision-proxy max-images-per-call ` | `PI_VISION_PROXY_MAX_IMAGES_PER_CALL` | `10` | 1–20 | | Max batch (auto) | `/vision-proxy max-batch ` | `PI_VISION_PROXY_MAX_BATCH` | `4` | 1–10 | | Result cache size | `/vision-proxy cache-size ` | `PI_VISION_PROXY_CACHE_SIZE` | `50` | 0–500 | | pHash similarity threshold | (`vision-proxy.json`) | `PI_VISION_PROXY_PHASH_THRESHOLD` | `0.80` | 0.0–1.0 | | Grounding-capable models | `/vision-proxy grounding-models …` | — | Tier 1 list (FR-4.1.1) | — | `vision-proxy.json` schema additions: ```json { "tool": "on", "maxImagesPerCall": 10, "maxBatch": 4, "cacheSize": 50, "pHashSimilarityThreshold": 0.80, "groundingModels": { "Qwen/Qwen2.5-VL-7B-Instruct": { "format": "qwen_pixels" }, "allenai/Molmo2-8B": { "format": "molmo_points" }, "deepseek-ai/deepseek-vl2": { "format": "deepseek_bbox" }, "OpenGVLab/InternVL3-8B": { "format": "internvl_pixels" } } } ``` (Shipped default is the full FR-4.7 mapping; example abbreviated.) Backwards compatibility: existing 1.3.0 `vision-proxy.json` files load unchanged. `groundingModels`, `tool`, `maxImagesPerCall`, `maxBatch`, `cacheSize` were absent in 1.3.0, so the new defaults take effect on first 1.4.0 launch without losing user state. --- ## Telemetry / observability Session entry types: - `vision_proxy.tool_call` — includes `crop_form` (`region`/`normalized`/`pixels`/none) and `crop_resolved_pixels` for analytics. - `vision_proxy.joint_description` — includes `filename_hint`, `phash_similarity`, and `dimensions` array. - `vision_proxy.command` - `vision_proxy.skip` Each entry includes timestamp, image hashes, model id, `grounding_format` used, latency_ms, provider-reported token usage, `cache_hit` where applicable. --- ## Security considerations - **Prompt injection.** Closing-tag neutralisation applied to all three fence types (``, ``, ``). Filename hints are basename-only — full paths never appear in fence attributes or hint strings. - **Path policy.** Defers to Pi's `read` tool plus `..`/symlink-escape rejection. - **Per-provider consent.** Tracked per provider. Consent for one provider does not carry over to a different provider. - **Crop as exfiltration vector.** Cropped region still sent to configured vision provider; documented in consent prompt. The three crop forms do not change the security posture — they just determine how the agent specifies what to send. - **Dimension metadata leak.** Image dimensions in fence attributes are derived from the user's own attached images and pose no additional risk beyond what the proxy already sends to the vision model. --- ## Rollout - **1.4.0-beta.1**: Feature 1 with `tool=off` default, all three crop forms supported, dimensions in description fence. - **1.4.0-beta.2**: Feature 3 with extended crop syntax. - **1.4.0-beta.3**: Feature 2 with `maxBatch=1` default; adaptive prompt, filename/pHash hints, joint-fence dimensions. - **1.4.0-beta.4**: Feature 4 with curated Tier 1 default list and `grounding_format` registry; opt-in flag itself off by default. - **1.4.0**: Flip defaults to `tool=on`, `maxBatch=4`. Grounding flag remains opt-in but list pre-populated. --- ## Out of scope (later versions) - Cross-session image library - Per-image TTL on cached descriptions - Vision model fallback chain - Streaming descriptions - Preprocessing (resize, format normalisation) before vision call - Dollar-cost telemetry - Proxy-side parsing/normalisation of grounding coordinates into a unified format (currently the agent reads `grounding_format` and decides) - Auto-detection of grounding capability and format from provider metadata --- ## Appendix A — References - Dong et al., *Training Multi-Image Vision Agents via End2End Reinforcement Learning* (IMAgent), arXiv 2512.08980, December 2025. - Chen et al., *MiCo: Multi-image Contrast for Reinforcement Visual Reasoning*, NeurIPS 2025, arXiv 2506.22434. - *LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning*, arXiv 2601.10129, January 2026. - Wang et al., *CiQi-Agent: Aligning Vision, Tools and Aesthetics in Multimodal Agent for Cultural Reasoning*, arXiv 2603.28474, March 2026. - Bai et al., *Qwen2.5-VL Technical Report*, 2025. Source for FR-4.7 `qwen_pixels` format. - Deitke et al., *Molmo2: Open Vision-Language Video Model*, arXiv 2601.10611, January 2026. Source for FR-4.7 `molmo_points` format. - DeepSeek AI, *DeepSeek-VL2: Mixture-of-Experts Vision-Language Models*, arXiv 2412.10302, December 2024. Source for FR-4.7 `deepseek_bbox` format. ## Appendix B — Tool contract ```typescript type AnalyzeImageInput = { images: string[]; question: string; model?: string; crop?: CropEntry[]; reason?: string; }; type CropEntry = { image_index: number; } & ( | { region: NamedRegion } | { normalized: { x: number; y: number; width: number; height: number } } | { pixels: { x: number; y: number; width: number; height: number } } ); type NamedRegion = | "top-left" | "top-right" | "bottom-left" | "bottom-right" | "top" | "bottom" | "left" | "right" | "center" | "top-half" | "bottom-half" | "left-half" | "right-half"; type AnalyzeImageOutput = string; ``` ## Appendix C — Slash commands ``` /vision-proxy tool on|off /vision-proxy max-images-per-call # 1..20 /vision-proxy max-batch # 1..10 /vision-proxy cache-size # 0..500 /vision-proxy grounding-models add|remove|list|reset [] [--format ] /vision-proxy describe ... [--question ""] [--crop :r= | n=,,, | p=,,,] [--model ] [--save] /vision-proxy redescribe [--model ] ``` ## Appendix D — Filename hint patterns (FR-2.5.1) Basename-only matching, case-insensitive. ``` before.* ∧ after.* → "before/after pair" old.* ∧ new.* → "old/new pair" {prefix}{version} (≥2, same prefix) → "versioned sequence" *_1.* ∧ *_2.* → "numbered sequence" *-1.* ∧ *-2.* → "numbered sequence" YYYY-MM-DD_*.* (sortable, ≥2 files) → "time-ordered sequence" (no match) → no hint emitted ``` **Version extraction rule:** From each basename, extract a `(prefix, version_number)` tuple by matching the rightmost occurrence of `[vV]?(\d+(?:\.\d+)?)` immediately before the extension. If ≥2 files share the same prefix but have different version numbers → "versioned sequence". Examples that match: - `mockup_v2.png ∧ mockup_v4.png` → prefix=`mockup_v`, versions {2, 4} - `draft_v1.1.png ∧ draft_v1.2.png` → prefix=`draft_v`, versions {1.1, 1.2} - `app2.png ∧ app3.png` → prefix=`app`, versions {2, 3} The "versioned sequence" rule is checked before the `_\d` and `-\d` numbered-sequence rules. ## Appendix E — Description fence reference All three fence types share a common metadata schema. Required attributes are always present; optional attributes appear when applicable. | Attribute | Required | Where applied | Notes | |---|---|---|---| | `image` | yes | `description`, `analysis` | sha256 hash; suffixed with `#crop:x,y,w,h` for cropped calls | | `images` | yes | `joint_description` | image count | | `dimensions` | yes | `joint_description` | JSON-encoded array, one entry per image | | `width` | yes | `description`, `analysis` | pixels of what the vision model saw (cropped or original) | | `height` | yes | `description`, `analysis` | as above | | `crop_origin` | optional | `description`, `analysis` | `"x,y"` pixels; present only on cropped calls; top-left corner of crop within original image | | `filename` | optional | all three | basename only; never full path | | `grounding_format` | optional | all three | one of FR-4.7's identifiers; absent or `"none"` when grounding off |