# Multi-Modal Composition — Universal `image` Block + Documents

LangChain 1.0 unifies the image-block shape across providers (P64). This
reference covers the universal shape, base64 vs URL tradeoffs, per-provider
size limits, and the `document` block used by Claude's citations API.

## The universal `image` block (1.0)

```python
{
    "type": "image",
    "source_type": "base64",         # or "url"
    "data": "<base64 string or URL>",
    "mime_type": "image/png",        # required for base64
}
```

Compose messages with this shape on **any** provider — the adapter
translates to the wire format Claude, OpenAI, or Gemini expects. Do not
hand-roll `{"type": "image_url", ...}` (OpenAI pre-1.0) or
`{"type": "image", "source": {"type": "base64", ...}}` (Anthropic
pre-1.0).

## Composing a text + image `HumanMessage`

```python
import base64
from pathlib import Path
from langchain_core.messages import HumanMessage

def image_block_from_file(path: str) -> dict:
    mime = {"png": "image/png", "jpg": "image/jpeg",
            "jpeg": "image/jpeg", "gif": "image/gif",
            "webp": "image/webp"}[Path(path).suffix.lstrip(".").lower()]
    data = base64.standard_b64encode(Path(path).read_bytes()).decode("ascii")
    return {
        "type": "image",
        "source_type": "base64",
        "data": data,
        "mime_type": mime,
    }

msg = HumanMessage(content=[
    {"type": "text", "text": "Describe what is broken in this screenshot."},
    image_block_from_file("screenshot.png"),
])
response = claude.invoke([msg])
```

Two rules that real code breaks:

1. **`content` must be a `list` when including non-text blocks.** Passing
   `content="Describe..."` and then trying to "attach" an image elsewhere
   silently drops the image. All blocks go in the list.
2. **Order matters for Claude.** Put the image block *before* the
   instruction text for best results — Claude attends most to the trailing
   tokens. Empirically, putting the instruction last improves
   follow-through on multi-image prompts.

## Base64 vs URL

| source_type | When to use | Tradeoffs |
|-------------|-------------|-----------|
| `"base64"` | Default. Private files, generated bytes, anything not publicly hosted. | Every byte ships in the request. For a 4MB PNG you pay ~5.4MB (base64 is 33% larger) of network + prompt tokens. |
| `"url"` | Public, stable URLs (CDN assets, public S3). | Zero upload cost, but the provider fetches the URL server-side — authentication headers you set are NOT forwarded. Private URLs behind auth: use base64 instead. |

Claude also accepts URLs via `source_type="url"` as of `anthropic >= 0.40`.
Earlier Anthropic SDKs required base64 only.

## Per-provider size limits

Exact limits, as of November 2024 / January 2026 docs — **these change;
always verify against provider docs before scaling**:

| Provider | Per-image limit | Per-request image count | Per-request total |
|----------|-----------------|--------------------------|--------------------|
| Anthropic (Claude 3.5/4.x) | 5 MB per image | up to 20 images | 8000 tokens max for images + text |
| OpenAI (GPT-4o) | 20 MB per image | no hard count | combined with text within context window (128k) |
| Google (Gemini 2.5) | 20 MB per request total | no hard count | 20 MB total including all images |

Exceeding these:

- Anthropic: `anthropic.BadRequestError: image exceeds 5 MB limit`
- OpenAI: HTTP 400 "image_url is too large"
- Gemini: `google.api_core.exceptions.InvalidArgument: Request payload size exceeds the limit`

Pre-resize before sending. For UX screenshots, 1024x1024 at JPEG quality
85 is typically < 500KB and preserves enough detail for Claude/GPT-4o
vision to read UI text.

## MIME type rules

| mime_type | Claude | GPT-4o | Gemini 2.5 |
|-----------|--------|--------|------------|
| `image/png` | yes | yes | yes |
| `image/jpeg` | yes | yes | yes |
| `image/gif` | yes (static frame only) | yes | yes |
| `image/webp` | yes | yes | yes |
| `image/bmp` | no | yes | no |
| `image/heic`, `image/heif` | no | no | yes |

If you have iPhone HEIC uploads and need Claude or OpenAI compatibility,
convert to JPEG at ingest. `pillow-heif` + `Pillow` handles this in
about four lines.

## What LangChain's adapter actually does

When you pass the universal `image` block to `ChatAnthropic.invoke()`,
the adapter in `langchain-anthropic` rewrites it to Anthropic's native
wire format:

```python
# Your code passes:
{"type": "image", "source_type": "base64",
 "data": "...", "mime_type": "image/png"}

# langchain-anthropic sends on the wire:
{"type": "image", "source":
 {"type": "base64", "media_type": "image/png", "data": "..."}}
```

On `ChatOpenAI`:

```python
# On the wire:
{"type": "image_url",
 "image_url": {"url": "data:image/png;base64,..."}}
```

On `ChatGoogleGenerativeAI`:

```python
# Translated to google.generativeai.protos.Part with inline_data
{"inline_data": {"mime_type": "image/png", "data": "..."}}
```

You should not see these wire shapes in your code. If you do, you are
bypassing the adapter (usually by importing `anthropic.Anthropic`
directly inside a chain). Keep the universal shape at the LangChain
boundary.

## `document` blocks (Claude citations API)

A parallel feature to images: `document` blocks let Claude attach
citations to specific output text. Input shape:

```python
from langchain_core.messages import HumanMessage

doc_block = {
    "type": "document",
    "source": {
        "type": "base64",
        "media_type": "application/pdf",
        "data": base64_pdf_bytes,
    },
    "title": "Q3 Earnings Report",
    "citations": {"enabled": True},
}

msg = HumanMessage(content=[
    doc_block,
    {"type": "text", "text": "What were the key revenue drivers?"},
])
```

Supported source types: PDF (`application/pdf`), plain text
(`text/plain`), and custom content (`content` array of text blocks).

**The output side.** Citations do NOT come back as top-level `citation`
blocks. They attach to `text` blocks as `citations` arrays:

```python
for block in response.content:
    if block.get("type") != "text":
        continue
    print(block["text"])
    for c in block.get("citations", []):
        print(f"  source: {c['document_title']}")
        print(f"  quoted: {c['cited_text']!r}")
        print(f"  location: chars {c['start_char_index']}-{c['end_char_index']}")
```

`msg.text()` strips the `citations` metadata. To surface citations you
must iterate `content` manually.

## Provider adapter checklist

Before composing a multi-modal message:

1. Is `content` a `list[dict]`? (Not a `str` with an attached image.)
2. Are all image blocks in the universal 1.0 shape (`source_type`, `data`, `mime_type`)?
3. Is each image under the provider's per-image limit? (5 MB Anthropic, 20 MB OpenAI, 20 MB total Gemini.)
4. Is the `mime_type` supported on the target provider? (See table above.)
5. If using `document` blocks: are you using `ChatAnthropic` Sonnet 4+? (No other provider supports them.)
6. If round-tripping a response: did you preserve any `thinking` blocks? (See `thinking-blocks.md`.)

## Errors you will hit

| Error | Cause | Fix |
|-------|-------|-----|
| `anthropic.BadRequestError: image exceeds 5 MB limit` | Un-resized screenshot | Pre-resize to < 5 MB; 1024x1024 JPEG 85 is typically < 500 KB |
| `openai.BadRequestError: Invalid image data` | Missing `data:image/png;base64,` prefix when hand-rolling | Use the universal block — LangChain adapts the prefix |
| `google.api_core.exceptions.InvalidArgument: Request payload size exceeds the limit` | Total request > 20 MB on Gemini | Chunk requests; Gemini's limit is per-request total |
| Content shows up as garbled text | `HumanMessage(content="...")` (string) with an attempted image attachment elsewhere | Put all blocks inside the `content` list |
| Image ignored by model | Image block after the instruction, and instruction says "If you see an image..." | Put the image block BEFORE the instruction; Claude attends most to trailing tokens |
| Citations missing from output | Read via `msg.text()` which strips metadata | Iterate `msg.content` and read `block["citations"]` on text blocks |

## References

- LangChain multimodal concepts: <https://python.langchain.com/docs/concepts/multimodality/>
- Anthropic vision: <https://docs.anthropic.com/en/docs/build-with-claude/vision>
- Anthropic citations: <https://docs.anthropic.com/en/docs/build-with-claude/citations>
- OpenAI vision: <https://platform.openai.com/docs/guides/vision>
- Gemini multimodal: <https://ai.google.dev/gemini-api/docs/vision>