# Sana (Image) + `/video` (Video Generation) Integration Plan

> **Status**: Plan only. Builds a complete, anchored handoff for a downstream implementation agent.
> **Goal**: (A) Promote NVIDIA Sana to the **primary image-generation model** for `/image` and the image-generation tool, (B) ship an entirely new **`/video` pipeline** modeled exactly on the existing `/image` and `/sound`-`/music` patterns, including its agent tool, Telegram public/private wiring, weight management menus, and TUI ASCII preview (video → thumbnail frame).
>
> **Reference inventory**: `/home/robit/Downloads/open_source_video_generation_models_agent_package.md`
> **Sana upstream**: <https://github.com/NVlabs/Sana>

---

## 0. TL;DR for the implementation agent

You are extending three parallel pipelines that already exist for `/image`, `/sound`, and `/music`. Replicate them for video. Do **not** invent new abstractions. Reuse:

1. The `ImageGenerateTool` (preset list, fallback ladder, Python venv bootstrapper, sidecar JSON, progress events, prompt expander, ASCII preview).
2. The `AudioGenerateTool` (kind discriminator — `sound`/`music`, per-kind defaults, project profile fallback).
3. The TUI command surface in [packages/cli/src/tui/commands.ts](packages/cli/src/tui/commands.ts) + [packages/cli/src/tui/command-registry.ts](packages/cli/src/tui/command-registry.ts).
4. The Telegram bridge wiring in [packages/cli/src/tui/telegram-bridge.ts](packages/cli/src/tui/telegram-bridge.ts) + [packages/cli/src/tui/telegram-creative-tools.ts](packages/cli/src/tui/telegram-creative-tools.ts).
5. Media classification + Telegram routing in [packages/cli/src/tui/media-routing.ts](packages/cli/src/tui/media-routing.ts) (already supports `kind: "video"` → `sendVideo`).

**Two deliverables**:

| # | Deliverable | Files (new / edited) |
| - | --- | --- |
| 1 | Make Sana the primary image generator | edit [packages/execution/src/tools/image-generate.ts](packages/execution/src/tools/image-generate.ts) (presets, ladder, defaults, runner) |
| 2 | New `/video` pipeline | new [packages/execution/src/tools/video-generate.ts](packages/execution/src/tools/video-generate.ts); edits to [packages/execution/src/index.ts](packages/execution/src/index.ts), [packages/cli/src/tui/commands.ts](packages/cli/src/tui/commands.ts), [packages/cli/src/tui/command-registry.ts](packages/cli/src/tui/command-registry.ts), [packages/cli/src/tui/omnius-directory.ts](packages/cli/src/tui/omnius-directory.ts), [packages/cli/src/tui/telegram-bridge.ts](packages/cli/src/tui/telegram-bridge.ts), [packages/cli/src/tui/telegram-creative-tools.ts](packages/cli/src/tui/telegram-creative-tools.ts) |

---

## 1. Architectural Anchors — How the existing `/image` pipeline is shaped

These line refs are the contract you replicate for `/video` and the spots where Sana becomes the primary model.

### 1.1 Tool surface — `ImageGenerateTool`

File: [packages/execution/src/tools/image-generate.ts](packages/execution/src/tools/image-generate.ts) (2115 lines)

| Anchor | Lines | What it is |
| --- | --- | --- |
| Tool class header | [1206-1215](packages/execution/src/tools/image-generate.ts#L1206-L1215) | `ImageGenerateTool implements Tool`, `name = "generate_image"`, description used by the LLM. |
| Tool parameters schema | [1216-1279](packages/execution/src/tools/image-generate.ts#L1216-L1279) | JSON-schema for `prompt`, `model`, `backend`, `aspect_ratio`, `width`, `height`, `steps`, `guidance`, `seed`, `action`, `fallback`, `strict_model`, `expand_prompt`. |
| Constructor + defaults wiring | [1281-1305](packages/execution/src/tools/image-generate.ts#L1281-L1305) | `defaults: ImageGenerateToolDefaults` carries model/backend/promptExpander from settings/Telegram. |
| `execute(args)` entry | [1325-1390](packages/execution/src/tools/image-generate.ts#L1325-L1390) | Dispatch to list/setup/prewarm/generate; calls `imageGenerationFallbackCandidates`. |
| Backend dispatch | [1459-1463](packages/execution/src/tools/image-generate.ts#L1459-L1463) | `ollama` / `sdcpp` / `diffusers` per candidate. |
| Sidecar writer | [1502-1534](packages/execution/src/tools/image-generate.ts#L1502-L1534) | Writes `<image>.json` alongside the PNG with original/expanded prompt, model, backend, seed, dimensions. |
| Python runner template | [507-641](packages/execution/src/tools/image-generate.ts#L507-L641) | `DIFFUSERS_RUNNER` heredoc; auto-pick FluxPipeline / SD3Pipeline / AutoPipelineForText2Image. |
| Venv bootstrap | [1077-1130](packages/execution/src/tools/image-generate.ts#L1077-L1130) | `ensurePythonFor("diffusers" | "sdcpp")`. |
| Cache layout | [1037-1052](packages/execution/src/tools/image-generate.ts#L1037-L1052) | HF/Torch/pip caches pinned to `.omnius/image-gen/{huggingface,torch,cache,pip-cache}`. |
| Preset list | [147-486](packages/execution/src/tools/image-generate.ts#L147-L486) | `IMAGE_GENERATION_MODEL_PRESETS` (Z-Image Turbo, FLUX1 dev variants, SD3.5, Hunyuan, SDXL Turbo, SD-Turbo, LCM, Sana Sprint, sdcpp). |
| Quality ladder | [488-501](packages/execution/src/tools/image-generate.ts#L488-L501) | `IMAGE_GENERATION_QUALITY_LADDER` — ordered IDs that drive `imageGenerationFallbackCandidates` (lines [832-868](packages/execution/src/tools/image-generate.ts#L832-L868)). |
| Default constants | [59-60](packages/execution/src/tools/image-generate.ts#L59-L60) | `DEFAULT_DIFFUSERS_IMAGE_MODEL = "stabilityai/sdxl-turbo"` (**this becomes Sana**), `DEFAULT_OLLAMA_IMAGE_MODEL = "x/z-image-turbo"`. |

### 1.2 TUI surface — `/image`

File: [packages/cli/src/tui/commands.ts](packages/cli/src/tui/commands.ts) (15065 lines)

| Anchor | Lines | What it is |
| --- | --- | --- |
| Dispatcher case | [1590-1592](packages/cli/src/tui/commands.ts#L1590-L1592) | `case "image": return handleImageCommand(...)`. |
| `handleImageCommand` | [9169-9246](packages/cli/src/tui/commands.ts#L9169-L9246) | Parses args, runs menu/list/setup/prewarm/generate, renders ASCII preview. |
| `parseImageCommand` | [8759-8779](packages/cli/src/tui/commands.ts#L8759-L8779) | Flag/positional parser used by `/image`. |
| `showImageModelsMenu` | [9054-9145](packages/cli/src/tui/commands.ts#L9054-L9145) | Hardware-aware fit menu with Del-to-delete-weights. |
| `renderImageModelList` | [8877-8911](packages/cli/src/tui/commands.ts#L8877-L8911) | `/image list` output. |
| `prewarmImageModel` | [9147-9167](packages/cli/src/tui/commands.ts#L9147-L9167) | Runs `ImageGenerateTool#execute({action: "prewarm"})`. |
| ASCII preview | [9226-9244](packages/cli/src/tui/commands.ts#L9226-L9244) | `buildImageAsciiPreview` + `extractSavedImagePath` from [image-ascii-preview.ts](packages/cli/src/tui/image-ascii-preview.ts). |
| Progress formatter | [9248-9260](packages/cli/src/tui/commands.ts#L9248-L9260) | Renders the `omnius_progress` JSON stream. |

### 1.3 Command registry surface

File: [packages/cli/src/tui/command-registry.ts](packages/cli/src/tui/command-registry.ts)

| Anchor | Lines | What it is |
| --- | --- | --- |
| Slash help entries | [136-152](packages/cli/src/tui/command-registry.ts#L136-L152) | `/image`, `/sound`, `/music` signatures + descriptions. |
| Category overrides | [344-435](packages/cli/src/tui/command-registry.ts#L344-L435) | `image: "media"` — video must be added here too. |
| User-only set | [479-544](packages/cli/src/tui/command-registry.ts#L479-L544) | Both `/image` and `/sound`-`/music` are user-only; video should join. |
| Networked set | [547-576](packages/cli/src/tui/command-registry.ts#L547-L576) | `image` is networked because Ollama/HF pulls hit the network. Same applies to video. |
| Telegram bot command export | [681-709](packages/cli/src/tui/command-registry.ts#L681-L709) | `buildTelegramBotCommands` — adding `/video` here automatically wires the Telegram BotFather command. |

### 1.4 Settings persistence

File: [packages/cli/src/tui/omnius-directory.ts](packages/cli/src/tui/omnius-directory.ts)

| Anchor | Lines | What it is |
| --- | --- | --- |
| `OmniusSettings` interface | [171-235](packages/cli/src/tui/omnius-directory.ts#L171-L235) | Adds `imageModel/imageBackend/soundModel/soundBackend/musicModel/musicBackend` — extend with `videoModel`/`videoBackend`/`videoKind` (default sub-kind). |
| Project/Global save+merge | [251-291](packages/cli/src/tui/omnius-directory.ts#L251-L291) | Already merges arbitrary keys — no changes needed beyond the interface. |

### 1.5 Telegram bridge surface

File: [packages/cli/src/tui/telegram-bridge.ts](packages/cli/src/tui/telegram-bridge.ts) (8798 lines)

| Anchor | Lines | What it is |
| --- | --- | --- |
| Tool imports | [85-104](packages/cli/src/tui/telegram-bridge.ts#L85-L104) | Imports `ImageGenerateTool`, `AudioGenerateTool`, `TtsGenerateTool` — extend with `VideoGenerateTool`. |
| Admin-DM tool list | [6119-6184](packages/cli/src/tui/telegram-bridge.ts#L6119-L6184) | `new ImageGenerateTool(...)`, `new AudioGenerateTool(...)` — add `new VideoGenerateTool(...)`. |
| Quota tag map | [6246-6253](packages/cli/src/tui/telegram-bridge.ts#L6246-L6253) | `generate_image|generate_audio|generate_tts|create_audio_file` → `"generation"` quota. Extend regex to include `generate_video`. |
| Public/group creative tool factory call | [6205-6213](packages/cli/src/tui/telegram-bridge.ts#L6205-L6213) | Invokes `buildTelegramCreativeTools` — the place that scopes creative tools to a per-chat workspace. |
| Defaults helpers | [7346-7368](packages/cli/src/tui/telegram-bridge.ts#L7346-L7368) | `imageGenerationDefaultsForRepo` / `audioGenerationDefaultsForRepo` — mirror with `videoGenerationDefaultsForRepo`. |
| Generated-image sidecar reader | [8095-8158](packages/cli/src/tui/telegram-bridge.ts#L8095-L8158) | Reads `<image>.json` after sendPhoto to enable reply-context awareness — re-use for `<video>.json` after sendVideo. |
| Multipart upload | [8050-8093](packages/cli/src/tui/telegram-bridge.ts#L8050-L8093) | `sendMediaReference` — already routes `kind: "video"` to `sendVideo` via `mediaTelegramMethod` in [media-routing.ts](packages/cli/src/tui/media-routing.ts) (lines [129-144](packages/cli/src/tui/media-routing.ts#L129-L144)). |

### 1.6 Telegram public/group creative workspace

File: [packages/cli/src/tui/telegram-creative-tools.ts](packages/cli/src/tui/telegram-creative-tools.ts)

| Anchor | Lines | What it is |
| --- | --- | --- |
| Tool factory | [139-157](packages/cli/src/tui/telegram-creative-tools.ts#L139-L157) | `buildTelegramCreativeTools` returns scoped instances of FileWrite/FileEdit/StructuredFile/Image/Audio/Tts + CreativeAudioFile. **Add `VideoGenerateTool` here.** |
| Scoped tool wrapper | [159-293](packages/cli/src/tui/telegram-creative-tools.ts#L159-L293) | Path guarding, encrypted blob storage, attachment notice. The `generate_image|generate_audio|generate_tts` regex on line 174 must include `generate_video`. |
| Manifest pickup of generated artifacts | [116-137](packages/cli/src/tui/telegram-creative-tools.ts#L116-L137) | `collectGeneratedArtifactPathsFromText` parses lines like `Image generated: <path>` and `Sound generated: ...` — extend marker regex to also catch `Video generated:`. |
| Public artifact policy | [99-105](packages/cli/src/tui/telegram-creative-tools.ts#L99-L105) | Blocks executable extensions. `.mp4/.webm/.mov/.mkv` are not on the blocklist — videos are allowed. |

### 1.7 Media routing (kind classification + Telegram method)

File: [packages/cli/src/tui/media-routing.ts](packages/cli/src/tui/media-routing.ts) (already complete for video)

| Anchor | Lines | Status |
| --- | --- | --- |
| `VIDEO_EXT` set | [27](packages/cli/src/tui/media-routing.ts#L27) | `.mp4 .mov .mkv .webm .avi .m4v` — already covers all model outputs. |
| `classifyMedia` | [117-127](packages/cli/src/tui/media-routing.ts#L117-L127) | Returns `"video"` for the above extensions. |
| `mediaTelegramMethod("video")` | [139-140](packages/cli/src/tui/media-routing.ts#L139-L140) | Routes to `sendVideo`. **No edits needed.** |

### 1.8 Execution package exports

File: [packages/execution/src/index.ts](packages/execution/src/index.ts)

Currently exports `ImageGenerateTool`, `ImageGenerateToolDefaults`, `AudioGenerateTool`, `AudioGenerateToolDefaults`, `VideoUnderstandTool`. **Add** exports for `VideoGenerateTool`, `VideoGenerateToolDefaults`, `VideoGenerationBackend`, `VideoGenerationPreset`, `VIDEO_GENERATION_MODEL_PRESETS`, `inferVideoGenerationBackend`, `videoGenerationSetupPlan`, `videoGenerationDir`, `videoGenerationQualityLadder`, `DEFAULT_DIFFUSERS_VIDEO_MODEL`.

---

## 2. Deliverable 1 — Sana as primary image model

### 2.1 What "primary" means here

The `/image` quality ladder ([packages/execution/src/tools/image-generate.ts:488](packages/execution/src/tools/image-generate.ts#L488-L501)) defines fallback order. Today `SECONDARY_FLUX_DEV_MODEL` sits at slot 0. With Sana installed, FLUX still requires gated HF access and 12GB+ VRAM; Sana 1.6B runs on 8-12GB VRAM under Apache/NSCL with **no gating**, has a Diffusers pipeline class shipping in `diffusers>=0.32`, and was already a fallback rung as `Sana Sprint 0.6B`. We promote the **SANA-1.5 1.6B 1024px** variant to slot 0 and add **SANA-1.5 4.8B** as a higher-quality option above it.

### 2.2 Sana model variants to register

Verified against <https://github.com/NVlabs/Sana> and HF model cards.

| ID | Pipeline | Steps | CFG | Resolution | dtype | Min VRAM | Notes |
| --- | --- | --- | --- | --- | --- | --- | --- |
| `Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers` | `SanaPipeline` | 20 | 4.5 | 1024×1024 | bfloat16 | 8 GB (CPU offload) / 12 GB recommended | **New primary.** Apache+NSCL/Gemma terms. Text enc.: `google/gemma-2-2b-it`. |
| `Efficient-Large-Model/SANA1.5_4.8B_1024px_diffusers` | `SanaPipeline` | 20 | 4.5 | 1024×1024 | bfloat16 | 16 GB / 24 GB recommended | Higher-fidelity Sana 1.5. |
| `Efficient-Large-Model/Sana_1600M_1024px_MultiLing` | `SanaPipeline` | 20 | 4.5 | 1024×1024 | bfloat16 | 8 GB / 12 GB | Multilingual variant (CN/emoji/EN). |
| `Efficient-Large-Model/Sana_1600M_2Kpx_BF16_diffusers` | `SanaPipeline` | 20 | 4.5 | 2048×2048 | bfloat16 | 16 GB / 22 GB tiled | 2K quality. |
| `Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers` | `SanaPipeline` | 20 | 4.5 | 4096×4096 | bfloat16 | 22 GB tiled | 4K — gated by tiling. |
| `Efficient-Large-Model/Sana_Sprint_0.6B_1024px_diffusers` | `SanaSprintPipeline` (or `AutoPipelineForText2Image`) | 4 | 0 | 1024×1024 | float16 | 6 GB | **Existing preset** — keep as the smallest/fastest path. |

### 2.3 Edits to `image-generate.ts`

1. Update `DEFAULT_DIFFUSERS_IMAGE_MODEL` (line [59](packages/execution/src/tools/image-generate.ts#L59)):
   ```ts
   export const DEFAULT_DIFFUSERS_IMAGE_MODEL = "Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers";
   ```
2. **Insert new presets** between the existing Sana Sprint preset (line [453](packages/execution/src/tools/image-generate.ts#L453-L469)) and the SDXL Turbo block. Use the same `ImageGenerationPreset` shape — copy from Sana Sprint and adjust `id/label/install/category/sizeClass/quality/minVramGB/recommendedVramGB/steps/guidance/width/height/note`.
3. **Update `IMAGE_GENERATION_QUALITY_LADDER`** (line [488](packages/execution/src/tools/image-generate.ts#L488-L501)) so Sana 1.5 4.8B → Sana 1.5 1.6B → existing FLUX trio → Z-Image → … Sana 1.5 1.6B should appear ahead of the FLUX entries because it has no gating risk; the rest of the ladder stays as a safety net.
4. **Patch `_pipeline_class`** in `DIFFUSERS_RUNNER` heredoc (line [529-538](packages/execution/src/tools/image-generate.ts#L529-L538)): add an explicit branch for Sana, since `AutoPipelineForText2Image` already picks `SanaPipeline` in diffusers ≥ 0.32 but Sana 1.5 needs a `text_encoder` dtype cast to bf16 (per Sana HF docs). Append after the `flux` branch:
   ```python
   if "sana" in lowered:
       from diffusers import SanaPipeline
       return SanaPipeline
   ```
   And in the call site, after `pipe = pipeline_cls.from_pretrained(...)`:
   ```python
   if "sana" in args.model.lower() and hasattr(pipe, "text_encoder"):
       try:
           pipe.text_encoder.to(torch.bfloat16)
       except Exception:
           pass
   ```
5. **Bump `_large_model` heuristic** (line [540-542](packages/execution/src/tools/image-generate.ts#L540-L542)) to enable `enable_model_cpu_offload` for the 4.8B Sana and the 2K/4K variants:
   ```python
   return any(token in lowered for token in [
       "flux.1", "flux.2", "stable-diffusion-3.5", "hunyuan", "z-image", "janus",
       "sana1.5_4.8b", "sana_1600m_2kpx", "sana_1600m_4kpx",
   ])
   ```
6. **Description string** update at line [1209-1215](packages/execution/src/tools/image-generate.ts#L1209-L1215): swap "SDXL Turbo default" for "**Sana 1.5 1.6B default**".
7. **Add Sana to fallback alternates**: when a FLUX preset fails (gated/oom), Sana should be a target. Set `fallbackFor: ["lllyasviel/flux1-dev-bnb-nf4", "ChuckMcSneed/FLUX.1-dev"]` on the new Sana 1.5 1.6B preset entry.

### 2.4 No new venv needed

`SanaPipeline` ships in `diffusers` (already in `DIFFUSERS_PYTHON_PACKAGES` at line [130-140](packages/execution/src/tools/image-generate.ts#L130-L140)). Sana 1.5 also pulls `google/gemma-2-2b-it` — already covered by `transformers` + `sentencepiece` + `protobuf`. **No requirements changes.**

### 2.5 First-run prewarm (Sana)

Plan and run automatically via the existing `prewarmImageModel(ctx, "Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers", "diffusers")` ([commands.ts:9147](packages/cli/src/tui/commands.ts#L9147)) path. The runner already streams `omnius_progress` JSON, which the TUI/Telegram surfaces consume identically.

---

## 3. Deliverable 2 — `/video` pipeline

### 3.1 Scope

- TUI: `/video`, `/video <prompt>`, `/video --model <id> <prompt>`, `/video setup <runtime>`, `/video list`, `/video prewarm`.
- Sub-kinds: `text-to-video` (T2V, default) and `image-to-video` (I2V, when an `--image` flag or attached image is present).
- Agent tool: `generate_video` (Diffusers-first; Ollama is not a path because no current Ollama model emits video). Telegram public + admin contexts.
- Telegram BotFather command `/video` auto-registered via `buildTelegramBotCommands` since the command registry covers it.
- Outputs: MP4 saved under `.omnius/videos/vid-<timestamp>-<rand>.mp4` with thumbnail PNG at `<video>.png` and sidecar JSON at `<video>.json`.
- Preview: TUI renders ASCII preview of the thumbnail; Telegram auto-attaches video via existing `sendVideo` routing.

### 3.2 New types — `VideoGenerationBackend`, `VideoGenerationKind`, `VideoGenerationPreset`

To live in [packages/execution/src/tools/video-generate.ts](packages/execution/src/tools/video-generate.ts) (new file).

```ts
export type VideoGenerationKind = "t2v" | "i2v";
export type VideoGenerationBackend = "auto" | "diffusers" | "comfyui";

export interface VideoGenerationPreset {
  id: string;
  label: string;
  kinds: VideoGenerationKind[];     // models can support t2v, i2v, or both
  backend: Exclude<VideoGenerationBackend, "auto">;
  pipelineClass:                    // signals which Diffusers class to use
    | "WanPipeline"
    | "MochiPipeline"
    | "CogVideoXPipeline"
    | "CogVideoXImageToVideoPipeline"
    | "LTXPipeline"
    | "LTXConditionPipeline"
    | "HunyuanVideoPipeline"
    | "AutoPipelineForText2Video";
  install: string;                  // canonical pip/cli line shown in setup
  category: string;
  sizeClass: string;
  quality: string;
  output: string;                   // e.g. "5s 720p MP4 at 24fps"
  minVramGB: number;
  recommendedVramGB: number;
  deployment: string;
  steps: number;
  guidance?: number;
  numFrames: number;
  fps: number;
  width: number;
  height: number;
  dtype: "bfloat16" | "float16";
  needsCpuOffload: boolean;         // forces enable_model_cpu_offload + tiling
  needsVae?: { repoSubfolder: string }; // Wan2.2 separate AutoencoderKLWan
  fallbackFor?: string[];
  note: string;
}

export const DEFAULT_DIFFUSERS_VIDEO_MODEL = "Wan-AI/Wan2.2-TI2V-5B-Diffusers";
```

### 3.3 Video preset registry (Phase 1 + Phase 2 from the reference doc)

These are the verified Diffusers-runnable models. Each entry uses the shape above.

| ID | Pipeline | Kinds | dtype | Frames | FPS | W×H | Steps | CFG | Min/Rec VRAM | License |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **`Wan-AI/Wan2.2-TI2V-5B-Diffusers`** | `WanPipeline` (+`AutoencoderKLWan`) | t2v, i2v | bfloat16 | 121 | 24 | 1280×704 | 50 | 5.0 | 24 / 80 | Apache 2.0 |
| `genmo/mochi-1-preview` | `MochiPipeline` | t2v | bfloat16 (variant `bf16`) | 84 | 30 | 848×480 | 64 | 4.5 | 22 (offload) / 42 | Apache 2.0 |
| `zai-org/CogVideoX-5b` | `CogVideoXPipeline` | t2v | bfloat16 | 49 | 8 | 720×480 | 50 | 6.0 | 5 (offload) / 15 | Apache 2.0 |
| `zai-org/CogVideoX-2b` | `CogVideoXPipeline` | t2v | bfloat16 | 49 | 8 | 720×480 | 50 | 6.0 | 4 / 8 | Apache 2.0 |
| `Lightricks/LTX-Video` | `LTXPipeline` | t2v | bfloat16 | 121 | 24 | 832×480 | 30 | n/a | 12 / 20 | LTX Open-Weights (non-commercial) |
| `Lightricks/LTX-Video-0.9.8-dev` | `LTXConditionPipeline` | i2v | bfloat16 | 96 | 24 | 832×480 | 30 | n/a | 16 / 24 | LTX Open-Weights |
| `tencent/HunyuanVideo` | `HunyuanVideoPipeline` | t2v | bfloat16 | 129 | 24 | 1280×720 | 50 | 6.0 | 60 / 80 | Tencent Hunyuan Community |
| `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | `WanPipeline` | i2v | bfloat16 | 121 | 24 | 1280×720 | 50 | 5.0 | 40 / 80 | Apache 2.0 |
| `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | `WanPipeline` | t2v | bfloat16 | 121 | 24 | 1280×720 | 50 | 5.0 | 40 / 80 | Apache 2.0 |

**Default**: `Wan-AI/Wan2.2-TI2V-5B-Diffusers` — practical 24GB VRAM target, supports both T2V and I2V, becomes the agent default.

**Quality ladder** (consumer-first, gated/biggest last):
```text
1. Wan2.2 TI2V 5B            (primary; both t2v+i2v)
2. LTX-Video 0.9.8 (T2V)     (fast iteration)
3. CogVideoX-5B              (5 GB VRAM with offload)
4. Mochi 1 Preview           (permissive)
5. CogVideoX-2B              (smoke-test fallback)
6. Wan2.2 I2V A14B           (premium I2V)
7. Wan2.2 T2V A14B           (premium T2V)
8. HunyuanVideo              (cinematic, datacenter)
```

### 3.4 Python runner template (`DIFFUSERS_VIDEO_RUNNER`)

Mirror `DIFFUSERS_RUNNER` from `image-generate.ts`. Key differences:

- Selects pipeline class from `model` via the same lowered-string heuristics (`wan`, `mochi`, `cogvideox`, `ltx`, `hunyuanvideo`). For Wan, additionally instantiate `AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)` and pass `vae=vae`.
- For I2V, accept `--image <path>` and feed via `image=load_image(args.image)`.
- Use `from diffusers.utils import export_to_video` to write MP4 at the model's preferred fps.
- After export, run `ffmpeg -hide_banner -loglevel error -ss 00:00:00 -i <out.mp4> -frames:v 1 -q:v 2 <out.mp4>.png` (or use OpenCV) to produce a thumbnail; emit thumbnail path in the final JSON line.
- Emit the same `{"omnius_progress": true, "stage": ..., "message": ..., "percent": ...}` lines on stderr so the TUI/telegram progress formatter keeps working.
- Output final JSON line: `{"ok": true, "path": "<mp4>", "thumbnail": "<mp4>.png", "frames": N, "width": W, "height": H, "fps": F, "duration_seconds": D, "model": "<id>", "device": "cuda"}`.

```python
def _video_pipeline(model):
    lowered = model.lower()
    if "wan" in lowered:
        from diffusers import WanPipeline, AutoencoderKLWan
        return ("wan", WanPipeline, AutoencoderKLWan)
    if "mochi" in lowered:
        from diffusers import MochiPipeline
        return ("mochi", MochiPipeline, None)
    if "cogvideox" in lowered:
        from diffusers import CogVideoXPipeline, CogVideoXImageToVideoPipeline
        return ("cogvideox", CogVideoXPipeline, CogVideoXImageToVideoPipeline)
    if "ltx" in lowered:
        from diffusers import LTXPipeline, LTXConditionPipeline
        return ("ltx", LTXPipeline, LTXConditionPipeline)
    if "hunyuanvideo" in lowered:
        from diffusers import HunyuanVideoPipeline
        return ("hunyuan", HunyuanVideoPipeline, None)
    from diffusers import AutoPipelineForText2Video
    return ("auto", AutoPipelineForText2Video, None)
```

The runner script lives at `.omnius/video-gen/diffusers_text2video.py` and is written by `ensureRunner(repoRoot, "video-diffusers")` (new variant of [image-generate.ts:1132-1141](packages/execution/src/tools/image-generate.ts#L1132-L1141)).

### 3.5 Python deps

```ts
const DIFFUSERS_VIDEO_PACKAGES = [
  "torch",
  "torchvision",
  "diffusers",
  "transformers",
  "accelerate",
  "safetensors",
  "pillow",
  "sentencepiece",
  "protobuf",
  "imageio",
  "imageio-ffmpeg",
  "ftfy",     // common Hunyuan/CogVideoX requirement
  "einops",   // LTX
  "av",       // Wan VAE
];
```

Cache layout mirrors image-gen at `.omnius/video-gen/{huggingface,torch,cache,pip-cache,.venv}`.

### 3.6 `VideoGenerateTool` class skeleton

Direct mirror of `ImageGenerateTool`. Required surface:

```ts
export class VideoGenerateTool implements Tool {
  name = "generate_video";
  description =
    "Generate a short video from a text prompt or text+image using a local Diffusers video model. " +
    "Default model: Wan-AI/Wan2.2-TI2V-5B-Diffusers (24GB-class GPU, both text-to-video and image-to-video). " +
    "Pass mode='t2v' (default) or mode='i2v' with image=<path|url>. Optional duration_seconds, fps, aspect_ratio, " +
    "negative_prompt, seed. Saves an MP4 under .omnius/videos and returns the file path. Outputs a thumbnail PNG " +
    "next to the MP4 so chat surfaces can render a preview.";
  parameters = { /* prompt, image, mode, model, backend, duration_seconds, fps,
                    num_frames, width, height, aspect_ratio, steps, guidance,
                    negative_prompt, seed, action, fallback, strict_model, expand_prompt */ };
  setProgressCallback(...) { ... }
  setPromptExpander(...) { ... }    // optional — reuses the same expander contract
  async execute(args): Promise<ToolResult> { ... }
}
```

Key parameter notes:

- `mode`: `"t2v" | "i2v"`. If `image` is passed and `mode` is unset, infer `"i2v"`.
- `duration_seconds` + `fps` derive `num_frames` if not provided.
- `aspect_ratio`: same `W:H` parser as image — resolves around the preset's longest side, snapped to a multiple of 16 (LTX requires multiples of 32; for those models snap to 32 in the runner-side validation).
- `negative_prompt`: forwarded to all models that accept it; ignored otherwise.
- `strict_model`/`fallback`: same semantics as `ImageGenerateTool`.

Reuse helpers from image-gen by lifting them into a tiny shared module (`packages/execution/src/tools/internal/generative-runtime.ts` if you want cleanliness, otherwise duplicate locally to keep the change surface tight):

- `runProcess`, `cleanProgressText`, `parsePercent`, `parseStructuredProgress`, `parseRunnerJson`
- `ensurePythonFor` (parameterize by `kind: "video-diffusers"` + dir + import-check string)
- `imageGenerationPythonEnv` → generalize as `pythonGenerationEnv(rootDir)`

> **Recommendation**: duplicate locally for the first cut to avoid touching the image-generate file beyond Sana edits. A follow-up PR can extract `internal/generative-runtime.ts`.

### 3.7 Sidecar JSON for video

Exact same shape as image sidecar ([image-generate.ts:1502-1534](packages/execution/src/tools/image-generate.ts#L1502-L1534)), extended fields:

```json
{
  "version": 1,
  "kind": "video-generation",
  "video_path": "/repo/.omnius/videos/vid-…mp4",
  "thumbnail_path": "/repo/.omnius/videos/vid-…mp4.png",
  "original_prompt": "...",
  "expanded_prompt": "...",
  "prompt_was_expanded": true,
  "mode": "t2v",
  "image_input": null,
  "model": "Wan-AI/Wan2.2-TI2V-5B-Diffusers",
  "backend": "diffusers",
  "width": 1280,
  "height": 704,
  "num_frames": 121,
  "fps": 24,
  "duration_seconds": 5.04,
  "seed": null,
  "aspect_ratio": "16:9",
  "created_at": "2026-…Z"
}
```

`recordOutboundGeneratedImagePrompt` in [telegram-bridge.ts:8102-8129](packages/cli/src/tui/telegram-bridge.ts#L8102-L8129) needs a parallel `recordOutboundGeneratedVideoPrompt` (or generalize the helper to take `kind`). The existing `sendMediaReference` path already accepts a `sourcePromptPath` option (see [telegram-bridge.ts:8087-8088](packages/cli/src/tui/telegram-bridge.ts#L8087-L8088)) — extend the conditional from `media.kind === "image"` to `media.kind === "image" || media.kind === "video"`.

### 3.8 TUI command — `/video`

Edit [packages/cli/src/tui/commands.ts](packages/cli/src/tui/commands.ts):

1. Add dispatcher case beside `/image` ([line 1590-1600](packages/cli/src/tui/commands.ts#L1590-L1600)):
   ```ts
   case "video": {
     return handleVideoCommand(ctx, arg, hasLocal);
   }
   ```
2. Implement `handleVideoCommand`, `showVideoModelsMenu`, `renderVideoModelList`, `prewarmVideoModel`, `rateVideoPresetForHardware`, `videoModelDiskStats`, `deleteVideoModelWeights`, `formatVideoGenerationProgress`. Mirror the image helpers at [commands.ts:9054-9260](packages/cli/src/tui/commands.ts#L9054-L9260) — keep names parallel for grep-ability.
3. Reuse `parseImageCommand` ([line 8759-8779](packages/cli/src/tui/commands.ts#L8759-L8779)) as-is — it is already generic over flags. Add `--image <path>` recognition through the existing flag pass-through.
4. For ASCII preview, after generation completes, the runner emits a `thumbnail` field in its final JSON. Pipe that into `buildImageAsciiPreview` and render with a "Generated video" label and `Tap to play: <path>` line. Use `extractSavedImagePath`-style helper `extractSavedVideoPath(text, repoRoot)` matching `Video generated: <path>`.

### 3.9 Command registry — `/video`

Edit [packages/cli/src/tui/command-registry.ts](packages/cli/src/tui/command-registry.ts):

1. Slash help block (after the `/music` entries at [line 147-151](packages/cli/src/tui/command-registry.ts#L147-L151)):
   ```ts
   ["/video", "Open video-generation model/setup menu"],
   ["/video <prompt>", "Generate a video from a prompt and show an ASCII thumbnail"],
   ["/video --model <model> <prompt>", "Generate with an explicit video model"],
   ["/video --image <path|url> <prompt>", "Image-to-video animation"],
   ["/video setup <diffusers|comfyui>", "Show setup commands for a video-generation backend"],
   ["/video list", "List video models by category, quality, size, and hardware fit"],
   ```
2. Add `video: "media"` to `CATEGORY_OVERRIDES` ([line 344-435](packages/cli/src/tui/command-registry.ts#L344-L435)).
3. Add `"video"` to the `USER_ONLY` and `NETWORKED` sets ([line 479-600](packages/cli/src/tui/command-registry.ts#L479-L600)).

### 3.10 Settings interface

Edit [packages/cli/src/tui/omnius-directory.ts](packages/cli/src/tui/omnius-directory.ts) at [line 228-229](packages/cli/src/tui/omnius-directory.ts#L228-L229):
```ts
/** Preferred video-generation model for /video and video-generation tool defaults */
videoModel?: string;
/** Preferred video-generation backend for /video */
videoBackend?: "auto" | "diffusers" | "comfyui";
/** Default sub-kind for /video when image input is absent */
videoKind?: "t2v" | "i2v";
```

### 3.11 Telegram bridge wiring

Edit [packages/cli/src/tui/telegram-bridge.ts](packages/cli/src/tui/telegram-bridge.ts):

1. Imports at [line 85-104](packages/cli/src/tui/telegram-bridge.ts#L85-L104) — add `VideoGenerateTool` + `VideoGenerateToolDefaults`.
2. Admin tool list at [line 6165-6168](packages/cli/src/tui/telegram-bridge.ts#L6165-L6168) — insert `new VideoGenerateTool(repoRoot, videoDefaults)`.
3. Compute `videoDefaults` in the same place as `imageDefaults`/`audioDefaults` (search for `imageGenerationDefaultsForRepo`). Add:
   ```ts
   private videoGenerationDefaultsForRepo(repoRoot: string): VideoGenerateToolDefaults {
     const settings = resolveSettings(repoRoot);
     return {
       model: typeof settings.videoModel === "string" && settings.videoModel.trim()
         ? settings.videoModel
         : undefined,
       backend: settings.videoBackend,
       defaultKind: settings.videoKind,
     };
   }
   ```
4. Quota regex at [line 6249](packages/cli/src/tui/telegram-bridge.ts#L6249):
   ```ts
   if (/^(generate_image|generate_audio|generate_video|generate_tts|create_audio_file)$/.test(toolName)) return "generation";
   ```
5. **Quota limits**: video is heavy. Add a dedicated `TelegramPublicQuotaKind` `"video-generation"` with a tighter window (e.g. 2/hour) in `TELEGRAM_PUBLIC_TOOL_QUOTAS` (search the same file for its declaration) and route `generate_video` to it explicitly.
6. Public-context creative tool factory call at [line 6205-6212](packages/cli/src/tui/telegram-bridge.ts#L6205-L6212) — pass `videoDefaults` into the new factory signature.
7. Outbound sidecar pickup at [line 8087-8093](packages/cli/src/tui/telegram-bridge.ts#L8087-L8093) — extend the condition so a successful `sendVideo` also calls `recordOutboundGeneratedMediaPrompt(chatId, messageId, videoPath, caption, "video")`. Read the `<video>.json` sidecar and store it on the chat history entry just like images today.
8. Telegram BotFather command list — `/video` will register automatically via `listCommandRegistry` consumed by `buildTelegramBotCommands` ([command-registry.ts:681-709](packages/cli/src/tui/command-registry.ts#L681-L709)).
9. System-prompt strings at [line 5555-5556](packages/cli/src/tui/telegram-bridge.ts#L5555-L5556): add a parallel sentence telling the model "For video generation requests, decide whether `generate_video` is appropriate (image-to-video if the user attached an image)."

### 3.12 Telegram public/group creative tools

Edit [packages/cli/src/tui/telegram-creative-tools.ts](packages/cli/src/tui/telegram-creative-tools.ts):

1. Imports at [line 22-30](packages/cli/src/tui/telegram-creative-tools.ts#L22-L30) — add `VideoGenerateTool` + `VideoGenerateToolDefaults`.
2. Factory signature at [line 139-145](packages/cli/src/tui/telegram-creative-tools.ts#L139-L145):
   ```ts
   export function buildTelegramCreativeTools(
     repoRoot: string,
     chatId: TelegramChatId | undefined,
     backendUrl?: string,
     imageDefaults: ImageGenerateToolDefaults = {},
     audioDefaults: AudioGenerateToolDefaults = {},
     videoDefaults: VideoGenerateToolDefaults = {},
   ): Tool[] {
   ```
3. Factory body at [line 148-157](packages/cli/src/tui/telegram-creative-tools.ts#L148-L157) — append `scopedTool(new VideoGenerateTool(root, videoDefaults), root, "generate")`.
4. Scoped wrapper regex at [line 174](packages/cli/src/tui/telegram-creative-tools.ts#L174):
   ```ts
   if (base.name === "generate_image" || base.name === "generate_audio" || base.name === "generate_video" || base.name === "generate_tts") {
   ```
5. `collectGeneratedArtifactPathsFromText` marker regex at [line 128](packages/cli/src/tui/telegram-creative-tools.ts#L128):
   ```ts
   line.match(/(?:Image generated|Sound generated|Music generated|Video generated|TTS generated|Created [A-Z]+ file|Created|Overwrote|Saved to):\s*([^\n\r(]+)/i);
   ```
6. **No blocklist edits needed** — `.mp4/.webm/.mov/.mkv/.m4v` are not in `PUBLIC_EXECUTABLE_ARTIFACT_EXTENSIONS` at [line 64-74](packages/cli/src/tui/telegram-creative-tools.ts#L64-L74), so videos already pass policy.

### 3.13 Execution package exports

Edit [packages/execution/src/index.ts](packages/execution/src/index.ts) — alongside existing `ImageGenerateTool` / `AudioGenerateTool` exports, re-export the entire `VideoGenerateTool` surface (class, defaults type, presets, backend type, ladder helper, setup plan, dir helper, default constant).

### 3.14 First-run prewarm strategy

Wan2.2 TI2V 5B is ~10 GB on disk and pulls T5 encoder + AutoencoderKLWan in addition to the transformer. The first `/video` call (or first `/video setup diffusers`) will:

1. Create `.omnius/video-gen/.venv`.
2. `pip install` `DIFFUSERS_VIDEO_PACKAGES`.
3. Pull `Wan-AI/Wan2.2-TI2V-5B-Diffusers` via `WanPipeline.from_pretrained(...)` and `AutoencoderKLWan.from_pretrained(..., subfolder="vae")`.
4. Run a `--prewarm` pass that loads the pipeline + VAE and exits. The runner must emit the same `omnius_progress` JSON for percent/stage so the TUI bar and Telegram updates work unchanged.

Document this in the existing `--help`-style command output for `/video setup diffusers`.

### 3.15 Hardware fit heuristic for video

Add `rateVideoPresetForHardware(preset, specs)` in `commands.ts`. Use the same VRAM-based scoring as `rateImagePresetForHardware` but include a "frames * resolution / dtype" penalty so 4K Wan and 720p Hunyuan land in red on a 16GB GPU. Reuse `imageFitIcon`. Keep three labels: `excellent / comfortable / offload-only / heavy-cloud`.

### 3.16 Tool description for the LLM (`generate_video` system surfacing)

Critical for autonomous use. Make the tool description list:

- Default model + that it runs both T2V and I2V.
- That **I2V** is triggered when an `image` is provided (path or URL).
- That `duration_seconds` ≤ 6 is recommended on consumer GPUs.
- That the tool **takes long enough that the agent should set expectations** (≥ 2 minutes on 24GB even for the small Wan); see Telegram quota in §3.11.
- That a thumbnail is auto-generated and attached for previews.
- That the sidecar JSON enables reply-context — the agent can reference "what prompt made this video?".

This matches the existing `ImageGenerateTool` description style ([image-generate.ts:1208-1215](packages/execution/src/tools/image-generate.ts#L1208-L1215)).

### 3.17 Public Telegram safety review

- **Public users cannot exfiltrate the repo**: `VideoGenerateTool` writes into the per-chat workspace root passed to `scopedTool` (the wrapper enforces `--output` is rewritten under `rootAbs` exactly like image+audio). Pattern: see [telegram-creative-tools.ts:171-235](packages/cli/src/tui/telegram-creative-tools.ts#L171-L235).
- **Public users cannot bomb GPU**: `generate_video` lands under `"generation"` (or new `"video-generation"`) quota.
- **Public users cannot pull arbitrary models**: agent gets `videoDefaults.model = settings.videoModel` and the description discourages exploring other models; admin can still override.
- **Public users cannot use I2V to leak files**: scoped wrapper already materializes `image` arg through `materializeTelegramCreativeArtifactForSend` for any local-path inputs ([telegram-creative-tools.ts:187-203](packages/cli/src/tui/telegram-creative-tools.ts#L187-L203)). Add the same TTS-style key list for video — recognized image-input keys: `image`, `image_path`, `init_image`, `source_image`, `reference_image`.

### 3.18 Thumbnail strategy

Emit thumbnail from the runner with one of:

- `ffmpeg -hide_banner -loglevel error -y -i <out.mp4> -frames:v 1 -q:v 2 <out.mp4>.png`
- Or `imageio.get_reader(out_path).get_data(0).save(thumb_path)` if ffmpeg is missing.

Set TUI ASCII preview to `<video>.png` since `image-ascii-preview.ts` already handles PNGs.

### 3.19 ComfyUI path (optional, deferred to follow-up)

The reference doc names ComfyUI as a strong runtime for Wan/Hunyuan. The plan defers ComfyUI to Phase 2 — implement the backend type now (`"comfyui"`) but `videoGenerationSetupPlan("comfyui", ...)` returns a "not yet supported in this build; use diffusers" message until a ComfyUI worker is wired. This keeps the menu layout future-proof without blocking shipping.

### 3.20 LTX license note (HIGH attention)

`Lightricks/LTX-Video` ships under the LTX Open-Weights License — **non-commercial**. The preset entry's `note` MUST surface this, and the LLM tool description should advise that the agent default (Wan2.2 TI2V 5B, Apache 2.0) is the safer baseline. We do not promote LTX to default; FLUX-style gating logic ([image-generate.ts:1031-1033](packages/execution/src/tools/image-generate.ts#L1031-L1033)) inspires the LTX error path: detect `LTX-Video-Open-Weights-License` files in the HF response and surface a license-accept reminder.

### 3.21 HunyuanVideo gating note

Tencent HunyuanVideo requires HF auth (license click-through). Reuse the existing gated-repo detection at [image-generate.ts:1031-1033](packages/execution/src/tools/image-generate.ts#L1031-L1033) — generalize it into `formatVideoFailure`.

---

## 4. Cross-cutting checklist for the downstream agent

### 4.1 Build / type pipeline

- `tsc --build` from the monorepo root must stay green. The new exports in `packages/execution/src/index.ts` need the new file to compile cleanly.
- Update `packages/execution/tsconfig.tsbuildinfo` consumers automatically; no manual edits.
- Add unit tests next to the image tests (look for `packages/cli/tests/image-ascii-preview.test.ts`-style pattern) — at minimum, register a `video-generate.preset.test.ts` that asserts every preset entry parses and the ladder is non-empty.

### 4.2 Smoke tests (post-merge)

Run the three reference doc smoke prompts (§8 in the package) via:

```
omnius /video setup diffusers
omnius /video "A locked-off cinematic shot of a matte black cube on wet asphalt at night..."
omnius /video --image .omnius/images/<sample>.png "Animate this image with subtle camera push-in..."
```

Then exercise via Telegram in (1) admin DM, (2) public group with bot added (use `aiwg-status` style ad-hoc check). Confirm:

- ASCII thumbnail renders in TUI.
- `sendVideo` delivers the file with caption containing the prompt.
- Reply-to-video carries `originalPrompt` into the next turn (because of the sidecar pickup at §3.11.7).
- Public quota blocks the third call within the window.

### 4.3 Telegram BotFather refresh

After shipping, run `aiwg ralph-status`-equivalent: the bot's command set updates via `setMyCommands` on startup. Confirm `/video` appears in the in-app menu by sending `/start` to the bot.

### 4.4 Docs to drop alongside the PR

- Update [README.md](README.md) media section (search for `/image` mention) to reference `/video`.
- Add a one-line bullet to [docs/voice-flow-architecture.md](docs/voice-flow-architecture.md) or create a peer doc `docs/video-flow-architecture.md` summarizing the runner shape.

### 4.5 Risk register

| Risk | Mitigation |
| --- | --- |
| Disk: Wan 5B = ~10 GB; Hunyuan ≈ 60 GB | Document in `/video setup` plan output; never trigger Hunyuan pull on a fresh install — keep it behind explicit selection. |
| Diffusers pipeline class drift between versions | Lock `diffusers>=0.32` in `DIFFUSERS_VIDEO_PACKAGES`; surface clear "update diffusers" failure note in `formatVideoFailure`. |
| ffmpeg missing for thumbnail | Fall back to `imageio` in the runner; bridge gracefully if both fail (skip preview, send video without ASCII). |
| Public spam (video is expensive) | Dedicated `video-generation` quota (≤ 2/hour/user); Telegram message acknowledges "this may take 2-5 minutes" before kickoff. |
| User runs `/video` with no GPU | `rateVideoPresetForHardware` returns "too heavy" / red icon; `prewarmVideoModel` refuses if `specs.gpuVramGB === 0` and points to cloud. |
| Aspect ratio not divisible by required factor | Runner snaps to multiple-of-32 for LTX, multiple-of-16 elsewhere; emit a `_progress("setup", "snapping ...")` line so the user sees the adjustment. |

---

## 5. Suggested implementation order

1. **Sana primary swap** in `image-generate.ts` (one file, no new tool surface). Build, smoke `/image "test"`.
2. **Skeleton video tool** in `packages/execution/src/tools/video-generate.ts` — types, preset registry, runner heredoc, no Tool class yet. `tsc` must compile.
3. **`VideoGenerateTool` class** with `execute({ action: "list" })` only. Wire export. Build.
4. **CogVideoX-2B path** — the smallest model — to validate the full Diffusers stack: prewarm + generate + thumbnail + sidecar. This is the equivalent of "Phase 1 smoke test" from the reference doc.
5. **Wan2.2 TI2V 5B path** — verify T2V and I2V on a 24 GB-class GPU (or note CPU-offload expectation).
6. **TUI surface** — dispatcher case, `handleVideoCommand`, menu, list, prewarm, ASCII preview hook.
7. **Command registry** entries.
8. **Settings interface** entries.
9. **Telegram admin DM** wiring — add to `adminTools`, defaults helper, quota regex.
10. **Telegram public/group** wiring — `buildTelegramCreativeTools` signature + factory body + marker regex.
11. **Outbound sidecar pickup** for `sendVideo`.
12. **License/gating warnings** for LTX + Hunyuan.
13. **Tests** — preset shape, ladder, parseImageCommand reused for video flags.
14. **Docs** — README + plan-link.

Each step is independently shippable; do not bundle 1+2+3 into one commit.

---

## 6. File-by-file changelist

| File | Change | Owner |
| --- | --- | --- |
| [packages/execution/src/tools/image-generate.ts](packages/execution/src/tools/image-generate.ts) | Promote Sana 1.5 1.6B to default; add 4.8B/2K/4K presets; patch `_pipeline_class`; update description; update ladder. | Sana swap |
| [packages/execution/src/tools/video-generate.ts](packages/execution/src/tools/video-generate.ts) **(new)** | Full `VideoGenerateTool` class + presets + runner template + helpers. | Video |
| [packages/execution/src/index.ts](packages/execution/src/index.ts) | Re-export Video* surface. | Video |
| [packages/cli/src/tui/commands.ts](packages/cli/src/tui/commands.ts) | `case "video"`, `handleVideoCommand`, `showVideoModelsMenu`, `renderVideoModelList`, `prewarmVideoModel`, `formatVideoGenerationProgress`, `rateVideoPresetForHardware`, `videoModelDiskStats`, `deleteVideoModelWeights`, `extractSavedVideoPath`. | Video |
| [packages/cli/src/tui/command-registry.ts](packages/cli/src/tui/command-registry.ts) | `/video` signatures; `CATEGORY_OVERRIDES.video = "media"`; add `video` to `USER_ONLY` + `NETWORKED`. | Video |
| [packages/cli/src/tui/omnius-directory.ts](packages/cli/src/tui/omnius-directory.ts) | `videoModel`/`videoBackend`/`videoKind` on `OmniusSettings`. | Video |
| [packages/cli/src/tui/telegram-bridge.ts](packages/cli/src/tui/telegram-bridge.ts) | Import; admin tool list; `videoGenerationDefaultsForRepo`; quota regex; outbound sidecar pickup; system prompt phrasing; new quota class. | Video |
| [packages/cli/src/tui/telegram-creative-tools.ts](packages/cli/src/tui/telegram-creative-tools.ts) | Import; signature; factory body; scoped-wrapper regex; marker regex. | Video |
| [packages/cli/tests/](packages/cli/tests/) | New `video-generate.preset.test.ts`; extend image preset test if shared registry helper introduced. | Video |
| [README.md](README.md) | One-line `/video` doc near `/image`. | Video |
| [docs/sana-and-video-generation-integration-plan.md](docs/sana-and-video-generation-integration-plan.md) **(this doc)** | Reference. | — |

---

## 7. Open decisions for the user

1. **Default video model**: Wan2.2 TI2V 5B (recommended — 24GB-class, handles both T2V and I2V, Apache 2.0). Alternate: CogVideoX-5B if 16GB target is preferred for the broader install base.
2. **Whether to also expose ComfyUI runtime now**: this plan defers it to a follow-up. Wire the type but not the backend.
3. **Public Telegram quota for video**: suggested 2/hour/user. Configurable.
4. **Whether the new tool should also generate **audio-with-video** (Wan2.2 S2V / LTX-2.3)**: out of scope for v1; document as a Phase 4 follow-up.

---

## 8. Reference excerpts (verified)

- Wan2.2 TI2V 5B Diffusers: `WanPipeline` + `AutoencoderKLWan` subfolder VAE; 50 steps, CFG 5.0, 121 frames @ 24fps, 1280×704; min 24GB; Apache 2.0. (HF `Wan-AI/Wan2.2-TI2V-5B-Diffusers`)
- CogVideoX-5B: `CogVideoXPipeline`; 50 steps, CFG 6.0, 49 frames @ 8fps, 720×480; min ~5GB with `enable_model_cpu_offload`. (HF `zai-org/CogVideoX-5b`)
- Mochi 1 Preview: `MochiPipeline`; variant `bf16`; 64 steps, CFG 4.5 schedule, 84 frames @ 30fps, 848×480; min 22GB with offload; Apache 2.0. (HF `genmo/mochi-1-preview`)
- LTX-Video: `LTXPipeline` / `LTXConditionPipeline`; 30 steps; frames divisible by 8+1; LTX Open-Weights non-commercial. (HF `Lightricks/LTX-Video`)
- Sana 1.5 1.6B: `SanaPipeline`; 20 steps, CFG 4.5, 1024×1024, bfloat16; text encoder `google/gemma-2-2b-it`; NSCL v2-custom + Gemma terms; not gated. (HF `Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers`, <https://github.com/NVlabs/Sana>)

---

## 9. Quick map for the downstream agent (line-by-line cheat sheet)

```text
NEW    packages/execution/src/tools/video-generate.ts                 (full file, ~1500 LOC mirroring image-generate.ts)
EDIT   packages/execution/src/tools/image-generate.ts
         L59  default = "Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers"
         L147 prepend 4 Sana 1.5 preset entries (1.6B / 4.8B / 1600M MultiLing / 2Kpx)
         L488 ladder: place SANA1.5_4.8B then SANA1.5_1.6B above FLUX entries
         L529 _pipeline_class: add Sana branch
         L540 _large_model: enable offload for 4.8B / 2Kpx / 4Kpx
         L1209 description: swap SDXL-Turbo for Sana 1.5 1.6B
EDIT   packages/execution/src/index.ts                                (add Video* re-exports near L165-197)
EDIT   packages/cli/src/tui/commands.ts
         L1590 add `case "video"`
         L8742 introduce ParsedVideoCommand parity OR reuse ParsedImageCommand
         L9054 add showVideoModelsMenu mirror
         L9147 add prewarmVideoModel mirror
         L9169 add handleVideoCommand mirror
EDIT   packages/cli/src/tui/command-registry.ts
         L147 insert /video signatures
         L344 add video: "media"
         L506 add "video" to USER_ONLY
         L555 add "video" to NETWORKED
EDIT   packages/cli/src/tui/omnius-directory.ts
         L228 append videoModel/videoBackend/videoKind
EDIT   packages/cli/src/tui/telegram-bridge.ts
         L85   add VideoGenerateTool import
         L6165 add new VideoGenerateTool(repoRoot, videoDefaults)
         L6205 pass videoDefaults to buildTelegramCreativeTools
         L6249 add generate_video to quota regex (or new "video-generation" kind)
         L7346 add videoGenerationDefaultsForRepo helper
         L8087 broaden generated-media sidecar pickup beyond images
         L5555 system-prompt note about generate_video
EDIT   packages/cli/src/tui/telegram-creative-tools.ts
         L24   add VideoGenerateTool / VideoGenerateToolDefaults imports
         L128  add "Video generated" to marker regex
         L139  add videoDefaults to factory signature
         L152  add scopedTool(new VideoGenerateTool(...))
         L174  add generate_video to the generation-routing regex
```

End of plan.