# Motion-overlay — `vclaw video motion-overlay`

Turn an existing **talking-head video** into a polished reel with **motion-graphics
overlays synced to the speech**, driven by Google Flow's **Omni Flash V2V**
transport (which paints kinetic typography / icon / metaphor animations on top of
footage while preserving the original voice).

The prompt IP — retention principles, a reel-anatomy load map, a concept→animation
metaphor map, and two hard-won guardrails (no metadata leaking as on-screen text;
no human portraits or the policy filter trips) — lives in deterministic,
offline-testable code under `src/video/motion-overlay/`, not in markdown.

> **Plan/dry by default. No provider spend** until you pass `--execute
> --confirm-spend`. The plan renders into the preview portal for human review
> first.

## At a glance

```
vclaw video motion-overlay --input <video-path> (--project <slug> | --output-dir <path>)
  [--layout split|overlay|motion-only|avatar-host]   # default: split
  [--style apple-clean|editorial-dark|knowledge-tool] # default: apple-clean
  [--accent <hex>]                                    # configurable accent (replaces a hardcoded brand colour)
  [--lang <code>]                                     # STT + on-screen-text language (default: auto)
  [--max-take-seconds 10]                             # Omni hard per-clip limit
  [--transcript <path>]                               # optional: bring-your-own transcript JSON
  [--gb-character <Name:ID>]                          # avatar-host layout ONLY: the locked host character
  [--preview]                                         # also render review/review.html
  [--root <path>]
  [--execute --confirm-spend]                         # render (spend-gated)
```

## What happens

**Default (dry):** ingest → STT → slice → compose → write work folder + manifest
+ (optionally) preview HTML. **No provider is called.**

**`--execute` (requires `--confirm-spend`, else exit-3 `spend_confirmation_required`):**
each take runs through omni-flash V2V, the original take audio is restored, and
the takes are restitched into the final reel.

### Work folder layout

```
<output>/                            # projects/<slug>/artifacts/motion-overlay/ OR --output-dir
├── motion-overlay-plan.json         # the manifest (schemaVersion 1)
├── README.md
├── source/  original.mp4 · audio.aac · transcript.json
├── takes/   take-01_0s-10s.mp4 ...  # frame-accurate cuts
├── prompts/ take-01_0s-10s.md ...   # one ready-to-send prompt per take
├── frames/  frame_01.jpg ...
├── host/    take-01_0s-10s.mp4 ...  # avatar-host ONLY: generated speaking host clips
├── v2v/     take-01_0s-10s.mp4 ...  # omni-flash overlay output (execute)
├── restored/ take-01_0s-10s.mp4 ... # audio-restored takes (execute)
├── review/  review.html             # preview-portal approval surface (--preview)
└── motion-overlay-reel.mp4          # the final stitched reel (execute)
```

## Layouts

- **split** (default) — motion graphics in the top half, speaker untouched in the bottom half.
- **overlay** — motion graphics composited over the speaker, safe areas around face/hands.
- **motion-only** — speaker removed; full-frame motion graphics narrated by their voice (keynote style).
- **avatar-host** (Layout D) — like `motion-only`, but the frame is filled by an
  **identity-locked character host** delivering the lines, with synced overlays on
  top. **Requires `--gb-character <Name:ID>`.** This is the heaviest path; see below.

## Avatar-host (Layout D)

The `avatar-host` layout replaces the speaker with an identity-locked character.
There are **two host engines** (`--host-engine`), trading identity-lock against
native voice — pick per project:

| `--host-engine` | Identity | Voice | Best for |
|---|---|---|---|
| `omni-r2v` (default) | loose (mitigated, see below) | **native + lip-synced** | a presenter who actually *speaks* |
| `veo-i2v` | **pixel-tight** (same still every take) | silent (captioned / add VO) | brand-critical identity, talking-head B-roll |

There is **no scriptable path that gives both at once** (Flow's `@Character` system
locks identity+voice but is web-UI only), so this is a genuine fork.

### `omni-r2v` (default) — the talking avatar

```
character reference image  +  the take's line  +  a voice preset
   → omni-flash R2V (ingredients:) generates the audio AND lip-synced video together
   → host/take-NN.mp4   (the avatar SPEAKING that line, genuinely lip-synced)
   → stitch the host clips in order        →  avatar-reel.mp4
   → re-STT the avatar's own speech + local lower-thirds  →  motion-overlay-avatar.mp4
```

Because Omni produces the speech and the mouth movement jointly, the avatar is
truly lip-synced — **no separate TTS, no muxing, no lip-sync model**. And unlike
the V2V "add-overlay" edit, R2V generation is **not moderation-blocked**.

### `veo-i2v` — the tight-identity (silent) avatar

```
the SAME character still  (image: literal first frame)  →  Veo 3.1 I2V, per take
   → host/take-NN.mp4   (frame 0 of EVERY take is pixel-identical → identity locked)
   → stitch  →  avatar-reel.mp4
   → caption from the planned script, timed to each take's snapped duration
                                          →  motion-overlay-avatar.mp4 (silent)
```

Because every take starts from the identical still, identity/wardrobe/backdrop are
locked frame-to-frame and take-to-take — far tighter than R2V — but the clip carries
no speech (add a voiceover or soundtrack separately, or let the captions carry it).
Veo I2V is **landscape-only** server-side. It shares the same retry + resume + scratch
plumbing as `omni-r2v` (a slow provider queue can time a take out; just re-run to
resume from it — completed takes are reused).

Flags: `--gb-character Name:ID` (label + go-bananas id, split on the final `:`),
`--gb-character-image <path>` (the reference still — **required** at `--execute`;
it is the R2V ingredient for `omni-r2v` and the literal first frame for `veo-i2v`),
`--host-engine omni-r2v|veo-i2v` (default `omni-r2v`),
`--gb-voice <preset>` (a Flow voice preset, default `Puck`; `omni-r2v` only),
`--host-retries <n>` (per-take generation attempts, default 3),
`--host-look <text>` (a stable appearance + setting descriptor — pinned on every
take by both engines), `--no-host-chain` (disable cross-take chaining; `omni-r2v` only).
Captions are timed to the avatar's *actual* speech (it is re-transcribed because
Omni's pacing differs from the source); the caption pass is best-effort — the
lip-synced reel stands on its own if it is skipped.

**Character consistency (the hard part).** omni-flash R2V treats the reference image
as a *loose* influence, so each independently-generated take re-invents the face,
wardrobe, and backdrop — the avatar drifts across a multi-take reel. There is **no
API-scriptable mechanism today** that locks identity *and* generates native voice
together (Google Flow's `@Character` system does, but it is web-UI only). Two levers
reduce the drift on the native-voice path:

- **`--host-look <text>`** pins a stable appearance + *fixed* setting descriptor that
  is repeated on **every** take's prompt (e.g. the character's go-bananas base
  prompt + one locked studio). This is what stops the backdrop/wardrobe from jumping
  take-to-take. Without it the prompt uses a neutral default. (The old build
  hardcoded a Bunty-flavoured "friendly cartoon presenter in a sleek modern studio",
  which actively *caused* drift for any other character — that is now gone.)
- **Chaining (on by default, `--no-host-chain` to disable)** seeds each take's R2V
  with the **previous take's last frame** as an extra `ingredients:` reference, so the
  face/wardrobe carry forward. The last frame is extracted with ffmpeg to a hidden
  `host/.chain-NN.png` and chained into take *N+1*. Chaining survives resume (a
  resumed take re-seeds from its existing clip) and is best-effort (a failed frame
  extract just drops that one seed). For tight pixel-level identity instead, use the
  silent go-bananas-character → Veo-I2V path (no native voice).

**Robust across multiple takes.** The Flow safety filter rejects a benign R2V
generation *probabilistically*, so on a multi-take reel one unlucky take would
otherwise fail-fast the whole run. Each take is therefore **retried up to
`--host-retries`** (a plain retry of the same line usually clears), and generation
is **resumable** — a `host/take-NN.mp4` already on disk from a prior run is reused,
never regenerated, so re-running after a mid-reel failure never re-spends on the
takes that already succeeded.

The generation runs **only under `--execute --confirm-spend`**; in plan/dry mode
the work folder is laid out and the host clips are merely enumerated.

## On-screen text guardrails (CRITICAL RULES)

Every composed prompt emits three standing rules:

1. **AUDIO** — pass the original voiceover through; the visual layer only.
2. **NO METADATA ON SCREEN** — never render px/ms/hex/font-name/easing/stroke as
   visible text; only quoted text renders.
3. **TEXT ONLY, NO PORTRAITS** — no drawn human silhouettes/portrait
   outlines/avatar icons (policy-filter guardrail). The `avatar-host` host base
   layer is exempt because it is generated separately and never described to Omni
   as a portrait.

## Transcription

Gemini STT (reuses the Gemini key pool) by default, or bring your own with
`--transcript <path>`:

```json
{ "language": "pt", "segments": [ { "start": 0.0, "end": 3.2, "text": "..." } ] }
```

## Module map (`src/video/motion-overlay/`)

| Module | Purpose | Purity |
|---|---|---|
| `types.ts` | shared contract for the whole family | types |
| `slice.ts` | segments + `maxTakeSeconds` → sentence-boundary ≤10s splits | pure |
| `analyze-reel.ts` | per-take reel anatomy + anchor words + pauses | pure |
| `motion-style.ts` | style registry + concept→animation metaphor map; accent/brand resolution | pure |
| `compose-prompt.ts` | **the IP** — per-take prompt (CRITICAL RULES + layout block + style + shots) | pure |
| `plan.ts` | assemble the `MotionOverlayPlan` manifest | pure |
| `transcribe.ts` | Gemini STT (injectable transport) | side-effect, thin |
| `ingest.ts` | ffmpeg probe / extract audio / cut takes / extract frames | side-effect, thin |
| `write.ts` | emit folder + README + manifest + per-take prompts | side-effect |
| `preview.ts` | render the plan into the preview-portal review surface | side-effect |
| `execute.ts` | per-take V2V → audio-restore → stitch (injectable `MotionOverlayStepRunner`) | side-effect, tested via injection |
| `avatar-host.ts` | avatar-host: per-take speaking-host orchestration (injectable `MotionOverlayHostGenerator`) | side-effect, tested via injection |
| `avatar-host-transport.ts` | real omni-flash R2V+voice host generator (shells `flow.ts`); pure prompt/arg builders tested, spawn injectable | side-effect, pure builders tested |
| `v2v-transport.ts` | real Google Flow omni-flash V2V transport (upload → submit → poll → download); pure request helpers tested, network path injectable | side-effect, tested via injected fetch |
| `render-local.ts` | native local overlay renderer — segment → lower-third SVG → `sharp` PNG → ffmpeg `overlay`; SVG/filtergraph builders pure, sharp+ffmpeg injectable (`--render local`) | side-effect, pure builders tested |
| `run.ts` | plan orchestrator — `planMotionOverlay`, the dry/plan front door wired into `vclaw video motion-overlay` | orchestrator |
| `motifs.ts` | concept → visual motif layer — surfaces a matching icon/stat motif at the moment a concept is spoken | pure |
| `animate.ts` | animation timing math (clamp + easing curves) for the per-frame renderer | pure |
| `animate-render.ts` | per-frame **animated** overlay renderer (`--animate`) — animated SVG frames instead of the static fade path; builders pure, sharp+ffmpeg injectable | side-effect, pure builders tested |
| `flow-pack.ts` | Flow hand-off pack + restitch — the productised "manual Omni" loop (write prompts → upload each take → paste back → restitch) | side-effect |

Every side effect — STT, ffmpeg, V2V, and the omni-flash R2V host
generation — is behind an injectable interface, so the whole pipeline is unit- and
e2e-tested offline with **no network and no spend**.

## Schema

`schemas/video/artifacts/motion-overlay-plan.schema.json` is the canonical
`MotionOverlayPlan` contract (allowlisted in the artifact-schema-coverage check).

## Live execution & moderation (`--execute`)

`--execute --confirm-spend` runs each take through the Google Flow **omni-flash
V2V** transport (`v2v-transport.ts`). Requires `USEAPI_API_TOKEN` and
`USEAPI_ACCOUNT_EMAIL` in the environment (export them from `.env`); missing creds
fail fast as `provider_unreachable` before any network call.

Proven request recipe (validated live 2026-06-05 against useapi.net):

1. **Upload** the take — `POST {base}/google-flow/assets/{email}` with
   `Content-Type: video/mp4` and the raw bytes →
   `mediaGenerationId.mediaGenerationId` (this becomes `referenceVideo_1`).
2. **Submit** — `POST {base}/google-flow/videos` with
   `{ model: "omni-flash", prompt, referenceVideo_1, startFrameIndex_1: 0,
   endFrameIndex_1: <frameCount>, aspectRatio, email, async: true }`. The
   response field is **`jobid`** (lowercase `i`), not `jobId`.
3. **Poll** — `GET {base}/google-flow/jobs/{jobId}` with the job id passed
   **RAW / unencoded** → `status: created → completed | failed`.
4. **Result** — `response.media[].videoUrl`; download to the take's output path.

Then `execute.ts` restores the take's original audio (`ffmpeg -map 0:v -map 1:a`)
and clip-stitches the takes into the final reel.

> **Moderation gotcha.** The Flow safety filter rejects some *input videos* for
> editing with `PUBLIC_ERROR_UNSAFE_GENERATION` / `FINISH_REASON_INPUT_VIDEO_EDIT`
> — this is **input-specific and prompt-independent** (a benign prompt on a
> flagged clip still fails; realistic-face footage is the usual trigger). The
> transport retries a failed verdict a few times (it is probabilistic), then
> errors with an actionable message. Use stylized / non-photoreal source clips.
> The sync `bun run flow.ts --ref-video` path returns the generic "All operations
> failed" and omits the frame-window params — prefer the async path above.

## Delivery targets (`--delivery`)

`--delivery` is the high-level "where the animation comes from" knob (it sets the
render mode + flow-pack defaults; `--render`/`--emit-flow-pack` stay as the low-level
controls):

| `--delivery` | What it does | Notes |
|---|---|---|
| `local` | render the finished reel locally | reliable, free, on your exact footage (default product path) |
| `flow-web` | render local **and** emit a Flow hand-off pack | the local reel is the instant fallback; the pack is for the *real* Omni animation, pasted into Flow's web editor by hand |
| `flow-api` | omni-flash V2V via the API | gated — needs `--execute --confirm-spend`; Flow currently safety-blocks the V2V edit for all inputs |

### The Flow hand-off loop (`--emit-flow-pack` / `--restitch`)

The source skill's real pipeline is *write prompts → upload each take + paste its
prompt into Flow → download → stitch*. The Omni step was never automated and Flow's
API blocks the V2V edit, so the hand-off stays manual — but it's now a first-class
artifact instead of ad-hoc scripts (`flow-pack.ts`):

- **`--emit-flow-pack`** writes `<work>/flow-pack/`: each take's clip + a clean,
  ready-to-paste prompt `.txt` (the prompt body extracted from `prompts/take-NN.md`)
  + a README of the workflow.
- **`--restitch <flow-outputs-dir>`** takes the operator's downloaded animated takes,
  re-muxes each with its **original audio** (CRITICAL RULE 1), and concatenates the
  finished reel (`<work>/motion-overlay-reel.mp4`). Terminal — no render/spend.

```
# 1. produce a local reel now + the Flow pack to upgrade it
vclaw video motion-overlay --input clip.mp4 --output-dir out/ --delivery flow-web --style editorial-dark
# 2. (you upload each out/flow-pack/take-NN.mp4 + paste take-NN.txt in Flow, download to out/done/)
# 3. stitch the animated takes back, audio restored
vclaw video motion-overlay --input clip.mp4 --output-dir out/ --restitch out/done/
```

The pure pieces (prompt extraction, README, output pairing, concat/restore args) are
unit-tested; fs + ffmpeg are injectable seams.

### `--render local` (no spend, no moderation) — the reliable path

Because the V2V "add-overlay" edit is moderation-blocked for most inputs, the
**local renderer is the recommended way to actually produce a reel**:

```
vclaw video motion-overlay --input clip.mp4 --output-dir out/ --render local [--kicker "BRAND"]
```

`--render local` (vs the default `--render v2v`) runs `renderLocalOverlay`
(`render-local.ts`): each transcript segment becomes a broadcast lower-third —
an **SVG rasterised to PNG via `sharp`** (resvg text shaping; NO system
freetype / ffmpeg `drawtext`) — composited onto the source with one ffmpeg
`overlay` filtergraph (each card fades + slides up on its window). Accent words
reuse the prompt composer's anchor-word detector; `--kicker` sets the small
brand label. It is **free** (no provider, no `--confirm-spend` gate) and the
original audio is preserved. The pure SVG/filtergraph builders are unit-tested;
sharp + ffmpeg are injectable seams.

#### Card styles (`--style`) and giant headlines (`--headlines`)

`--render local` honours `--style`, rendering three **structurally** distinct card
looks faithful to the source skill's catalog (not just a colour swap — different
panel geometry, label treatment and typography so they read as different over busy
footage). The same `--style`/`--accent`/`--kicker`/`--headlines` also drive the
**avatar-host** caption pass.

| `--style` | Card look |
|---|---|
| `apple-clean` (default) | **floating inset** rounded frosted card · thin accent left bar · accent-dot + uppercase kicker · clean sans-serif sentence case |
| `editorial-dark` | **full-bleed** edge-to-edge near-black poster bar · thick accent **top rule** · solid accent **tag block** label · bold condensed **UPPERCASE** |
| `knowledge-tool` | **left-anchored** narrower cool-dark study card · thick lavender left rule · **`[[ MONO ]]`** bracket label · editorial **serif** body |

Accent words render in the accent colour **and** get the skill's signature accent
**`*` beat-marker** after them.

`--headlines` adds the skill's **kinetic-typography beat**: each detected anchor word
also explodes to a frame-filling centred headline with the accent `*` beat-marker,
synced to the moment the word is spoken, composited above the lower-third. Headline
moments are capped (≤6, one per caption) so the reel doesn't strobe. Pure
`lowerThirdSvg`/`anchorHeadlineSvg` builders are unit-tested.

`--icons` adds the skill's **"visualize what's being said" layer** (`motifs.ts`):
the transcript is scanned and, at the moment a concept is spoken, a matching
**hairline icon** (money → `$`, team → people, launch → rocket, AI → node graph,
time → clock, …) or an **infographic stat callout** (a number, or a percentage with
a **ring gauge**) pulses in the upper-right corner like a broadcast stat bug — in the
style's accent, capped to ≤5 per reel. Icons are vector geometry (no portraits, no
bitmaps), so they render reliably. Both `--headlines` and `--icons` work on
`--render local` and the `avatar-host` caption pass. `detectMotifs` + the
`iconMotifSvg`/`statMotifSvg` builders are pure and unit-tested.

#### `--animate` — real motion graphics (per-frame)

By default the local overlays fade and slide in, then hold. `--animate` swaps in a
**per-frame motion-graphics layer** (`animate.ts` + `animate-render.ts`): every overlay
is evaluated as a function of time `t` and **one transparent frame is rasterised per
output frame** (30fps), then ffmpeg overlays the sequence on the footage. The motions:

- **lower-third** — fade + slide-up entrance, then a **word-by-word reveal** (words
  appear across the first ~60% of the take; the panel holds full size so it never
  reflows — later words render at `fill-opacity 0`);
- **stat callout** — the number **counts up** 0→N and the **% gauge fills** (eased);
- **icons / headlines** — **pop in** with a small overshoot (spring).

All the timing is pure math (`easeOutCubic`/`easeOutBack`, `revealWordCount`,
`countUpValue`, `gaugeProgress`) and unit-tested; the sharp rasterise + ffmpeg run are
injectable. It costs more (N frames rasterised — a 16s clip ≈ 480 frames, ~20s on a
laptop) but stays fully local/free. Works with any `--style` and composes with
`--icons`/`--headlines`.

```
vclaw video motion-overlay --input clip.mp4 --output-dir out/ \
  --render local --style apple-clean --icons --animate --kicker "BRAND"
```

```
vclaw video motion-overlay --input clip.mp4 --output-dir out/ \
  --render local --style editorial-dark --headlines --kicker "BRAND"
```