# Canvas Frame Scrub — Scroll-Driven Video

> **Scope:** frontend-scroll-driven
> **Layer:** 3
> **Keywords:** frame-scrub, canvas, scroll-driven video, scrubbing, ffmpeg, apple-style, poster fallback
> **Load When:** ui-designer active, or the brief says scroll controls a video

**Verified against:** GSAP 3.13 ScrollTrigger + Lenis 1.x + ffmpeg 6.x, shipped to production. Last-verified: 2026-08-21.

---

Reference for building a hero (or any section) where scrolling advances a video frame by frame, drawn into a `<canvas>`. This is the Apple product-page mechanic. It covers why canvas beats `<video>`, frame extraction with ffmpeg, the focal-point cover draw, the resize-on-visible bug, cache busting, the mobile poster fallback, and how to verify the result with motion instead of a still.

---

## Section 1: Canvas, Not `<video currentTime>`

Two failure modes rule out driving a `<video>` element from scroll position:

1. **`<video currentTime>` is unreliable on iOS Safari.** Decode throttling and autoplay restrictions make seeking stutter, and swapping the source can flash a black frame.
2. **An autoplaying `<video>` is not a scroll animation at all.** It reads as a demo running by itself rather than something the user drives. If the requirement is "scroll controls the video", a looping autoplay video does not satisfy it.

A canvas fed by a preloaded image sequence works cross-browser and never flashes.

A self-playing ambient loop is the exception, not the default: use it only for a secondary background plane, never for the primary hero.

---

## Section 2: Frame Extraction

Extract the approved `.mp4` into a JPG sequence:

```bash
ffmpeg -i source.mp4 -vf "fps=30,scale=1280:-1" -q:v 5 frames/frame_%03d.jpg
```

| Knob | Guidance |
|---|---|
| Duration and fps | 4 to 10s at 30fps yields roughly 120 to 300 frames. Around 145 is a good balance of smooth and light. |
| `-q:v` | 5 balances quality against weight. |
| Numbering | `%03d` (frame_001) matches the 1-based `framePath` contract in Section 4. |
| Format | JPG is the safe default (fast decode, universal). If the sequence gets heavy (tens of MB), also try WebP via `-c:v libwebp -q:v 70`, which usually cuts total weight at similar quality. |

**Treat total sequence weight as a performance budget.** It shows up directly in Lighthouse. If it is too heavy, cut frame count (120 instead of 145) or resolution before cutting quality.

**Mirror the footage when the subject collides with the copy.** If the subject sits on the side where text goes, flip it during extraction:

```bash
ffmpeg -i source.mp4 -vf "hflip,fps=30,scale=1664:-1" -q:v 5 frames/frame_%03d.jpg
```

**Always extract a poster.** Pick a frame with the subject well composed and save it as `poster.jpg`. It backs the mobile and reduced-motion paths.

---

## Section 3: Cache Busting Is Mandatory

Re-extracting frames (for example going from 480p to 1080p) while keeping the same filenames means the browser serves the stale ones and nothing appears to change.

Always version the frame URL and bump `N` on every re-extraction:

```js
const framePath = (i) => `/hero-frames/frame_${String(i).padStart(3, '0')}.jpg?v=2`
```

---

## Section 4: The Component

```jsx
import { useEffect, useRef } from 'react'
import { gsap, prefersReducedMotion } from '../lib/motion'

/**
 * props:
 *   frameCount  number of extracted frames
 *   framePath   (i) => url for frame i (1-based). ALWAYS cache-busted with ?v=N
 *   focusX/Y    0..1 focal point preserved when cover-cropping. default 0.5
 *   poster      static poster url (mobile + reduced-motion + fallback)
 */
export default function ScrollScrub({
  frameCount, framePath, focusX = 0.5, focusY = 0.5,
  poster, posterAlt = '', className = 'scrub', children,
}) {
  const sectionRef = useRef(null)
  const canvasRef = useRef(null)

  useEffect(() => {
    // Mobile and reduced-motion never scrub. See Section 6.
    const smallScreen = typeof window !== 'undefined'
      && window.matchMedia('(max-width: 860px)').matches
    if (prefersReducedMotion || smallScreen) return

    const canvas = canvasRef.current
    const ctx = canvas.getContext('2d')
    const images = []
    let loaded = 0, failed = false, current = -1
    const state = { frame: 0 }
    const dpr = Math.min(window.devicePixelRatio || 1, 2)

    // Cover with a focal point: keeps the subject visible when the cover crop
    // eats the edges (16:9 footage in a wide, short viewport).
    const drawCover = (img, cw, ch) => {
      const ir = img.width / img.height
      const cr = cw / ch
      let dw, dh, dx, dy
      if (ir > cr) { dh = ch; dw = ch * ir; dy = 0; dx = (cw - dw) * focusX }
      else { dw = cw; dh = cw / ir; dx = 0; dy = (ch - dh) * focusY }
      ctx.drawImage(img, dx, dy, dw, dh)
    }

    const draw = (idx, force) => {
      const i = Math.max(0, Math.min(frameCount - 1, Math.round(idx)))
      if (i === current && !force) return
      current = i
      const img = images[i]
      const r = canvas.getBoundingClientRect()
      if (img && img.complete && img.naturalWidth) drawCover(img, r.width, r.height)
    }

    const resize = () => {
      const r = canvas.getBoundingClientRect()
      canvas.width = Math.max(1, Math.floor(r.width * dpr))
      canvas.height = Math.max(1, Math.floor(r.height * dpr))
      ctx.setTransform(dpr, 0, 0, dpr, 0, 0)
      draw(current < 0 ? 0 : current, true)
    }

    // Preload every frame before scrubbing.
    for (let i = 0; i < frameCount; i++) {
      const img = new Image()
      img.onload = () => {
        loaded++
        // See Section 5: resize once the canvas is actually visible.
        if (loaded === 1) { canvas.style.display = 'block'; resize() }
      }
      img.onerror = () => { failed = true }
      img.src = framePath(i + 1)
      images[i] = img
    }

    const ctxGsap = gsap.context(() => {
      gsap.to(state, {
        frame: frameCount - 1,
        ease: 'none',
        snap: 'frame',
        scrollTrigger: {
          trigger: sectionRef.current,
          start: 'top top', end: 'bottom bottom', scrub: 0.4,
        },
        onUpdate: () => { if (!failed) draw(state.frame) },
      })
    }, sectionRef)

    window.addEventListener('resize', resize)
    const t = setTimeout(resize, 60) // first measurement after layout settles
    return () => {
      clearTimeout(t)
      window.removeEventListener('resize', resize)
      ctxGsap.revert()
    }
  }, [])

  return (
    <section className={className} ref={sectionRef}>
      <div className={`${className}__sticky`}>
        <div className={`${className}__fallback`}><img src={poster} alt={posterAlt} /></div>
        {/* starts hidden so resize-on-visible fires correctly */}
        <canvas className={`${className}__canvas`} ref={canvasRef} style={{ display: 'none' }} />
        <div className={`${className}__scrim`} />
        <div className={`${className}__copy`}>{children}</div>
      </div>
    </section>
  )
}
```

Use one instance per scrub section, varying `framePath`, `focusX/focusY` and the overlaid copy.

---

## Section 5: The 1x1 Canvas Bug

**Symptom:** the whole section renders as one stretched color or blur instead of the footage.

**Cause:** if the section starts outside the viewport with `display:none`, the initial resize runs while `getBoundingClientRect()` returns 0. The canvas becomes 1x1 and then draws a single pixel stretched to full size.

**Fix (already in the component):** keep the canvas hidden, and on the first loaded frame set `display:block` and re-run `resize()` with real measurements.

If it reappears, the section is almost certainly being measured while hidden.

---

## Section 6: Responsive and Reduced Motion

**Mobile does not scrub.** Landscape footage crops badly in portrait, and downloading dozens of 1080p frames costs 10 to 25 MB. The section collapses to a static 100vh poster.

```css
.scrub { position: relative; height: 320vh; }  /* height is the scrub duration */
.scrub__sticky { position: sticky; top: 0; height: 100vh; overflow: hidden; }
.scrub__fallback { position: absolute; inset: 0; }
.scrub__fallback img { width: 100%; height: 100%; object-fit: cover; }
.scrub__canvas { position: absolute; inset: 0; width: 100%; height: 100%; }
.scrub__copy { position: relative; z-index: 3; }

@media (max-width: 860px) {
  .scrub { height: 100vh; }
  .scrub__sticky { position: relative; }
  .scrub__canvas { display: none !important; }
}
@media (prefers-reduced-motion: reduce) {
  .scrub { height: 100vh; }
  .scrub__sticky { position: relative; }
  .scrub__canvas { display: none !important; }
}
```

The JS guard and the CSS guard must agree. The component returns early on the same conditions the media queries cover.

---

## Section 7: Text Beats Over the Scrub

Copy that enters and exits at scroll milestones uses a separate GSAP timeline on the same trigger. Positions are fractions of the timeline (0 to 1):

```js
const tl = gsap.timeline({
  scrollTrigger: { trigger: sectionRef.current, start: 'top top', end: 'bottom bottom', scrub: 0.4 },
})
// pointerEvents tracks each beat so a beat is never clickable while invisible
tl.to(beat1Ref.current, { opacity: 0, yPercent: -18, duration: 0.16 }, 0.24)
tl.set(beat1Ref.current, { pointerEvents: 'none' }, 0.24)
tl.set(beat2Ref.current, { pointerEvents: 'auto' }, 0.40)
tl.fromTo(beat2Ref.current, { opacity: 0, yPercent: 18 }, { opacity: 1, yPercent: 0, duration: 0.12 }, 0.40)
tl.to(beat2Ref.current, { opacity: 0, yPercent: -18, duration: 0.12 }, 0.62)
tl.set(beat2Ref.current, { pointerEvents: 'none' }, 0.62)
```

**An invisible beat still swallows clicks.** The beats stack in the same area with `position: absolute` and differ only by opacity, so without `pointer-events` the beat that sits last in the DOM wins the hit test even at `opacity: 0`. The visible beat renders fine and its CTA is simply dead. Set the resting state in CSS as well, so the first paint is correct before the timeline runs:

```css
.hero-beat--2, .hero-beat--3 { opacity: 0; pointer-events: none; }
```

**Check what else is `position: fixed` over the hero.** A cookie banner or a floating bar at a high z-index intercepts the CTA until it is dismissed, and on mobile it can take 250 to 270px of height. Push the beats up while the bar is on screen instead of lowering its z-index, and write the selector so it survives a library that hides the node with an inline style rather than removing it from the DOM:

```css
body:has(.cookie-bar:not([style*="display: none"])) .hero-beat { bottom: clamp(190px, 24vh, 260px); }
```

Text over footage needs a directional scrim plus text-shadow to stay readable. See `frontend/design-system/premium-finish.md`.

---

## Section 8: Verify With Motion, Never With a Still

**This is the highest-leverage check in the whole build.** A static screenshot does not reveal a broken scrub: the canvas can be 1x1 drawing garbage, or the frames can be ignoring scroll entirely, and the screenshot still looks plausible.

Drive the scroll and capture the sequence, using the exposed Lenis instance (see `frontend/scroll-driven/smooth-scroll.md`):

```js
// browser console, or via Playwright / Chrome DevTools MCP
const lenis = window.__lenis
const H = document.body.scrollHeight - innerHeight
for (const p of [0, .15, .3, .45, .6, .8, 1]) {
  lenis.scrollTo(H * p, { immediate: true })
  await new Promise(r => setTimeout(r, 350))  // let scrub:0.4 settle
  // screenshot here
}
```

The `setTimeout` matters: `scrub: 0.4` does not jump instantly, so without the wait you photograph the previous frame.

Confirm three things: different frames appear at different scroll positions, the subject stays framed, and the copy stays legible over every captured frame.

---

## Checklist

- [ ] Frames extracted with explicit fps, scale and `-q:v`; total weight measured
- [ ] Poster frame extracted and wired to the fallback
- [ ] `framePath` carries a `?v=N` cache-bust, bumped on every re-extraction
- [ ] Canvas starts hidden and resizes on the first loaded frame
- [ ] `focusX/focusY` set so the cover crop keeps the subject
- [ ] Mobile and reduced-motion collapse to the poster in both JS and CSS
- [ ] Inactive text beats carry `pointer-events: none` in the CSS and in the timeline
- [ ] Verified by driving the scroll and comparing frames, not with a single screenshot