feat(engine): preserve alpha channel on <video> frame extraction

[leopeak]

Alpha-channel preservation for <video> extraction

Problem

When a HyperFrames composition contains a <video> with an alpha channel —
WebM VP8/VP9 yuva420p, HEVC yuva444p, PNG-in-AVI rgba, QuickTime ProRes
4444, etc. — the alpha is silently dropped before the frame ever reaches
Chrome. The offending step is the ffmpeg pre-extract:

ffmpeg -i alpha.webm -vf fps=30 -q:v ... frame_%05d.jpg

1. Default output is JPEG, which has no alpha.
2. Even when output is forced to PNG (via --format webm / --format mov),
   the command omits -pix_fmt rgba, so ffmpeg may downconvert yuva → yuv and
   write an opaque RGB PNG.

Downstream the compositor works correctly — <img> data URIs preserve alpha,
Chrome's compositor handles blending — so a one-line extract fix unlocks
transparency for every output container, including opaque MP4.

Approach

Added opt-in + auto-detect alpha preservation at the extraction stage. No
new CLI flag, no changes to the renderer / encoder.

1. New data-alpha attribute on <video> elements:
   • data-alpha="true" — force RGBA PNG extraction.
   • data-alpha="false" — force opaque (overrides auto-detect).
   • Absent / "auto" — auto-detect from ffprobe's pix_fmt.
2. Auto-detection via new pixelFormatHasAlpha() util. Flags the
   yuva-family, rgba/argb/bgra/abgr, rgba64, and ya8/ya16 gray-alpha formats.
3. FFmpeg invocation is switched to format=rgba + -pix_fmt rgba for
   alpha-bearing videos, with PNG forced. Other videos stay on JPEG at full
   speed — zero regression for existing compositions.
4. Readiness wait bypass: frameCapture.initializeSession no longer
   blocks on video.readyState >= 1 for data-alpha="true" elements, since
   Chrome often can't decode the source codec (e.g. PNG-in-AVI) and the
   renderer swaps the <video> for an <img> before capture anyway.

Files changed

File	Summary
packages/engine/src/utils/ffprobe.ts	Added pixelFormat + hasAlpha to VideoMetadata; new pixelFormatHasAlpha() util.
packages/engine/src/services/videoFrameExtractor.ts	Parse data-alpha into VideoElement.hasAlpha; force PNG+rgba extraction when alpha is in play; thread hasAlpha through extractAllVideoFrames.
packages/engine/src/services/videoFrameExtractor.test.ts	New tests for data-alpha parsing.
packages/engine/src/services/frameCapture.ts	Skip the video-readyState gate for data-alpha="true" elements in both screenshot and BeginFrame paths.
packages/engine/src/index.ts	Re-export pixelFormatHasAlpha.
packages/producer/src/services/htmlCompiler.ts	Thread hasAlpha through browser-discovered media metadata.
packages/producer/src/services/renderOrchestrator.ts	Carry hasAlpha into new VideoElement entries discovered via DOM probe.

Usage

<!-- Alpha-preserving video with explicit opt-in -->
<video
  id="lion"
  class="clip"
  data-start="0"
  data-duration="5.92"
  data-track-index="0"
  data-alpha="true"
  muted
  playsinline
  src="assets/lion_rgba.avi"
></video>

For well-tagged sources (e.g. WebM with yuva420p correctly advertised) the
data-alpha="true" attribute is optional — ffprobe auto-detection will set
the flag at extraction time.

Test results

• Unit: bun run test for @hyperframes/engine — all 4 parseVideoElements
  tests pass, including 2 new ones for data-alpha="true" / "false".
• Smoke test: 800×600 composition with a single PNG-in-AVI lion
  (data-alpha="true") over a red div → output.mp4 shows the red background
  through every transparent pixel of the lion. No black box, no fringe.
• Integration: 97.32 s / 1080p / 30 fps / 2920-frame VSL composition
  (vsl-hf-reproduction/index.html) with 13 alpha-tagged lion overlays
  composited on top of ~13 opaque background clips — rendered end-to-end in
  13m4s to a 79.6 MB MP4. Alpha preserved across all lion shots; no regression
  on the opaque background videos.

Non-goals

• No changes to the --format webm / --format mov transparent-output path.
  Those already worked; this fix is about alpha on the input side.
• No new CLI flag. The attribute-based API is consistent with existing
  data-has-audio, data-media-start, etc.
• No engine-wide pix_fmt override — RGBA is applied only to the specific
  videos that opt in or auto-detect as alpha-bearing.

[jrusso1020]

Thanks for the contribution! Nice, well-scoped feature and a genuinely excellent PR description — the problem analysis and the "fix only the extract stage, downstream already handles alpha" framing made this very easy to review.

Looks good

• VideoToolbox hwaccel skipped for alpha (videoFrameExtractor.ts:161) — non-obvious catch, VT does strip alpha on decode.
• Belt-and-braces RGBA: vf format=rgba + -pix_fmt rgba on the PNG encoder. Either alone can misbehave on some ffmpeg builds; doing both is the right call.
• Auto-detect via ffprobe pix_fmt as default, explicit data-alpha="true" opt-in for misreported sources — right layering, and the pixelFormatHasAlpha() pixel-format coverage looks correct (yuva*, rgba/argb/bgra/abgr, rgba64*, ya8/ya16*).
• Opaque videos keep the JPEG fast path — zero regression for existing compositions.
• Lint + engine tests pass on the branch (287/287).

Needs fixing

1. Conflicts with skipReadinessVideoIds — please consolidate mechanisms

Since this branch was cut, main landed #346 which adds skipReadinessVideoIds on CaptureOptions (packages/engine/src/types.ts:108). The producer populates it with native-HDR video IDs and frameCapture.ts:260 / :339 filter those out of the readyState>=1 gate. This PR solves the same problem — "skip the video readiness gate because frames come from ffmpeg out-of-band" — with a different mechanism (DOM data-alpha="true" filter), and they conflict on both gate sites.

Suggested path: drop the frameCapture.ts changes entirely and instead populate skipReadinessVideoIds from the producer when a video has hasAlpha: true, next to the existing nativeHdrVideoIds plumbing (renderOrchestrator.ts:1001,1184,1245). One mechanism, one code path; both cases ("native HDR" and "alpha") become instances of the same pattern.

2. ffprobe.ts merge must preserve main's stillImageMeta fallback

Main added a PNG cICP chunk fallback: colorSpace: ffprobeColorSpace ?? stillImageMeta?.colorSpace ?? null. The PR region reverts to the old hasColorInfo check. The rebase needs to keep main's stillImage fallback and add pixelFormat + hasAlpha alongside it — naively picking the PR side breaks HDR PNG detection.

3. Please drop IMPLEMENTATION_PLAN.md and PR_DRAFT.md from the repo root

These read like author scratch files. Repo root hasn't carried planning docs before; they'll rot and confuse future contributors. Either delete, or move under notes/ (already gitignored). The .gitignore entry for /test-alpha/ is fine to keep.

Minor / style (non-blocking)

• extractVideoFramesRange now takes 8 positional args with the new overrides param. Fold hasAlpha into options instead.
• videoFrameExtractor.test.ts:25 weakens toEqual → toMatchObject, which silently allows extra fields. Prefer keeping toEqual with explicit hasAlpha: undefined.
• pixelFormat: "" as absent-marker is inconsistent with colorSpace: VideoColorSpace | null. string | null would be more in line with the rest of the file.
• No automated end-to-end test asserting alpha survives through ffmpeg (the new tests cover parser only). A small fixture-based test reading one non-255 alpha byte from the output would lock in the behavior — low priority, but nice to have for a feature this "silent" in its failure mode.

Thanks again — once (1)–(3) are sorted, this is a solid addition.

[leopeak]

Thanks for the thoughtful review — the feedback on consolidating around skipReadinessVideoIds (#346) makes a lot of sense. The existing mechanism absolutely subsumes what this PR was doing in frameCapture.ts, so going through the producer-side plumbing is the cleaner path.

Planned updates (likely in the next few days):

1. Drop the frameCapture.ts changes; populate skipReadinessVideoIds from the producer when hasAlpha: true, alongside nativeHdrVideoIds in renderOrchestrator.ts.
2. Rebase ffprobe.ts to preserve the stillImageMeta?.colorSpace fallback added in main, and layer pixelFormat + hasAlpha on top.
3. Remove IMPLEMENTATION_PLAN.md and PR_DRAFT.md from the repo root.
4. Minor: fold hasAlpha into options for extractVideoFramesRange, tighten videoFrameExtractor.test.ts back to toEqual, switch pixelFormat: "" → string | null, and add a small fixture-based e2e test that asserts alpha survives the ffmpeg round-trip.

Will ping once the updates are pushed. Thanks again for the detailed look!