feat(hdr): HDR image support, HDR10 metadata, and regression test harness

[vanceingalls]

Summary

Top of the HDR stack. Lands native HDR <img> rendering plus the polish needed to ship HDR end-to-end: HDR10 container metadata, sRGB→BT.2020 overlay color, encoder/render fixes, and a regression test harness (hdr-smoke) with three fixtures so the stack stays green.

What's in this PR

1. HDR image support — <img> rendering with full HDR fidelity

249675dc feat(hdr): add HDR image support — probe, extract, and composite

• ImageElement type + parseImageElements() in videoFrameExtractor.ts collects <img> elements with data-start/data-duration from compositions.
• Image probing — when --hdr is set, each image's src is probed with ffprobe for HDR color space (bt2020 + PQ/HLG). Runs in parallel alongside video probes.
• Single-frame extraction — HDR images are extracted once via ffmpeg -frames:v 1 -pix_fmt rgb48le at the element's layout dimensions. Output goes to hdr_<id>/frame_0001.png — same directory structure as video frame extraction, just one frame.
• Compositing — reuses the existing blitHdrVideoLayer path. The image's single frame gets blitted every frame the image is visible. Cross-transfer conversion (HLG↔PQ) applies automatically.
• CompiledComposition.images — htmlCompiler.ts now collects images into a typed array alongside videos and audios.

The HDR video pipeline already handled z-ordered layer compositing, GSAP transforms, border-radius masks, sRGB→HDR LUT conversion, and cross-transfer conversion — an HDR image is just a video with one frame, so the entire blit path reuses unchanged.

2. HDR10 container metadata for player compatibility

10bea303 feat(hdr): inject mdcv/clli MP4 container boxes for HDR10 outputs

YouTube, QuickTime, and Safari only treat a file as HDR10 when the MP4 carries mdcv (Mastering Display Color Volume) and clli (Content Light Level Info) boxes. We now inject both during muxing — without them PQ files were detected as SDR by most consumer players. Includes mp4HdrBoxes.test.ts with 398 lines of round-trip coverage.

3. Color-accurate DOM overlays in HDR composites

b78a8df2 fix(engine): convert sRGB DOM overlays to BT.2020 primaries before HDR composite

DOM overlays were being baked into HDR frames with their sRGB primaries unchanged, so reds/greens shifted noticeably under PQ encoding. Overlays now go through a 3×3 sRGB→BT.2020 primary conversion before the OETF, matching what real HDR content looks like next to text/icons.

4. Engine stability

• 8e3dc4bd fix(engine): install __name shim before page scripts run — esbuild-emitted bundles call a global __name() helper for class display names. Pages that loaded before our injection threw __name is not defined. Shim is now installed in the early stub.
• 5e90cd5e fix(engine): handle stdin EINVAL on streaming encoder pipe close — when ffmpeg exits between frames the next write was crashing the producer; now we detect EINVAL/EPIPE, log once, and let the encoder shutdown path handle it.
• 039dda8c fix(producer): add HDR image decode cache + promote blit failure to warn — HDR images were being decoded once per frame (~4 ms × 30 fps × 5 s = 600 ms wasted per clip). Cached now. Blit failures degrade to a warning so a single bad layer doesn't kill the whole render.

5. Regression test harness — hdr-smoke

161d4dbb test(producer): add hdr-smoke regression script + blitHdrVideoLayer logger test

packages/producer/scripts/hdr-smoke.ts — single entry point that renders every fixture in tests/hdr-regression/, verifies the output with ffprobe (yuv420p10le, smpte2084, bt2020, requireHdrSideData), and reports per-fixture pass/fail. Used both locally and in CI.

Three fixtures cover the HDR stack:

Fixture	What it exercises
hdr-feature-stack	6 scenes covering layered compositing, transforms, masks, transitions, and the new image path. Real HDR/SDR clips from YouTube. White Scene 6 with transform-only yoyo.
opacity-mixed-fade	Side-by-side HDR + SDR clips running an entry fade-in (0→1) followed by an opacity yoyo (1↔0.15). Locks in the SDR-opacity fix landed on feat/hdr-layered-compositing.
parseImageElements coverage	749cfd96 — extra unit-test cases for the new image parser.

6. Cleanup

• 5279501e fix: restore .gitignore entries removed during rebase
• 5a8ba3d7 fix(hdr): address review feedback across stack
• 5893c6f3 chore: ignore studio scene thumbnails cache

Files changed (vs. feat/hdr-layered-compositing)

 .gitignore                                                |  43 +-
 packages/engine/src/index.ts                              |   6 +-
 packages/engine/src/services/frameCapture.ts              |  16 +
 packages/engine/src/services/streamingEncoder.ts          |  34 +-
 packages/engine/src/services/videoFrameExtractor.ts       |  37 +-
 packages/engine/src/services/videoFrameExtractor.test.ts  |  57 +-
 packages/engine/src/services/videoFrameInjector.ts        |  10 +
 packages/engine/src/utils/alphaBlit.ts                    | 201 ++-
 packages/engine/src/utils/alphaBlit.test.ts               | 149 ++-
 packages/engine/src/utils/mp4HdrBoxes.ts                  | 520 +++++++
 packages/engine/src/utils/mp4HdrBoxes.test.ts             | 398 ++++++
 packages/producer/scripts/hdr-smoke.ts                    | 329 +++++
 packages/producer/src/services/htmlCompiler.ts            |   6 +
 packages/producer/src/services/renderOrchestrator.ts      | 169 +++
 packages/producer/src/services/renderOrchestrator.test.ts | 224 ++-
 packages/producer/tests/hdr-regression/...                | (fixture files)
 31 files changed, 2999 insertions(+), 99 deletions(-)

How to test

# Render every regression fixture and verify HDR side data
bunx tsx packages/producer/scripts/hdr-smoke.ts

# Or render one fixture by hand
bunx hyperframes render \
  packages/producer/tests/hdr-regression/hdr-feature-stack/src \
  --hdr --duration 30 --fps 30 -o /tmp/feature-stack.mp4

For ad-hoc HDR image testing:

1. Create a 16-bit HDR PNG: ffmpeg -ss 2 -i hdr-video.mp4 -frames:v 1 -pix_fmt rgb48le -c:v png hdr-photo.png
2. Use it in a composition: <img id="bg" class="clip" data-start="0" data-duration="5" src="hdr-photo.png" style="..." />
3. Render with --hdr: verify H.265 10-bit PQ/BT.2020 output, image visible at HDR quality
4. Render without --hdr: verify SDR output, image renders through Chrome, no probe overhead

Stack position

7 of 7 — top of the HDR stack. Stacked on #268 (layered compositing).

Full stack: #258 → #265 → #288 → #289 → #290 → #268 → #314

🤖 Generated with Claude Code

[vanceingalls]

• feat(hdr): HDR image support, HDR10 metadata, and regression test harness #314 👈 (View in Graphite)
• feat(hdr): layered HDR compositing, shader transitions, and HDR image support #268
• feat(hdr): GSAP transforms and border-radius masks on HDR video #290
• feat(hdr): z-ordered multi-layer compositing with PQ support #289
• feat(engine): add HDR two-pass compositing — DOM layer + native HLG video #288
• feat(engine): add HDR video output pipeline #265
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[miguel-heygen]

Automated review (Claude Code)

Reviewed against CLAUDE.md. Stack-level meta on #268.

Critical

• Silent failure on HDR image extraction — packages/producer/src/services/renderOrchestrator.ts:~1236-1244. If runFfmpeg fails (non-HDR codec, multi-frame GIF/AVIF, unsupported pix_fmt), the code only log.debugs and still calls hdrFrameDirs.set(img.id, frameDir) pointing at an empty directory. Downstream blit reads missing frame_0001.png and either crashes or silently skips depending on consumer. Promote to log.warn, and either skip hdrFrameDirs.set() on failure or throw. At minimum remove the id from nativeHdrVideoIds so the blit path doesn't try to use it.
• Silent failure when HDR image path doesn't exist — renderOrchestrator.ts:~878-881. if (!existsSync(imgPath)) return; with no log. A typo in src silently drops the image from HDR detection and the render proceeds as SDR with no user-facing indication. Add log.warn.

Important

• Promise.all short-circuits on one bad image — same probe block. extractVideoMetadata(imgPath) is not wrapped in try/catch — one corrupt image rejects the whole Promise.all, aborting the render. Existing video path probes tolerate this; match the behavior.
• images on CompiledComposition / CompositionMetadata is a breaking change — packages/producer/src/services/htmlCompiler.ts:45, renderOrchestrator.ts:221. The fields are non-optional. Any downstream TS consumer constructing these (tests, external plugins) fails to compile. Make images?: ImageElement[] optional or default to [] and document.
• Multi-frame source (GIF/AVIF) silently picks frame 1 — -frames:v 1 on an animated source yields frame 0. DOM plays the animation, composite renders a static poster → desync invisible in logs. Detect nb_frames > 1 at probe time and warn/refuse, or document "HDR img supports still images only".

Suggestions

• hdrVideoStartTimes naming now misleading — images share the map. Rename to hdrLayerStartTimes.
• Test coverage gaps on parseImageElements — currently covers happy path + no-duration + auto-id. Missing: zero-<img> case; data-duration="0" / negative / NaN (the <= 0 guard is correct but untested); missing data-start (defaults to 0 but untested); dedupe-by-id when two imgs share an id.
• Dir collision hdr_<id> — both extractHdrVideoFramesAtTimes and the new image block write to framesDir/hdr_<id>/. parseImageElements generates hf-img-N but a video author could write id="hf-img-0". Namespace to hdr_img_<id>.
• Decode caching — blitting the same decoded 16-bit PNG every frame the image is visible is wasteful. Decode-once + memcpy. Worth a TODO; will dominate CPU for long stills.
• parseImageElements mutates DOM via el.setAttribute("id", id) but the rewritten tree is discarded — the mutation has no effect. Either remove the side effect, or have the function return the rewritten HTML (mirrors what parseVideoElements needs for id stability).

Strengths

• Reuses extractVideoMetadata + isHdrColorSpace + blitHdrVideoLayer without introducing a parallel probe/blit code path — correct.
• skipSdrConversion flag with inline "why Chrome's sRGB misinterpretation matters" comment is exactly the kind of "why" comment that saves future debugging.
• Parallel probing via Promise.all is right for I/O-bound ffprobe calls.
• Tests cover happy path + two important edge cases (no data-duration, auto-id).

🤖 Automated review. See stack meta-comment on #268.

[vanceingalls]

Thanks. Fixes landed.

Addressed:

• Silent failure on HDR image extraction — promoted from log.debug to log.warn with stderr tail. On failure the id is removed from nativeHdrVideoIds and hdrFrameDirs.set is skipped, so the blit path can't read an empty dir.
• Silent failure when image path doesn't exist — log.warn with id + source path.
• Promise.all short-circuits on one bad image — per-image probe wrapped in try/catch with log.warn. One corrupt image no longer aborts the whole render; mirrors video path behavior.
• images non-optional breaking change — CompiledComposition.images?: is now optional. Consumer code uses compiled.images ?? [] at the single assembly site.
• Multi-frame source silently picks frame 1 — probe now warns if the image reports fps > 0 or durationSeconds > 0.05s (animated GIF/AVIF), naming HDR img as stills-only for now.
• hdrVideoStartTimes misleading name — renamed to hdrLayerStartTimes everywhere. The map holds both video and image layer start times since HDR image support landed.
• parseImageElements test coverage — added: empty input, zero-data-duration, negative, NaN, missing data-start. Caught and fixed a real bug: NaN <= 0 is false, so data-duration="abc" was slipping through — now explicitly gated on Number.isFinite.
• Dir collision hdr_<id> — namespaced to hdr_img_<id>. A video author naming a video hf-img-0 no longer clobbers image extraction output.
• parseImageElements dead setAttribute mutation — removed. The parsed tree was discarded, so the mutation had no effect on the rendered HTML.
• Decode caching — HDR image layers now cache decoded rgb48le buffers in a render-scoped Map keyed by framePath::sourceTransfer::targetTransfer. Videos are excluded (per-frame paths differ). Entries store post-convertTransfer buffers and are cloned on hit to prevent cache corruption.

Nothing from the original findings on this PR is unaddressed.

[vanceingalls]

CI fix: typecheck regression in renderOrchestrator.ts (ab78ab4e)

After the layered-compositing typecheck fix landed downstack, this branch still failed Build / Typecheck with:

src/services/renderOrchestrator.ts(1470,35): error TS2304: Cannot find name 'hdrVideoStartTimes'.

The fix is the opposite direction from the one on feat/hdr-layered-compositing, and that asymmetry is intentional. On this branch, commit eac396a3 renamed the start-time map from hdrVideoStartTimes → hdrLayerStartTimes stack-wide as part of the image-support generalization (videos and images both become "HDR layers"). The reference at line 1470 was missed by that rename, so it kept the old name and broke compilation.

Single-line fix: hdrVideoStartTimes.get(layer.element.id) → hdrLayerStartTimes.get(layer.element.id). No semantic change.

Why two seemingly opposite fixes for "the same" typo:

• Downstack (feat/hdr-layered-compositing): variable is canonically hdrVideoStartTimes; one log line misused the not-yet-introduced hdrLayerStartTimes name.
• Here (feat/hdr-image-support): variable was renamed to hdrLayerStartTimes stack-wide; one reference was missed and still used the old hdrVideoStartTimes name.

Verification:

• bunx tsc --noEmit clean in packages/producer.
• oxlint + oxfmt --check clean on the touched file.
• CI run 24639989755: Build / Typecheck / Test / Lint / Format all green. Regression shards still running.

[vanceingalls]

Cleanup (rebase, no functional change)

Per request, dropped two stray commits that didn't belong on this PR:

• 99d4129c docs(hdr): document regression test gaps and follow-up plan — added packages/producer/tests/hdr-regression/FOLLOWUP.md. The regression-coverage discussion belongs in an issue, not a tracked doc on this branch.
• 5628afcd chore: ignore studio scene thumbnails cache — added .thumbnails/ to .gitignore. Redundant: my-video/ (where the cache lives during local dev) is already globally ignored a few lines above, so the entry was a no-op.

Performed via git rebase --onto 9d82de8b 99d4129c feat/hdr-image-support (backup tag backup/hdr-image-support-pre-cleanup for safety). The CI typecheck fix at line 1470 (hdrLayerStartTimes rename) was preserved and replayed as 63a1723a.

Local typecheck + lint + format checks pass on the rebased head. Pushed and waiting on CI to confirm regression shards are still green.