feat(engine): add HDR two-pass compositing — DOM layer + native HLG video

[vanceingalls]

Summary

Compositions with HDR video AND DOM overlays (text, graphics, SDR video) couldn't render both correctly — either HDR data was lost (Chrome captures sRGB only) or DOM overlays were missing (FFmpeg pass-through skips Chrome). This PR adds in-memory alpha compositing that combines both.

What it does

Per-frame two-pass capture:

1. DOM pass — Chrome screenshots the page with a transparent background (CDP alpha). HDR videos are hidden, leaving transparent holes where they go.
2. HDR pass — Pre-extracted native HLG/PQ frames (16-bit PNG from FFmpeg) are read from disk.
3. Composite — DOM pixels (sRGB RGBA8) are alpha-composited over HDR pixels (rgb48le) in Node.js memory, with sRGB→HLG/PQ conversion via a 256-entry lookup table.

Key components:

• decodePng() / decodePngToRgb48le() — Pure Node.js PNG decoders (no native dependencies). Support all 5 PNG filter types.
• blitRgba8OverRgb48le() — Alpha composite with per-pixel sRGB→HDR LUT conversion. Fast paths for alpha=0 (skip) and alpha=255 (overwrite).
• initTransparentBackground() + captureAlphaPng() — Split CDP transparent background setup (once) from per-frame screenshot capture (eliminates 2 CDP round-trips per frame).
• Single-pass FFmpeg extraction — All HDR frames extracted in one sequential FFmpeg run (avoids duplicate frames from per-frame -ss fast seek).

Key design decisions

Decision	Why
In-memory compositing (not FFmpeg overlay)	Eliminates ~2400 process spawns + temp files per render. Pure pixel math is 10x faster.
16-bit PNG intermediate	Raw -f rawvideo loses color metadata, causing moiré artifacts. PNG is self-describing.
sRGB→HLG LUT (256 entries)	DOM content is sRGB. Without conversion, it appears orange-shifted in HLG stream.
Native HDR detection before extraction	extractAllVideoFrames converts SDR→HDR. Pre-extraction probe identifies original HDR sources so only truly-HDR videos get native extraction.

Files changed

File	What changed
packages/engine/src/utils/alphaBlit.ts	NEW — PNG decode, sRGB→HDR LUT, alpha compositing (14 tests)
packages/engine/src/services/screenshotService.ts	Transparent background CDP, captureAlphaPng()
packages/engine/src/services/videoFrameInjector.ts	hideVideoElements() / showVideoElements()
packages/engine/src/services/streamingEncoder.ts	Input color space tags for rgb48le
packages/producer/src/services/renderOrchestrator.ts	Two-pass HDR capture loop, native HDR detection

How to test

Render a composition with an HDR video background and text overlays. Both should be visible — HDR video at full quality, text crisp with correct colors (not orange-shifted).

Stack position

3 of 6 — Stacked on #265 (HDR output pipeline). This is the foundation for all layered compositing that follows.

🤖 Generated with Claude Code

[vanceingalls]

• feat(hdr): HDR image support, HDR10 metadata, and regression test harness #314
• feat(hdr): layered HDR compositing, shader transitions, and HDR image support #268
• feat(hdr): GSAP transforms and border-radius masks on HDR video #290
• feat(hdr): z-ordered multi-layer compositing with PQ support #289
• feat(engine): add HDR two-pass compositing — DOM layer + native HLG video #288 👈 (View in Graphite)
• feat(engine): add HDR video output pipeline #265
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[miguel-heygen]

Automated review (Claude Code)

Reviewed against CLAUDE.md. Stack-level meta on #268.

Critical

• PNG decoder silently ignores interlace bit — packages/engine/src/utils/alphaBlit.ts:~74-82. IHDR byte 12 is never read or validated. An Adam7-interlaced PNG is silently mis-decoded (filter bytes at wrong positions → garbage) rather than thrown. Even if Chrome never emits interlaced, decodePngToRgb48le accepts ffmpeg PNGs too. Assert interlace === 0 or throw.
• DOM-layer decode error swallowed — renderOrchestrator.ts:~1883. try { decodePng(...); blitRgba8OverRgb48le(...) } catch {}. Violates CLAUDE.md ("no try/catch fallbacks"). Every frame ships with no DOM layer → silent black-overlay video.
• HDR frame-extraction failure swallowed — renderOrchestrator.ts:~1793. try { execSync(ffmpeg ...) } catch {} with comment "loop handles gracefully" — the loop substitutes Buffer.alloc(width*height*6) (pure black HDR) per frame with no warning.
• HDR frame decode swallowed — renderOrchestrator.ts:~1865. Third instance of the same pattern.
• Frame-index off-by-source-fps — renderOrchestrator.ts:1850. videoFrameIndex = Math.round((time - video.start) * fps) + 1. ffmpeg -r ${fps} drops/duplicates to hit rate; source fps ≠ job fps can yield rounded indices beyond extracted count → file not found → silent black. No bounds check against dir size. (Partially mitigated in #290's freeze-on-last-frame fallback.)

Important

• captureScreenshotWithAlpha restore not in finally (screenshotService.ts:87-107) — if Page.captureScreenshot throws, setDefaultBackgroundColorOverride({}) is never called; subsequent opaque captures keep a transparent background. Wrap restore in finally, or drop the helper (orchestrator uses initTransparentBackground + captureAlphaPng instead).
• Multiple active HDR videos collapse to one — renderOrchestrator.ts:1843 picks activeBounds[0]. Overlapping HDR sources → second dropped silently. Log the limitation.
• sRGB→HLG LUT convention undocumented — alphaBlit.ts:1276-1282 assumes "linear signal in [0,1] relative to SDR white" (no OOTF). Correct for overlay-over-HLG-video, but it's a non-obvious convention; the endpoint-only test does not capture it.
• Fast-path desync risk — α=0 skip / α=255 overwrite (alphaBlit.ts:1310-1338) have no test comparing them against the slow path at boundaries. Add a parametric test at α=254.
• Test coverage gaps — makePng test helper exercises only the None PNG filter (byte 0). Sub / Up / Average / Paeth: zero behavioral coverage. Chrome's libpng picks filters heuristically — this is exactly where regressions hide. Also: no multi-chunk IDAT test (Chrome splits at ~8KB); decodePngToRgb48le has no tests even though it's the HDR-frame hot path.
• parseTransformMatrix returns NaN silently — non-numeric captures via match.slice(1,7).map(Number) → NaN. det = NaN, checks pass, writeUInt16LE(NaN) throws deep in the blit. Validate with Number.isFinite → return null.

Suggestions

• Comment "GSAP transforms are NOT applied (future work)" was swapped for "position/opacity applied via queried bounds", but transform/opacity aren't applied here — blitRgb48leAffine isn't called from the orchestrator in this diff. Tighten to "position queried; transform/opacity lands in later PR".
• "optimizeForSpeed: false // must be false to preserve alpha" — cite CDP docs or drop the justification.
• decodePngRaw silently accepts files missing IHDR → empty data. Throw on missing IHDR.
• blitRgb48leRegion computes canvasHeight = canvas.length / (canvasWidth*6) — assumes tight packing. Fragile if a sliced Buffer is ever passed. Explicit canvasHeight param would future-proof.
• 3× page.evaluate round-trips per frame (bounds, hide, show) on top of captureAlphaPng. At 60fps×30s = 1800 frames — consolidate.

Strengths

• decodePngRaw shares chunk-parse logic between 8-bit and 16-bit paths — real abstraction (matches CLAUDE.md).
• 681-line test file is genuinely behavioral — pixel-value assertions, not type probes.
• initTransparentBackground / captureAlphaPng split correctly eliminates 2× CDP round-trips per frame.
• Replacing per-frame ffmpeg spawn with in-memory decode is a large perf win that justifies writing a PNG decoder.
• parseTransformMatrix correctly rejects matrix3d.

🤖 Automated review. See stack meta-comment on #268.

[vanceingalls]

Thanks for the review. Fixes landed.

Addressed:

• PNG decoder interlace bit — decodePngRaw now asserts interlace === 0 (rejects Adam7-interlaced PNGs explicitly). Added test.
• Missing IHDR chunk — decodePngRaw throws with a clear message instead of inflating empty data. Added test.
• DOM-layer decode swallow / HDR extract swallow / HDR decode swallow — all three are now log.warn with frame time or element id, and the HDR-image extraction failure path now drops the image id from nativeHdrVideoIds instead of silently pointing at an empty frame dir.
• captureScreenshotWithAlpha restore not in finally — wrapped Page.captureScreenshot in try/finally so the opaque background is restored even on throw.
• Multiple active HDR videos collapse to one (activeBounds[0]) — superseded by the feat(hdr): layered HDR compositing, shader transitions, and HDR image support #268 refactor to blitHdrVideoLayer which composites each HDR layer independently via z-ordered layers.
• parseTransformMatrix NaN silently — validates with Number.isFinite on all 6 values; returns null on any non-finite input so the identity fallback path takes over.
• PNG filter test gap — added parametric round-trip tests for None/Sub/Up/Average/Paeth against a 3×2 unique-pixel fixture, plus explicit-rejection tests for Adam7 and missing IHDR.
• Multi-chunk IDAT — added test that splits a compressed stream across two IDATs and verifies the decoder concatenates correctly.
• decodePngToRgb48le tests — added: byte-order swap (BE wire → LE buffer), 16-bit precision preservation, and row-major multi-pixel layout.
• sRGB→HLG LUT convention — LUT doc now spells out the scene-light-in-[0,1]-relative-to-SDR-white convention and why there's no OOTF (overlay composites on top of HLG, shares signal space).
• Fast-path α=254 boundary — added test verifying α=254 and α=255 produce nearly identical output (guards the α=255 fast-path branch).
• Frame-index off-by-source-fps — added an explicit bounds check against the actual frame-dir size. A module-level frameDirMaxIndexCache reads the max index once via readdirSync, then orchestrator clamps frameIndex > maxIndex to the last available frame (preserving the freeze-on-last-frame behavior without spuriously requesting unfilled indices).
• blitRgb48leRegion explicit canvasHeight param — function signature now takes canvasHeight explicitly instead of inferring it from canvas.length / (canvasWidth * 6). All call sites updated.

Deferred:

• 3× page.evaluate consolidation — architectural; worth a dedicated perf pass.

[vanceingalls]

Merge activity

• Apr 19, 10:47 PM UTC: Graphite couldn't merge this PR because it had merge conflicts.