fix: prevent nested composition videos from autoplaying on seek

[miguel-heygen]

Problem

Studio seek could still wake nested composition media even when the transport itself stayed paused.

In the real repro from apple-presentation, scrubbing to 0:29 without pressing play lands on the slide-translation composition. That composition contains Multilingual_Journey.mp4 inside the composition host. On the broken path:

• the main Studio transport remained paused
• the nested video advanced and stayed playing anyway
• the user saw autoplay-like behavior even though the only action was a seek

That was especially confusing because the seek was otherwise correct: the timeline moved to the right point, but the nested media stopped obeying the paused transport state.

What this fixes

Nested media now participates in runtime media sync

• the runtime media cache no longer assumes only video[data-start] / audio[data-start] are relevant
• nested media inside a composition host can now be included in the same timed-media sync pass even when the inner media element does not carry its own authored data-start

Nested media timing is resolved in the host composition window

• nested media start time is resolved against the enclosing composition host instead of falling back to scene-local 0
• nested media duration is clamped to the enclosing composition window so it stays aligned with the authored host clip timing

Paused seeks land on the right frame and stay paused

• after seeking into a nested composition, the inner media is now seeked to the correct frame relative to the host timeline
• because it is now part of the managed media set, the runtime also keeps it paused when the transport is paused instead of letting it continue playing on its own

Regression coverage

• adds a runtime regression test that covers a nested composition video with no local data-start
• the test verifies that player.seek(29) leaves the nested video paused while landing it at the expected currentTime

Root cause

The bug came from a mismatch between deterministic timeline seeking and media ownership.

1. The runtime only managed media with direct timing attrs

refreshRuntimeMediaCache() only collected video[data-start] and audio[data-start]. That works for root-level timed media, but not for media embedded inside a composition host where timing is inherited from the host composition rather than duplicated onto the inner media node.

2. Nested composition seek could still advance inner media

The runtime intentionally rearms sibling timelines during deterministic seek so nested timelines land on the right local offsets. That part is necessary and correct.

But because the nested video was not part of the managed media cache, it could advance during that seek path without being brought back under the paused transport state afterward.

3. The runtime had no way to reconcile the two

So the system had an inconsistent split:

• timeline seek knew about the nested composition timeline
• media sync did not know about the nested media inside it

The fix closes that split by resolving nested media start/duration from the enclosing composition context and running it through the same sync logic as other managed media.

Verification

Local checks

• bun run --filter @hyperframes/core typecheck
• bun run test -- src/runtime/init.test.ts src/runtime/media.test.ts src/runtime/player.test.ts in packages/core
• bunx oxlint packages/core/src/runtime/media.ts packages/core/src/runtime/init.ts packages/core/src/runtime/init.test.ts
• bunx oxfmt --check packages/core/src/runtime/media.ts packages/core/src/runtime/init.ts packages/core/src/runtime/init.test.ts

Browser verification

Verified against a repo-backed local Studio preview of apple-presentation:

• opened http://127.0.0.1:3014/#project/apple-presentation
• seeked to 0:29 without pressing play
• confirmed the visible composition switched to slide-translation
• confirmed Multilingual_Journey.mp4 landed at a non-zero currentTime (3.067 in the verified run)
• confirmed the nested video stayed paused and its currentTime remained stable across a follow-up check instead of autoplaying

Notes

• the local browser proof artifacts under qa-artifacts/autoplay-seek/ are verification-only and are not part of this PR
• this PR is intentionally scoped to nested media ownership during paused seek; it does not broaden into unrelated runtime media refactors beyond bringing inherited nested media under the existing sync contract

[jrusso1020]

Verdict: approve — bug reproduced on main, fix cleanly closes the ownership split

Local verification

Bug reproduced on main (6b21ead7). Cherry-picked the new init.test.ts case onto main and ran via vitest:

FAIL  src/runtime/init.test.ts > initSandboxRuntimeModular > pauses nested media that is outside the timed-media cache after a seek
AssertionError: expected false to be true // Object.is equality
  expect(video.paused).toBe(true);

That's the real autoplay-on-seek: player.seek(29) leaves the nested <video> at paused: false because the pre-PR refreshRuntimeMediaCache only tracked video[data-start] / audio[data-start], so an inner media element without its own data-start was invisible to the sync pass.

Fix verified on 97f888be via vitest:

• bun run test -- src/runtime/init.test.ts src/runtime/media.test.ts src/runtime/player.test.ts → 74/74 pass (the new regression case + 2 pre-existing init.test.ts + 42 media.test.ts + 29 player.test.ts)
• bun run --filter @hyperframes/core typecheck → clean

Staff review — architecture + correctness

The PR identifies a single conceptual split and closes it at the right abstraction layer. Prior contract: "media ownership requires data-start." That's correct for root-level timed media, wrong for media that inherits its timing from an enclosing composition host. The fix extends refreshRuntimeMediaCache with explicit ownership predicates (shouldIncludeElement, resolveDurationSeconds) so the media helper stays generic and the init.ts caller owns the policy. Clean.

The inheritance math is right for the current pipeline:

• shouldIncludeElement: hasAttribute("data-start") || closest("[data-composition-id]"). Every <video>/<audio> inside any composition host now participates.
• resolveStartSeconds: resolveStartForElement(element, inheritedStart ?? 0). The fallback only kicks in when the element has no data-start of its own — in which case we pick up the host's start.
• resolveDurationSeconds: computes both sourceDuration = element.duration - mediaStart and hostRemaining = inheritedStart + inheritedDuration - start, then returns min(sourceDuration, hostRemaining). Source-longer-than-host clamps to the host window; source-shorter-than-host keeps the natural duration. Correct in both directions.

Why the "global-data-start" assumption holds. For a nested <video> WITH its own data-start: the Hyperframes compiler already offsets scene-local data-start values to global during sub-composition inlining (that's the whole point of parseSubCompositions + the offset logic, recently reinforced by the hf#476 template-unwrap fix for the same data flow). So by the time the runtime sees the HTML, resolveStartForElement returns a global start even when the author wrote a scene-local one. The fallback-to-inherited in this PR only ever fires for media that legitimately had no data-start at all.

Transport-state sync. The paused transition flows through syncMediaForCurrentState → syncRuntimeMedia (the existing path), and the new elements get included in mediaClips with correct start/end bounds. The regression test proves the paused-after-seek path works; the playing path is covered by the same sync logic (just with a non-paused transport) so it should work by construction.

Non-blocking observations

1. Test gap — positive play-state path. The new regression test covers paused-after-seek. A complementary test for "transport playing, seek into nested composition, expect nested video playing + advancing" would guard against a future change that silently re-breaks the play path. The current single test is necessary but not sufficient coverage. Not blocking since the shared syncRuntimeMedia handles both transport states.

2. Test gap — source-longer-than-host clamp. resolveDurationSeconds returns min(sourceDuration, hostRemaining). In the regression fixture, element.duration = 20 and hostRemaining = 16 (host at [20, 36], media starts at 20 → remaining 16), so the clamp fires (min(20, 16) = 16), but the test asserts currentTime === 9 which doesn't exercise the end-of-window behavior. A positive test that seeks past the host edge and verifies the nested video stops would validate the clamp. Not blocking.

3. includeAuthoredTimingAttrs: true here vs. false in hf#464's visibility path. This file's resolveMediaCompositionContext uses true to resolve the host's duration, meaning the AUTHORED data-hf-authored-duration wins over the live child timeline's .duration(). In hf#464's 9754c3b3 the composition-host visibility path was flipped to false to prefer the live timeline. The two paths diverge intentionally — visibility needs live truth (so a shrinking sub-composition hides its children early); media sync needs authored stability (so a short child timeline doesn't clamp the media window smaller than the author intended). Correct asymmetry, but a one-line comment at this call site noting "we intentionally use authored duration here because media window is stable, unlike visibility cutoff" would save future archaeology.

4. Semantic broadening of "timed media." shouldIncludeElement now brings every <video>/<audio> inside any [data-composition-id] element under transport sync, not just those with explicit data-start. That aligns with the framework contract ("the framework manages media playback") and matches the PR's stated intent. If the codebase ever grows a legitimate use case for a composition-local video that's meant to play independently of the transport (e.g., an ambient loop), the new shouldIncludeElement lambda gives a clean extension point to exclude it. For now, no such case exists.

5. Performance. refreshRuntimeMediaCache now runs closest("[data-composition-id]") + resolveStartForElement + resolveDurationForElement per media element per sync tick. At Studio preview scale (tens of media elements, seeks a few times per second) this is unmeasurable. At scale (hundreds of media elements in a large composition) it could become a noticeable fraction of a sync tick. Future optimization: memoize composition context per-element, invalidate on DOM mutation. Flag only.

CI state

On 97f888be — green where completed: Format / Typecheck / Lint / Test / Build / Test: runtime contract / Smoke / CodeQL / Perf (fps/scrub/drift/load). In-progress: Perf: parity, Tests on windows-latest, Render on windows-latest, all 10 regression-shards. No blockers; hold merge on those settling.

Ship once CI finishes.

Review by Rames Jusso

[miguel-heygen]

Merge activity

• Apr 24, 8:10 PM UTC: @miguel-heygen merged this pull request with Graphite.