Reduce the intensity of the gray "flash" when enabling the BackgroundProcessor

[1egoman]

Background

Previously, the BackgroundProcessor track processor would show a brief gray flash right after being enabled on chrome, firefox, and safari - see the below video (recorded in chrome) for an example:

chrome-old.mov

As this behavior is not ideal, I wanted to understand why it happened and if there was anything I could do to fix it, or at least make it less noticeable to an end user.

Behavior

As far as I can tell, what the "gray" state actually represents is that the media pipeline hasn't yet had a frame make its way end to end so it can be shown to a user. The gray state starts right after the pipedStream.pipeTo(...) here under the stream processor path, and after the render loop starts here under the fallback path, and in both cases it ends once the transform handler here finishes.

That brings up an important question though - why does it take so long (sometimes upwards of 100ms) for the first frame to be processed? The reason is because of the mediapipe vision segmentForVideo call here, and to a lesser degree, the drawFrame call here. Both calls are synchronous and block the event loop - in addition, segmentForVideo for some reason takes especially long to run for the first frame (most frames it takes a couple ms at max, but the first frame could be nearly 100ms (!), accounting for most of the gray state time).

What happens on the first frame looks something like this:

1. The ProcessorWrapper media pipeline stack gets constructed (either implementation), and the output video turns gray because no frames have been processed yet.
2. transform here gets called, and execution starts.
3. This segmentPromise line gets hit, which seems like from the way the code was written should be async, but is actually synchronous, so execution halts for tens+ of milliseconds.
   a. Importantly, the promise isn't being awaited here, but that doesn't actually matter - new Promise kicks off the callback immediately no matter what.
   b. Because this is blocking the event loop, it grinds the rest of the page to a halt - importantly, actually blocking rendering of other frames / etc to the screen as well!
4. This drawFrame call occurs and also blocks the event loop for some time while it issues a bunch of webgl commands.
5. Finally, controller.enqueue(frame) here is hit, and the frame is passed along to infrastructure further down the line, and the gray state goes away.

Solution / Workaround

I think ideally, all these relatively long-running (at least in a graphics rendering context) synchronous functions would be offloaded to something like a webworker, but this proved to be challenging since webgl objects can't be passed around via a postMessage.

To work around the problem, I've opted to send along the first frame unprocessed to get rid of the "gray state", which makes it less obvious, sort of similar to how ios uses a splash image to cover up initial app loading.

What this looks like in practice:

1. (same as above - media stack is constructed)
2. (same as above - transform called for the first time)
3. On the first call of transform, isFirstFrame is true, which goes down a special first-frame-only path:
   a. Enqueue the unprocessed frame, directly from the input
   b. Wait for the browser to paint that frame to the screen before continuing
4. The initial unprocessed frame is shown, clearing the "gray state" now that there is a frame available to render
5. Segmentation, drawFrame, etc are performed - these sync operations still block the event loop, but now because of the brief pause in 3b, they don't block rendering
6. controller.enqueue(frame) is called, and a processed frame is sent along

It's also worth mentioning that this approach still doesn't 100% fix the issue - there's some delay at the start of the media pipeline that it seems is inevitable given the abstractions that exist right now. This maybe is something to consider doing better as part of a future refactor - what I think would need to happen in addition to this change is that all the media pipeline logic would have to be always wired up even if no track processor was set, and then adding a new track processor with setProcessor would just swap out the transform function run by the pipeline.

Testing

This is obviously a substantial change, and probably needs some testing. I've tested on chrome, safari, and firefox on my macbook pro with success, though more is probably required - videos below:

Chrome:

chrome.mov

Safari:

safari.mov

Firefox:

firefox.mov

[changeset-bot]

⚠️ No Changeset found

Latest commit: 6d39915

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[lukasIO]

ah, would be good to add a changeset before merging

[1egoman]

Weirdly, I did - I don't know what's up with the bot: https://github.com/livekit/track-processors-js/pull/96/files#diff-46b4f760f286903ca3692f9807e2935882ea5e4b9e87b9113cd96a4487dde624

[lukasIO]

huh, all good then