Add video texture sample

[shaoboyan]

This sample shows how to upload video frame to webgpu and rendering on
canvas.
It can be easily modified to check the uploading perf of
copyImageBitmapToTexture.

[shaoboyan]

@austinEng Hi, Austin,
I use this sample (with some modification) to check copyImageBitmapToTexture performance. Do you think it is Ok to add this sample into webgpu sample collections?

[austinEng]

I think it would be great to add! I'm going to be making a few refactorings across the whole samples repo, so if you don't mind, I'd like to land that first. It should make some of your code here simpler or not require ts-ignore suppressions. I commented about what would be changing.

[austinEng]

mm... I don't think we should be doing an await instead rAF. Instead we should probably do the await elsewhere, and when it's done, set a boolean flag which indicates to this rAF loop that the ImageBitmap is ready to use.

[shaoboyan]

Or do you think change this to the form like Promise.then will be Ok ? Like createImageBitmap(video).then(function(imageBitmap) {// Do the uploading and rendering}).

[austinEng]

Lastly, the video is a bit big.. could we link to it instead of checking it in?

[shaoboyan]

@austinEng Thanks for reviewing! I'll wait for your refactory.

Lastly, the video is a bit big.. could we link to it instead of checking it in?

Ack.

[austinEng]

It landed! so you should be good to update/rebase this

[shaoboyan]

@austinEng Rebased and addressed most comments except this one:

mm... I don't think we should be doing an await instead rAF. Instead we should probably do the await elsewhere, and when
it's done, set a boolean flag which indicates to this rAF loop that the ImageBitmap is ready to use.

Instead I'm using Promise.then. (I'm not quite clear about how Promise.then or await works), so if it is not suitable, I'll find another way to address it :)

[austinEng]

Here's a suggestion, though @kainino0x might have other ideas on how to do this.

  // When the video timestamp updates, call |updateVideoFrame|
  video.addEventListener('timeupdate', updateVideoFrame, true);

  let newVideoFrame = undefined;
  function updateVideoFrame() {
    // get the image bitmap, and then store it in |newVideoFrame|
    createImageBitmap(video).then(videoFrame => {
       newVideoFrame = videoFrame;
    });
  }
  
  const videoTexture = device.createTexture( .... );

  function frame() {
    if (newVideoFrame) {
      // get the video frame and then set it to undefined so we only do it once per video frame change
      const videoFrame = newVideoFrame;
      newVideoFrame = undefined;
     
      // copy the new frame into |videoTexture|
      device.defaultQueue.copyImageBitmapToTexture(
          imageBitmap: videoFrame, texture: videoTexture, ...);
    }
 
    const outputView = swapChain.getCurrentTexture().createView();
    
    // Copy / sample from |videoTexture| into |outputView|
    // ...
    // ...

    requestAnimationFrame(frame);
  }
  requestAnimationFrame(frame);

[kainino0x]

I didn't look at the demo, but it depends on whether you want to render between video frames or not (i.e. whether you want to render at the display framerate or the video framerate). If you want to render at the video framerate, you could chain the requestAnimationFrame off of the createImageBitmap().then. Otherwise, that code looks pretty reasonable to me. I think could just do the copyImageBitmapToTexture in the createImageBitmap().then though?

  function updateVideoFrame() {
    // get the image bitmap, and then copy the new frame into |videoTexture|
    createImageBitmap(video).then(videoFrame => {
      device.defaultQueue.copyImageBitmapToTexture(videoFrame, videoTexture, ...);
    });
  }
  
  const videoTexture = device.createTexture( .... );

  // When the video timestamp updates, call |updateVideoFrame|
  video.addEventListener('timeupdate', updateVideoFrame, true);

  function frame() {
    const outputView = swapChain.getCurrentTexture().createView();
    
    // Copy / sample from |videoTexture| into |outputView|
    // ...
    // ...

    requestAnimationFrame(frame);
  }
  requestAnimationFrame(frame);

[shaoboyan]

@austinEng and @kainino0x thanks for explaining. I think introduce 'timeupdate' event listener will help reduce unnecessary imageBitmap creation and copyIB. Agree with Kai on the updateVideoFrame form.

PTAL!

[shaoboyan]

@austinEng softly ping on this :)

[austinEng]

Thanks for the update @shaoboyan. It looks good! though I just tried it and the video isn't very smooth. It looks like the timeupdate event only fires every 250ms. Seems like we have to call updateVideoFrame every frame then.

To make things better, there's a requestVideoFrameCallback which is still in draft https://wicg.github.io/video-rvfc/.

I suggest we do the following:

interface VideoFrameMetadata {
  presentationTime: DOMHighResTimeStamp;
  expectedDisplayTime: DOMHighResTimeStamp;
  width: number;
  height: number;
  mediaTime?: number;
  presentedFrames: number;
  processingDuration?: number;
  captureTime?: DOMHighResTimeStamp;
  receiveTime?: DOMHighResTimeStamp;
  rtpTimestamp?: number;
}

declare global {
  interface HTMLVideoElement {
    // optional because it's still draft.
    requestVideoFrameCallback?:
      (callback: (now: DOMHighResTimeStamp, metadata: VideoFrameMetadata) => void) => void
  }
}

// ...
// ...


  const hasRequestVideoFrameCallback = !!video.requestVideoFrameCallback;
  if (hasRequestVideoFrameCallback) {
    const vfc = () => {
      updateVideoFrame();
      video.requestVideoFrameCallback(vfc);
    }
    video.requestVideoFrameCallback(vfc);
  }

  return function frame() {
    if (!hasRequestVideoFrameCallback) {
      updateVideoFrame();
    }

    // ...
  }

[kainino0x]

I think you can still skip an updateVideoFrame if the timestamp hasn't changed.

[shaoboyan]

@austinEng Thanks for trying this. I also find it doesn't work smooth when the video plays in first time. But if I load it from local or the video plays in second time. It become much smooth. I thought it was due to the downloading. But for a 2M video, it tooks so long(But since I'm behind the proxy, I thought this might be my special case). But since you also hit this, do you have any thoughts on this?
And I think if it is related to the decode speed or network, skip updateVideoFrame won't help, maybe ?

[shaoboyan]

BTW, I'll try the vdieo.requestVideoFrameCallback.

[kainino0x]

I think we should avoid it because it's nonstandard and not available in browsers except Chromium.

[shaoboyan]

@austinEng @kainino0x I find that three.js make their video related example with local video
e.g. https://threejs.org/examples/webgl_video_panorama_equirectangular.html
https://threejs.org/examples/webgl_materials_video.html
(but not sure about babylon.js). Do you think we can commit a smaller video clip to make this sample smooth?

[kainino0x]

I think it would definitely okay to use a small test video file. Agree it would be good to make it more reliable/local.

[shaoboyan]

(Sry for backing to this a bit late because working on reshape CTS)
@austinEng @kainino0x I've did some local test and find that the smooth video needs:

1. local video clip
2. Don't rely on timeupdated event.
   So I did some changes:
3. commit a smaller video clip( from three.js)
4. check video.currentTime but not timeupdated event to trigger updating.

After these changes, I can have a smooth video in my local. Pls have a double check if you have interets.

[austinEng]

For me, this condition was true every time. Is that also true for you? If it is, we should just call updateVideoFrame every frame.

[shaoboyan]

This condition was true every time on my desktop too (I'm a bit surprise about this) Remove it.

[shaoboyan]

@austinEng Make some changes, PTAL