RTP Bridges Interfere With Connection of Chrome Clients

[TravisBowers]

OS: Windows 10 (Virtualized with libvirt)
UE Engine Version: 4.25.4
GPU: Nvidia 2080 Ti
GPU Driver: 461.72

If an RTP bridge is connected to the cirrus server before Google Chrome tries to connect, the UE4 Application crashes once video begins streaming to Google Chrome. The following sequence of events occur:

1. Cirrus is started
2. The pixel streaming UE4 app is started
3. An RTP bridge is started an connected to Cirrus

• RTP is being streamed and can be tuned

4. Google Chrome is started on a remote client

• navigate to cirrus server URL
• click start
• "waiting for video" message appears in Chrome
• "Play" triangle appears
• Click play Triangle
• A single video frame appears in chrome

5. UE4 App crashes with popup reiterating log output

The log contains the following:

[2021.03.18-21.13.02:006][399]LogThreadingWindows: Error: Runnable thread NvVideoEncoder crashed.
[2021.03.18-21.13.02:007][399]LogWindows: Error: === Critical error: ===
[2021.03.18-21.13.02:007][399]LogWindows: Error: 
[2021.03.18-21.13.02:007][399]LogWindows: Error: Assertion failed: State == EFrameState::Capturing [File:D:/Build/++UE4/Sync/Engine/Source/Runtime/AVEncoder/Private/Microsoft/Windows/NvVideoEncoder.cpp] [Line: 695] 
[2021.03.18-21.13.02:007][399]LogWindows: Error: Buffer 1 : Expected state 2, but found 3
[2021.03.18-21.13.02:007][399]LogWindows: Error: 
[2021.03.18-21.13.02:007][399]LogWindows: Error: 
[2021.03.18-21.13.02:007][399]LogWindows: Error: 
[2021.03.18-21.13.02:007][399]LogWindows: Error: Crash in runnable thread NvVideoEncoder
[2021.03.18-21.13.02:058][402]LogExit: Executing StaticShutdownAfterError
[2021.03.18-21.13.02:058][402]LogWindows: FPlatformMisc::RequestExit(1)
[2021.03.18-21.13.02:058][402]LogCore: Engine exit requested (reason: Win RequestExit)
[2021.03.18-21.13.02:068][403]Log file closed, 03/18/21 17:13:02

If Google Chrome is connected to the cirrus server before the rtp bridge, everything seems to work smoothly. i.e., the following:

1. Start Cirrus
2. Start pixel streaming UE4 app
3. Open google chrome on remote client and begin interactive pixel streaming session.
4. Start RTP bridge on host
5. Tune RTP stream on remote client

However, if Google Chrome is disconnected while the RTP bridge remains connected and then a Chrome instance attempts to reconnect, it cannot resume an interactive pixel streaming session until the RTP bridge is disconnected from the Cirrus server. When this occurs, the "click to start" message is displayed on chrome. Once clicked, the "WebRTC connected, waiting for video" message appears until the RTP bridge is terminated. Once the bridge is not longer running, Chrome can be refreshed and an interactive pixel streaming session can resume. At this point, subsequent RTP bridges can be instantiated.

Generally, Google Chrome cannot create a new pixel streaming session if existing RTP bridges are connected to Cirrus. If RTP bridges are connected the first time that a Google Chrome client attempts to connect, the Unreal application will crash.

[lukehb]

Good investigation, I wish all issues were presented to me so beautifully! Thank you, I will dig in further to see if I can find the root cause.

[andrewdunndev]

For reference, this is something that still exists in 4.27.

[lukehb]

Yeah, I can confirm this still repros (well sort of) for latest versions of the engine (4.27 branch from Github) - though not as catastrophically as in previous version of Pixel Streaming. In the latest version of the UE4 from Github the bridge can connect first and then Chrome with no problems; however, the reverse case, Chrome first and then the bridge - the bridge does not work.

The failure to connect no longer crashes the engine, which is good.

I am pretty sure the failure to connect is to do some missing WebRTC information. This looks "not too bad" to fix in the latest version of Pixel Streaming.

[lukehb]

I think I found the problem. Default video payload the bridge was using/expecting did not match what Chrome was using - so they could not share a single stream depending who set it up.

Fix is here (and pushed): 40eef6d

Additionally, I noticed FFPlay will sometimes be unable to start decoding the stream until a keyframe is sent. The easiest way to force a keyframe to be sent is to connect another peer (i.e. another browser or bridge) - but this highlights something I have been looking into lately anyway, which is that keyframes in Pixel Streaming should be sent on regular interval anyway (and currently are not).

Let me know how this goes for you :)

[lukehb]

I forgot to update this, from UE 4.27.2 onwards there is now a new console variable/command line switch:

https://github.com/EpicGames/UnrealEngine/blob/d94b38ae3446da52224bedd2568c078f828b4039/Engine/Plugins/Media/PixelStreaming/Source/PixelStreaming/Private/PixelStreamingSettings.cpp#L99-L103

-PixelStreamingEncoderKeyframeInterval=

By default in 4.27.2 it is set to

-PixelStreamingEncoderKeyframeInterval=300

Meaning Pixel Streaming will send a keyframe every 300 frames in addition to whatever the browser is asking for. So in the case of the RTP bridge, at worst there will be a 300 frame delay on startup before video can be decoded.