Crash on startup of packaged unreal build

[monodop]

Hello, I've packaged my unreal 5.3.2 project and I'm getting an intermittent crash immediately after launching the executable. This only seems to happen ~25% of the time. Things seem to work fine when running from the editor or launching a standalone process from the editor. I'm suspecting there's some kind of race condition here, but I'm not really sure how to debug this:

[2024.05.14-03.03.31:283][186]Message dialog closed, result: Ok, title: The UE-ThePoorlyVisible Game has crashed and will close, text: Fatal error!


[2024.05.14-03.03.31:283][186]LogAudioMixer: Error: Command Queue has grown to 1355kb, containing 8997 cmds, last complete pump was 2.40472 seconds ago.
[2024.05.14-03.03.31:283][186]LogAudioMixer: Warning: Stall in AudioRenderThread Phase: 'Finished'
[2024.05.14-03.03.31:283][186]LogAudioMixer: Warning: ***** ThreadStackWalkAndDump for ThreadId(121300) ******
0x00007ff9cb78f9d4 ntdll.dll!UnknownFunction []
0x00007ff9c8d9427e KERNELBASE.dll!UnknownFunction []
0x00007ff62c2c47d0 ThePoorlyVisible.exe!ReportCrash() []
0x00007ff636bceabe ThePoorlyVisible.exe!_guard_xfg_dispatch_icall_nop() []
0x00007ff9c0d5f73f VCRUNTIME140.dll!UnknownFunction []
0x00007ff9cb7949ff ntdll.dll!UnknownFunction []
0x00007ff9cb70e466 ntdll.dll!UnknownFunction []
0x00007ff9cb7939ee ntdll.dll!UnknownFunction []
0x00007ff9c0d61241 VCRUNTIME140.dll!UnknownFunction []
0x00007ff63467ed36 ThePoorlyVisible.exe!FAcousticsSpatializerReverb::ProcessMixedAudio() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\ProjectAcoustics\Source\ProjectAcousticsSpatializer\Private\AcousticsSpatializerReverb.cpp:73]
0x00007ff632759c42 ThePoorlyVisible.exe!FSoundEffectSubmix::ProcessAudio() []
0x00007ff630e4e146 ThePoorlyVisible.exe!Audio::FMixerSubmix::GenerateEffectChainAudio() []
0x00007ff630e644a2 ThePoorlyVisible.exe!Audio::FMixerSubmix::ProcessAudio() []
0x00007ff630e63532 ThePoorlyVisible.exe!Audio::FMixerSubmix::ProcessAudio() []
0x00007ff630e0cff1 ThePoorlyVisible.exe!Audio::FMixerDevice::OnProcessAudioStream() []
0x00007ff62e8c1fcb ThePoorlyVisible.exe!Audio::FOutputBuffer::MixNextBuffer() []
0x00007ff62e8c3298 ThePoorlyVisible.exe!Audio::IAudioMixerPlatformInterface::RunInternal() []
0x00007ff62e8c31b9 ThePoorlyVisible.exe!Audio::IAudioMixerPlatformInterface::Run() []
0x00007ff62c340582 ThePoorlyVisible.exe!?Run@FRunnableThreadWin@@AEAAIXZ() []
0x00007ff62c333fd7 ThePoorlyVisible.exe!FRunnableThreadWin::GuardedRun() []
0x00007ff9c986257d KERNEL32.DLL!UnknownFunction []

[2024.05.14-03.03.31:283][186]LogThreadingWindows: Error: Runnable thread AudioMixerRenderThread(1) crashed.
[2024.05.14-03.03.31:283][186]LogWindows: Error: === Critical error: ===
[2024.05.14-03.03.31:283][186]LogWindows: Error: 
[2024.05.14-03.03.31:283][186]LogWindows: Error: Fatal error!
[2024.05.14-03.03.31:283][186]LogWindows: Error: 
[2024.05.14-03.03.31:283][186]LogWindows: Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x000001750bfb0ed0
[2024.05.14-03.03.31:283][186]LogWindows: Error: 
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff9c0d61241 VCRUNTIME140.dll!UnknownFunction []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff63467ed36 ThePoorlyVisible.exe!FAcousticsSpatializerReverb::ProcessMixedAudio() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\ProjectAcoustics\Source\ProjectAcousticsSpatializer\Private\AcousticsSpatializerReverb.cpp:73]
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff632759c42 ThePoorlyVisible.exe!FSoundEffectSubmix::ProcessAudio() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff630e4e146 ThePoorlyVisible.exe!9(￣+￶�() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff630e644a2 ThePoorlyVisible.exe!Audio::FMixerSubmix::ProcessAudio() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff630e63532 ThePoorlyVisible.exe!Audio::FMixerSubmix::ProcessAudio() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff630e0cff1 ThePoorlyVisible.exe!Audio::FMixerSubmix::ProcessAudio() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff62e8c1fcb ThePoorlyVisible.exe!Audio::FOutputBuffer::MixNextBuffer() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff62e8c3298 ThePoorlyVisible.exe!Audio::IAudioMixerPlatformInterface::RunInternal() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff62e8c31b9 ThePoorlyVisible.exe!Audio::IAudioMixerPlatformInterface::Run() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff62c340582 ThePoorlyVisible.exe!FRunnableThreadWin::Run() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff62c333fd7 ThePoorlyVisible.exe!?GuardedRun@FRunnableThreadWin@@AEAAIXZ() []
[2024.05.14-03.03.31:283][186]LogWindows: Error: [Callstack] 0x00007ff9c986257d KERNEL32.DLL!UnknownFunction []
[2024.05.14-03.03.31:283][186]LogWindows: Error: 
[2024.05.14-03.03.31:283][186]LogWindows: Error: Crash in runnable thread AudioMixerRenderThread(1)
[2024.05.14-03.03.31:295][187]LogExit: Executing StaticShutdownAfterError
[2024.05.14-03.03.31:296][187]LogWindows: FPlatformMisc::RequestExit(1, FRunnableThreadWin::GuardedRun.ExceptionHandler)
[2024.05.14-03.03.31:296][187]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3, FRunnableThreadWin::GuardedRun.ExceptionHandler)
[2024.05.14-03.03.31:296][187]LogCore: Engine exit requested (reason: Win RequestExit)

Looking at the plugin source, it looks like something is wrong with this memcpy.

            for (auto OutputChannel : DeinterleaveViewOutput)
            {
                for (auto HrtfChannel : DeinterleaveViewHrtf)
                {
                    if (HrtfChannel.ChannelIndex == OutputChannel.ChannelIndex)
                    {
                        FMemory::Memcpy(OutputChannel.Values.GetData(), HrtfChannel.Values.GetData(), outputBufferLength * sizeof(float));
                    }
                }
            }

Do you have any suggestions on what I can do to further troubleshoot this?

[NoelCross]

@monodop We haven't seen this issue before although there were some crashes that we noticed when the Google Resonance plugin was enabled at the same time as the Project Acoustics plugins. The Resonance plugin was fixed at some point to avoid the memory corruption, but if you do have this plugin enabled, you might want to disable it to see if the crashes go away.

[monodop]

@monodop We haven't seen this issue before although there were some crashes that we noticed when the Google Resonance plugin was enabled at the same time as the Project Acoustics plugins. The Resonance plugin was fixed at some point to avoid the memory corruption, but if you do have this plugin enabled, you might want to disable it to see if the crashes go away.

Hi @NoelCross, thanks for looking into this. I've disabled that plugin and repackaged, but it looks like I'm still getting the same crash. Is there any other information I can gather/provide to make this easier to troubleshoot? e.g. showing full plugin list, changing some logging settings, etc?

[NoelCross]

@monodop when you disabled the plugin, did you disable from the plugins page, or did you unselect the plugin in the package settings? Unfortunately, just having the plugin enabled even if it isn't in use, was a problem in the past.

Make sure that Project Acoustics is the plugin set for all three plugin types (spatialization, reverb and SDO):

You might also try to swap to another spatializer/reverb combo and see if you get different results.

If the memory is getting corrupted and the memcpy is crashing, it is going to be challenging to narrow this down without a local repro. If you have a minimum project you can share, that would help with tracking this down.

[monodop]

@NoelCross I disabled the plugin via the plugins page, and it does not appear to be mounted in the log file:

I'm not sure what you're referring to about unselecting the plugin in the package settings though.

As far as plugin types for the Windows platform:

I can't really share the full project, but I can try to put together a new project to repro the issue - it'll just take some time.

[NoelCross]

@monodop Yes, it looks like the Windows platform settings are correct. One thing about the disabling of the Resonance plugin is that you need to make sure you restart the project to ensure that it is really unloaded. I'm assuming that you did this before you started the packaging process.

A small project would be helpful for tracking this down, otherwise, we haven't seen this issue so we are not sure what the root cause would be.

[monodop]

@NoelCross I created a new c++ project based on the first person template, followed the setup guide for unreal, and I'm getting the crash still. I zipped the folder and put it in a dropbox folder: https://www.dropbox.com/scl/fi/l0bkx9sdthhjaenkl981b/PACrashRepro.zip?rlkey=x3wyuy6ady3kigfmdz4ypvxzm&st=j7vwz2fn&dl=0

[NoelCross]

@monodop I downloaded the project and was able to get a packaged Windows build in the Shipping configuration. This was after I copied the ACE file into the Content/Acoustics folder since the file was located in the Content folder and wasn't found when running in PIE.

After running the packaged application about 10 times, I wasn't able to repro the reported crash. I only had the Project Acoustics plugin installed in my UE5.3 instance.

As a test, can you uninstall the Resonance plugin from your engine instance from the launcher and not just disable in the Plugin settings? Maybe that will change the behavior.

There might be some other plugin that is also causing this so I would suggest to temporarily uninstall all plugins except for the PA one and see if there is an interaction problem.

Depending upon your debug chops, you can put a breakpoint on the ProcessMixedAudio() routine and see which buffers are not valid before the copy operation and then start tracking the memory with "write breakpoints" to see when it gets overwritten.

Another option would be to take a time travel trace of the repro and share the trace file. This would allow for stepping forward and backwards in time to see when memory gets overwritten.

[monodop]

@NoelCross

This was after I copied the ACE file into the Content/Acoustics folder since the file was located in the Content folder and wasn't found when running in PIE.

Oh, oops! I didn't notice that in the repro example. I fixed that locally and still have the same issue. It was also configured properly in my main project.

I downloaded the project and was able to get a packaged Windows build in the Shipping configuration.

After running the packaged application about 10 times, I wasn't able to repro the reported crash. I only had the Project Acoustics plugin installed in my UE5.3 instance.

Is shipping configuration required here? If so, I might have missed that in the docs somewhere. I just tried a build on shipping, and I don't see the same crash at nearly the same frequency. I've been building with the development configuration. A copy of the packaged development build exe is available in the Build/Windows folder in the zip file I linked.

As a test, can you uninstall the Resonance plugin from your engine instance from the launcher and not just disable in the Plugin settings? Maybe that will change the behavior.

There might be some other plugin that is also causing this so I would suggest to temporarily uninstall all plugins except for the PA one and see if there is an interaction problem.

I uninstalled all other plugins, but I don't know how to completely uninstall the resonance plugin, as it appears to be installed by default. This doesn't appear to make a difference either.

Depending upon your debug chops, you can put a breakpoint on the ProcessMixedAudio() routine and see which buffers are not valid before the copy operation and then start tracking the memory with "write breakpoints" to see when it gets overwritten.

Another option would be to take a time travel trace of the repro and share the trace file. This would allow for stepping forward and backwards in time to see when memory gets overwritten.

I can't look into these tonight, but I can work on it through this week if needed.

[NoelCross]

@monodop I've tried launching your pre-packaged binary on two different machines 10 times and didn't see a crash. Could you try your packaged build on a different machine? Maybe there is something going on with the audio hardware you have installed in your config that is sending us down a bad path. Are you using standard motherboard audio on your PC or do you have an adapter that exposes advanced output configurations?

[monodop]

@NoelCross Thanks for taking the time to do that!

I think I discovered a piece of the problem: the SteelSeries GG App that comes with my headset. I'm not entirely sure if it's a specific feature (like Sonar or Moments), or the app itself. I've tried disabling as much as I can, but I haven't been able to identify anything in particular that's causing the crashes. I'm going to keep investigating to see if I kind figure out anything more.

[NoelCross]

@monodop After installing the SteelSeries GG App, I've been able to repro what you are seeing. This application exposes an 8ch output device that it hitting code that isn't well tested so I'm not surprised there is an issue there. I'll look into the root cause and provide a fix. Thanks for reporting this.

[NoelCross]

@monodop We'll look at getting this fixed in the marketplace plugin. For now, you should be able to copy the plugin source from the engine folders and place into your project, then edit AcousticsSpatializerReverb.cpp:

        else if (OutData.NumChannels > 2)
        {
            // If the output buffer has more than 2 channels we copy the HRTF-processed signal into the first 2 channels
            Audio::TAutoDeinterleaveView<float, Audio::FAudioBufferAlignedAllocator> DeinterleaveViewHrtf(outputBuffer, m_ScratchBufferHrtf, 2);
            float* OutputBufferPtr = OutData.AudioBuffer->GetData();

            for (auto HrtfChannel : DeinterleaveViewHrtf)
            {
                auto inData = HrtfChannel.Values.GetData();
                auto inDataCount = HrtfChannel.Values.Num();
                for (int32 i = 0; i < inDataCount; ++i)
                {
                    const int32 output_offset = i * OutData.NumChannels + HrtfChannel.ChannelIndex;
                    OutputBufferPtr[output_offset] = inData[i];
                }
            }
        }

[monodop]

@NoelCross That appears to have done the trick! I really appreciate all of your help here. Looking forward to hearing when this hits the marketplace.