[Dota2] fossilize eats all RAM and makes the system unresponsive as a result.

[DarkDefender]

Specs:

OS: Gentoo Linux
CPU: i7-9700K
GPU: AMD 5700xt (mesa drivers)
RAM: 16GB

In the new steam beta after every update to Dota2 (doesn't seem to matter how small of an update), fossilize will rebuild the vulkan shaders when launching Dota2.

When it is rebuilding it eats all of my ram (16gb) and makes my computer swap. Making it unresponsive for around 1-2 minutes while it is building the shader cache.

Perhaps there could be some mechanism in place to make sure fossilize doesn't go overboard with memory usage if it will go over the available amount of memory?

I don't know if this is the correct place to report this. Sorry if it is not.

[HansKristian-Work]

Fossilize should already be limiting its memory usage fairly aggressively as this is a known problem. Not sure why this happens in Steam.

[DarkDefender]

Are you able to reproduce it on your end? (With the steam beta)

[HansKristian-Work]

Haven't triaged it, but I'll look at it.

[HansKristian-Work]

Ok, I tried replaying locally and I can reproduce rather large memory consumption when replaying on the 5700xt. This does not happen when I run with a dummy device, so the memory consumption must happen in the driver. I suspect it's retaining a lot of memory in the disk caches, and multiply that over lots of processes, it becomes a problem. I've seen this issue before, but to my knowledge it was fixed in Mesa a while back.

Seems like I might need to poke around to see where it's hogging so much memory ...

[DarkDefender]

If it is any help, I am currently using and building mesa from the git master branch.
So I am seeing this with the absolutely bleeding edge drivers.

[rLy07]

Hey, I didn't know that this has been tracked here before I opened my report at steam for linux. As you can see this effects Shadow of the Tomb Raider as well. Also I'm not sure if the following should be treated as a separate issue (or if is an issue at all) but background processing happens every time I start steam even when there are no updates to the games. On top of that today SotTR run twice(few other games run between the 2) without restarting steam.

[Samega7Cattac]

Noticed from time to time it starts to release some memory (intended, crash or terminated?) but the consumed memory is by far higher than the released so it will still happen.

Arch Linux
kernel version: 5.7.2-arch1-1
mesa version: 20.1.1-1
Graphic card: AMD R7 m265

How many processes and threads should it be using in normal conditions?

[lewisdiamond]

I'm also on Arch Linux:

Operating System Version:
    "Arch Linux" (64 bit)
    Kernel Name:  Linux
    Kernel Version:  5.6.15-arch1-1
Video Card:
    Driver:  X.Org Radeon RX 580 Series (POLARIS10, DRM 3.36.0, 5.6.15-arch1-1, LLVM 10.0.0)
    Driver Version:  4.6 (Compatibility Profile) Mesa 20.1.1
    OpenGL Version: 4.6
    Desktop Color Depth: 24 bits per pixel
Number of Monitors:  2
    Number of Logical Video Cards:  1
    Primary Display Resolution:  1920 x 1200
    Desktop Resolution: 4480 x 1440
    Primary Display Size: 20.39" x 12.76" (24.02" diag)
                                            51.8cm x 32.4cm (61.0cm diag)
    Primary VRAM: 8192 MB

Memory:
    RAM:  16007 Mb

I have installed earlyoom to kill anything that is not steam/dota2 and almost all other processes get killed and processing vulkan shaders takes the full 16GB, barely anything left for the system to run.

[arcsaber]

I'm having the same issue on KDE neon with an RX570 (Mesa 19.2.8). Doesn't even help to launch the game with -gl, already got two abandons because I didn't manage to restart Dota within five minutes after a crash, even though I had killed all steam processes in htop.
How to fix this mess?

[rLy07]

I'm having the same issue on KDE neon with an RX570 (Mesa 19.2.8). Doesn't even help to launch the game with -gl, already got two abandons because I didn't manage to restart Dota within five minutes after a crash, even though I had killed all steam processes in htop.
How to fix this mess?

You can probably turn off background shader processing for the time being.

[arcsaber]

You can probably turn off background shader processing for the time being.

I never had it turned on in the first place. I updated to latest oibaf drivers, let the pregame shader processing run through and am going to test this in low priority now...

[rLy07]

You can probably turn off background shader processing for the time being.

I never had it turned on in the first place. I updated to latest oibaf drivers, let the pregame shader processing run through and am going to test this in low priority now...

I don't have Dota so I don't know it then. Does it have a separate thing for shader processing from the steam wide one that was introduced recently? Or does the shader processing supposed to run at the start of a game? I haven't played SotTR for a while so I don't know if it would run at the start and I haven't really noticed it with other games but those usually only take a few seconds to compile.