Valve needs to rethink Fossilize

[jkrhu]

I think the amount of data that is being replayed is really excessive and unnecessary. I've been testing out Deadlock lately and the amount of shader pipelines to be compiled is really staggering. I tried to get through it, but it was just to much. I thought of checking the background compile and it felt like it was churning throught these for hours without end in sight. That behaviour is not exclusive to Deadlock, but many popular games have really huge amounts of replayable fragments.

Even after going through it, receiving more fragments with updates, it still takes way to long to validate everything, when you ultimately have almost 700k shader fragments. Most of which are probably duplicates, that only get catched when merging shader binaries.
[2024-09-07 21:48:58] Still replaying 1422450 (99%, 677994/691375).

I believe if it takes you more than 5 minutes to compile most of shader binaries on a decent CPU, then there's something not right with the implementation. Thats usually how PSOs work in many new titles and I think it's a good idea to emulate the behaviour of it. Yes, I know you can skip at any time, but some people will sit down for 30 minutes or even hours on lower end hardware (like this issue #251) thinking that it will make a difference, when in reality there probably wouldn't be any at this point.

I think Valve needs to rethink how the system works. In my opinion, I believe there needs to be some sort of relevancy system.

• It needs to serve users the most relevant or commonly recorded shaders.
• It needs to perform cleanup of old shader fragments that haven't been used in a while, but only when in the mean time commonly received other fragments. That would make sure we're not discarding fragments just because a game is not popular enough.

I think those options would greatly reduce compile time. Another good idea is to set a timer for 3-5 minutes and automatically kill the replaying job. With an option for the user to allow compilation of all fragments. Then you don't have to care how long it takes to complete. If the relevancy system works, then I believe after 5 minutes, they should be most of the way there anyway. Maybe it would be beneficial to collect stats for how long it takes users to replay shader fragments and perform cleanup based on that.

I personally believe Fossilize is a great system, but most users either:

• skip compilation early on because of how long it took for them before
• disable the feature altogether as a result
• will wait for 30 minutes or an hour+ thinking it has to be done to provide a stutter-less and meaningful difference to the experience.

I'm open to suggestions, but I believe Valve needs to do something for non-Deck users in this regard. I know Valve ships binaries to Deck users and that will not be possible for all configurations. Replay system is great, but it needs improvement to show users it's worth doing.

[MissPotato]

I completely agree that the system is overkill, and needs to be toned down. However, to address why I waited the full time instead of skipping midway was because even after skipping early the hitches in deadlock were unplayable. Skipping entirely, the match would quit before I could even fully join.

Considering the first 80% was so quick, I figured the last 20% couldn't take that long. Had I known it would take that long I wouldn't have even bothered.

[kakra]

I think the problem is that fossilize splits the work into X equal chunks before feeding it to the CPU cores for processing (where X is the amount of cores used for processing). 80% of the chunks are processed very fast, and the last 20% then remain processing on a few or only one core. If fossilize wants to improve here, maybe it should consider splitting the work into a lot more small fixed-size chunks and then feed those to whatever core has finished processing the previous chunk. This would also allow to more easily and dynamically change the amount of cores to adapt for foreground workloads. Currently, fossilize does it by pausing processing on a core, which only makes this observation worse, because if the process continues, it potentially has a lot of work left to do.