CUDA out of memory when trying to load 70B models.

[gradientputr1]

Heya!

Made a github for this. The script worked fine for 7b and 13bs, but I am getting out of memory errors when trying to load in a 70b model.

I tried using CPU mode, and I tried loading it in with 2 or 3 a100 80gb gpus.

I'm not sure what to do next, hope you can fix it :)

I don't know code, only know how to run scripts, and this and your blockmerge was easy to run.

[Gryphe]

Hi! I haven't really had a chance to use it on a 70b scale yet (nor had a need to, seeing as my own hardware support doesn't go beyond 13b) but there could be a couple limitations going on in the background.

How much RAM does the machine you're using have? Two copies will be loaded in memory and during the cycle a third copy is made before being moved to the GPU.

For 70b that'd put the RAM requirements at 420 GB and the GPU requirement at 140 GB VRAM. (Not counting additional overhead, which I can't predict)

I'm fully aware these are crazy numbers but due to the nature of this process there's no way to get around these requirements, hence the use of the word "monster" in Merge Monster. ;-)

[gradientputr1]

512gb of ram, I'm using runpod here.

Usually it'll either fail upon trying to load the first checkpoint, or OOM soon after managing to load the first model.

[Gryphe]

I'll do a thorough review of the process today - There might be points where both RAM and VRAM requirements are exceptionally high that I might be able to tackle.