Implement mblock flag toggle

[pawzoflion]

It would be nice if we had -mlock argument like on llama.cpp
Model using my whole swap instead of RAM.

[henk717]

While I see no harm in a toggle, with the current default I am curious how you come to that conclusion, which model you are trying to load and what amount of ram you have available to you before running the model.
I am a windows system administrator myself and I know this argument is related to Windows, it is correct that task manager shows this in a weird way where the ram usage isn't correctly listed to the program itself. But when I manually check the size of the page file the page file is not large enough to contain the model indicating its not using the page file for this.

So I think as a default the current setup is going to benefit most users since it automatically creates a hybrid between a disk cache and RAM usage without actually hitting the users page file and even allows multiple instances without having the model in memory twice.

If you are noticing different results I am curious how I can best reproduce it. One really good and reliable way to check what is being used is checking your disk's activity when it is generating especially when it is parsing the prompt. If the page file is not being used you should see no disk activity related to the program. If it merely streams the model from disk you should see high read speeds but no write speeds, and if you are actually using the page file you should see read and write speeds.

[LostRuins]

Henk is right. In any case, mlock is usually only properly supported on OSX, which I do not have. My dev machine does not support it.

Usually after loading the model once, the second load happens in under 3 seconds for me, no changes needed,.

[pawzoflion]

Maybe it was extremely stupid question. I have 8gb ram and system is arch. Maybe I don't understand how Linux manages memory, its just weird to me llama.cpp uses only 50% from free Ram and loading the rest in swap, so I am usually forcing it to load in ram with mlock.

[LostRuins]

There was a bunch of changes to the model loader, you can now try with the latest version, there is also a way to turn off mmap with --nommap