[Feature request] Ability to cache context between runs for faster initial generation of the same history (after app restart)

[aleksusklim]

I mean, the context of "current" generation, that is super-fast for regeneration of latest action, and for minor editing of history.

I propose adding an option/param, that points to local binary file. If existed, koboldcpp should read the context from there.
While this option is active, koboldcpp should update this file with each new context after generation. (Read once at start, rewrite/append during runtime).

I can name two main reasons, why this will be extremely useful:

• I have a long initial context (personality) in my template, that I want to play from. But for larger models (most notably Llama 2 70B) it takes for too long to digest it! After that, I can just cut the history to start the new story – and that would be instantly. Only the initial loading suffers. I believe, even if the cache-context file would be updated after each turn – that worth the speed for the fast start.
• Sometimes, my computer crashes for various reasons while I'm playing with large history turn-by-turn. Each turn by itself is fast, because the context is cached. But after a crash – the browser remembers everything from history, but the first generation will be REALLY slow, much slower than my normal gameplay speed (sometimes taking up to 30 minutes if I was using a heavy model and my context was almost at its limit).

Yes, I understand that any accidental move – and I can easily destroy the cache (loading wrong history, adding a space from the start of text, etc.) which would ultimately lead to full regeneration. But! If I would play locally and alone, that would be only my own fault. Avoiding that, I can restart my system anytime, and continue playing instantly later.

Also, for use-case about big story templates, you might give an additional option for this context to be read-only, as in ggml-org#1640

[aleksusklim]

Follow-up: is it technically possible to "download" the current context history from the koboldcpp server? As text (render is server-side, if is stored as tokens in memory).

If so, can we have an action (somewhere in settings window of Lite) to replace current history in browser by its fresh copy from the server?
This is needed for three cases:

1. If browser lagged badly and somehow destroyed the history (fox example when the tab gets discarded due to low RAM conditions; I saw that myself! History was blank after tab unexpected reload mid-generation; then I grabbed whatever was printed on console and stuffed it in conversation .json file, that worked and the context cache was not regenerated)
2. If the browser was disconnected before the result could be sent. Current way to recover the model response is to copy it from the console manually. Simple case: tab reload (due to low RAM again) or browser restart mid-generation.
3. If user accidentally messed up with history, which would lead to cached context mismatch and full regeneration. Downloading the actual history ensures that the next submit won't lead to BLAS re-evaluation. Also this might be useful when changing modes (from Story to Instruct and back) and be sure that no occasional non-stripped spaces would lead to context mismatch again.

Cautions:

• For multi-user mode, this should not let one user to download history of another user. As simple measure, this feature can be completely disabled for multi-user mode if there is no reasonable way to support it. (For example, I want to open new fresh browser and "download" current history whatever it is in koboldcpp – that might be logically impossible for multi-user session, since I am a new user with no personal history for now; but that kills the core of this feature).
• Smart Context. This "cache-download" feature should return the full history as it was before smart context took in effect, otherwise it would trim the history from the beginning, ultimately leading to data loss and re-evaluation.

Overall, this would be super-useful with my original on-topic suggestion about caching contexts to files: then, you could load a binary context from file when starting the server – and then get its actual text in browser to continue flawlessly!

[LostRuins]

This is probably not very suitable for KoboldCpp - the context would be massive (few gigabytes) and the cost of writing it to disk from memory or storing it somewhere will be prohibitive and slow, for something that becomes useless shortly after. Have you tried using GPU acceleration? Processing speeds increase greatly when using Cublas or CLBlast compared to without.

[YulienPohl]

This could also be beneficial for group chats where all tokens are reprocessed every round.
Or am I doing something wrong?

[aleksusklim]

the context would be massive (few gigabytes) and the cost of writing it to disk from memory or storing it somewhere will be prohibitive and slow

Then, it should be rewritten partially, so only the last response is written at the end of the file from proper offset…
Now I understand that it will require quite massive amount of code to implement properly. Along with a custom file format.

You think it is not worth it?

(Yes, I know about CLBlast, but I have different machines, including weak 16 Gb RAM + 4 Gb VRAM where full re-evaluation is a pain!)

[Vladonai]

I too think this feature could be very useful for huge models. Implementation could be like this: -usesavecache parameter at startup = save cache (by current model name) at exit (by Ctrl+C). When restarting the program - check by model name and if cache and model names coincide, then load the saved cache. This will not load the system at each request, but the speed of the first response will SIGNIFICANTLY increase.

[aleksusklim]

save cache (by current model name) at exit (by Ctrl+C).

This won't help with system crash: the cache won't be saved. I'll rather opt-in to save cache before each generation, unless there is nothing changed (e.g. regeneretion of the last query).

check by model name and if cache and model names coincide, then load the saved cache

It is not enough to check only the name, rather metadata evaluation would be required (e.g. mismatched koboldcpp versions if the cache was left from old one). And still, the file format should be defined and implemented, which is not trivial.

I've checked LM Studio (https://lmstudio.ai/) and looks like they've implemented full cache saving in runtime for each dialog separately, printing used disk space.

[Vladonai]

save cache (by current model name) at exit (by Ctrl+C).

This won't help with system crash: the cache won't be saved. I'll rather opt-in to save cache before each generation, unless there is nothing changed (e.g. regeneretion of the last query).

Yes, but to be honest my Koboldcpp doesn't crash most of the time. It's a very stable program. So some light tuning is certainly desirable, but writing cache after each request will cost too much IMHO.

check by model name and if cache and model names coincide, then load the saved cache

It is not enough to check only the name, rather metadata evaluation would be required (e.g. mismatched koboldcpp versions if the cache was left from old one). And still, the file format should be defined and implemented, which is not trivial.

Well, yes, but the details of implementation are best left to the author :)

[aleksusklim]

I didn't say that it's koboldcpp who is crashing, but my system!
On one particular low-RAM machine I experience hangs and crashes after using koboldcpp (or any other engine to run GGML models).

For example, I can load 13B model with 16 Gb of RAM just fine (takes, for example, 15.7/16.0 used total by all processes), then close koboldcpp, and then get system hang after opening Firefox because something in memory was already corrupted!
(Already tried changing my RAM, booting from another Windows disk, using Safe mode, etc.; previously it happened with 7B models on 8 Gb or RAM)
Or it could crash after my workstation locks during to inactivity, unable to login again when the model is loaded in RAM.

I know that this is a completely different problem, but having the context cached would made those crashes not so painful.

On the other hand, on my hi-RAM machine (64 Gb plus 12 VRAM) not only it is not crashing whatsoever, but the full context re-evaluation is pretty fast. So, yeah… "Just get a higher-tier PC" is the solution here too.

[Vladonai]

On the other hand, on my hi-RAM machine (64 Gb plus 12 VRAM) not only it is not crashing whatsoever, but the full context re-evaluation is pretty fast. So, yeah… "Just get a higher-tier PC" is the solution here too.

Starting with the 70B-models, even on a "higher level PC" the first response at a context size of 4K has to wait too long. This is not the limit. GGUF models support context sizes up to 16K and such a context may require some solution even for a 13K model. In general, we must admit that the problem does exist.

[e576082c]

From the perspective of a simple user, it "feels like", that a lot more time is being spent on "prompt ingestion", "prompt processing" or whatever is the first step before the actual text generation process begins.

The actual text generation speed, with streaming enabled, never "feels like" it's slow, because I can spend time actively reading the text while it is being generated.

However, if I do change tasks, scenarios, characters, or do any edit in the chat history, then the next generation will always have a very noticeable delay lag, like as if the process is hanging and frozen, it does always "feel like" it's slow, even with cuBLAS, openBLAS, CLBLAS, or whatever magical accelerator backend I choose. If the context or history is changed, then the beginning of the next generation will always "feel slow".

I don't know how could this be solved or improved, but some miracle would be good to happen. lol

Might be related:
[Enhancement] Multiple instances of koboldcpp at a time using the same model, also using different models simultaneously. #163

[Question] Confused about how / why full BLAS processing happens #386

When editing response, reprocessing prompt length varies randomly #385

[LostRuins]

In the next version, there will be a brand new feature for context shifting that should allow you to significantly reduce processing time.

[e576082c]

In the next version, there will be a brand new feature for context shifting that should allow you to significantly reduce processing time.

Thank you very much! That would be great. :)

[aleksusklim]

I can reply to this from what I know. (Let's pretend that the new "context shifting" is not exists yet; still, everything I'll say will be useful!)

1. Any model has "context window" measured in tokens. For llama 1 it was 2048, for llama 2 it is 4096.
2. The model physically cannot process anything beyond the pre-allocated context memory. To claim more space, you have to allocate more from the start.
3. Most models do not perform well when their context window is enlarged beyond their native length. The quality degrades quickly.
4. There are several methods of extending context length are proposed, it is called "RoPE". One of its variant requires to fine-tune models on higher contexts to be able to use it.
5. Luckily, for GGUF models the correct parameters for extending the context are determined automatically, that's why for koboldcpp it is as simple as changing your desired context length when starting the server.

Notes:

1. You cannot easily see, how long is your current context. (Why? What is "token budget" displayed in the corner? Isn't it better to display there the actual token counter and limit, since it is already exposed in API?) Good news, recent versions of koboldcpp are printing this after each generation as ContextLimit: <used>/<total>
2. Your client (browser) knows the context length too, and can adjust it. For example, your server may be running 4096 while your client is still on 2048 tokens. (Why? Isn't it better to always resort to the server value? What is the point in having less context size in Lite?) You should always open Settings and move the Max Ctx. Tokens to the right, making sure numbers above and below its right part are equal.
3. The slider Amount to Gen. is actually paying from total context size, because it is reserving some room for the model to answer. For example, if your context is 2048 but you set Amount to generate = 512, then your effective context for history will be 1536, even if the model stops early and never writes that much of text.

What will happen if you run out of context?

1. Each normal turn consist of two stages: processing your prompt ("BLAS") and generating response ("streaming"). Everything that was already processed is caching in memory, so when you regenerate the last output without changing anything – there would be no processing/BLAS part (it will show up as 1 token), but only generation right away.
2. Processing takes long time and is invisible in browser. The longer the text of your turn is, the longer you'll have to wait for generation to start (but various BLAS optimizations can make this pretty quick, especially with good GPU).
3. When the context has no room for the next line, koboldcpp will trim the history from the beginning. This means that some of your earlier turns will effectively disappear from the memory completely. The model would never see them from now on.
4. If you have anything in Memory tab, then it will be preserved as the beginning of the history. Actually, Memory it just a way to split the history into "keep this always" and "delete if needed" parts, so whenever you run out of context only a few of topmost lines are deleted (so there would be a gap between "memory" and "dialogue" as if a book was missing a page right after introduction of characters).
5. Since the context was trimmed and recreated, it has to be BLAS-processed from the start. There is now way for the model to magically shift it without re-evaluation (YET!?). And since your context was already full, this will take the longest possible time, just as if you have restarted the server. It is more than 20 times slower than your regular play speed, provided your normal turns are about 100-150 tokens, while the history is somewhere at 2000-3000 and must be processed from the scratch.

What can you do about it?

1. You may continue paying as-is. Each your turn will trigger the full context reevaluation, but you might be fine with it, especially if the model can fit in GPU entirely. Regenerations of the last reply would still be instant, only your own turn will become slow.
2. You can manually delete something from the beginning of your story. For example, you can describe a half of actions as an extended introduction, and remove the redundant lines. This way you could empty a lot of context room yourself, in full control of what the model would know.
3. You can enable Use SmartContext when starting the server. It will split the history for you the next time you run out of context (and otherwise won't hurt if you'll never overflow). Actually, it will simply cut the history in two, deleting the first half (and preserving the Memory), so that each time you will continue from half-empty context window. This way you could more or less play normally, but experience full evaluations only occasionally. Just be warned that the model might forget many of older events.
4. You can restart the server and increase Context Size. Personally, I see no quality degradation for llama2 models if I run them with 8192 (no Custom RoPE Config!) instead of native 4096 right from the beginning. But if you do, fine, play with 4096 and restart in 8196 when you run out of context. This way you will double your window for free, and the model would still write coherent texts because it has a good history to mimic. If you hit 8192 – increase to 12288, it will still work, as long as you continue playing and not starting a new game (where it would be noticeably worse). If you hit 12288, you can try going to 16384, but unfortunately you will have to regenerate a lot, since the degradation would become obvious.

To protect yourself from an accidental context reevaluation, you can:

1. Keep your "amount to generate" low in browser (I prefer 96). Even if the model sometimes won't have a chance to finish its last sentence – you can just let it continue again, just make sure Trim Sentences is unchecked in Advanced tab.
2. Do not use "Author's Note" (in Memory). Instead, explain your intent for the model yourself, for example some models like mythalion respect <|system|> messages which you can embed at your will (for example, by adding to the end of last output), and delete if they are not needed anymore (this will re-evaluate everything below your edition, of course).
3. Look at the console when you feel the end is close. Maybe edit-out some unnecessary dialogue lines, which intent was already settled and won't literally affect your planned story later.
4. Be careful to not edit anything in the textbox unintentionally, always undo your accidental changes.

As for my own confusion: LostRuins, can you explain why we see a rough token budget instead of precise token counter and context size?
And also, what's the point of adjusting Max Ctx. Tokens in Lite, if the server will ignore too large value, and there is no practical point in having it less than what is available currently in koboldcpp? Or is there?