Question: Is there any way to increase the max number of tokens generated from 512?

[dogjamboree]

Maybe this is a dumb question, but why is the limit 512? The model I'm using has a limit that's much higher (vicuna), if not unlimited. If this something hard-coded in the program or could it be modified?

[LostRuins]

It can be increased in the webui settings. It is not fixed.

[dogjamboree]

It can be increased in the webui settings. It is not fixed.

It can be set to a maximum of 512 while the models support more. Is there a way to change this via coding or is this a hard-coded limit?

[LostRuins]

Perhaps you are editing the wrong field. You should be changing the "Max Context Length" which has a slider range to 2048, and not the "Amount to Generate".

It's also possible to override the max slider values by manually editing the text number on the input.

[dogjamboree]

You're right,the "Max Context Length" slider does go to 2048 and I have it to that maximum number, but I'm interested in getting the "Amount to Generate" as high as possible.

For "Amount to Generate", I just realized what you meant about manually editing that number and again, you're right, you can override the slider number of 512 by manually typing a number there but it even if I type 5000 or 10000 or some other number that I know is supported through both the model and llama.cpp it cuts it back to 1024 when I go back into the settings again. (llama.cpp even allows -1 for no limit whatsoever and I can attest that sometimes it returns a ridiculous amount of text, which is great as someone who is generating stories).

So I guess my question remains -- is there a way you as a developer can get past that limit of 1024 or is it something hardcoded into Kobaldai?

Thanks for all your work on this BTW!!

[LostRuins]

Uh yeah, I hard limited it to cap at 1024 as larger numbers may cause issues running OOM. I can probably increase that limit further - but is there a reason why simply hitting the "Generate" button a second time to continue is less preferred?

[dogjamboree]

When I'm writing a story and using llama.cpp, I set the limit to -1 (infinite), which means sometimes it will generate a ridiculous amount of text, like 5000 tokens, 10,000 tokens, or just keep going forever (and I have to control-c the program of course)

You're right, this does cause my swap to go crazy BUT there's a real advantage to this over hitting generate again and that is the entire story is completely coherent within the context of the original prompt. Because I use very long prompts and so eventually that context is lost and I start getting worse stories.

Anyway, Unfortunately I'm still using llama.cpp instead of your excellent implementation because of this limitation. Because I love having more control over all the parameters.

Which brings up something else which I should prob just open another question / suggestion for, but is there any way to pass llama.cpp flags into kobaldai? Because as you know there a several more esoteric flags that can''t be triggered via the kobaldai interaface like --keep, repeat last, etc (Unless I'm just missing them?).

[LostRuins]

KoboldAI handles context is a different way from llama.cpp. While they have a few options to control the textgen, almost everything we have regarding prompt shaping is powered from within the UI itself. So a --keep option would not make sense, that is already done (at a dynamic level) by editing the "Memory" field within the UI.

As for your example, actually if you look through the llama.cpp source code for infinite generation and long prompts you'll realize it's no different too - it simply cuts away the first half of the story from the context, shifts the rest up, and regenerates everything from that point.

[dogjamboree]

Thanks for the explanation. Obviously I don't understand how KobaldAI works, I was just guessing about how both programs worked based on what I observed.

Is there documentation for this stuff anywhere? I quick search didn't turn anything up. I would love to know how that memory field works for example (because it didn't seem to do anything when I trired it).

Maybe you could tell me if you get a free second, how to accomplish what I need. Basically my prompts are quite long and consists of a short paragraph with a few sentences, followed by a long numbered list of plot elements of things I want to include. My problem is I find that many of those plot elements get ignored by the model (not every time but randomly) regardless of whether I use straight llama.cpp or KobaldAI, although I have to say I seem to have slightly better luck with llama.cpp.

So if there's something I could be doing differntly I would love to hear it. Like I said, I tried using that memory field before by putting my list there and just giving a very short prompt but it didn't take ANY of those elements into consideration when it generated the story. Not sure if maybe it's only used in certain modes?

Anyway, sorry for the long comment, and thanks again for all your work!

[LostRuins]

Hmm may I suggest joining the KoboldAI discord? We have plenty of helpful guides on the best ways to format your prompts/stories, and the people here are generally helpful and glad to assist.

https://koboldai.org/discord

[Midaychi]

Alpacino is an example of an odd edge case that would make sense to uncap the generation output for.
It is a model optimized for a very specific use of instruct, caps its own generation off, and set up for for "~680 or greater" token output using godlike, or storywriter+1.1 temp outputs.
And I suppose there's nothing stopping you from just continuing the prompts; it's just annoying because of how the instruct mode is setup.

(here's the base model for the modelcard, and they only recently added their own ggml variant, but there are also other ggml variants on huggingface that work fine) https://huggingface.co/digitous/Alpacino30b

[LostRuins]

You can definitely set the output to 680 tokens, just manually edit the number, which allows you to set it up to 1024.

Note that the main KoboldAI client also crashes if generated tokens > context length, so that's also something to be careful of.

[Midaychi]

I legit did not know that was an editable field until you pointed it out.

[LostRuins]

It's usually not meant to be, the sliders represent the recommend range of values. But there is some leeway for people who wish to push the limit.

[Midaychi]

Works great at 1024 tokens and doesn't even tend to generate all of them. Though this is all kind of moot because I guess I mis-read the instructions and its meant to be used in chat mode; just with an instruct setup... instructions unclear reboot cucumber.

[gustrd]

Works great at 1024 tokens and doesn't even tend to generate all of them. Though this is all kind of moot because I guess I mis-read the instructions and its meant to be used in chat mode; just with an instruct setup... instructions unclear reboot cucumber.

@Midaychi, give a look at my PR and fork, I implemented parameters to upper cap tokens and context. I was able the run at KoboldAI max.