PoC: add chat template heuristics

[kallewoof]

The fallback chat template adapter of Vicuna is not ideal in some cases (e.g. a test against a sub-portion of the BBC news classification task on Kaggle gave an 82% accuracy with Vicuna and 88% with the official ChatML format for a q4_k_m Qwen 2.5 3B-Instruct gguf).

This PR adds a proof of concept simple heuristic which looks at the chat template and upgrades the adapter when it is able to.

• Determine where this should go. koboldcpp.py seems big enough as it is, but the project does not seem to split into .py files often/at all.
• Extend the cases to cover more chat templates, e.g. Llama 3.1, Mistral versions, etc.

Alternative approach: expose the llama chat template mechanism and use that.

[LostRuins]

The problem is the chat template cannot be trusted. It is set by unknown third parties, and very often straight up incorrect or misleading - that's the whole reason why I didn't use the jinja template to begin with. I'm alright with frontends using the /props endpoint to make their own pick, but I'm not sure overwriting the default is a good idea.

The reason why the default is Alpaca (not vicuna in fact) has to do with the fact that \n### Instruction:\n is a tokenizer resistant format. When using ChatML, for example, if the vocab doesn't have a <|im_start|> and <|im_end|> added tokens, the sequence gets split into an awful mess of tokens like [<,|,im,_,start,|,>] which severely degrades the output quality. Meanwhile, Alpaca, being plain English, is relatively unaffected, and the newline ensures that tokenizers with huge vocabs like Llama3 don't end up merging part of the instruct sequence into the input text tokens.

[kallewoof]

I see. When I was evaluating a bunch of different models against a test, I was completely unaware of the fact they were all using Alpaca (sorry, mixed them up) and when I switched, there was a significant increase in accuracy across the board. I understand not wanting to trust third parties willy nilly. Perhaps there can be a flag that lets users choose whether or not they want to trust it or not?

Edit: We can also add built in guard rails to deal with fuckery like unknown tokens. I.e. for every chat template profile, we include a list of tokens that must be present in the tokenizer. If any are missing, the chat template is not adopted.

[LostRuins]

Well, I think there's a nice way to do it.

What I can suggest instead is we create a new dummy file called AutoGuess.json as a built-in-adapter for chat completion adapters. This can be optionally selected by the user when they wish to apply heuristics.

(they can also simply type AutoGuess)

Then, the heuristics will apply.

[kallewoof]

Sounds good. I assume this can be selected via the Open AI API chat completion endpoint as well?

I do think we may want to include an option to default to using it as well, but that's not a blocker (I just have to add it to my requests). Edit: it looks like people can do --chatcompletionsadapter AutoGuess which is fine I suppose.

[kallewoof]

I added two new prints on startup, which come after the text model load:

1. Notifies user about failure to detect the chat completion adapter, and that Alpaca will be used. This is when heuristics are enabled, but no applicable heuristic was found.
2. Notifies user about Alpaca being used for OAI API chat completions, for when no --chatcompletionsadapter was provided.

[kallewoof]

Giving this some thought, I think this would do well as a json file with search string array -> chat template name + params. Then a for loop that checks each in the code. We could probably even use the AutoGuess.json file for this.

[kallewoof]

Better! Any adapter json file can now be a list with dicts with search, name, and adapter keys which will be searched for. Not sure how useful that is, but hey. This seems like a pretty seamless integration.

[LostRuins]

lgtm