[Feature Request] Additions to the OpenAI-compatible API

[AMDBartek]

First off, thank you for the great work! I've been using koboldcpp for a while now and it works pretty well.

I have a few feature suggestions for the APIs though:

• API keys: This would be great for when you want to expose the endpoint to the web. It would be good if multiple API keys could be used, perhaps they could be stored in a user specified file.
• Endpoint stuff: The ability to be able to select the endpoint type(s) you want, whether to only expose the Kobold API, the OpenAI-compatible API, or both.
• Chat template config file (and the ability to force it, ignoring the adapter object passed to the API and just using the config file): This would be really great so the code on the client side is cleaner, not having to specify the adapter (OpenAI compat API adapter #466).

I know this probably isn't koboldcpp's intended use case but this would be really nice to have. It would be a seamless replacement for the official OpenAI API.

Also, small question, with that adapter, would it be possible to use LLaMA 2 Chat templating as shown below?

[INST] <<SYS>>
You are a helpful assistant.
<</SYS>>

{prompt 0} [/INST] {response 0} </s><s>[INST] {prompt 1} [/INST] {response 1} </s><s>...[INST] {prompt N} [/INST]

[teddybear082]

On the endpoint one I disagree I think it’s actually cool that all the endpoints are available all the time with kobold without having to add a bunch of options to enable or disable them. You can hit it with a native kobold API request, a kobold API streaming request and a openai API chat/completions request in a row and it will respond to each.

I also like the adapter angle because it keeps kobold’s code more streamlined to maintain and not have to constantly worry about new templates. I originally had the same idea as you-build in all the chat templates - but models are coming out all the time and actually sometimes some models perform better or differently with different types of prompts so it’s nice the end user at this point can send their request with whatever exact prompt they want or otherwise it defaults to alpaca which so far in my experience gives great responses with the vast majority of models including most llama2.

Yes, I do believe you can replicate that prompt format you noted for llama 2 though with the current adapter because it has fields for system start and end, user start and end and response start and end but I haven’t specifically tried it yet because the default has been fine for me.

[LostRuins]

The adapter object is entirely optional, skipping it will just use the default tags which will work for most cases. I'd like to keep the API relatively straightforward without too many unnecessary bells and whistles if possible. Again, I do encourage power users to go for the Kobold API approach as it is far more versatile.

Yes, you can certainly approximate LLAMA 2 template. Check what Kobold Lite has for an example.

Handling authentication and keys is probably beyond the scope of KoboldCpp directly.

Instead, you might want to check out the Khanon OAI reverse proxy which is a proxy that can handle authentication between the web and another OAI compatible endpoint e.g. KoboldCpp, bridging and forwarding requests, while handling key management and authentication based on your setup.

[AMDBartek]

Thanks a lot for pointing me in the right direction! That reverse proxy looks very good and is definitely something I was looking for. I'll just force the LLaMA 2 template by modifying the reverse proxy to include my desired features such as automatically including its own adapter/various config objects to the koboldcpp request and some misc security features here and there.

All of this is because I use koboldcpp as a replacement for the OpenAI chat completions API in various applications or sharing it with other people with my own fine-tuned model, so the templating was quite important for me. So for my use case, the Kobold API isn't really something I used or could use even though it's better in some places. Before I was using a very heavily modified version of anon998/simple-proxy-for-tavern, but Khanon's OAI proxy does look way better for my purpose.

So yeah, that's this solved for me, and once again thank you for creating this project. Feel free to close the issue if you feel that implementing this directly into koboldcpp is out of scope, etc.

[LostRuins]

Update: API key support has been added