How to integrate this into our own applications?

[trevtravtrev]

chatGPT has an API that allows you to easily integrate it into your own applications. alpaca.cpp currently runs in the terminal, but is there an existing way to call it from our own programs?

The only way I've seen this to be possible currently is via a terminal command similar to below (#103):
.\Release\chat.exe -p "write a python script that prints the current time" --interactive

The problem with this solution is that it requires restarting/reloading the entire model for every prompt you ask it. This becomes increasingly time consuming and compute intensive the larger the weight you use as well.

Is there a better way to do this currently? If not, how can it be added?

[frenchmustard]

There's two parts to your question, the integration part I can't really answer due to my lack of knowledge on C++ but I can say that you can probably (and this is very inefficient mind you) open the executable as a process from a different script like use Popen in Python and then read/write to the chat.

As for the restarting/reloading: ChatGPT stores parts of the conversation that is relevant I believe, so in order for this to do the same you'd need to either implement some hefty algorithm that summarizes important parts of the convo (tokens are heavy) or, way less efficient, give back the history of the convo in the background with lots of "You said" and "I responded".

[dzid26]

Maybe ggml-org#278 (comment)