Feature Request: Support for Vicuna finetuned model

[lolxdmainkaisemaanlu]

https://vicuna.lmsys.org/ - "Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality"

This model works amazing, It also has 2048 context size! But it needs the prompt formatted in this format:

" A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.

Human: Hello, Assistant.

Assistant: Hello. How may I help you today?

Human: Please tell me the largest city in Europe.

Assistant: Sure. The largest city in Europe is Moscow, the capital of Russia. "

Right now the instruct mode seems to be hardcoded with Alpaca style formatting of ### Instruction: and ### Response:. Would really appreciate if this feature was added. Thanks in advance.

[LostRuins]

This can be done - can you provide a template for how Vicuna's instruct mode works? Would also be good if you had one for OpenAssistant - I hear there are many different formats with unique templates and I cannot keep track of them all.

[lolxdmainkaisemaanlu]

Sir the instruct mode of Vicuna works in a similar manner to that of Alpaca but instead of ### Instruction and ### Response , it is ### Human and ### Assistant like in my initial post. And the system message is: A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.

[LostRuins]

This has been added in the latest version, instruct prompt is now customizable

[dogjamboree]

I confirmed that this fix works -- thanks!

BTW, is there any way you can make it so the instruct mode has a streaming option like some of the other modes do? I mean so the tokens appear as they're generated?

[LostRuins]

It should work in instruct mode too. Just make sure the url has a ?streaming=1 e.g. http://localhost:5001?streaming=1

[lolxdmainkaisemaanlu]

Thank you for implementing support but the system message is still Below is an instruction that describes a task. Write a response that appropriately completes the request. instead of A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions..

I feel Vicuna responds more accurately to the 2nd system message. It would be great if you could add this feature too.

[lolxdmainkaisemaanlu]

In the new Vicuna 1.1 model, there is </s> (the EOS token) marking the end of an exchange.

Old format:

"### Human: blahblah blah

'### Assistant: blahblah blah"

New format:

"USER: blahblah blah

ASSISTANT: blahblah blah</s>"

Can you please implement this feature? thanks in advance

[LostRuins]

In the next release, the Alpaca instruction header will not be included by default if the "Memory" field of the UI is populated. Thus, you will be able to setup a custom instruction header by filling your own text e.g. from Vicuna into the memory input.

As for the EOS token, it is banned by default and will not be generated, unless a specific model has changed the EOS token ID.