Add support for chat completion API

[peakji]

Checklist

• Regardless of whether the last role in messages is user or assistant, the response will always be assistant.
• When steam=true, the first returned event will always be {"role": "assistant"}.
• When steam=true, the specific finish_reason will be yielded as a separate event.

Compatibility

Parameter	Basaran	OpenAI	Default Value	Maximum Value
model	○	●	-	-
messages	●	●	[]	CHAT_MAX_PROMPT
min_tokens	●	○	0	CHAT_MAX_TOKENS
max_tokens	●	●	512	CHAT_MAX_TOKENS
temperature	●	●	1.0	-
top_p	●	●	1.0	-
n	●	●	1	CHAT_MAX_N
stream	●	●	false	-
stop	○	●	-	-
presence_penalty	○	●	-	-
frequency_penalty	○	●	-	-
logit_bias	○	●	-	-
user	○	●	-	-

Examples

Chat completion (n=1, stream=false)

{
    "id": "chatcmpl-6z5sqEUkSdUyWqsNFRyyJ1s7kCzpM",
    "object": "chat.completion",
    "created": 1680018644,
    "model": "gpt-3.5-turbo-0301",
    "usage": {
        "prompt_tokens": 12,
        "completion_tokens": 14,
        "total_tokens": 26
    },
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "I was created by a team of developers and engineers at OpenAI."
            },
            "finish_reason": "stop",
            "index": 0
        }
    ]
}

Chat completion (n=2, stream=false)

{
    "id": "chatcmpl-6z5sOxJ318FNEWVkDsRbsZPS7wexL",
    "object": "chat.completion",
    "created": 1680018616,
    "model": "gpt-3.5-turbo-0301",
    "usage": {
        "prompt_tokens": 12,
        "completion_tokens": 34,
        "total_tokens": 46
    },
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "I was created by a team of developers at OpenAI."
            },
            "finish_reason": "stop",
            "index": 0
        },
        {
            "message": {
                "role": "assistant",
                "content": "I was created by a team of developers at OpenAI, using advanced artificial intelligence and natural language processing technologies."
            },
            "finish_reason": "stop",
            "index": 1
        }
    ]
}

Chat completion (n=1, stream=true)

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"role":"assistant"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"I"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" was"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" created"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" by"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" a"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" team"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" of"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" developers"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" at"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" Open"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"AI"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"."},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: [DONE]

Chat completion (n=2, stream=true)

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"role":"assistant"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"I"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"role":"assistant"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"I"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" was"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" was"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" created"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" created"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" by"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" by"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" a"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" Open"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"AI"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" team"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"."},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" of"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" programmers"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" and"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" developers"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" at"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" Open"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"AI"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"."},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{},"index":1,"finish_reason":"stop"}]}

data: [DONE]

Text completion (n=1, stream=false)

{
    "id": "cmpl-6z60oeEN6Wf4O3Ad0OvAMtA92ELQd",
    "object": "text_completion",
    "created": 1680019138,
    "model": "text-davinci-003",
    "choices": [
        {
            "text": "\n\nThis is indeed a test",
            "index": 0,
            "logprobs": null,
            "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 5,
        "completion_tokens": 7,
        "total_tokens": 12
    }
}

[fardeon]

Prompt templating is available in v4.34:

We've added a new template feature for chat models. This allows the formatting that a chat model was trained with to be saved with the model, ensuring that users can exactly reproduce that formatting when they want to fine-tune the model or use it for inference. For more information, see our template documentation.