Grammar support for Web Server

[dthuerck]

Created this small PR which just adds a grammar parameter to the completions endpoint, therefore closing #778 . Of course this violates OpenAI's API definition, but the parameter is optional. Here's an example for it to test (use the json.gnbf from llama.cpp):

import json
import requests

if __name__ == "__main__":

  prompt = "Please provide me with a valid JSON that encodes the following " \
    "Address: Testine Testerson, 420 Imaginary St, 12345 Star City, USA. " \
    "Make sure yu respect data types and only use one per field. "

  print("######")
  print("# WITHOUT GRAMMAR")
  print("######")

  # simple post request asking for a code problem
  res = requests.post(
    url="http://localhost:8000/v1/completions",
    json={
      "prompt" : f"<s>[INST]{prompt}[/INST]",
      "stop" : ["</s>"],
      "max_tokens" : 256,
      "stream" : True,
      "temperature" : 0.9
    },
    stream=True
  )

  for line in res.iter_lines(decode_unicode=True):
    if line:
      try:
        line = line[6:]
        data = json.loads(line)
        print(data["choices"][0]["text"], end="", flush=True)
      except Exception as e:
        pass

  print("")

  print("######")
  print("# WITH GRAMMAR")
  print("######")

  # now try with using the grammar
  with open("json.gnbf") as f:
    grammar = f.read()

  res = requests.post(
    url="http://localhost:8000/v1/completions",
    json={
      "prompt" : f"<s>[INST]{prompt}[/INST]",
      "stop" : ["</s>"],
      "max_tokens" : 256,
      "stream" : True,
      "temperature" : 0.8,
      "grammar" : grammar
    },
    stream=True
  )

  for line in res.iter_lines(decode_unicode=True):
    if line:
      try:
        line = line[6:]
        data = json.loads(line)
        print(data["choices"][0]["text"], end="", flush=True)
      except Exception as e:
        pass

[Freed-Wu]

@dthuerck you should close #788 not #778

[aabbi]

Thanks for adding this. The parameter can be passed as an extra_body parameter when using the openai client. Would it make sense to add this in the documentation somewhere (I had to dig a little bit to figure out how to do this) ?
Example usage

llm_uri = "http://localhost:8000/v1"
llm_client = OpenAI(base_url=llm_uri, api_key="xx")
modelname = "llama-2-7b-chat.Q6_K.gguf"
with open("json.gnbf") as f:
    grammar = f.read()
resp = llm_client.chat.completions.create(messages=prompt, model=modelname, extra_body={"grammar": grammar})