feat(exllama2): Add support to exllama2

[mudler]

exllama2 has just been released: https://github.com/turboderp/exllamav2

This card is a tracker to add it to LocalAI, similarly to exllama

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

Sure! I can help you with that.
To add Exllama2 as a backend in LocalAI, you need to follow these steps:

1. Install Exllama2 on your machine if you are building LocalAI locally. You can do this by following the instructions on the official Exllama2 GitHub page: https://github.com/turboderp/exllamav2#installation
2. Download the model as a folder inside the model directory and create a YAML file specifying the Exllama2 backend. For instance with TheBloke/WizardLM-7B-uncensored-GPTQ model:

$ git lfs install
$ cd models && git clone https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GPTQ
$ ls models/                                                                 
.keep                        WizardLM-7B-uncensored-GPTQ/ exllama.yaml
$ cat models/exllama.yaml                                                     
name: exllama2
parameters:
  model: WizardLM-7B-uncensored-GPTQ
backend: exllama2
# ...

3. Update the models directory to include the Exllama2 backend in the exllama.yaml file. You can do this by adding a new line to the Yaml file with the name of the backend and its parameters:

name: exllama2
parameters:
  model: WizardLM-7B-uncensored-GPTQ
backend: exllama2

4. Build the LocalAI container image with the Exllama2 backend included. You can do this by running the following command in your terminal:

docker build -t localai/exllama2 .

This will create a new container image for LocalAI with the Exllama2 backend included.
5. Test the model using the new container image. You can do this by running the following command in your terminal:

docker run -v /path/to/models:/models -v /path/to/data:/data localai/exllama2

Replace /path/to/models and /path/to/data with the paths to your own LocalAI models and data directories, respectively. This will start a new container instance of LocalAI with the Exllama2 backend and load the WizardLM-7B-uncensored-GPTQ model. You can then use this instance to train or run inference on your own data.
I hope that helps! Let me know if you have any further questions.

Sources:

• https://localai.io/basics/news/index.html
• https://localai.io/model-compatibility/exllama/index.html
• https://localai.io/integrations/index.html

[Aisuko]

We already have the example of exllama. I will try this one.

[Aisuko]

ExLlama2 needs CUDA 117 or 118. I have been tried to install it on older coda, but it cannot work well. So, this one maybe need to work on these two environment.