Compatibility with Llama

[cycloarcane]

Is it possible to add out of the box functionality with OS models like llama 405b?

[2187Nick]

I tested it with 405b and works fine. Nvidia gives 1000 free credits. You can see here how to adjust the OpenAI client. https://build.nvidia.com/explore/discover#llama-3_1-405b-instruct

[Dang-Tu-lang]

@2187Nick do you give a example with using llama 3.1?

[2187Nick]

@dangyuuki123 Here is an example using Gemini.
https://github.com/2187Nick/ADAS/tree/main/examples

I will work on llama 3.1 version next.

[vmayoral]

@dangyuuki123 Here is an example using Gemini. https://github.com/2187Nick/ADAS/tree/main/examples

I will work on llama 3.1 version next.

That'd be lovely and would help receive further contributions from our side. As a side note, recommending using ollama, as a popular provider for llama-like models and general alignment with a wider community of users.

[2187Nick]

@dangyuuki123 @vmayoral Llama 405b is not producing consistent JSON. You can use my new example that uses free Nvidia hosted models which offers Llama 405b. If you are able to modify the model settings or prompt to get consistent JSON output, please let me know.

Here is a working example that uses Nvidia hosted Mistral Large 2:
https://github.com/2187Nick/ADAS/tree/main/examples/nvidia_arc/mistral_large2

I found Codestal-22b will also work.

[Bilgeworth]

Can be quick and dirty swapped if using local ollama, search and replace

client = openai.OpenAI()

change to

client = openai.OpenAI(
    base_url = 'http://localhost:11434/v1',
    api_key='ollama',
)

and change the parser.add_argument models from default to what you want, i.e.

parser.add_argument('--model',
                        type=str,
                        default='llama3.1',
                        choices=['mistral-nemo', 'gemma2', 'llama3.1'])

llama3.1 8B has inconsistencies that make the output iffy at best, gest caught on json format errors every single iteration.

You can run the optional models with for example:

python _arc/search.py --model gemma2