Draft: feat: Support Llama 3 model (#478)

[reneleonhardt]

Wow, fresh of the press, seems to be one of the best models now!! 🚀

Waiting for llama.cpp support... 😅
ggml-org/llama.cpp#6747
ggml-org/llama.cpp#6751

[carlrobertoh]

Is there any more work to be expected here, or can we perhaps remove the "Draft" prefix?

[reneleonhardt]

As far as I can see it's finished (can you double-check the InfillPrompt?), I'm just waiting for the llama.cpp server support 😅
Or can you check it locally? In LM Studio I did see no big problems, the model is amazing, I just have to ask it to tell the truth when it fantasized to me about non-existing Kotlin operators 😉
7B Q8 is super fast on M1 Max 🚀

[carlrobertoh]

Nice! It looks like the model doesn't support infilling, or at least I couldn't find anything. Maybe we can just remove the new infill template for now.

[reneleonhardt]

Nice! It looks like the model doesn't support infilling, or at least I couldn't find anything. Maybe we can just remove the new infill template for now.

Really? All main calls I saw contained --in-prefix and suffix ggml-org/llama.cpp#6747 (comment) that's why I added it 😅
But if you think it's a problem I can remove it for now 🙂

[carlrobertoh]

Hmm, their model card doesn't specify anything related to that matter - https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3

[reneleonhardt]

All I can see is that there is a lot of confusion about the model, some people have many problems, some have none... I guess it depends on how you use it, and some HF models also seem to contain problems.
phymbert mentioned a special infill endpoint ggml-org/llama.cpp#6747 (comment), is CodeGPT using that?

[carlrobertoh]

No, but we were using it for a while, I think. However, I can't remember the reason why we switched back.