Support Llama3.1 tokenization #56
Description
Llama3.1's tokenizer is also BPE-based. We already ship this library in VS Code's Copilot Chat extension and it would be great if Llama3.1 was supported so that we do not need to ship another library.
{{ message }}