[ignore_merges] Fix offsets

[ArthurZucker]

>>> import tokenizers
>>> from tokenizers import Tokenizer
>>> t = Tokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
>>> t.encode("are you ok").offsets
[(0, 0), (0, 3), (3, 7), (7, 10)]

fixes #1553, fixes #1517, fixes huggingface/transformers#33675, fixes #1620

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[thepowerfuldeez]

Hi @ArthurZucker can you push a hotfix to pip as a 0.20.1 version? Seems like installing from source is complicated for tokenizers :)

[ArthurZucker]

On it!