Fix + Test

[LysandreJik]

Fix an edge case of the blenderbot-90 tokenizer.

Closes #8029

Context

If the blenderbot-90 tokenizer is used to tokenize the following sequence:

sequence = "Ok ."

It will split it in two tokens at first:

transformers/src/transformers/tokenization_blenderbot.py

Line 221 in 8bbe824

split_tokens.extend([t for t in self.bpe(token).split(" ")])

Those two tokens will be ['Ok', '.']

The issue is that, when passed the second token, the bpe method will convert it from '.' to ' .' here:

transformers/src/transformers/tokenization_blenderbot.py

Line 160 in 8bbe824

token = re.sub("([.,!?()])", r" \1", token)

This then gets split on spaces here:

transformers/src/transformers/tokenization_blenderbot.py

Line 166 in 8bbe824

tokens = token.split(" ")

This is where the issue lies, as it creates two strings: ["", "."], the first one being empty.

It then crashes a bit further as we try to index the empty string:

transformers/src/transformers/tokenization_blenderbot.py

Line 171 in 8bbe824

word = tuple(list(word[:-1]) + [word[-1] + "</w>"])

Proposal

Ensure that the token has a length > 0 before trying to manage it, otherwise ignore that token.

Added a test.

[sshleifer]

Great catch!

[sshleifer]

if not token also works

[LysandreJik]

You're right!