[Tokenization] fix edge case for bert tokenization

[patrickvonplaten]

This PR fixes #3502 .

The reason why the tests fail in #3502 is because of an edge case.

If the input to tokenizer.batch_encode_plus() consists of a tokenized string that results in a list of exactly two strings ([[16], [.]] in issue #3502) then it is treated as a pair of input sequences (=> [CLS] input_sequence_1 [SEP] input_sequence_2 [SEP]) but this behavior should only happen if the input list consists of two untokenized strings.

[patrickvonplaten]

@mfuntowicz @n1t0 @LysandreJik - could you check? :-)

[n1t0]

Does this mean that batch_encode_plus is supposed to handle "pre-tokenized" inputs? I thought this was something introduced by #3185 with a specific flag is_pretokenized (cc @mfuntowicz)

[codecov-io]

Codecov Report

Merging #3517 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3517      +/-   ##
==========================================
+ Coverage   78.03%   78.05%   +0.01%     
==========================================
  Files         104      104              
  Lines       17708    17709       +1     
==========================================
+ Hits        13819    13822       +3     
+ Misses       3889     3887       -2     

Impacted Files	Coverage Δ
src/transformers/tokenization_utils.py	85.78% <100.00%> (+0.01%)	⬆️
src/transformers/modeling_utils.py	92.23% <0.00%> (+0.12%)	⬆️
src/transformers/modeling_tf_utils.py	93.28% <0.00%> (+0.16%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5aa8a27...3bde162. Read the comment docs.

[patrickvonplaten]

Does this mean that batch_encode_plus is supposed to handle "pre-tokenized" inputs? I thought this was something introduced by #3185 with a specific flag is_pretokenized (cc @mfuntowicz)

@mfuntowicz showed me the is_pretokenized flag for tokenizers v3.0.0 so this makes everything much easier

[LysandreJik]

Woa that flag is useful!