Fix KeyError in convert_to_native_format for dict vocab

[weiguangli-io]

Fix KeyError in convert_to_native_format for dict vocab

Fixes #44451

Problem

AutoTokenizer.from_pretrained("vesteinn/ScandiBERT") raises KeyError: 0 in convert_to_native_format.

ScandiBERT's tokenizer_config.json specifies tokenizer_class: "XLMRobertaTokenizer", which has model = Unigram. When convert_to_native_format reads the tokenizer.json, it enters the Unigram branch and executes:

if vocab and isinstance(vocab[0], (list, tuple)):

However, the vocab from tokenizer.json is a dict ({"token": score, ...}), not a list. Indexing a dict with [0] raises KeyError because there is no key 0.

Fix

Add an isinstance(vocab, list) check before attempting to index vocab[0]:

if isinstance(vocab, list) and vocab and isinstance(vocab[0], (list, tuple)):

This is consistent with the guards used in the other branches (e.g., BPE/WordPiece branch on line 163). When vocab is already a dict, no conversion is needed and it is passed through as-is.

Regression

Introduced in v5 by #42894 (use TokenizersBackend).

[Rocketknight1]

cc @itazap @ArthurZucker since it seems like a tokenizer regression (see also the original issue at #44451)

[itazap]

good catch, thank you!

[AngledLuffa]

Thank you, looking forward to it! This was the best model I'd found so far for annotation tasks on DA and IS. (Will keep exploring other options anyway)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[AngledLuffa]

Can this be merged? It would be great to use this transformer again with newer versions of the package

[Rocketknight1]

Looks like it was ready to go and just had automerge issues, will try to merge it.

[AngledLuffa]

That's great, thank you!

[AngledLuffa]

Is this fixable?