Deepseek tokenizer produces incorrect results as of v5 (works in v4)

[xenova]

System Info

• transformers version: 5.3.0
• Platform: Linux-6.6.113+-x86_64-with-glibc2.35
• Python version: 3.12.12
• Huggingface_hub version: 1.6.0
• Safetensors version: 0.7.0
• Accelerate version: 1.13.0
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (accelerator?): 2.10.0+cpu (NA)
• Using distributed or parallel set-up in script?:

Who can help?

@ArthurZucker and @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('deepseek-ai/DeepSeek-R1')

text = "How are you doing?"
print(tokenizer.encode(text))
print(tokenizer.tokenize(text))
print(tokenizer.decode(tokenizer.encode(text)))

produces

[4117, 591, 12829, 62552, 33]
['How', 'are', 'you', 'doing', '?']
Howareyoudoing?

Expected behavior

Downgrading to transformers==4.57.6, running the same code as above produces

[0, 4117, 477, 440, 4843, 33]
['How', 'Ġare', 'Ġyou', 'Ġdoing', '?']
<｜begin▁of▁sentence｜>How are you doing?

[xenova]

possibly related:
loading and saving again causes these differences between: original and re-saved.

diff original.json new.json

7369c7369
<   "normalizer": { "type": "Sequence", "normalizers": [] },
---
>   "normalizer": null,
7371,7399c7371,7374
<     "type": "Sequence",
<     "pretokenizers": [
<       {
<         "type": "Split",
<         "pattern": { "Regex": "\\p{N}{1,3}" },
<         "behavior": "Isolated",
<         "invert": false
<       },
<       {
<         "type": "Split",
<         "pattern": { "Regex": "[一-龥぀-ゟ゠-ヿ]+" },
<         "behavior": "Isolated",
<         "invert": false
<       },
<       {
<         "type": "Split",
<         "pattern": {
<           "Regex": "[!\"#$%&'()*+,\\-./:;<=>?@\\[\\\\\\]^_`{|}~][A-Za-z]+|[^\r\n\\p{L}\\p{P}\\p{S}]?[\\p{L}\\p{M}]+| ?[\\p{P}\\p{S}]+[\r\n]*|\\s*[\r\n]+|\\s+(?!\\S)|\\s+"
<         },
<         "behavior": "Isolated",
<         "invert": false
<       },
<       {
<         "type": "ByteLevel",
<         "add_prefix_space": false,
<         "trim_offsets": true,
<         "use_regex": false
<       }
<     ]
---
>     "type": "Metaspace",
>     "replacement": "▁",
>     "prepend_scheme": "always",
>     "split": false
7402,7419c7377,7380
<     "type": "TemplateProcessing",
<     "single": [
<       { "SpecialToken": { "id": "<｜begin▁of▁sentence｜>", "type_id": 0 } },
<       { "Sequence": { "id": "A", "type_id": 0 } }
<     ],
<     "pair": [
<       { "SpecialToken": { "id": "<｜begin▁of▁sentence｜>", "type_id": 0 } },
<       { "Sequence": { "id": "A", "type_id": 0 } },
<       { "SpecialToken": { "id": "<｜begin▁of▁sentence｜>", "type_id": 1 } },
<       { "Sequence": { "id": "B", "type_id": 1 } }
<     ],
<     "special_tokens": {
<       "<｜begin▁of▁sentence｜>": {
<         "id": "<｜begin▁of▁sentence｜>",
<         "ids": [0],
<         "tokens": ["<｜begin▁of▁sentence｜>"]
<       }
<     }
---
>     "type": "ByteLevel",
>     "add_prefix_space": true,
>     "trim_offsets": false,
>     "use_regex": true
7422,7425c7383,7389
<     "type": "ByteLevel",
<     "add_prefix_space": true,
<     "trim_offsets": true,
<     "use_regex": true
---
>     "type": "Sequence",
>     "decoders": [
>       { "type": "Replace", "pattern": { "String": "▁" }, "content": " " },
>       { "type": "ByteFallback" },
>       { "type": "Fuse" },
>       { "type": "Strip", "content": " ", "start": 1, "stop": 0 }
>     ]
7433,7434c7397,7398
<     "fuse_unk": false,
<     "byte_fallback": false,
---
>     "fuse_unk": true,
>     "byte_fallback": true,

[sukhmansaran]

I investigated this issue and can confirm it is caused by a tokenizer pipeline mismatch introduced when the tokenizer is re-saved in transformers v5.

Root cause

The original DeepSeek tokenizer uses a ByteLevel-based pipeline:

• pre_tokenizer: Sequence(Split..., ByteLevel)
• decoder: ByteLevel
• post_processor: TemplateProcessing (with BOS token)

However, in transformers v5 this gets replaced with:

• pre_tokenizer: Metaspace(split=False)
• decoder: Sequence(Replace + ByteFallback + Fuse + Strip)
• post_processor: changed to ByteLevel

This change is incompatible with the existing vocabulary.

Specifically, the vocab does not contain metaspace-prefixed tokens:

• 'How' → valid token
• '▁How' → not in vocab

Because of this, spaces are never encoded in the token stream, and the decoder cannot reconstruct them, leading to outputs like:

Howareyoudoing?

Why decoder-only fixes don’t work

This is not just a decoding issue. Since the metaspace pre-tokenizer never produces tokens aligned with the vocab, the whitespace information is already lost before decoding. Any decoder-level fix will be insufficient.

Fix

Restoring the original ByteLevel pipeline resolves the issue completely. The required changes are:

• Restore pre_tokenizer to Sequence(Split..., ByteLevel)
• Restore decoder to ByteLevel
• Restore TemplateProcessing (BOS handling)

After applying this, decoding becomes lossless again:

decode(encode(text)) ==text

across:

• punctuation
• whitespace
• mixed inputs

Minimal patch

from tokenizers import Regex
from tokenizers.pre_tokenizers import Sequence, Split, ByteLevel as BLPreTok
from tokenizers.processors import TemplateProcessing
from tokenizers.decoders import ByteLevel as BLDecoder

def fix_deepseek_tokenizer(tokenizer):
    backend = tokenizer.backend_tokenizer

    backend.pre_tokenizer = Sequence([
        Split(pattern=Regex(r"\p{N}{1,3}"), behavior="isolated"),
        Split(pattern=Regex(r"[一-龥぀-ゟ゠-ヿ]+"), behavior="isolated"),
        Split(
            pattern=Regex(
                r"""[!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~][A-Za-z]+"""
                r"""|[^\r\n\p{L}\p{P}\p{S}]?[\p{L}\p{M}]+"""
                r"""| ?[\p{P}\p{S}]+[\r\n]*"""
                r"""|\s*[\r\n]+"""
                r"""|\s+(?!\S)"""
                r"""|\s+"""
            ),
            behavior="isolated",
        ),
        BLPreTok(add_prefix_space=False, trim_offsets=True, use_regex=False),
    ])

    bos = "<｜begin▁of▁sentence｜>"
    bos_id = tokenizer.convert_tokens_to_ids(bos)
    backend.post_processor = TemplateProcessing(
        single=f"{bos}:0 $A:0",
        pair=f"{bos}:0 $A:0 {bos}:1 $B:1",
        special_tokens=[(bos, bos_id)],
    )

    backend.decoder = BLDecoder(
        add_prefix_space=True,
        trim_offsets=True,
        use_regex=True,
    )

    return tokenizer


### Result

With this fix applied:

- decode(encode(text)) matches the original input (modulo BOS token)

- behavior matches transformers v4 and the original tokenizer.json

[ArthurZucker]

Yeah this is just a remote code model to be added to

transformers/src/transformers/models/auto/tokenization_auto.py

Line 342 in 3491f11

MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS: set[str] = {

[ArthurZucker]

cc @itazap as I think you are working on extanding the list with a test script>?