Bug: SPM tokenization breaks in at least one specific case.

[snichols]

What happened?

Consider this code snippet:

auto chat_ml_tokens = llama_tokenize(model, "<|im_start|><|im_end|>\n", false, true);
std::cout << "chat_ml_tokens found:";
for(const auto t : chat_ml_tokens) {
    std::cout << " " << t;
}

With the latest version this is generating the following output:

chat_ml_tokens found: 32001 32000 28705 13

In an earlier version of llama.cpp, the correct tokenization was generated:

chat_ml_tokens found: 32001 32000 13

This work is based on https://huggingface.co/cognitivecomputations/dolphin-2.8-mistral-7b-v02.

If I tokenize each component separately, I get the correct results for each token. However, tokenizing <|im_end|>\n results in an extra 28705 token in the output. Interestingly enough, <|im_start|>\n is also correct. There's something extra special about this <|im_end|>. I haven't methodically gone over previous commits to see when this problem was introduced. Let me know if that'll help narrow the cause down.

I'm pretty confident that I can work around this problem just by tokenizing each element separately. I'll do that and run the model through some tests. That being said, there may be some other tokenization issues in the code that are being surfaced by this.

Name and Version

This is a custom app using tag b3040 of llama.cpp. https://github.com/ggerganov/llama.cpp/releases/tag/b3040

What operating system are you seeing the problem on?

Linux

Relevant log output

No response

[ggerganov]

The 2 tokens <|im_start|> and <|im_end|> in this model have a different special property - one is true the other is false:

https://huggingface.co/cognitivecomputations/dolphin-2.8-mistral-7b-v02/blob/6e0cd64b2c341a2328de00af2cd9d74cfcc89b74/tokenizer_config.json#L30-L45

Looks like a mistake in the model configuration to me

[snichols]

That is strange. I'll convert the model again with that fixed.

I'm not sure why that'd cause this problem though. Because it's <|im_end|>\n that's exhibiting the bug and that token is marked special.

[jaime-m-p]

I found more known problems. Fails with "</s> a".
The property rstrip is different.

phi-3/tokenizer_config.json:

    "2": {
      "content": "</s>",
      "lstrip": false,
      "normalized": false,
      "rstrip": true,  <--
      "single_word": false,
      "special": false
    },

dolphin-2.8-mistral-7b-v02/tokenizer_config.json:

    "2": {
      "content": "</s>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,  <--
      "single_word": false,
      "special": true
    },

In phi-3 all added tokens have rstrip: true except UNK <unk>, BOS <s> and EOS <|endoftext|>.
in dolphin-2.8-mistral-7b-v02 all added tokens have rstrip: false,
Also llama-spm has all added tokens rstrip to false, but seems to works.

I don't see any way to solve this correctly without having per-token flags/properties.

The only thing I can do is to initialize rstrip to false (this should revert to previous behavior).
Then use a harcoded IF model_name == "phy-3: ..." to fix other models.

[snichols]

It seems that if we want to be able to handle arbitrary models then we need to level up the tokenizer to take all of the data into account. Hard-coding model names in the tokenizer might be a quick fix, but it's gonna need ongoing maintenance. That's fragile and messy. But, it's better than it just plain not working I guess.

In my case, I'm skipping llama_tokenize and going right to the low-level tokenizer to take control over the process.

[jaime-m-p]

Hard-coding model names in the tokenizer might be a quick fix, but it's gonna need ongoing maintenance. That's fragile and messy. But, it's better than it just plain not working I guess.

I agree, meanwhile #7685.

[giladgd]

@snichols I think #7697 will solve the issue you encountered.
Can you please check that?

[jaime-m-p]

@snichols Should work now (3b38d48).

[snichols]

@jaime-m-p @giladgd I'm verifying now.

[snichols]

The problem persists in 3b38d4. No change.

[jaime-m-p]

In an earlier version of llama.cpp, the correct tokenization was generated:
chat_ml_tokens found: 32001 32000 13

@snichols Actually this is the unexpected output.

The correct output seems to have an additional space:

model = "./models/tokenizers/dolphin-2.8-mistral-7b-v02"
tokenizer = AutoTokenizer.from_pretrained(model)

text = "<|im_start|><|im_end|>\n"
ids  = tokenizer.encode(text, add_special_tokens=False)
re   = tokenizer.decode(ids)

print(repr(text))  # '<|im_start|><|im_end|>\n'
print(repr(re))    # '<|im_start|><|im_end|> \n'
print(ids)         # [32001, 32000, 28705, 13]

[snichols]

@jaime-m-p Well, then all is as expected. Woot!