Add GPT 4.1 to Tiktoken Tokenizer

[fanyang-mono]

I would like to see GPT 4.1 being added to Tiktoken Tokenizer.

[stephentoub]

@tarekgh, looks like OpenAI stated yesterday that gpt-4.1 uses the same tokenizer as 4o.
https://community.openai.com/t/whats-the-tokenization-algorithm-gpt-4-1-uses/1245758/2

[tarekgh]

@stephentoub I was waiting for OpenAI to officially add it to their tokenizer library openai/tiktoken#395.

[fanyang-mono]

@tarekgh Thanks for adding the support. I wonder when this will be available to be consumed?

[tarekgh]

@fanyang-mono the change will be included in our next preview (hopefully in 10 days or so). But you don't have to be blocked on that as you can still create the tokenizer using other compatible model name. Like

Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4o");

or you can create it from the encoding name like:

Tokenizer tokenizer = TiktokenTokenizer.CreateForEncoding("o200k_base");

Both ways should create a tokenizer that can be used with gpt4.1 model. Let me know if you have any question.