build: Extend tokenizer capabilities

[benITo47]

Description

This PR introduces rebuilt binaries that contain new, updated tokenizers.
This iteration features support for more tokenisation models (i.e. unigram, worldlevel) as well as bunch of previously unsupported pre-tokenisers, decoders, post-processors.

Introduces a breaking change?

• Yes
• No

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Other (chores, tests, code style improvements etc.)

Tested on

• iOS
• Android

Testing instructions

Before merging, test all demo applications. See if all models that proved problematic during bumps in the past are working (i.e. kokoro, multi-method models)
Check all LLM models, see if output is working.

• LLM app on iOS
• LLM app on Android
• Speech app on iOS
• Speech app on Android
• Text Embeddings on iOS
• Text Embeddings on Android

Screenshots

Related issues

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

[msluszniak]

Ok, tested all Android demo apps and they worked. Unfortunately, I don't have any iOS with me. We need someone to test iOS as well and then, I think we are ready to ship it.

[chmjkb]

Ok, tested all Android demo apps and they worked. Unfortunately, I don't have any iOS with me. We need someone to test iOS as well and then, I think we are ready to ship it.

ill take a look tomorrow

[chmjkb]

is there any particular tokenizer this should be tested with?

[msluszniak]

is there any particular tokenizer this should be tested with?

Yes, unigram. You can test it by running model from this PR: #1115

[msluszniak]

🚀