Add Byte Pair Encoding (BPE) class for subword tokenization

[Cydral]

Description:

This PR introduces a new bpe_tokenizer class to Dlib, implementing the Byte Pair Encoding (BPE) algorithm for subword tokenization. The BPE tokenizer is a widely used technique in natural language processing (NLP) for handling out-of-vocabulary words and reducing vocabulary size while maintaining text representation capabilities.

Key Features:

• BPE Algorithm: Implements the BPE algorithm as described in Sennrich et al., 2016.
• Special Tokens: Supports predefined special tokens (e.g., <text>, <url>, <image>) for marking specific elements in the text.
• Training and Encoding: Provides methods for training the tokenizer on a text corpus and encoding/decoding text into subword tokens.
• Serialization: Supports saving and loading the tokenizer model and vocabulary for reuse.
• Thread-Safe: Utilizes multi-threading for efficient frequency statistics computation during training.

Usage:

dlib::bpe_tokenizer tokenizer;
tokenizer.train(corpus_text, target_vocab_size, true); // Train on a text corpus
std::vector<int> tokens = tokenizer.encode("Sample text to tokenize."); // Encode text
std::string decoded_text = tokenizer.decode(tokens); // Decode tokens back to text

[davisking]

Nice, this is great. Sorry it took so long for me to get back to this.

[Cydral]

No problem. I think this is another great new feature for our library.

[davisking]

No problem. I think this is another great new feature for our library.

Indeed 😁