BPE pretokenizer - add support for command-r-plus and command-r models

[sealad886]

Fixes #7030
Fixes #7040

Added both command-r and command-r-plus pretokenizer regex's to the model, updated the files as recommended in (120cf37).

Tests compiled at ./tests/test-tokenizer-o passed for both models.

[dranger003]

There is already a PR for this #7033.

[github-actions]

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 550 iterations 🚀

Expand details for performance related PR only

• Concurrent users: 8, duration: 10m
• HTTP request : avg=8495.23ms p(95)=20849.85ms fails=, finish reason: stop=479 truncated=71
• Prompt processing (pp): avg=96.13tk/s p(95)=378.82tk/s
• Token generation (tg): avg=32.63tk/s p(95)=47.83tk/s
• ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=master commit=0d1a0164e7a86b63053230ae962abcc3cdaf6df7

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1714648467 --> 1714649097
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 619.38, 619.38, 619.38, 619.38, 619.38, 503.23, 503.23, 503.23, 503.23, 503.23, 533.6, 533.6, 533.6, 533.6, 533.6, 596.15, 596.15, 596.15, 596.15, 596.15, 633.36, 633.36, 633.36, 633.36, 633.36, 640.06, 640.06, 640.06, 640.06, 640.06, 657.46, 657.46, 657.46, 657.46, 657.46, 676.7, 676.7, 676.7, 676.7, 676.7, 696.37, 696.37, 696.37, 696.37, 696.37, 698.87, 698.87, 698.87, 698.87, 698.87, 736.94, 736.94, 736.94, 736.94, 736.94, 771.44, 771.44, 771.44, 771.44, 771.44, 823.53, 823.53, 823.53, 823.53, 823.53, 830.21, 830.21, 830.21, 830.21, 830.21, 834.12, 834.12, 834.12, 834.12, 834.12, 831.56, 831.56, 831.56, 831.56, 831.56, 847.0, 847.0, 847.0, 847.0, 847.0, 848.17, 848.17, 848.17, 848.17, 848.17, 853.2, 853.2, 853.2, 853.2, 853.2, 853.26, 853.26, 853.26, 853.26, 853.26, 852.58, 852.58, 852.58, 852.58, 852.58, 855.25, 855.25, 855.25, 855.25, 855.25, 849.78, 849.78, 849.78, 849.78, 849.78, 851.58, 851.58, 851.58, 851.58, 851.58, 852.04, 852.04, 852.04, 852.04, 852.04, 861.19, 861.19, 861.19, 861.19, 861.19, 859.48, 859.48, 859.48, 859.48, 859.48, 859.47, 859.47, 859.47, 859.47, 859.47, 863.12, 863.12, 863.12, 863.12, 863.12, 863.69, 863.69, 863.69, 863.69, 863.69, 861.9, 861.9, 861.9, 861.9, 861.9, 857.38, 857.38, 857.38, 857.38, 857.38, 852.8, 852.8, 852.8, 852.8, 852.8, 860.08, 860.08, 860.08, 860.08, 860.08, 858.72, 858.72, 858.72, 858.72, 858.72, 857.89, 857.89, 857.89, 857.89, 857.89, 855.89, 855.89, 855.89, 855.89, 855.89, 857.93, 857.93, 857.93, 857.93, 857.93, 858.22, 858.22, 858.22, 858.22, 858.22, 858.07, 858.07, 858.07, 858.07, 858.07, 830.46, 830.46, 830.46, 830.46, 830.46, 828.38, 828.38, 828.38, 828.38, 828.38, 808.55, 808.55, 808.55, 808.55, 808.55, 807.84, 807.84, 807.84, 807.84, 807.84, 806.15, 806.15, 806.15, 806.15, 806.15, 810.81, 810.81, 810.81, 810.81, 810.81, 810.44, 810.44, 810.44, 810.44, 810.44, 813.46, 813.46, 813.46, 813.46, 813.46, 815.81, 815.81, 815.81, 815.81, 815.81, 819.77, 819.77, 819.77, 819.77, 819.77, 822.75, 822.75, 822.75, 822.75, 822.75, 823.54, 823.54, 823.54, 823.54, 823.54, 825.21, 825.21, 825.21, 825.21, 825.21, 829.8, 829.8, 829.8, 829.8, 829.8, 830.9, 830.9, 830.9, 830.9, 830.9, 829.77, 829.77, 829.77, 829.77, 829.77, 831.07, 831.07, 831.07, 831.07, 831.07, 832.6, 832.6, 832.6, 832.6, 832.6, 833.32, 833.32, 833.32, 833.32, 833.32, 835.55, 835.55, 835.55, 835.55, 835.55, 835.87]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1714648467 --> 1714649097
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 42.47, 42.47, 42.47, 42.47, 42.47, 33.89, 33.89, 33.89, 33.89, 33.89, 32.43, 32.43, 32.43, 32.43, 32.43, 35.81, 35.81, 35.81, 35.81, 35.81, 35.12, 35.12, 35.12, 35.12, 35.12, 35.75, 35.75, 35.75, 35.75, 35.75, 36.7, 36.7, 36.7, 36.7, 36.7, 36.83, 36.83, 36.83, 36.83, 36.83, 36.66, 36.66, 36.66, 36.66, 36.66, 36.19, 36.19, 36.19, 36.19, 36.19, 35.94, 35.94, 35.94, 35.94, 35.94, 35.0, 35.0, 35.0, 35.0, 35.0, 34.13, 34.13, 34.13, 34.13, 34.13, 33.47, 33.47, 33.47, 33.47, 33.47, 33.69, 33.69, 33.69, 33.69, 33.69, 34.01, 34.01, 34.01, 34.01, 34.01, 33.5, 33.5, 33.5, 33.5, 33.5, 33.49, 33.49, 33.49, 33.49, 33.49, 33.16, 33.16, 33.16, 33.16, 33.16, 32.77, 32.77, 32.77, 32.77, 32.77, 32.65, 32.65, 32.65, 32.65, 32.65, 32.72, 32.72, 32.72, 32.72, 32.72, 32.61, 32.61, 32.61, 32.61, 32.61, 32.62, 32.62, 32.62, 32.62, 32.62, 32.65, 32.65, 32.65, 32.65, 32.65, 32.59, 32.59, 32.59, 32.59, 32.59, 32.25, 32.25, 32.25, 32.25, 32.25, 32.27, 32.27, 32.27, 32.27, 32.27, 32.46, 32.46, 32.46, 32.46, 32.46, 32.58, 32.58, 32.58, 32.58, 32.58, 32.66, 32.66, 32.66, 32.66, 32.66, 32.78, 32.78, 32.78, 32.78, 32.78, 32.59, 32.59, 32.59, 32.59, 32.59, 32.49, 32.49, 32.49, 32.49, 32.49, 32.32, 32.32, 32.32, 32.32, 32.32, 32.1, 32.1, 32.1, 32.1, 32.1, 32.09, 32.09, 32.09, 32.09, 32.09, 32.14, 32.14, 32.14, 32.14, 32.14, 32.29, 32.29, 32.29, 32.29, 32.29, 32.3, 32.3, 32.3, 32.3, 32.3, 32.43, 32.43, 32.43, 32.43, 32.43, 32.12, 32.12, 32.12, 32.12, 32.12, 31.81, 31.81, 31.81, 31.81, 31.81, 31.69, 31.69, 31.69, 31.69, 31.69, 30.39, 30.39, 30.39, 30.39, 30.39, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.51, 30.51, 30.51, 30.51, 30.51, 30.54, 30.54, 30.54, 30.54, 30.54, 30.64, 30.64, 30.64, 30.64, 30.64, 30.65, 30.65, 30.65, 30.65, 30.65, 30.62, 30.62, 30.62, 30.62, 30.62, 30.36, 30.36, 30.36, 30.36, 30.36, 30.35, 30.35, 30.35, 30.35, 30.35, 30.42, 30.42, 30.42, 30.42, 30.42, 30.52, 30.52, 30.52, 30.52, 30.52, 30.63, 30.63, 30.63, 30.63, 30.63, 30.74, 30.74, 30.74, 30.74, 30.74, 30.75, 30.75, 30.75, 30.75, 30.75, 30.81, 30.81, 30.81, 30.81, 30.81, 30.81]
                    

Loading

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1714648467 --> 1714649097
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.17, 0.17, 0.17, 0.17, 0.17, 0.23, 0.23, 0.23, 0.23, 0.23, 0.11, 0.11, 0.11, 0.11, 0.11, 0.2, 0.2, 0.2, 0.2, 0.2, 0.13, 0.13, 0.13, 0.13, 0.13, 0.11, 0.11, 0.11, 0.11, 0.11, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.14, 0.14, 0.14, 0.14, 0.14, 0.19, 0.19, 0.19, 0.19, 0.19, 0.22, 0.22, 0.22, 0.22, 0.22, 0.31, 0.31, 0.31, 0.31, 0.31, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.25, 0.25, 0.25, 0.25, 0.25, 0.26, 0.26, 0.26, 0.26, 0.26, 0.23, 0.23, 0.23, 0.23, 0.23, 0.17, 0.17, 0.17, 0.17, 0.17, 0.13, 0.13, 0.13, 0.13, 0.13, 0.3, 0.3, 0.3, 0.3, 0.3, 0.1, 0.1, 0.1, 0.1, 0.1, 0.18, 0.18, 0.18, 0.18, 0.18, 0.16, 0.16, 0.16, 0.16, 0.16, 0.25, 0.25, 0.25, 0.25, 0.25, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.11, 0.11, 0.11, 0.11, 0.11, 0.17, 0.17, 0.17, 0.17, 0.17, 0.19, 0.19, 0.19, 0.19, 0.19, 0.32, 0.32, 0.32, 0.32, 0.32, 0.16, 0.16, 0.16, 0.16, 0.16, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.36, 0.36, 0.36, 0.36, 0.36, 0.52, 0.52, 0.52, 0.52, 0.52, 0.49, 0.49, 0.49, 0.49, 0.49, 0.55, 0.55, 0.55, 0.55, 0.55, 0.21, 0.21, 0.21, 0.21, 0.21, 0.22, 0.22, 0.22, 0.22, 0.22, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.21, 0.21, 0.21, 0.21, 0.21, 0.25, 0.25, 0.25, 0.25, 0.25, 0.29, 0.29, 0.29, 0.29, 0.29, 0.17, 0.17, 0.17, 0.17, 0.17, 0.2, 0.2, 0.2, 0.2, 0.2, 0.12, 0.12, 0.12, 0.12, 0.12, 0.09, 0.09, 0.09, 0.09, 0.09, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.23]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1714648467 --> 1714649097
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0]
                    

Loading

[JoanFM]

just curious, aren't they the exact same hash?

[teleprint-me]

Yes, they are.

[LostRuins]

Is this duplicated in #7063 ?

[dranger003]

@LostRuins Yes, this is a duplicate.