Fix per token atrributes bits by jaime-m-p · Pull Request #7749 · ggml-org/llama.cpp

jaime-m-p · 2024-06-04T17:08:53Z

github-actions · 2024-06-04T20:15:26Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 551 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8521.08ms p(95)=21321.49ms fails=, finish reason: stop=497 truncated=54
Prompt processing (pp): avg=95.77tk/s p(95)=424.22tk/s
Token generation (tg): avg=37.95tk/s p(95)=47.37tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=per-token-attribs commit=f95c7dc7104fb4516a1dacbada5546e979194323

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 551 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717531488 --> 1717532120
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 612.59, 612.59, 612.59, 612.59, 612.59, 710.39, 710.39, 710.39, 710.39, 710.39, 737.8, 737.8, 737.8, 737.8, 737.8, 792.75, 792.75, 792.75, 792.75, 792.75, 806.86, 806.86, 806.86, 806.86, 806.86, 802.1, 802.1, 802.1, 802.1, 802.1, 805.47, 805.47, 805.47, 805.47, 805.47, 817.88, 817.88, 817.88, 817.88, 817.88, 834.28, 834.28, 834.28, 834.28, 834.28, 859.82, 859.82, 859.82, 859.82, 859.82, 873.02, 873.02, 873.02, 873.02, 873.02, 826.48, 826.48, 826.48, 826.48, 826.48, 867.87, 867.87, 867.87, 867.87, 867.87, 875.41, 875.41, 875.41, 875.41, 875.41, 877.9, 877.9, 877.9, 877.9, 877.9, 879.56, 879.56, 879.56, 879.56, 879.56, 879.64, 879.64, 879.64, 879.64, 879.64, 900.04, 900.04, 900.04, 900.04, 900.04, 887.75, 887.75, 887.75, 887.75, 887.75, 892.35, 892.35, 892.35, 892.35, 892.35, 890.83, 890.83, 890.83, 890.83, 890.83, 892.93, 892.93, 892.93, 892.93, 892.93, 885.09, 885.09, 885.09, 885.09, 885.09, 876.54, 876.54, 876.54, 876.54, 876.54, 876.77, 876.77, 876.77, 876.77, 876.77, 888.83, 888.83, 888.83, 888.83, 888.83, 886.35, 886.35, 886.35, 886.35, 886.35, 884.07, 884.07, 884.07, 884.07, 884.07, 883.47, 883.47, 883.47, 883.47, 883.47, 886.87, 886.87, 886.87, 886.87, 886.87, 883.98, 883.98, 883.98, 883.98, 883.98, 886.55, 886.55, 886.55, 886.55, 886.55, 885.59, 885.59, 885.59, 885.59, 885.59, 883.68, 883.68, 883.68, 883.68, 883.68, 882.81, 882.81, 882.81, 882.81, 882.81, 861.53, 861.53, 861.53, 861.53, 861.53, 854.27, 854.27, 854.27, 854.27, 854.27, 853.34, 853.34, 853.34, 853.34, 853.34, 859.13, 859.13, 859.13, 859.13, 859.13, 859.24, 859.24, 859.24, 859.24, 859.24, 858.48, 858.48, 858.48, 858.48, 858.48, 860.4, 860.4, 860.4, 860.4, 860.4, 849.4, 849.4, 849.4, 849.4, 849.4, 847.23, 847.23, 847.23, 847.23, 847.23, 850.88, 850.88, 850.88, 850.88, 850.88, 852.74, 852.74, 852.74, 852.74, 852.74, 852.05, 852.05, 852.05, 852.05, 852.05, 851.2, 851.2, 851.2, 851.2, 851.2, 854.84, 854.84, 854.84, 854.84, 854.84, 855.43, 855.43, 855.43, 855.43, 855.43, 857.43, 857.43, 857.43, 857.43, 857.43, 850.59, 850.59, 850.59, 850.59, 850.59, 854.79, 854.79, 854.79, 854.79, 854.79, 856.31, 856.31, 856.31, 856.31, 856.31, 855.86, 855.86, 855.86, 855.86, 855.86, 854.98, 854.98, 854.98, 854.98, 854.98, 855.6, 855.6, 855.6, 855.6, 855.6, 855.38, 855.38, 855.38, 855.38, 855.38, 857.36, 857.36, 857.36, 857.36, 857.36, 857.37, 857.37, 857.37, 857.37, 857.37, 857.72, 857.72]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 551 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717531488 --> 1717532120
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 44.85, 44.85, 44.85, 44.85, 44.85, 28.7, 28.7, 28.7, 28.7, 28.7, 28.5, 28.5, 28.5, 28.5, 28.5, 30.22, 30.22, 30.22, 30.22, 30.22, 30.93, 30.93, 30.93, 30.93, 30.93, 32.24, 32.24, 32.24, 32.24, 32.24, 32.87, 32.87, 32.87, 32.87, 32.87, 33.69, 33.69, 33.69, 33.69, 33.69, 33.85, 33.85, 33.85, 33.85, 33.85, 34.27, 34.27, 34.27, 34.27, 34.27, 34.28, 34.28, 34.28, 34.28, 34.28, 34.03, 34.03, 34.03, 34.03, 34.03, 33.2, 33.2, 33.2, 33.2, 33.2, 32.77, 32.77, 32.77, 32.77, 32.77, 31.45, 31.45, 31.45, 31.45, 31.45, 30.34, 30.34, 30.34, 30.34, 30.34, 30.4, 30.4, 30.4, 30.4, 30.4, 30.57, 30.57, 30.57, 30.57, 30.57, 30.29, 30.29, 30.29, 30.29, 30.29, 30.22, 30.22, 30.22, 30.22, 30.22, 30.21, 30.21, 30.21, 30.21, 30.21, 30.41, 30.41, 30.41, 30.41, 30.41, 30.65, 30.65, 30.65, 30.65, 30.65, 30.64, 30.64, 30.64, 30.64, 30.64, 30.77, 30.77, 30.77, 30.77, 30.77, 30.79, 30.79, 30.79, 30.79, 30.79, 30.6, 30.6, 30.6, 30.6, 30.6, 30.76, 30.76, 30.76, 30.76, 30.76, 30.99, 30.99, 30.99, 30.99, 30.99, 31.13, 31.13, 31.13, 31.13, 31.13, 31.27, 31.27, 31.27, 31.27, 31.27, 31.37, 31.37, 31.37, 31.37, 31.37, 31.42, 31.42, 31.42, 31.42, 31.42, 31.16, 31.16, 31.16, 31.16, 31.16, 31.06, 31.06, 31.06, 31.06, 31.06, 30.96, 30.96, 30.96, 30.96, 30.96, 30.58, 30.58, 30.58, 30.58, 30.58, 30.81, 30.81, 30.81, 30.81, 30.81, 30.92, 30.92, 30.92, 30.92, 30.92, 30.98, 30.98, 30.98, 30.98, 30.98, 31.16, 31.16, 31.16, 31.16, 31.16, 30.76, 30.76, 30.76, 30.76, 30.76, 30.54, 30.54, 30.54, 30.54, 30.54, 30.26, 30.26, 30.26, 30.26, 30.26, 29.66, 29.66, 29.66, 29.66, 29.66, 29.6, 29.6, 29.6, 29.6, 29.6, 29.54, 29.54, 29.54, 29.54, 29.54, 29.39, 29.39, 29.39, 29.39, 29.39, 29.42, 29.42, 29.42, 29.42, 29.42, 29.47, 29.47, 29.47, 29.47, 29.47, 29.48, 29.48, 29.48, 29.48, 29.48, 29.43, 29.43, 29.43, 29.43, 29.43, 29.43, 29.43, 29.43, 29.43, 29.43, 29.36, 29.36, 29.36, 29.36, 29.36, 29.3, 29.3, 29.3, 29.3, 29.3, 29.32, 29.32, 29.32, 29.32, 29.32, 29.42, 29.42, 29.42, 29.42, 29.42, 29.55, 29.55, 29.55, 29.55, 29.55, 29.66, 29.66, 29.66, 29.66, 29.66, 29.66, 29.66, 29.66, 29.66, 29.66, 29.72, 29.72]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 551 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717531488 --> 1717532120
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.26, 0.26, 0.26, 0.26, 0.26, 0.24, 0.24, 0.24, 0.24, 0.24, 0.17, 0.17, 0.17, 0.17, 0.17, 0.16, 0.16, 0.16, 0.16, 0.16, 0.25, 0.25, 0.25, 0.25, 0.25, 0.09, 0.09, 0.09, 0.09, 0.09, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.27, 0.27, 0.27, 0.27, 0.27, 0.21, 0.21, 0.21, 0.21, 0.21, 0.24, 0.24, 0.24, 0.24, 0.24, 0.32, 0.32, 0.32, 0.32, 0.32, 0.35, 0.35, 0.35, 0.35, 0.35, 0.32, 0.32, 0.32, 0.32, 0.32, 0.19, 0.19, 0.19, 0.19, 0.19, 0.13, 0.13, 0.13, 0.13, 0.13, 0.25, 0.25, 0.25, 0.25, 0.25, 0.23, 0.23, 0.23, 0.23, 0.23, 0.24, 0.24, 0.24, 0.24, 0.24, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.33, 0.33, 0.33, 0.33, 0.33, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.39, 0.39, 0.39, 0.39, 0.39, 0.16, 0.16, 0.16, 0.16, 0.16, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.31, 0.31, 0.31, 0.31, 0.31, 0.21, 0.21, 0.21, 0.21, 0.21, 0.26, 0.26, 0.26, 0.26, 0.26, 0.34, 0.34, 0.34, 0.34, 0.34, 0.15, 0.15, 0.15, 0.15, 0.15, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.15, 0.15, 0.15, 0.15, 0.15, 0.41, 0.41, 0.41, 0.41, 0.41, 0.56, 0.56, 0.56, 0.56, 0.56, 0.5, 0.5, 0.5, 0.5, 0.5, 0.38, 0.38, 0.38, 0.38, 0.38, 0.21, 0.21, 0.21, 0.21, 0.21, 0.29, 0.29, 0.29, 0.29, 0.29, 0.31, 0.31, 0.31, 0.31, 0.31, 0.22, 0.22, 0.22, 0.22, 0.22, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.28, 0.28, 0.28, 0.28, 0.28, 0.27, 0.27, 0.27, 0.27, 0.27, 0.22, 0.22, 0.22, 0.22, 0.22, 0.27, 0.27, 0.27, 0.27, 0.27, 0.32, 0.32, 0.32, 0.32, 0.32, 0.17, 0.17, 0.17, 0.17, 0.17, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.21, 0.21, 0.21, 0.21, 0.21, 0.22, 0.22]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 551 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717531488 --> 1717532120
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 1.0, 1.0, 1.0, 1.0, 1.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 1.0, 1.0, 1.0, 1.0, 1.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0]

jaime-m-p and others added 12 commits June 1, 2024 19:42

Add per token attrib enum

cec6a3b

Using phi-3 for testing 'rstrip'

3ead1b9

bugfix: assertions, wrong special token list

33de247

Implement 'rstrip' properly

ada961c

Refactor + add 'jina-v2' for testing 'lstrip'

01c9229

Update phi-3 GGUF file (obsolete since 917dc8c)

8564c19

Update brute force test: testing 'lstrip' and 'rstrip'

83cac0e

Fix previous commit

54e9f23

Replace llama_token_type with llama_token_attribs

ac40ff0

Rename token attributes

18f5fc7

Fix per token atrributes bits

f95c7dc

Merge branch 'master' into per-token-attribs

6a0f3db

jaime-m-p mentioned this pull request Jun 4, 2024

Per token attributes #7685

Merged

ggerganov approved these changes Jun 4, 2024

View reviewed changes

jaime-m-p merged commit c90dbe0 into ggml-org:master Jun 4, 2024

jaime-m-p mentioned this pull request Jun 23, 2024

Detokenizer fixes #8039

Merged

4 tasks

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

Fix per token atrributes bits (ggml-org#7749)

baffcf1

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

Fix per token atrributes bits (ggml-org#7749)

b443dc4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix per token atrributes bits#7749

Fix per token atrributes bits#7749
jaime-m-p merged 12 commits intoggml-org:masterfrom
jaime-m-p:per-token-attribs

jaime-m-p commented Jun 4, 2024

Uh oh!

github-actions Bot commented Jun 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jaime-m-p commented Jun 4, 2024

Uh oh!

github-actions Bot commented Jun 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants