imatrix : migrate to gpt_params by ggerganov · Pull Request #7771 · ggml-org/llama.cpp

ggerganov · 2024-06-05T13:36:33Z

ggml-ci

github-actions · 2024-06-05T15:34:23Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 548 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8522.29ms p(95)=20526.31ms fails=, finish reason: stop=492 truncated=56
Prompt processing (pp): avg=98.51tk/s p(95)=419.57tk/s
Token generation (tg): avg=32.7tk/s p(95)=47.65tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/imatrix-cli-update commit=a9e4addb7607bed44713f2c5d6ea48623433f024

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 548 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717617178 --> 1717617798
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 714.32, 714.32, 714.32, 714.32, 714.32, 896.02, 896.02, 896.02, 896.02, 896.02, 870.55, 870.55, 870.55, 870.55, 870.55, 896.84, 896.84, 896.84, 896.84, 896.84, 944.28, 944.28, 944.28, 944.28, 944.28, 940.89, 940.89, 940.89, 940.89, 940.89, 934.97, 934.97, 934.97, 934.97, 934.97, 945.1, 945.1, 945.1, 945.1, 945.1, 942.42, 942.42, 942.42, 942.42, 942.42, 950.98, 950.98, 950.98, 950.98, 950.98, 958.2, 958.2, 958.2, 958.2, 958.2, 932.13, 932.13, 932.13, 932.13, 932.13, 930.44, 930.44, 930.44, 930.44, 930.44, 923.46, 923.46, 923.46, 923.46, 923.46, 930.46, 930.46, 930.46, 930.46, 930.46, 928.64, 928.64, 928.64, 928.64, 928.64, 926.06, 926.06, 926.06, 926.06, 926.06, 923.06, 923.06, 923.06, 923.06, 923.06, 941.53, 941.53, 941.53, 941.53, 941.53, 939.4, 939.4, 939.4, 939.4, 939.4, 933.33, 933.33, 933.33, 933.33, 933.33, 936.85, 936.85, 936.85, 936.85, 936.85, 938.06, 938.06, 938.06, 938.06, 938.06, 948.38, 948.38, 948.38, 948.38, 948.38, 945.32, 945.32, 945.32, 945.32, 945.32, 944.5, 944.5, 944.5, 944.5, 944.5, 951.92, 951.92, 951.92, 951.92, 951.92, 951.88, 951.88, 951.88, 951.88, 951.88, 951.5, 951.5, 951.5, 951.5, 951.5, 949.96, 949.96, 949.96, 949.96, 949.96, 949.96, 949.96, 949.96, 949.96, 949.96, 947.59, 947.59, 947.59, 947.59, 947.59, 949.42, 949.42, 949.42, 949.42, 949.42, 951.83, 951.83, 951.83, 951.83, 951.83, 953.53, 953.53, 953.53, 953.53, 953.53, 955.58, 955.58, 955.58, 955.58, 955.58, 955.47, 955.47, 955.47, 955.47, 955.47, 951.78, 951.78, 951.78, 951.78, 951.78, 952.43, 952.43, 952.43, 952.43, 952.43, 953.77, 953.77, 953.77, 953.77, 953.77, 952.08, 952.08, 952.08, 952.08, 952.08, 943.55, 943.55, 943.55, 943.55, 943.55, 924.2, 924.2, 924.2, 924.2, 924.2, 923.47, 923.47, 923.47, 923.47, 923.47, 922.23, 922.23, 922.23, 922.23, 922.23, 916.39, 916.39, 916.39, 916.39, 916.39, 920.1, 920.1, 920.1, 920.1, 920.1, 921.71, 921.71, 921.71, 921.71, 921.71, 920.06, 920.06, 920.06, 920.06, 920.06, 923.4, 923.4, 923.4, 923.4, 923.4, 922.44, 922.44, 922.44, 922.44, 922.44, 924.24, 924.24, 924.24, 924.24, 924.24, 927.54, 927.54, 927.54, 927.54, 927.54, 926.91, 926.91, 926.91, 926.91, 926.91, 919.38, 919.38, 919.38, 919.38, 919.38, 919.78, 919.78, 919.78, 919.78, 919.78, 918.98, 918.98, 918.98, 918.98, 918.98, 916.62, 916.62, 916.62, 916.62, 916.62, 916.64, 916.64, 916.64, 916.64, 916.64, 913.42, 913.42, 913.42, 913.42, 913.42, 913.09]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 548 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717617178 --> 1717617798
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 42.53, 42.53, 42.53, 42.53, 42.53, 38.44, 38.44, 38.44, 38.44, 38.44, 29.16, 29.16, 29.16, 29.16, 29.16, 30.53, 30.53, 30.53, 30.53, 30.53, 31.47, 31.47, 31.47, 31.47, 31.47, 31.03, 31.03, 31.03, 31.03, 31.03, 32.62, 32.62, 32.62, 32.62, 32.62, 33.22, 33.22, 33.22, 33.22, 33.22, 33.46, 33.46, 33.46, 33.46, 33.46, 33.49, 33.49, 33.49, 33.49, 33.49, 33.24, 33.24, 33.24, 33.24, 33.24, 33.03, 33.03, 33.03, 33.03, 33.03, 32.36, 32.36, 32.36, 32.36, 32.36, 32.25, 32.25, 32.25, 32.25, 32.25, 31.57, 31.57, 31.57, 31.57, 31.57, 31.03, 31.03, 31.03, 31.03, 31.03, 29.63, 29.63, 29.63, 29.63, 29.63, 30.03, 30.03, 30.03, 30.03, 30.03, 29.82, 29.82, 29.82, 29.82, 29.82, 29.67, 29.67, 29.67, 29.67, 29.67, 29.52, 29.52, 29.52, 29.52, 29.52, 29.43, 29.43, 29.43, 29.43, 29.43, 29.5, 29.5, 29.5, 29.5, 29.5, 29.81, 29.81, 29.81, 29.81, 29.81, 29.85, 29.85, 29.85, 29.85, 29.85, 29.99, 29.99, 29.99, 29.99, 29.99, 30.23, 30.23, 30.23, 30.23, 30.23, 30.01, 30.01, 30.01, 30.01, 30.01, 30.02, 30.02, 30.02, 30.02, 30.02, 30.32, 30.32, 30.32, 30.32, 30.32, 30.61, 30.61, 30.61, 30.61, 30.61, 30.78, 30.78, 30.78, 30.78, 30.78, 30.94, 30.94, 30.94, 30.94, 30.94, 31.07, 31.07, 31.07, 31.07, 31.07, 30.93, 30.93, 30.93, 30.93, 30.93, 30.71, 30.71, 30.71, 30.71, 30.71, 30.53, 30.53, 30.53, 30.53, 30.53, 30.65, 30.65, 30.65, 30.65, 30.65, 30.82, 30.82, 30.82, 30.82, 30.82, 30.98, 30.98, 30.98, 30.98, 30.98, 31.07, 31.07, 31.07, 31.07, 31.07, 31.14, 31.14, 31.14, 31.14, 31.14, 30.8, 30.8, 30.8, 30.8, 30.8, 30.59, 30.59, 30.59, 30.59, 30.59, 30.31, 30.31, 30.31, 30.31, 30.31, 29.04, 29.04, 29.04, 29.04, 29.04, 29.04, 29.04, 29.04, 29.04, 29.04, 29.05, 29.05, 29.05, 29.05, 29.05, 29.08, 29.08, 29.08, 29.08, 29.08, 29.12, 29.12, 29.12, 29.12, 29.12, 29.13, 29.13, 29.13, 29.13, 29.13, 29.17, 29.17, 29.17, 29.17, 29.17, 29.13, 29.13, 29.13, 29.13, 29.13, 29.2, 29.2, 29.2, 29.2, 29.2, 29.21, 29.21, 29.21, 29.21, 29.21, 29.29, 29.29, 29.29, 29.29, 29.29, 29.43, 29.43, 29.43, 29.43, 29.43, 29.55, 29.55, 29.55, 29.55, 29.55, 29.69, 29.69, 29.69, 29.69, 29.69, 29.75, 29.75, 29.75, 29.75, 29.75, 29.76]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 548 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717617178 --> 1717617798
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.21, 0.21, 0.21, 0.21, 0.21, 0.32, 0.32, 0.32, 0.32, 0.32, 0.16, 0.16, 0.16, 0.16, 0.16, 0.19, 0.19, 0.19, 0.19, 0.19, 0.28, 0.28, 0.28, 0.28, 0.28, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.24, 0.24, 0.24, 0.24, 0.24, 0.29, 0.29, 0.29, 0.29, 0.29, 0.2, 0.2, 0.2, 0.2, 0.2, 0.16, 0.16, 0.16, 0.16, 0.16, 0.38, 0.38, 0.38, 0.38, 0.38, 0.41, 0.41, 0.41, 0.41, 0.41, 0.33, 0.33, 0.33, 0.33, 0.33, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.25, 0.25, 0.25, 0.25, 0.25, 0.33, 0.33, 0.33, 0.33, 0.33, 0.18, 0.18, 0.18, 0.18, 0.18, 0.22, 0.22, 0.22, 0.22, 0.22, 0.16, 0.16, 0.16, 0.16, 0.16, 0.22, 0.22, 0.22, 0.22, 0.22, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.24, 0.24, 0.24, 0.24, 0.24, 0.09, 0.09, 0.09, 0.09, 0.09, 0.11, 0.11, 0.11, 0.11, 0.11, 0.1, 0.1, 0.1, 0.1, 0.1, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.22, 0.22, 0.22, 0.22, 0.22, 0.25, 0.25, 0.25, 0.25, 0.25, 0.19, 0.19, 0.19, 0.19, 0.19, 0.3, 0.3, 0.3, 0.3, 0.3, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.11, 0.11, 0.11, 0.11, 0.11, 0.17, 0.17, 0.17, 0.17, 0.17, 0.33, 0.33, 0.33, 0.33, 0.33, 0.52, 0.52, 0.52, 0.52, 0.52, 0.54, 0.54, 0.54, 0.54, 0.54, 0.63, 0.63, 0.63, 0.63, 0.63, 0.28, 0.28, 0.28, 0.28, 0.28, 0.15, 0.15, 0.15, 0.15, 0.15, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.23, 0.22, 0.22, 0.22, 0.22, 0.22, 0.19, 0.19, 0.19, 0.19, 0.19, 0.15, 0.15, 0.15, 0.15, 0.15, 0.21, 0.21, 0.21, 0.21, 0.21, 0.23, 0.23, 0.23, 0.23, 0.23, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.1, 0.1, 0.1, 0.1, 0.1, 0.13, 0.13, 0.13, 0.13, 0.13, 0.19, 0.19, 0.19, 0.19, 0.19, 0.21, 0.21, 0.21, 0.21, 0.21, 0.16]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 548 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717617178 --> 1717617798
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 1.0, 1.0, 1.0, 1.0, 1.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 1.0, 1.0, 1.0, 1.0, 1.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0]

* imatrix : migrate to gpt_params ggml-ci * imatrix : add --save-frequency cli arg * common : fix --no-ppl

imatrix : migrate to gpt_params

cbe51d7

ggml-ci

github-actions Bot added examples server labels Jun 5, 2024

imatrix : add --save-frequency cli arg

901b86b

ggerganov mentioned this pull request Jun 5, 2024

Bug: Unable to generate imatrix data using imatrix.exe from release 3089 #7765

Closed

bartowski1182 reviewed Jun 5, 2024

View reviewed changes

Comment thread common/common.cpp

common : fix --no-ppl

a9e4add

ggerganov merged commit f83351f into master Jun 6, 2024

ggerganov deleted the gg/imatrix-cli-update branch June 6, 2024 13:31

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

imatrix : migrate to gpt_params (ggml-org#7771)

fb01a32

* imatrix : migrate to gpt_params ggml-ci * imatrix : add --save-frequency cli arg * common : fix --no-ppl

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

imatrix : migrate to gpt_params (ggml-org#7771)

c2a2806

* imatrix : migrate to gpt_params ggml-ci * imatrix : add --save-frequency cli arg * common : fix --no-ppl

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

imatrix : migrate to gpt_params#7771

imatrix : migrate to gpt_params#7771
ggerganov merged 3 commits intomasterfrom
gg/imatrix-cli-update

ggerganov commented Jun 5, 2024

Uh oh!

github-actions Bot commented Jun 5, 2024 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ggerganov commented Jun 5, 2024

Uh oh!

github-actions Bot commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jun 5, 2024 •

edited

Loading