ggml : generalize GGML_OP_CONCAT by ggerganov · Pull Request #7563 · ggml-org/llama.cpp

ggerganov · 2024-05-27T11:15:32Z

Support concat across all dimensions:

https://pytorch.org/docs/stable/generated/torch.cat.html#torch.cat

ggml-ci

github-actions · 2024-05-27T11:53:30Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 540 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8676.14ms p(95)=22491.83ms fails=, finish reason: stop=483 truncated=57
Prompt processing (pp): avg=94.05tk/s p(95)=381.08tk/s
Token generation (tg): avg=45.19tk/s p(95)=45.12tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/concat commit=df80e03899bfcb0e2213231dc4d68ca4bd56ce78

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 540 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1716882871 --> 1716883501
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 301.67, 301.67, 301.67, 301.67, 301.67, 738.48, 738.48, 738.48, 738.48, 738.48, 764.74, 764.74, 764.74, 764.74, 764.74, 780.4, 780.4, 780.4, 780.4, 780.4, 806.59, 806.59, 806.59, 806.59, 806.59, 864.43, 864.43, 864.43, 864.43, 864.43, 850.69, 850.69, 850.69, 850.69, 850.69, 863.45, 863.45, 863.45, 863.45, 863.45, 873.71, 873.71, 873.71, 873.71, 873.71, 886.12, 886.12, 886.12, 886.12, 886.12, 883.73, 883.73, 883.73, 883.73, 883.73, 896.91, 896.91, 896.91, 896.91, 896.91, 862.68, 862.68, 862.68, 862.68, 862.68, 839.32, 839.32, 839.32, 839.32, 839.32, 858.3, 858.3, 858.3, 858.3, 858.3, 865.65, 865.65, 865.65, 865.65, 865.65, 866.63, 866.63, 866.63, 866.63, 866.63, 871.33, 871.33, 871.33, 871.33, 871.33, 866.91, 866.91, 866.91, 866.91, 866.91, 886.88, 886.88, 886.88, 886.88, 886.88, 886.75, 886.75, 886.75, 886.75, 886.75, 884.16, 884.16, 884.16, 884.16, 884.16, 889.92, 889.92, 889.92, 889.92, 889.92, 891.06, 891.06, 891.06, 891.06, 891.06, 886.79, 886.79, 886.79, 886.79, 886.79, 875.56, 875.56, 875.56, 875.56, 875.56, 876.88, 876.88, 876.88, 876.88, 876.88, 870.0, 870.0, 870.0, 870.0, 870.0, 864.36, 864.36, 864.36, 864.36, 864.36, 861.57, 861.57, 861.57, 861.57, 861.57, 863.58, 863.58, 863.58, 863.58, 863.58, 865.83, 865.83, 865.83, 865.83, 865.83, 863.91, 863.91, 863.91, 863.91, 863.91, 864.36, 864.36, 864.36, 864.36, 864.36, 862.52, 862.52, 862.52, 862.52, 862.52, 863.72, 863.72, 863.72, 863.72, 863.72, 861.72, 861.72, 861.72, 861.72, 861.72, 864.95, 864.95, 864.95, 864.95, 864.95, 855.62, 855.62, 855.62, 855.62, 855.62, 853.63, 853.63, 853.63, 853.63, 853.63, 857.09, 857.09, 857.09, 857.09, 857.09, 860.13, 860.13, 860.13, 860.13, 860.13, 866.95, 866.95, 866.95, 866.95, 866.95, 863.39, 863.39, 863.39, 863.39, 863.39, 854.45, 854.45, 854.45, 854.45, 854.45, 853.86, 853.86, 853.86, 853.86, 853.86, 852.42, 852.42, 852.42, 852.42, 852.42, 855.99, 855.99, 855.99, 855.99, 855.99, 857.7, 857.7, 857.7, 857.7, 857.7, 856.48, 856.48, 856.48, 856.48, 856.48, 862.56, 862.56, 862.56, 862.56, 862.56, 861.35, 861.35, 861.35, 861.35, 861.35, 862.49, 862.49, 862.49, 862.49, 862.49, 865.75, 865.75, 865.75, 865.75, 865.75, 866.97, 866.97, 866.97, 866.97, 866.97, 869.68, 869.68, 869.68, 869.68, 869.68, 870.03, 870.03, 870.03, 870.03, 870.03, 869.5, 869.5, 869.5, 869.5, 869.5, 870.19, 870.19, 870.19, 870.19, 870.19, 871.56, 871.56, 871.56, 871.56, 871.56, 872.32, 872.32, 872.32, 872.32, 872.32, 872.32]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 540 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1716882871 --> 1716883501
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 34.23, 34.23, 34.23, 34.23, 34.23, 38.46, 38.46, 38.46, 38.46, 38.46, 25.94, 25.94, 25.94, 25.94, 25.94, 27.73, 27.73, 27.73, 27.73, 27.73, 28.67, 28.67, 28.67, 28.67, 28.67, 29.35, 29.35, 29.35, 29.35, 29.35, 30.17, 30.17, 30.17, 30.17, 30.17, 30.94, 30.94, 30.94, 30.94, 30.94, 31.85, 31.85, 31.85, 31.85, 31.85, 32.04, 32.04, 32.04, 32.04, 32.04, 32.2, 32.2, 32.2, 32.2, 32.2, 32.24, 32.24, 32.24, 32.24, 32.24, 32.3, 32.3, 32.3, 32.3, 32.3, 31.48, 31.48, 31.48, 31.48, 31.48, 31.32, 31.32, 31.32, 31.32, 31.32, 30.42, 30.42, 30.42, 30.42, 30.42, 28.87, 28.87, 28.87, 28.87, 28.87, 28.32, 28.32, 28.32, 28.32, 28.32, 28.52, 28.52, 28.52, 28.52, 28.52, 28.57, 28.57, 28.57, 28.57, 28.57, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.55, 28.55, 28.55, 28.55, 28.55, 28.85, 28.85, 28.85, 28.85, 28.85, 29.15, 29.15, 29.15, 29.15, 29.15, 29.1, 29.1, 29.1, 29.1, 29.1, 29.32, 29.32, 29.32, 29.32, 29.32, 29.52, 29.52, 29.52, 29.52, 29.52, 29.35, 29.35, 29.35, 29.35, 29.35, 29.43, 29.43, 29.43, 29.43, 29.43, 29.65, 29.65, 29.65, 29.65, 29.65, 29.78, 29.78, 29.78, 29.78, 29.78, 29.99, 29.99, 29.99, 29.99, 29.99, 30.06, 30.06, 30.06, 30.06, 30.06, 30.25, 30.25, 30.25, 30.25, 30.25, 30.16, 30.16, 30.16, 30.16, 30.16, 30.11, 30.11, 30.11, 30.11, 30.11, 30.0, 30.0, 30.0, 30.0, 30.0, 29.53, 29.53, 29.53, 29.53, 29.53, 29.45, 29.45, 29.45, 29.45, 29.45, 29.65, 29.65, 29.65, 29.65, 29.65, 29.74, 29.74, 29.74, 29.74, 29.74, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.36, 29.36, 29.36, 29.36, 29.36, 29.15, 29.15, 29.15, 29.15, 29.15, 28.51, 28.51, 28.51, 28.51, 28.51, 28.46, 28.46, 28.46, 28.46, 28.46, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.59, 28.59, 28.59, 28.59, 28.59, 28.61, 28.61, 28.61, 28.61, 28.61, 28.72, 28.72, 28.72, 28.72, 28.72, 28.73, 28.73, 28.73, 28.73, 28.73, 28.66, 28.66, 28.66, 28.66, 28.66, 28.57, 28.57, 28.57, 28.57, 28.57, 28.64, 28.64, 28.64, 28.64, 28.64, 28.74, 28.74, 28.74, 28.74, 28.74, 28.84, 28.84, 28.84, 28.84, 28.84, 28.92, 28.92, 28.92, 28.92, 28.92, 29.02, 29.02, 29.02, 29.02, 29.02, 29.11]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 540 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1716882871 --> 1716883501
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.08, 0.08, 0.08, 0.08, 0.08, 0.36, 0.36, 0.36, 0.36, 0.36, 0.26, 0.26, 0.26, 0.26, 0.26, 0.22, 0.22, 0.22, 0.22, 0.22, 0.14, 0.14, 0.14, 0.14, 0.14, 0.23, 0.23, 0.23, 0.23, 0.23, 0.22, 0.22, 0.22, 0.22, 0.22, 0.17, 0.17, 0.17, 0.17, 0.17, 0.16, 0.16, 0.16, 0.16, 0.16, 0.19, 0.19, 0.19, 0.19, 0.19, 0.23, 0.23, 0.23, 0.23, 0.23, 0.19, 0.19, 0.19, 0.19, 0.19, 0.27, 0.27, 0.27, 0.27, 0.27, 0.3, 0.3, 0.3, 0.3, 0.3, 0.45, 0.45, 0.45, 0.45, 0.45, 0.39, 0.39, 0.39, 0.39, 0.39, 0.31, 0.31, 0.31, 0.31, 0.31, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.25, 0.25, 0.25, 0.25, 0.25, 0.22, 0.22, 0.22, 0.22, 0.22, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.3, 0.3, 0.3, 0.3, 0.3, 0.1, 0.1, 0.1, 0.1, 0.1, 0.16, 0.16, 0.16, 0.16, 0.16, 0.33, 0.33, 0.33, 0.33, 0.33, 0.21, 0.21, 0.21, 0.21, 0.21, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.32, 0.32, 0.32, 0.32, 0.32, 0.28, 0.28, 0.28, 0.28, 0.28, 0.48, 0.48, 0.48, 0.48, 0.48, 0.24, 0.24, 0.24, 0.24, 0.24, 0.07, 0.07, 0.07, 0.07, 0.07, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.31, 0.31, 0.31, 0.31, 0.31, 0.42, 0.42, 0.42, 0.42, 0.42, 0.45, 0.45, 0.45, 0.45, 0.45, 0.39, 0.39, 0.39, 0.39, 0.39, 0.33, 0.33, 0.33, 0.33, 0.33, 0.26, 0.26, 0.26, 0.26, 0.26, 0.23, 0.23, 0.23, 0.23, 0.23, 0.11, 0.11, 0.11, 0.11, 0.11, 0.21, 0.21, 0.21, 0.21, 0.21, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.29, 0.29, 0.29, 0.29, 0.29, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.23, 0.23, 0.23, 0.23, 0.23, 0.21, 0.21, 0.21, 0.21, 0.21, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.23]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 540 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1716882871 --> 1716883501
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 1.0]

ggml-ci

ggerganov · 2024-05-27T12:15:38Z

The CUDA implementation probably should be improved - I tried to keep the original logic, but I think it would be better to have a single kernel accepting dim as argument

ggml-ci

ggerganov · 2024-05-28T07:40:52Z

I realized there was a bug in yesterday's CPU/Metal implementation. Should be good now

* ggml : generalize GGML_OP_CONCAT (WIP) ggml-ci * tests : add dim != 2 tests * metal : generalize concat kernel * tests : naming * cuda : generalize concat kernel ggml-ci * sycl : add warning and assert * ggml : fix op params handling * metal : bugfix kernel ggml-ci * ggml : reimplement CPU and Metal * cuda : add asserts ggml-ci * ggml : fix ptrs ggml-ci

ggml : generalize GGML_OP_CONCAT (WIP)

0b52245

ggml-ci

ggerganov changed the title ~~ggml : generalize GGML_OP_CONCAT (WIP)~~ ggml : generalize GGML_OP_CONCAT May 27, 2024

ggerganov added 3 commits May 27, 2024 14:19

tests : add dim != 2 tests

0347657

metal : generalize concat kernel

acdc075

tests : naming

222a71c

ggerganov added 2 commits May 27, 2024 15:08

cuda : generalize concat kernel

ed0891f

ggml-ci

sycl : add warning and assert

ec96ee5

ggerganov marked this pull request as ready for review May 27, 2024 12:14

ggerganov requested a review from slaren May 27, 2024 12:15

github-actions Bot added testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language labels May 27, 2024

slaren approved these changes May 27, 2024

View reviewed changes

Comment thread ggml.c Outdated

Comment thread ggml.c

Comment thread ggml.c Outdated

ggerganov added 2 commits May 27, 2024 15:55

ggml : fix op params handling

b9a6363

metal : bugfix kernel

9d8f12c

ggml-ci

ggerganov marked this pull request as draft May 27, 2024 13:55

mofosyne added the Review Complexity : High Generally require indepth knowledge of LLMs or GPUs label May 28, 2024

ggml : reimplement CPU and Metal

94a0c76

ggerganov force-pushed the gg/concat branch from 444113b to 94a0c76 Compare May 28, 2024 07:30

ggerganov added 2 commits May 28, 2024 10:32

cuda : add asserts

6cd825f

ggml-ci

ggml : fix ptrs

df80e03

ggml-ci

ggerganov marked this pull request as ready for review May 28, 2024 07:40

ggerganov merged commit 0548a41 into master May 28, 2024

ggerganov deleted the gg/concat branch May 28, 2024 08:04

ggerganov added a commit that referenced this pull request May 28, 2024

sycl : fix assert (#7563)

6bd12ce

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

sycl : fix assert (ggml-org#7563)

ceb65e6

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

sycl : fix assert (ggml-org#7563)

5730e70

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ggml : generalize GGML_OP_CONCAT#7563

ggml : generalize GGML_OP_CONCAT#7563
ggerganov merged 11 commits intomasterfrom
gg/concat

ggerganov commented May 27, 2024 •

edited

Loading

Uh oh!

github-actions Bot commented May 27, 2024 •

edited

Loading

Uh oh!

ggerganov commented May 27, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov commented May 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ggerganov commented May 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented May 27, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov commented May 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ggerganov commented May 27, 2024 •

edited

Loading

github-actions Bot commented May 27, 2024 •

edited

Loading