ggml : fix YARN + add tests + add asserts by ggerganov · Pull Request #7617 · ggml-org/llama.cpp

ggerganov · 2024-05-29T11:33:26Z

There might be some issues with RoPE, will try to address here

github-actions · 2024-05-29T14:18:08Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 536 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8743.5ms p(95)=21549.19ms fails=, finish reason: stop=476 truncated=60
Prompt processing (pp): avg=104.95tk/s p(95)=516.49tk/s
Token generation (tg): avg=33.81tk/s p(95)=46.93tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/rope-tests commit=aa15e7d246ebf1de30148f29805111714b3cfa35

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 536 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717000522 --> 1717001150
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 416.17, 416.17, 416.17, 416.17, 416.17, 982.09, 982.09, 982.09, 982.09, 982.09, 955.89, 955.89, 955.89, 955.89, 955.89, 958.24, 958.24, 958.24, 958.24, 958.24, 999.79, 999.79, 999.79, 999.79, 999.79, 1003.31, 1003.31, 1003.31, 1003.31, 1003.31, 998.87, 998.87, 998.87, 998.87, 998.87, 1012.07, 1012.07, 1012.07, 1012.07, 1012.07, 996.55, 996.55, 996.55, 996.55, 996.55, 1006.4, 1006.4, 1006.4, 1006.4, 1006.4, 1021.61, 1021.61, 1021.61, 1021.61, 1021.61, 982.66, 982.66, 982.66, 982.66, 982.66, 960.88, 960.88, 960.88, 960.88, 960.88, 970.38, 970.38, 970.38, 970.38, 970.38, 971.74, 971.74, 971.74, 971.74, 971.74, 967.5, 967.5, 967.5, 967.5, 967.5, 968.15, 968.15, 968.15, 968.15, 968.15, 962.84, 962.84, 962.84, 962.84, 962.84, 976.62, 976.62, 976.62, 976.62, 976.62, 975.89, 975.89, 975.89, 975.89, 975.89, 980.13, 980.13, 980.13, 980.13, 980.13, 976.99, 976.99, 976.99, 976.99, 976.99, 978.53, 978.53, 978.53, 978.53, 978.53, 983.1, 983.1, 983.1, 983.1, 983.1, 983.75, 983.75, 983.75, 983.75, 983.75, 983.04, 983.04, 983.04, 983.04, 983.04, 960.93, 960.93, 960.93, 960.93, 960.93, 957.15, 957.15, 957.15, 957.15, 957.15, 953.69, 953.69, 953.69, 953.69, 953.69, 952.38, 952.38, 952.38, 952.38, 952.38, 954.74, 954.74, 954.74, 954.74, 954.74, 952.99, 952.99, 952.99, 952.99, 952.99, 948.72, 948.72, 948.72, 948.72, 948.72, 950.7, 950.7, 950.7, 950.7, 950.7, 944.28, 944.28, 944.28, 944.28, 944.28, 945.8, 945.8, 945.8, 945.8, 945.8, 946.08, 946.08, 946.08, 946.08, 946.08, 943.84, 943.84, 943.84, 943.84, 943.84, 940.77, 940.77, 940.77, 940.77, 940.77, 937.42, 937.42, 937.42, 937.42, 937.42, 938.91, 938.91, 938.91, 938.91, 938.91, 937.4, 937.4, 937.4, 937.4, 937.4, 936.86, 936.86, 936.86, 936.86, 936.86, 932.23, 932.23, 932.23, 932.23, 932.23, 927.46, 927.46, 927.46, 927.46, 927.46, 925.04, 925.04, 925.04, 925.04, 925.04, 922.51, 922.51, 922.51, 922.51, 922.51, 920.42, 920.42, 920.42, 920.42, 920.42, 924.08, 924.08, 924.08, 924.08, 924.08, 922.45, 922.45, 922.45, 922.45, 922.45, 927.27, 927.27, 927.27, 927.27, 927.27, 925.98, 925.98, 925.98, 925.98, 925.98, 927.82, 927.82, 927.82, 927.82, 927.82, 905.92, 905.92, 905.92, 905.92, 905.92, 904.76, 904.76, 904.76, 904.76, 904.76, 903.43, 903.43, 903.43, 903.43, 903.43, 901.51, 901.51, 901.51, 901.51, 901.51, 900.92, 900.92, 900.92, 900.92, 900.92, 901.28, 901.28, 901.28, 901.28, 901.28, 900.21, 900.21, 900.21, 900.21, 900.21, 900.75, 900.75, 900.75, 900.75, 900.75]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 536 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717000522 --> 1717001150
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 42.4, 42.4, 42.4, 42.4, 42.4, 29.41, 29.41, 29.41, 29.41, 29.41, 27.18, 27.18, 27.18, 27.18, 27.18, 29.64, 29.64, 29.64, 29.64, 29.64, 29.79, 29.79, 29.79, 29.79, 29.79, 31.38, 31.38, 31.38, 31.38, 31.38, 33.16, 33.16, 33.16, 33.16, 33.16, 34.0, 34.0, 34.0, 34.0, 34.0, 34.22, 34.22, 34.22, 34.22, 34.22, 34.16, 34.16, 34.16, 34.16, 34.16, 34.13, 34.13, 34.13, 34.13, 34.13, 34.33, 34.33, 34.33, 34.33, 34.33, 33.97, 33.97, 33.97, 33.97, 33.97, 33.63, 33.63, 33.63, 33.63, 33.63, 31.94, 31.94, 31.94, 31.94, 31.94, 29.61, 29.61, 29.61, 29.61, 29.61, 29.37, 29.37, 29.37, 29.37, 29.37, 29.47, 29.47, 29.47, 29.47, 29.47, 29.58, 29.58, 29.58, 29.58, 29.58, 29.4, 29.4, 29.4, 29.4, 29.4, 29.52, 29.52, 29.52, 29.52, 29.52, 29.74, 29.74, 29.74, 29.74, 29.74, 30.14, 30.14, 30.14, 30.14, 30.14, 30.02, 30.02, 30.02, 30.02, 30.02, 30.14, 30.14, 30.14, 30.14, 30.14, 30.27, 30.27, 30.27, 30.27, 30.27, 30.33, 30.33, 30.33, 30.33, 30.33, 30.24, 30.24, 30.24, 30.24, 30.24, 30.18, 30.18, 30.18, 30.18, 30.18, 30.45, 30.45, 30.45, 30.45, 30.45, 30.51, 30.51, 30.51, 30.51, 30.51, 30.53, 30.53, 30.53, 30.53, 30.53, 30.58, 30.58, 30.58, 30.58, 30.58, 30.71, 30.71, 30.71, 30.71, 30.71, 30.72, 30.72, 30.72, 30.72, 30.72, 30.66, 30.66, 30.66, 30.66, 30.66, 30.19, 30.19, 30.19, 30.19, 30.19, 29.84, 29.84, 29.84, 29.84, 29.84, 29.8, 29.8, 29.8, 29.8, 29.8, 29.68, 29.68, 29.68, 29.68, 29.68, 29.84, 29.84, 29.84, 29.84, 29.84, 29.96, 29.96, 29.96, 29.96, 29.96, 30.05, 30.05, 30.05, 30.05, 30.05, 30.03, 30.03, 30.03, 30.03, 30.03, 29.7, 29.7, 29.7, 29.7, 29.7, 29.31, 29.31, 29.31, 29.31, 29.31, 28.79, 28.79, 28.79, 28.79, 28.79, 28.31, 28.31, 28.31, 28.31, 28.31, 28.23, 28.23, 28.23, 28.23, 28.23, 28.32, 28.32, 28.32, 28.32, 28.32, 28.37, 28.37, 28.37, 28.37, 28.37, 28.41, 28.41, 28.41, 28.41, 28.41, 28.46, 28.46, 28.46, 28.46, 28.46, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.69, 28.69, 28.69, 28.69, 28.69, 28.8, 28.8, 28.8, 28.8, 28.8, 28.92, 28.92, 28.92, 28.92, 28.92, 28.98, 28.98, 28.98, 28.98, 28.98]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 536 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717000522 --> 1717001150
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.12, 0.12, 0.12, 0.12, 0.12, 0.4, 0.4, 0.4, 0.4, 0.4, 0.15, 0.15, 0.15, 0.15, 0.15, 0.1, 0.1, 0.1, 0.1, 0.1, 0.19, 0.19, 0.19, 0.19, 0.19, 0.1, 0.1, 0.1, 0.1, 0.1, 0.11, 0.11, 0.11, 0.11, 0.11, 0.18, 0.18, 0.18, 0.18, 0.18, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.2, 0.2, 0.2, 0.2, 0.2, 0.25, 0.25, 0.25, 0.25, 0.25, 0.3, 0.3, 0.3, 0.3, 0.3, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.27, 0.27, 0.27, 0.27, 0.27, 0.28, 0.28, 0.28, 0.28, 0.28, 0.19, 0.19, 0.19, 0.19, 0.19, 0.31, 0.31, 0.31, 0.31, 0.31, 0.13, 0.13, 0.13, 0.13, 0.13, 0.2, 0.2, 0.2, 0.2, 0.2, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.08, 0.08, 0.08, 0.08, 0.08, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.26, 0.26, 0.26, 0.26, 0.26, 0.2, 0.2, 0.2, 0.2, 0.2, 0.13, 0.13, 0.13, 0.13, 0.13, 0.18, 0.18, 0.18, 0.18, 0.18, 0.16, 0.16, 0.16, 0.16, 0.16, 0.21, 0.21, 0.21, 0.21, 0.21, 0.17, 0.17, 0.17, 0.17, 0.17, 0.15, 0.15, 0.15, 0.15, 0.15, 0.27, 0.27, 0.27, 0.27, 0.27, 0.35, 0.35, 0.35, 0.35, 0.35, 0.41, 0.41, 0.41, 0.41, 0.41, 0.35, 0.35, 0.35, 0.35, 0.35, 0.29, 0.29, 0.29, 0.29, 0.29, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.53, 0.53, 0.53, 0.53, 0.53, 0.63, 0.63, 0.63, 0.63, 0.63, 0.55, 0.55, 0.55, 0.55, 0.55, 0.43, 0.43, 0.43, 0.43, 0.43, 0.17, 0.17, 0.17, 0.17, 0.17, 0.23, 0.23, 0.23, 0.23, 0.23, 0.2, 0.2, 0.2, 0.2, 0.2, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.23, 0.23, 0.23, 0.23, 0.23, 0.26, 0.26, 0.26, 0.26, 0.26, 0.14, 0.14, 0.14, 0.14, 0.14, 0.24, 0.24, 0.24, 0.24, 0.24, 0.14, 0.14, 0.14, 0.14, 0.14, 0.11, 0.11, 0.11, 0.11, 0.11, 0.18, 0.18, 0.18, 0.18, 0.18, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 536 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717000522 --> 1717001150
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0]

ggml-ci

slaren · 2024-05-29T17:21:03Z

+                        ggml_is_contiguous_2(src0);
+                        ggml_is_contiguous_2(src1);


Missing GGML_ASSERT?

Thanks - fixing

slaren · 2024-05-29T17:21:49Z

+                        ggml_is_contiguous_2(src0);
+                        ggml_is_contiguous_2(src1);


* tests : add rope tests ggml-ci * ggml : fixes (hopefully) ggml-ci * tests : add non-cont tests ggml-ci * cuda : add asserts for rope/norm + fix DS2 ggml-ci * ggml : assert contiguousness * tests : reduce RoPE tests ggml-ci

mofosyne added the Review Complexity : Medium Generally require more time to grok but manageable by beginner to medium expertise level label May 29, 2024

ggerganov force-pushed the gg/rope-tests branch from 13bd89d to 20d70c6 Compare May 29, 2024 11:57

github-actions Bot added testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs Vulkan Issues specific to the Vulkan backend python python script changes ggml changes relating to the ggml tensor library for machine learning labels May 29, 2024

ggerganov force-pushed the gg/rope-tests branch from 20d70c6 to 3b0df03 Compare May 29, 2024 12:09

This was referenced May 29, 2024

ggml : add rope tests + fix neox ggml-org/ggml#844

Closed

Add support for DeepseekV2ForCausalLM #7519

Merged

ggerganov added 3 commits May 29, 2024 17:18

tests : add rope tests

9d5605f

ggml-ci

ggml : fixes (hopefully)

b822605

ggml-ci

tests : add non-cont tests

1e41f2f

ggml-ci

ggerganov force-pushed the gg/rope-tests branch from 4fa3b23 to 036007e Compare May 29, 2024 14:46

cuda : add asserts for rope/norm + fix DS2

5db268c

ggml-ci

ggerganov force-pushed the gg/rope-tests branch from 036007e to 5db268c Compare May 29, 2024 14:46

ggerganov changed the title ~~tests : add rope tests~~ ggml : fix YARN + add tests + add asserts May 29, 2024

ggml : assert contiguousness

a69935b

github-actions Bot added SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language Kompute https://github.com/KomputeProject/kompute/ labels May 29, 2024

tests : reduce RoPE tests

aa15e7d

ggml-ci

ggerganov marked this pull request as ready for review May 29, 2024 16:02

ggerganov merged commit fb76ec3 into master May 29, 2024

ggerganov deleted the gg/rope-tests branch May 29, 2024 17:17

This was referenced May 29, 2024

YaRN : correction to GPT-NeoX implementation #4093

Closed

fix YaRN for NeoX type ggml-org/ggml#839

Closed

slaren reviewed May 29, 2024

View reviewed changes

ggerganov added a commit that referenced this pull request May 29, 2024

metal : add missing asserts (#7617)

975ec63

ggerganov added a commit that referenced this pull request May 29, 2024

metal : remove invalid asserts (#7617)

55d6226

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

metal : add missing asserts (ggml-org#7617)

2ceac00

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

metal : remove invalid asserts (ggml-org#7617)

1126bf4

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

metal : add missing asserts (ggml-org#7617)

8b43fe0

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

metal : remove invalid asserts (ggml-org#7617)

6646c41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ggml : fix YARN + add tests + add asserts#7617

ggml : fix YARN + add tests + add asserts#7617
ggerganov merged 6 commits intomasterfrom
gg/rope-tests

ggerganov commented May 29, 2024

Uh oh!

github-actions Bot commented May 29, 2024 •

edited

Loading

Uh oh!

slaren May 29, 2024

Uh oh!

ggerganov May 29, 2024

Uh oh!

slaren May 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ggerganov commented May 29, 2024

Uh oh!

github-actions Bot commented May 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slaren May 29, 2024

Choose a reason for hiding this comment

Uh oh!

ggerganov May 29, 2024

Choose a reason for hiding this comment

Uh oh!

slaren May 29, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented May 29, 2024 •

edited

Loading