[SYCL] Disable iqx on windows as WA by airMeng · Pull Request #6435 · ggml-org/llama.cpp

airMeng · 2024-04-02T07:13:52Z

github-actions · 2024-04-02T07:27:32Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3: 500 iterations 🚀

Concurrent users: 8, duration: 10m
HTTP request : avg=9364.1ms p(90)=25949.07ms fails=0, finish reason: stop=500 truncated=0
Prompt processing (pp): avg=242.74tk/s p(90)=733.9tk/s total=198.88tk/s
Token generation (tg): avg=100.05tk/s p(90)=279.74tk/s total=129.83tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=sycl-win-crash commit=d100b7511cc7a92faca3c4e86db5806afb8da747

Time series

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 500 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1712047439 --> 1712048067
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 28.39, 28.39, 28.39, 28.39, 28.39, 314.59, 314.59, 314.59, 314.59, 314.59, 406.83, 406.83, 406.83, 406.83, 406.83, 462.57, 462.57, 462.57, 462.57, 462.57, 511.38, 511.38, 511.38, 511.38, 511.38, 515.2, 515.2, 515.2, 515.2, 515.2, 520.64, 520.64, 520.64, 520.64, 520.64, 548.47, 548.47, 548.47, 548.47, 548.47, 551.54, 551.54, 551.54, 551.54, 551.54, 562.5, 562.5, 562.5, 562.5, 562.5, 566.26, 566.26, 566.26, 566.26, 566.26, 588.59, 588.59, 588.59, 588.59, 588.59, 627.99, 627.99, 627.99, 627.99, 627.99, 618.17, 618.17, 618.17, 618.17, 618.17, 602.57, 602.57, 602.57, 602.57, 602.57, 612.92, 612.92, 612.92, 612.92, 612.92, 615.39, 615.39, 615.39, 615.39, 615.39, 614.39, 614.39, 614.39, 614.39, 614.39, 631.26, 631.26, 631.26, 631.26, 631.26, 632.36, 632.36, 632.36, 632.36, 632.36, 631.38, 631.38, 631.38, 631.38, 631.38, 630.94, 630.94, 630.94, 630.94, 630.94, 635.36, 635.36, 635.36, 635.36, 635.36, 639.04, 639.04, 639.04, 639.04, 639.04, 654.0, 654.0, 654.0, 654.0, 654.0, 654.29, 654.29, 654.29, 654.29, 654.29, 652.47, 652.47, 652.47, 652.47, 652.47, 653.99, 653.99, 653.99, 653.99, 653.99, 662.97, 662.97, 662.97, 662.97, 662.97, 661.23, 661.23, 661.23, 661.23, 661.23, 661.11, 661.11, 661.11, 661.11, 661.11, 664.77, 664.77, 664.77, 664.77, 664.77, 668.26, 668.26, 668.26, 668.26, 668.26, 667.93, 667.93, 667.93, 667.93, 667.93, 670.37, 670.37, 670.37, 670.37, 670.37, 674.8, 674.8, 674.8, 674.8, 674.8, 680.41, 680.41, 680.41, 680.41, 680.41, 690.85, 690.85, 690.85, 690.85, 690.85, 693.28, 693.28, 693.28, 693.28, 693.28, 691.89, 691.89, 691.89, 691.89, 691.89, 691.95, 691.95, 691.95, 691.95, 691.95, 692.66, 692.66, 692.66, 692.66, 692.66, 694.03, 694.03, 694.03, 694.03, 694.03, 701.46, 701.46, 701.46, 701.46, 701.46, 703.37, 703.37, 703.37, 703.37, 703.37, 702.3, 702.3, 702.3, 702.3, 702.3, 699.75, 699.75, 699.75, 699.75, 699.75, 697.16, 697.16, 697.16, 697.16, 697.16, 695.29, 695.29, 695.29, 695.29, 695.29, 693.84, 693.84, 693.84, 693.84, 693.84, 694.23, 694.23, 694.23, 694.23, 694.23, 699.22, 699.22, 699.22, 699.22, 699.22, 699.14, 699.14, 699.14, 699.14, 699.14, 703.75, 703.75, 703.75, 703.75, 703.75, 703.77, 703.77, 703.77, 703.77, 703.77, 700.97, 700.97, 700.97, 700.97, 700.97, 702.76, 702.76, 702.76, 702.76, 702.76, 701.76, 701.76, 701.76, 701.76, 701.76, 701.36, 701.36, 701.36, 701.36, 701.36, 704.34, 704.34, 704.34, 704.34, 704.34, 705.47, 705.47, 705.47, 705.47, 705.47]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 500 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1712047439 --> 1712048067
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 27.86, 27.86, 27.86, 27.86, 27.86, 16.4, 16.4, 16.4, 16.4, 16.4, 17.25, 17.25, 17.25, 17.25, 17.25, 17.46, 17.46, 17.46, 17.46, 17.46, 18.81, 18.81, 18.81, 18.81, 18.81, 20.09, 20.09, 20.09, 20.09, 20.09, 20.4, 20.4, 20.4, 20.4, 20.4, 20.41, 20.41, 20.41, 20.41, 20.41, 20.48, 20.48, 20.48, 20.48, 20.48, 20.47, 20.47, 20.47, 20.47, 20.47, 20.43, 20.43, 20.43, 20.43, 20.43, 20.37, 20.37, 20.37, 20.37, 20.37, 19.99, 19.99, 19.99, 19.99, 19.99, 19.46, 19.46, 19.46, 19.46, 19.46, 19.0, 19.0, 19.0, 19.0, 19.0, 18.96, 18.96, 18.96, 18.96, 18.96, 19.17, 19.17, 19.17, 19.17, 19.17, 19.15, 19.15, 19.15, 19.15, 19.15, 18.99, 18.99, 18.99, 18.99, 18.99, 18.81, 18.81, 18.81, 18.81, 18.81, 18.68, 18.68, 18.68, 18.68, 18.68, 18.58, 18.58, 18.58, 18.58, 18.58, 18.6, 18.6, 18.6, 18.6, 18.6, 18.67, 18.67, 18.67, 18.67, 18.67, 18.55, 18.55, 18.55, 18.55, 18.55, 18.63, 18.63, 18.63, 18.63, 18.63, 18.73, 18.73, 18.73, 18.73, 18.73, 18.73, 18.73, 18.73, 18.73, 18.73, 18.62, 18.62, 18.62, 18.62, 18.62, 18.54, 18.54, 18.54, 18.54, 18.54, 18.57, 18.57, 18.57, 18.57, 18.57, 18.63, 18.63, 18.63, 18.63, 18.63, 18.66, 18.66, 18.66, 18.66, 18.66, 18.8, 18.8, 18.8, 18.8, 18.8, 18.84, 18.84, 18.84, 18.84, 18.84, 18.77, 18.77, 18.77, 18.77, 18.77, 18.74, 18.74, 18.74, 18.74, 18.74, 18.64, 18.64, 18.64, 18.64, 18.64, 18.58, 18.58, 18.58, 18.58, 18.58, 18.61, 18.61, 18.61, 18.61, 18.61, 18.63, 18.63, 18.63, 18.63, 18.63, 18.65, 18.65, 18.65, 18.65, 18.65, 18.68, 18.68, 18.68, 18.68, 18.68, 18.62, 18.62, 18.62, 18.62, 18.62, 18.56, 18.56, 18.56, 18.56, 18.56, 18.47, 18.47, 18.47, 18.47, 18.47, 18.18, 18.18, 18.18, 18.18, 18.18, 18.02, 18.02, 18.02, 18.02, 18.02, 17.74, 17.74, 17.74, 17.74, 17.74, 17.57, 17.57, 17.57, 17.57, 17.57, 17.52, 17.52, 17.52, 17.52, 17.52, 17.56, 17.56, 17.56, 17.56, 17.56, 17.58, 17.58, 17.58, 17.58, 17.58, 17.62, 17.62, 17.62, 17.62, 17.62, 17.64, 17.64, 17.64, 17.64, 17.64, 17.65, 17.65, 17.65, 17.65, 17.65, 17.64, 17.64, 17.64, 17.64, 17.64, 17.61, 17.61, 17.61, 17.61, 17.61, 17.53, 17.53, 17.53, 17.53, 17.53, 17.56, 17.56, 17.56, 17.56, 17.56]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 500 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1712047439 --> 1712048067
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.01, 0.01, 0.01, 0.01, 0.01, 0.25, 0.25, 0.25, 0.25, 0.25, 0.17, 0.17, 0.17, 0.17, 0.17, 0.08, 0.08, 0.08, 0.08, 0.08, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.21, 0.21, 0.21, 0.21, 0.21, 0.16, 0.16, 0.16, 0.16, 0.16, 0.11, 0.11, 0.11, 0.11, 0.11, 0.23, 0.23, 0.23, 0.23, 0.23, 0.21, 0.21, 0.21, 0.21, 0.21, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.14, 0.14, 0.14, 0.14, 0.27, 0.27, 0.27, 0.27, 0.27, 0.3, 0.3, 0.3, 0.3, 0.3, 0.26, 0.26, 0.26, 0.26, 0.26, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.22, 0.22, 0.22, 0.22, 0.22, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.12, 0.12, 0.12, 0.12, 0.12, 0.33, 0.33, 0.33, 0.33, 0.33, 0.21, 0.21, 0.21, 0.21, 0.21, 0.12, 0.12, 0.12, 0.12, 0.12, 0.15, 0.15, 0.15, 0.15, 0.15, 0.11, 0.11, 0.11, 0.11, 0.11, 0.19, 0.19, 0.19, 0.19, 0.19, 0.14, 0.14, 0.14, 0.14, 0.14, 0.19, 0.19, 0.19, 0.19, 0.19, 0.14, 0.14, 0.14, 0.14, 0.14, 0.24, 0.24, 0.24, 0.24, 0.24, 0.19, 0.19, 0.19, 0.19, 0.19, 0.16, 0.16, 0.16, 0.16, 0.16, 0.22, 0.22, 0.22, 0.22, 0.22, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.28, 0.28, 0.28, 0.28, 0.28, 0.38, 0.38, 0.38, 0.38, 0.38, 0.48, 0.48, 0.48, 0.48, 0.48, 0.56, 0.56, 0.56, 0.56, 0.56, 0.5, 0.5, 0.5, 0.5, 0.5, 0.42, 0.42, 0.42, 0.42, 0.42, 0.27, 0.27, 0.27, 0.27, 0.27, 0.15, 0.15, 0.15, 0.15, 0.15, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.21, 0.21, 0.21, 0.21, 0.21, 0.31, 0.31, 0.31, 0.31, 0.31, 0.26, 0.26, 0.26, 0.26, 0.26, 0.18, 0.18, 0.18, 0.18, 0.18, 0.07, 0.07, 0.07, 0.07, 0.07, 0.11, 0.11, 0.11, 0.11, 0.11]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 500 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1712047439 --> 1712048067
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0]

NeoZhangJianyu · 2024-04-02T08:43:45Z

-            auto kmask_iq2xs_ptr_ct1 = kmask_iq2xs.get_ptr();
+            auto iq2xxs_grid_ptr_ct1 = &iq2xxs_grid[0];
+            auto ksigns_iq2xs_ptr_ct1 = &ksigns_iq2xs[0];
+            auto kmask_iq2xs_ptr_ct1 = &kmask_iq2xs[0];


remove kmask_iq2xs_ptr_ct1 and replaced by kmask_iq2xs in next code.

auto kmask_iq2xs_ptr_ct1 = &kmask_iq2xs[0];

It's same for other codes, like:
auto kmask_iq2xs_ptr_ct1 = &kmask_iq2xs[0];

remove them.

* disable iqx on windows as WA * array instead of global_memory

disable iqx on windows as WA

9d49a41

airMeng requested a review from NeoZhangJianyu April 2, 2024 07:15

airMeng added 3 commits April 2, 2024 07:56

fix typo

5ead10f

fix indent

d2ecac5

array instead of global_memory

d100b75

airMeng mentioned this pull request Apr 2, 2024

[SYCL] fail to load llama.dll compiled by icx with -DBUILD_SHARED_LIBS=on flag on Windows #6309

Closed

NeoZhangJianyu approved these changes Apr 3, 2024

View reviewed changes

airMeng merged commit 5260486 into master Apr 3, 2024

airMeng mentioned this pull request Apr 3, 2024

Enabling ollama to run on Intel GPUs with SYCL backend ollama/ollama#3278

Merged

airMeng deleted the sycl-win-crash branch April 3, 2024 07:34

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

[SYCL] Disable iqx on windows as WA (ggml-org#6435)

f9af101

* disable iqx on windows as WA * array instead of global_memory

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

[SYCL] Disable iqx on windows as WA (ggml-org#6435)

d29450b

* disable iqx on windows as WA * array instead of global_memory

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SYCL] Disable iqx on windows as WA#6435

[SYCL] Disable iqx on windows as WA#6435
airMeng merged 4 commits intomasterfrom
sycl-win-crash

airMeng commented Apr 2, 2024

Uh oh!

github-actions Bot commented Apr 2, 2024 •

edited

Loading

Uh oh!

NeoZhangJianyu Apr 2, 2024

Uh oh!

NeoZhangJianyu Apr 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

airMeng commented Apr 2, 2024

Uh oh!

github-actions Bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NeoZhangJianyu Apr 2, 2024

Choose a reason for hiding this comment

Uh oh!

NeoZhangJianyu Apr 2, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Apr 2, 2024 •

edited

Loading