Skip to content

[SYCL] fix multi-gpu issue on sycl#8554

Merged
airMeng merged 11 commits intoggml-org:masterfrom
ClarkChin08:multi_device
Jul 25, 2024
Merged

[SYCL] fix multi-gpu issue on sycl#8554
airMeng merged 11 commits intoggml-org:masterfrom
ClarkChin08:multi_device

Conversation

@ClarkChin08
Copy link
Copy Markdown
Contributor

@ClarkChin08 ClarkChin08 commented Jul 18, 2024 •
edited by airMeng
Loading

fix multi-gpu crash issue by filter the platforms of sycl.

fix multi-gpu issue on sycl
cd296fe 
Signed-off-by: Chen Xi <xi2chen@intel.com>
@github-actions github-actions Bot added the SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language label Jul 18, 2024
@ClarkChin08 ClarkChin08 changed the title fix multi-gpu issue on sycl [SYCL] fix multi-gpu issue on sycl Jul 18, 2024
Chen Xi added 3 commits July 18, 2024 07:13
fix some typo
6b4f7b2 
Signed-off-by: Chen Xi <xi2chen@intel.com>
remove unnecessary whitespace
d096de2 
Signed-off-by: Chen Xi <xi2chen@intel.com>
file format issue
bd71cda 
Signed-off-by: Chen Xi <xi2chen@intel.com>
@ClarkChin08 ClarkChin08 mentioned this pull request Jul 18, 2024
Closed
@airMeng
Copy link
Copy Markdown
Contributor

airMeng commented Jul 18, 2024

@luoyu-intel

@airMeng
Copy link
Copy Markdown
Contributor

airMeng commented Jul 18, 2024

@ClarkChin08 can you attach the measurements results? like llama3-70B on 8 GPUs, memory consumption on each GPU, performance?

@ClarkChin08
Copy link
Copy Markdown
Contributor Author

ClarkChin08 commented Jul 18, 2024

This is the llama2-70b memory consumption data and the performance data:
0. run command: ./build/bin/llama-cli -m ../llama-2-70b-chat.Q4_0.gguf -p "how to build a website?" -n 400 -e -ngl 81 -sm layer

  1. memory consumption
    image

image
2. performance
image

sycl::queue can directly use as shared_ptr
6160a76 
Signed-off-by: Chen Xi <xi2chen@intel.com>
OuadiElfarouki
OuadiElfarouki reviewed Jul 18, 2024
View reviewed changes
Comment thread ggml/src/ggml-sycl/dpct/helper.hpp Outdated
Comment thread ggml/src/ggml-sycl/dpct/helper.hpp Outdated
OuadiElfarouki
OuadiElfarouki reviewed Jul 18, 2024
View reviewed changes
Comment thread ggml/src/ggml-sycl/dpct/helper.hpp
@mofosyne mofosyne added Review Complexity : Medium Generally require more time to grok but manageable by beginner to medium expertise level Review Complexity : High Generally require indepth knowledge of LLMs or GPUs and removed Review Complexity : Medium Generally require more time to grok but manageable by beginner to medium expertise level labels Jul 19, 2024
@ClarkChin08
fix the perf issue of multi-device
e4b86a1 
Signed-off-by: Chen Xi <xi2.chen@intel.com>
@ClarkChin08
fix intel mkl
22c72c5 
Signed-off-by: Chen Xi <xi2.chen@intel.com>
@ClarkChin08
Copy link
Copy Markdown
Contributor Author

ClarkChin08 commented Jul 24, 2024

image
This is the new performance table with input=6 and output=32

@airMeng airMeng requested a review from OuadiElfarouki July 24, 2024 06:56
OuadiElfarouki
OuadiElfarouki approved these changes Jul 24, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@OuadiElfarouki OuadiElfarouki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good thank you!

@ClarkChin08
add doc for sycl multi-card
fcce873 
Signed-off-by: Chen Xi <xi2.chen@intel.com>
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 25, 2024
@ClarkChin08
add linux part change on doc
8fe8086 
Signed-off-by: Chen Xi <xi2.chen@intel.com>
@ClarkChin08
fix typo
fc76684 
Signed-off-by: Chen Xi <xi2.chen@intel.com>
@airMeng airMeng merged commit ed67bcb into ggml-org:master Jul 25, 2024
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@ClarkChin08 @airMeng
[SYCL] fix multi-gpu issue on sycl (ggml-org#8554)
03bb312 

---------

Signed-off-by: Chen Xi <xi2chen@intel.com>
Co-authored-by: Meng, Hengyu <hengyu.meng@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

4 more reviewers

@NeoZhangJianyu NeoZhangJianyu NeoZhangJianyu left review comments

@airMeng airMeng airMeng approved these changes

@OuadiElfarouki OuadiElfarouki OuadiElfarouki approved these changes

@luoyu-intel luoyu-intel luoyu-intel approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation Review Complexity : High Generally require indepth knowledge of LLMs or GPUs SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ClarkChin08 @airMeng @NeoZhangJianyu @OuadiElfarouki @luoyu-intel @mofosyne