Skip to content

Clarify default MMQ for CUDA and LLAMA_CUDA_FORCE_MMQ flag#8115

Merged
JohannesGaessler merged 2 commits intoggml-org:masterfrom
isaac-mcfadyen:mmq-readme-update
Jun 26, 2024
Merged

Clarify default MMQ for CUDA and LLAMA_CUDA_FORCE_MMQ flag#8115
JohannesGaessler merged 2 commits intoggml-org:masterfrom
isaac-mcfadyen:mmq-readme-update

Conversation

@isaac-mcfadyen
Copy link
Copy Markdown
Contributor

@isaac-mcfadyen isaac-mcfadyen commented Jun 25, 2024

Summary

  • In CUDA: use MMQ instead of cuBLAS by default #8075, MMQ was enabled by default on GPUs with int8 tensor core support.
  • A short description of the LLAMA_CUDA_FORCE_MMQ was added to the README. As it currently stands though, the message makes it seem like MMQ will not be used unless enabled with the flag.
  • The message says "flag forces MMQ to be enabled on GPUs without int8 support" but doesn't really say that it will be enabled by default on GPUs with int8 support.
  • This PR just adds a short blurb stating that MMQ is enabled by default on GPUs with int8 tensor core support, and that the flag forces it for all GPUs.

@mofosyne mofosyne added the Review Complexity : Low Trivial changes to code that most beginner devs (or those who want a break) can tackle. e.g. UI fix label Jun 25, 2024
@slaren slaren requested a review from JohannesGaessler June 25, 2024 20:13
JohannesGaessler
JohannesGaessler requested changes Jun 25, 2024
View reviewed changes
Comment thread README.md Outdated
@isaac-mcfadyen @JohannesGaessler
Add suggestions from review
37ff709 
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
@JohannesGaessler JohannesGaessler merged commit 8854044 into ggml-org:master Jun 26, 2024
Nexesenex pushed a commit to Nexesenex/croco.cpp that referenced this pull request Jun 28, 2024
@isaac-mcfadyen @JohannesGaessler @Nexesenex
Clarify default MMQ for CUDA and LLAMA_CUDA_FORCE_MMQ flag (ggml-org#…
8bdd3f4 
…8115)

* Add message about int8 support

* Add suggestions from review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Nexesenex pushed a commit to Nexesenex/croco.cpp that referenced this pull request Jun 28, 2024
@isaac-mcfadyen @JohannesGaessler @Nexesenex
Clarify default MMQ for CUDA and LLAMA_CUDA_FORCE_MMQ flag (ggml-org#…
8591377 
…8115)

* Add message about int8 support

* Add suggestions from review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
MagnusS0 pushed a commit to MagnusS0/llama.cpp-normistral-tokenizer that referenced this pull request Jul 1, 2024
@isaac-mcfadyen @JohannesGaessler @MagnusS0
Clarify default MMQ for CUDA and LLAMA_CUDA_FORCE_MMQ flag (ggml-org#…
22413b0 
…8115)

* Add message about int8 support

* Add suggestions from review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@isaac-mcfadyen @JohannesGaessler
Clarify default MMQ for CUDA and LLAMA_CUDA_FORCE_MMQ flag (ggml-org#…
107239a 
…8115)

* Add message about int8 support

* Add suggestions from review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@slaren slaren slaren approved these changes

@JohannesGaessler JohannesGaessler JohannesGaessler approved these changes

Assignees

No one assigned

Labels

Review Complexity : Low Trivial changes to code that most beginner devs (or those who want a break) can tackle. e.g. UI fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@isaac-mcfadyen @slaren @JohannesGaessler @mofosyne