Skip to content

Increase 3B scratch buffers.#1698

Merged
SlyEcho merged 1 commit intomasterfrom
fix-3b-mem-req
Jun 5, 2023
Merged

Increase 3B scratch buffers.#1698
SlyEcho merged 1 commit intomasterfrom
fix-3b-mem-req

Conversation

@SlyEcho
Copy link
Copy Markdown
Contributor

@SlyEcho SlyEcho commented Jun 5, 2023

The 128 MB was too optimistic.
Too bad it is not dynamically computed.

Ref: #1588 (comment)

@SlyEcho
Increase 3B scratch buffers.
dffd2a7 
The 128 MB was too optimistic.
Too bad it is not dynamically computed.
@SlyEcho SlyEcho requested a review from Green-Sky June 5, 2023 09:09
@SlyEcho
Copy link
Copy Markdown
Contributor Author

SlyEcho commented Jun 5, 2023

@LostRuins, you suggested increasing to 256MB.

Is that going to be enough? What is the best way to test it?

@SlyEcho
Copy link
Copy Markdown
Contributor Author

SlyEcho commented Jun 5, 2023

It seems to run for me with both Q4_0 and Q5_1 on context 2048 and batch 512 with only 128MB.

@LostRuins
Copy link
Copy Markdown
Collaborator

LostRuins commented Jun 5, 2023 •
edited
Loading

Hi @SlyEcho I guess the best way to test it is to download and run your OpenLLAMA 3B ggml quant (which I don't know if I am allowed to link here). But running it as q4_0 with 256MB scratch at batch size 512, for 2048 context, it seems to work for me. I dunno if there may be some boundary parameter that could still fail.

128MB crashes for me at around 1.5k token mark

LostRuins
LostRuins approved these changes Jun 5, 2023
View reviewed changes
Copy link
Copy Markdown
Collaborator

@LostRuins LostRuins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@SlyEcho
Copy link
Copy Markdown
Contributor Author

SlyEcho commented Jun 5, 2023

OK, I will merge it. Memory use is probably is also dependent on the user's system and build.

@SlyEcho SlyEcho merged commit 5220a99 into master Jun 5, 2023
@SlyEcho SlyEcho deleted the fix-3b-mem-req branch June 5, 2023 10:43
@Green-Sky
Copy link
Copy Markdown
Collaborator

Green-Sky commented Jun 5, 2023

download and run your OpenLLAMA 3B ggml quant (which I don't know if I am allowed to link here)

yes you are :) . openllama is an opensource reproduction. licensed under the apache2 license

@Bearsaerker Bearsaerker mentioned this pull request Mar 12, 2025
Closed
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@SlyEcho
Increase 3B scratch buffers. (ggml-org#1698)
517c83a 
The 128 MB was too optimistic.
Too bad it is not dynamically computed.
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
@SlyEcho
Increase 3B scratch buffers. (ggml-org#1698)
c0bd595 
The 128 MB was too optimistic.
Too bad it is not dynamically computed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@Green-Sky Green-Sky Green-Sky approved these changes

@LostRuins LostRuins LostRuins approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@SlyEcho @LostRuins @Green-Sky