Move model.half() before model.to(device) to avoid GPU copy of full model #484
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Like probably many others, I have a lot of different virtualenvs, one for each project. Most of them are handled by `pyenv`. After installing according to these instructions I had issues with ´pyenv`and `miniconda` fighting over the $PATH of my system. But then I stumbled upon this nice solution on SO: https://stackoverflow.com/a/73139031 , upon which I have based my suggested changes. It runs perfectly on my M1 setup, with the anaconda setup as a virtual environment handled by pyenv. Feel free to incorporate these instructions as you see fit. Thanks a million for all your hard work.
Co-authored-by: Henry van Megen <hvanmegen@gmail.com>
…odel Speeds up initial load by ~1.5s, as tested on 8GB nvidia eGPU setup. Only avoids initial copying, doen't improve possible image size. Still same max 704x512 before OOM.
Collaborator
|
I"m not sure if you were aware of it, but this fix reduces the VRAM requirements for loading the model by 50%, to 2.17 G (from 4.3G). This probably means we can now load the model and run image generation on 4 GB cards. Congratulations! Because the PR was directed at main rather than development, I found it was simpler to copy your fix into development and commit it separately. So I'll be closing this PR, but the commit message acknowledges you (@mh-dm) as the author. |
Contributor
Author
|
Thank you! And I appreciate the call out.
And to top it off, I found a sweet simple fix that ups the speed by ~11%
and reduces max VRAM to 3.37G for 512x512. I've limited VRAM to supposedly
under 4GB with torch.cuda.set_per_process_memory_fraction(0.46, 0) but
nvidia-smi still reports 4403MiB so not sure if 4 GB cards are in the clear
yet.
However, my change conflicts with the development branch. So I'm hoping
you'd allow merging both this #484 into main and my upcoming change into
main. As for the development branch, I find the
attention.py CrossAttention.forward() code to be quite complex and I'm
looking into another strategy to bring down VRAM.
…On Sat, Sep 10, 2022 at 5:08 PM Lincoln Stein ***@***.***> wrote:
I"m not sure if you were aware of it, but this fix reduces the VRAM
requirements for loading the model by 50%, to 2.17 G (from 4.3G). This
probably means we can now load the model and run image generation on 4 GB
cards. Congratulations!
Because the PR was directed at main rather than development, I found it
was simpler to copy your fix into development and commit it separately. So
I'll be closing this PR, but the commit message acknowledges you ***@***.***
<https://github.com/mh-dm>) as the author.
—
Reply to this email directly, view it on GitHub
<#484 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACJAB6YNADMEURGCP6VCIDV5SI6JANCNFSM6AAAAAAQJJLZJI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Speeds up initial load by ~1.5s, as tested on 8GB nvidia eGPU setup. Only avoids initial copying, doesn't improve possible image size. Still same max 704x512 before OOM.