Distributed optimizer infrastructure for FP8 parameters#1723
Merged
crcrpar merged 2 commits intoNVIDIA:masterfrom Sep 29, 2023
Merged
Distributed optimizer infrastructure for FP8 parameters#1723crcrpar merged 2 commits intoNVIDIA:masterfrom
crcrpar merged 2 commits intoNVIDIA:masterfrom
Conversation
Signed-off-by: Tim Moon <tmoon@nvidia.com>
timmoon10
force-pushed
the
distopt-fp8
branch
from
fbc1ab4 to
c71321f
Compare
crcrpar
reviewed
Sep 12, 2023
Co-authored-by: Masaki Kozuki <mkozuki@nvidia.com>
8 tasks
crcrpar
approved these changes
Sep 29, 2023
minitu
pushed a commit
to minitu/apex
that referenced
this pull request
Sep 29, 2023
* Add distopt support for param syncs with non-floating-point dtypes Signed-off-by: Tim Moon <tmoon@nvidia.com> * Update apex/contrib/optimizers/distributed_fused_adam.py Co-authored-by: Masaki Kozuki <mkozuki@nvidia.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Masaki Kozuki <mkozuki@nvidia.com>
crcrpar
added a commit
that referenced
this pull request
Sep 30, 2023
* Add update_scale_hysteresis * Fix compile errors * Massively reduce LayerNorm/RMSNorm GPU memory usage in modern networks by tricking torch autograd (#1715) * input grad checks out * adding clamp gamma * Both old and proposed implementation checks out * 2 tests not yet passed due to numerical issues * mem_eff works * fast-layer-norm done * Moving mem-eff to templates * Relax tolerance for memory efficient backward * Fix backward api of python * Distributed optimizer infrastructure for FP8 parameters (#1723) * Add distopt support for param syncs with non-floating-point dtypes Signed-off-by: Tim Moon <tmoon@nvidia.com> * Update apex/contrib/optimizers/distributed_fused_adam.py Co-authored-by: Masaki Kozuki <mkozuki@nvidia.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Masaki Kozuki <mkozuki@nvidia.com> * Add unit test * Fix comment in unit test * Remove unnecessary bits --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Jaemin Choi <jaeminc@nvidia.com> Co-authored-by: Rui Wang <rui@helixon.com> Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: Masaki Kozuki <mkozuki@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR does some refactoring that will enable distributed optimizer support for FP8 parameters in NeMo. It adds the option to do parameter all-gathers in integer dtypes and adds two member functions -
_check_params_shard_dtypesand_param_copy_fragments- to handle casting into and out of the all-gather buffer. For now these functions will either do a direct cast for floating-point dtypes or copy the most significant bytes for other dtypes. I plan to override these functions in the NeMo derived class so that it casts to FP8, performs the all-gather in UINT8, and unpacks into a custom FP8 tensor class.This PR depends on #1719 and #1721.