IFU-master-2021-11-23 #48
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Shaden Smith <shaden.smith@microsoft.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com> Co-authored-by: Reza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com> Co-authored-by: eltonzheng <eltonz@microsoft.com> Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
…edai#1411) Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
* Fix spelling errors in inference tutorial * Remove unused imports in inference tutorial * Fix inference tutorial code to work with 1 GPU
* Fix typos in docs/ * Fix typos in code comments and output strings * Fix typos in the code itself * Fix typos in tests/ Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
…eedai#1376) * Update checkpointing.py * Fix formatting * Add flexibility of pipeline module and engine * Separate PRs * Separate PRs * Update checkpointing.py * Update checkpointing.py * Reflect code review for contiguous activation checkpointing * remove useless condition Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
* Revise param_shapes to be a list of ordered dict * test i can push * add tests; split z2 and z3 into separate funcs Co-authored-by: Xiaopeng Li <xiaopel@amazon.com> Co-authored-by: Stas Bekman <stas@stason.org> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
HF tests `image-classification` requirements have been fixed to not require pt-1.9
* DeepSpeedInferenceConfig get epsilon value from config * epsilon -> layer_norm_eps to keep var name same as in DeepSpeedTransformerConfig * DeepSpeedTransformerConfig get epsilon value from config * configurabale stochastic_mode eg: 1. For LM pre-training True 2. For LM fine-tuning on task False * Updated replace_module.py checking layer_norm_eps is attribute of config default 1e-12 Co-authored-by: Reza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
…1397) * fix the workspace allocation for the transformer kernel * change layer-id type & rm one unit test due to OOM
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Thomas <thomas@Thomass-MacBook-Pro.local> Co-authored-by: Shaden Smith <shaden.smith@microsoft.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Tunji Ruwase <olruwase@microsoft.com> Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* CL+PP * add TODO
* update docs w. 530b info * add tutorial link
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: skpig <1900012999@pku.edu.cn>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
…1530) * Add warmup_type arguments in WarmupLR and WarmupDecayLR * Add warmup_type unit test * replace hardcoded constants with global vars Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
* [squash] Staging autotuning v4 Co-authored-by: Cheng Li <pistasable@gmail.com> Co-authored-by: Minjia Zhang <minjiaz@microsoft.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com> * add new extra, guard xgboost, cleanup dead files (deepspeedai#268) * Fix autotuning docs (deepspeedai#1553) * fix docs * rewording the goal * fix typos * fix typos (deepspeedai#1556) * fix typos * fix format * fix bug (deepspeedai#1557) * fix bug Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Minjia Zhang <minjiaz@microsoft.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
…eck if gradient data is sparse (deepspeedai#1562) Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
…dai#1564) * Enforce nccl/rccl alignment of start location of each shard * Making yapf happy Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
* guard tabulate package in case autotuning isn't installed * address comment
…pspeedai#960) Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
…epSpeedExamples to latest commit 206e48b7638d1a36e90466071bd0f50844a3002b after ROCm/DeepSpeedExamples#16 IFU
Collaborator
|
CI runs are reportedly still broken. And the GPT2 CI script needs to be updated to use the Megatron v1.1.5 script. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolved conflicts in setup.py, op_builder/init.py and pointed DeepSpeedExamples to the latest commit, 206e48b7638d1a36e90466071bd0f50844a3002b, after ROCm/DeepSpeedExamples#16 IFU
init.txt
setup.txt
I have verified Megatron-LM-v1.1.5 gpt2 and Bing BERT. Both work fine.
cc: @jithunnair-amd