Refactor ZeRO configs to use Pydantic#2004
Merged
tjruwase merged 34 commits intodeepspeedai:masterfrom Jul 27, 2022
Merged
Conversation
mrwyattii
changed the title
mrwyattii
changed the title
Collaborator
Collaborator
|
Wonderful! Thank you for the heads up, Jeff! Also probably extending hjson-support to the rest of DS config would be great, but doesn't have to be part of this PR - just a suggestion. |
Collaborator
|
Absolutely, I think adding hjson + pydantic support is the eventual goal. Once all of our configs are managed by pydantic the goal is to also replace the config-json page with a read-the-docs style page that is auto-generated from doc-strings that are easier to maintain and more accurate. However, one step at a time :) |
…into pydantic-zero
mrwyattii
requested review from
RezaYazdaniAminabadi,
ShadenSmith,
arashb,
awan-10,
cli99,
conglongli,
duli2012,
eltonzheng,
minjiaz,
samyam,
tjruwase,
xiaoxiawu-microsoft and
yaozhewei
as code owners
tjruwase
reviewed
Jul 11, 2022
tjruwase
reviewed
Jul 11, 2022
tjruwase
reviewed
Jul 11, 2022
tjruwase
reviewed
Jul 26, 2022
| prev_best_metric_val=0): | ||
| config_zero = tuning_space.get(ZERO_OPTIMIZATION, {}) | ||
| stage = config_zero.get(ZERO_OPTIMIZATION_STAGE, ZERO_OPTIMIZATION_STAGE_DEFAULT) | ||
| stage = config_zero.stage |
Contributor
There was a problem hiding this comment.
Isn't this a bug if config_zero equals {}, and leads to evaluation of {}.stage?
tjruwase
reviewed
Jul 26, 2022
| pydantic.BaseModel to allow for deprecated fields. To enable this feature, | ||
| add deprecated=True to pydantic.Field: | ||
|
|
||
| my_dep_field: int = Field(0, deprecated=True) |
Contributor
There was a problem hiding this comment.
Can you document first argument of Field(, i.e., 0.?
tjruwase
approved these changes
Jul 26, 2022
rahilbathwal5
added a commit
to neevaco/DeepSpeed
that referenced
this pull request
Aug 3, 2022
This reverts commit 5997589.
rahilbathwal5
added a commit
to neevaco/DeepSpeed
that referenced
this pull request
Aug 3, 2022
This reverts commit 5997589.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings Pydantic into the zero config:
DeepSpeedConfigModelclass which ZeRO and offload configs inherit. This class allows us to enforce extra params are not provided in the config and also implements logic for deprecated config paramsconstants.pyandconfig.pyinto a single Pydantic class that contains the default values for all parametersconstants.py