ZeRO 3 Offload#834

Merged

samyam merged 17 commits intomasterfrom

staging-zero3-release

Mar 8, 2021

Commits on Mar 3, 2021

Commits on Mar 4, 2021

formatting fix (#150)
Shaden Smith
authored
stage3 bugfix (API) update and simplified FP16 Z3 tests (#151)
Shaden Smith
authored
ZeRO-3 detach and race condition bugfixes (#149)
Shaden Smith
authored

Commits on Mar 5, 2021

Fix optimizer state_dict KeyError (#148)

tjruwase
and
jeffra
authored
fix for smaller SGS sizes, ensures each grad is backed by unique tensors (#152)
jeffra
authored

Commits on Mar 7, 2021

Simplifying the logic for getting averaged gradients (#153)
samyam
authored

Commits on Mar 8, 2021

skip for now
jeffra
committed
Z3 Docs redux (#154)
Shaden Smith
authored
removing some TODOs and commented code (#155)
Shaden Smith
authored
New Z3 defaults (#156)

Shaden Smith
and
jeffra
authored
formatting
jeffra
committed
megatron external params
Shaden Smith
committed
Merge branch 'master' into staging-zero3-release
jeffra
authored
Merge branch 'staging-zero3-release' of github.com:microsoft/DeepSpeed into staging-zero3-release
Shaden Smith
committed
Merge branch 'master' into staging-zero3-release
jeffra
authored