[shardformer] support sharded optimizer checkpointIO of HybridParallelPlugin#4540

Merged

Fridge003 merged 15 commits intohpcaitech:feature/shardformerfrom

Fridge003:feature/shardformer

Aug 31, 2023

Commits on Aug 31, 2023

implement sharded optimizer saving
Baizhou Zhang
committed
add more param info
Baizhou Zhang
committed
finish implementation of sharded optimizer saving
Baizhou Zhang
committed
fix bugs in optimizer sharded saving
Baizhou Zhang
committed
add pp+zero test
Baizhou Zhang
committed
param group loading
Baizhou Zhang
committed
greedy loading of optimizer
Baizhou Zhang
committed
fix bug when loading
Baizhou Zhang
committed
implement optimizer sharded saving
Baizhou Zhang
committed
add optimizer test & arrange checkpointIO utils
Baizhou Zhang
committed
fix gemini sharding state_dict
Baizhou Zhang
committed
add verbose option
Baizhou Zhang
committed
add loading of master params
Baizhou Zhang
committed
fix typehint
Baizhou Zhang
committed
fix master/working mapping in fp16 amp
Baizhou Zhang
committed