[gemini] support gradient accumulation#4869

Merged

Fridge003 merged 13 commits intohpcaitech:mainfrom

Fridge003:grad_accum

Oct 17, 2023

Commits on Oct 17, 2023

add test
Baizhou Zhang
committed
fix no_sync bug in low level zero plugin
Baizhou Zhang
committed
fix test
Baizhou Zhang
committed
add argument for grad accum
Baizhou Zhang
committed
add grad accum in backward hook for gemini
Baizhou Zhang
committed
finish implementation, rewrite tests
Baizhou Zhang
committed
fix test
Baizhou Zhang
committed
skip stuck model in low level zero test
Baizhou Zhang
committed
update doc
Baizhou Zhang
committed
optimize communication & fix gradient checkpoint
Baizhou Zhang
committed
modify doc
Baizhou Zhang
committed
cleaning codes
Baizhou Zhang
committed
update cpu adam fp16 case
Baizhou Zhang
committed