Develop/experiments#59

Merged

FrankLeeeee merged 14 commits intomainfrom

develop/experiments

Dec 9, 2021

Commits on Dec 9, 2021

Add gradient accumulation, fix lr scheduler

1SAA
authored and
FrankLeeeee
committed
fix FP16 optimizer and adapted torch amp with tensor parallel (#18)

ver217
authored and
FrankLeeeee
committed
fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes
FrankLeeeee
committed
fixed trainer
FrankLeeeee
committed
Revert "fixed trainer"
FrankLeeeee
committed
improved consistency between trainer, engine and schedule (#23)

FrankLeeeee
and
1SAA
committed
Split conv2d, class token, positional embedding in 2d, Fix random number in ddp

1SAA
authored and
FrankLeeeee
committed
Integrate 1d tensor parallel in Colossal-AI (#39)

WANG-CR
authored and
FrankLeeeee
committed
fixed 1D and 2D convergence (#38)
FrankLeeeee
committed
Feature/ddp (#49)

committed
Feature/pipeline (#40)

committed
optimized 3d layer to fix slow computation ; tested imagenet performance with 3d; reworked lr_scheduler config definition; fixed launch args; fixed some printing issues; simplified apis of 3d layer…

kurisusnowdeng
authored and
FrankLeeeee
committed
Update 2.5d layer code to get a similar accuracy on imagenet-1k dataset

BoxiangW
authored and
FrankLeeeee
committed
update api for better usability (#58)
FrankLeeeee
committed