Support TP-compatible Torch AMP and Update trainer API#27

Merged

ver217 merged 7 commits intomainfrom

develop/experiments

Nov 18, 2021

Commits on Nov 8, 2021

Commits on Nov 10, 2021

fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes
FrankLeeeee
committed

Commits on Nov 11, 2021

Commits on Nov 15, 2021

improved consistency between trainer, engine and schedule (#23)

FrankLeeeee
and
1SAA
authored