[shardformer] support pipeline and hybrid parallelism#4441

Merged

ver217 merged 80 commits intomainfrom

Aug 15, 2023

Commits on Aug 15, 2023

[cluster] add process group mesh (#4039)
ver217
committed
[pipeline] add stage manager (#4093)
ver217
committed
[pipeline] implement p2p communication (#4100)
ver217
committed
[pipeline] refactor 1f1b schedule (#4115)
ver217
committed
[pipeline]add pipeline policy and bert forward (#4130)

CjhHa1
authored and
ver217
committed
[pipeline] add stage manager (#4093)
ver217
committed
[pipeline]add pipeline policy and bert forward (#4130)

CjhHa1
authored and
ver217
committed
[pipeline] build bloom model and policy , revise the base class of policy (#4161)

CjhHa1
authored and
ver217
committed
[pipeline] update shardformer policy
ver217
committed
[pipeline] update shardformer docstring
ver217
committed
[test] update shardformer tests
ver217
committed
[test] add shard util tests
ver217
committed
[shardformer] rename policy file name
ver217
committed
[shardformer] fix type hint
ver217
committed
[pipeline] add bert_for_pretraining bert_lmhead forward and policy (#4172)

CjhHa1
authored and
ver217
committed
[pipeline] move bert related pipeline components to shardformer (#4187)

CjhHa1
authored and
ver217
committed
[shardformer] support lazy init (#4202)
ver217
committed
[pipeline] Bert pipeline for shardformer and its tests (#4197)

CjhHa1
authored and
ver217
committed
[pipeline] Llama pipeline (#4205)

CjhHa1
authored and
ver217
committed
[pipeline] Llama causal lm and llama for sequence classification pipeline (#4208)

CjhHa1
authored and
ver217
committed
[pipeline] add bloom model pipeline (#4210)

CjhHa1
authored and
ver217
committed
[pipeline] Add Pipeline Forward for GPT2Model Shardformer (#4224)

Baizhou Zhang
authored and
ver217
committed
[shardformer] fix base policy (#4229)
ver217
committed
[pipeline] add pipeline forward for variants of gpt2 (#4238)

Baizhou Zhang
authored and
ver217
committed
[pipeline] All bert models (#4233)

CjhHa1
authored and
ver217
committed
[pipeline] finish bloom models pipeline and tests (#4223)

CjhHa1
authored and
ver217
committed
[bugs] hot fix some testing bugs for new models (#4268)

CjhHa1
authored and
ver217
committed
[pipeline] support shardformer for GPT2ForQuestionAnswering & complete pipeline support for GPT2 (#4245)

Baizhou Zhang
authored and
ver217
committed
[shardformer] support inplace sharding (#4251)
ver217
committed
[pipeline] refactor gpt2 pipeline forwards (#4287)

Baizhou Zhang
authored and
ver217
committed
[pipeline] OPT model pipeline (#4258)

CjhHa1
authored and
ver217
committed
[hotfix] fix opt pipeline (#4293)

CjhHa1
authored and
ver217
committed
[pipeline] reformat for unified design (#4283)

CjhHa1
authored and
ver217
committed
[pipeline] add pipeline support for T5Stack/T5EncoderModel (#4300)

Baizhou Zhang
authored and
ver217
committed
[pipeline] test pure pipeline process using llama (#4218)

CjhHa1
authored and
ver217
committed
[pipeline] add pipeline support for all T5 models (#4310)

Baizhou Zhang
authored and
ver217
committed
[shardformer] support pipeline base vit model (#4284)

authored and
ver217
committed
[plugin] add 3d parallel plugin (#4295)
ver217
committed
[hotfix] fix gemini and zero test (#4333)
ver217
committed
[pipeline] fix return_dict/fix pure_pipeline_test (#4331)

Baizhou Zhang
authored and
ver217
committed
[pipeline] add unit test for 1f1b (#4303)

Gy-Lu
authored and
ver217
committed
[pipeline] refactor test pipeline and remove useless utils in pipeline (#4324)

CjhHa1
authored and
ver217
committed
[pipeline] support fp32 for HybridPlugin/merge shardformer test and pipeline test into one file (#4354)

Baizhou Zhang
authored and
ver217
committed
Feature/vit support (#4182)

klhhhhh
authored and
ver217
committed
[shardformer] support SAM (#4231)

FoolPlayer
authored and
ver217
committed
[shardformer] support whisper (#4212)

FoolPlayer
authored and
ver217
committed
Feature/chatglm (#4240)

klhhhhh
authored and
ver217
committed
[shardformer] added tests

klhhhhh
authored and
ver217
committed
[shardformer] vit test finish and support

klhhhhh
authored and
ver217
committed
import chatglm

klhhhhh
authored and
ver217
committed
[shardformer] add test kit in model zoo for chatglm

klhhhhh
authored and
ver217
committed
[sharformer] add first version of policy of chatglm

klhhhhh
authored and
ver217
committed
[shardformer] polish chatglm code

klhhhhh
authored and
ver217
committed
[shardformer] polish code

klhhhhh
authored and
ver217
committed
[shardformer] support chatglm without layernorm

klhhhhh
authored and
ver217
committed
[shardformer] delete some file

klhhhhh
authored and
ver217
committed
[shardformer] ChatGLM support layernorm sharding

klhhhhh
authored and
ver217
committed
[shardformer] register without auto policy

klhhhhh
authored and
ver217
committed
[shardformer] pre-commit check files

klhhhhh
authored and
ver217
committed
[shardformer] support ChatGLMForConditionalGeneration & add fusedlayernorm for vit

klhhhhh
authored and
ver217
committed
[shardformer] support Blip2 (#4243)

FoolPlayer
authored and
ver217
committed
update some module with new api version

FoolPlayer
authored and
ver217
committed
[test] skip some not compatible models

FoolPlayer
authored and
ver217
committed
[test] Hotfix/fix some model test and refactor check util api (#4369)

FoolPlayer
authored and
ver217
committed
[shardformer] add util functions for shardformer tests/fix sync_shared_param (#4366)

Baizhou Zhang
authored and
ver217
committed
[pipeline] add chatglm (#4363)

CjhHa1
authored and
ver217
committed
[Shardformer] Merge flash attention branch to pipeline branch (#4362)

authored and
ver217
committed
[pipeline] rewrite t5 tests & support multi-tensor transmitting in pipeline (#4388)

Baizhou Zhang
authored and
ver217
committed
[shardformer] update shardformer to use flash attention 2 (#4392)

flybird11111
authored and
ver217
committed
[shardformer] test all optimizations (#4399)

flybird11111
authored and
ver217
committed
[pipeline] rewrite bert tests and fix some bugs (#4409)

CjhHa1
authored and
ver217
committed
[shardformer]fix, test gpt2 for AMP+TP (#4403)

flybird11111
authored and
ver217
committed
[shardformer] rewrite tests for opt/bloom/llama/vit/chatglm (#4395)

Baizhou Zhang
authored and
ver217
committed
[shardformer] update tests for all optimization (#4413)

flybird11111
authored and
ver217
committed
[shardformer]update t5 tests for using all optimizations. (#4407)

flybird11111
authored and
ver217
committed
[shardformer] update bloom/llama/vit/chatglm tests (#4420)

flybird11111
authored and
ver217
committed
[misc] resolve code factor issues (#4433)
ver217
committed
[misc] update requirements
ver217
committed
[shardformer] fix embedding
ver217
committed
[shardformer] fix import
ver217
committed