[FP8] rebase main#5963

Merged

flybird11111 merged 132 commits intohpcaitech:feature/fp8_commfrom

flybird11111:feature/fp8_comm

Aug 6, 2024

Commits on Jun 28, 2024

Commits on Jul 5, 2024

Commits on Jul 8, 2024

[Feat] Diffusion Model(PixArtAlpha/StableDiffusion3) Support (#5838)
LRY89757
authored
[HotFix] CI,import,requirements-test for #5838 (#5892)

LRY89757
and
pre-commit-ci[bot]
authored

Commits on Jul 9, 2024

[Feature] Enable PP + SP for llama (#5868)

Edenzzzz
and
pre-commit-ci[bot]
authored

Commits on Jul 15, 2024

Commits on Jul 16, 2024

Commits on Jul 17, 2024

[release] update version (#5912)
ver217
authored

Commits on Jul 21, 2024

[FIX BUG] UnboundLocalError: cannot access local variable 'default_conversation' where it is not associated with a value (#5931)

zhurunhua
and
pre-commit-ci[bot]
authored

Commits on Jul 22, 2024

Commits on Jul 23, 2024

Add n_fused as an input from native_module (#5894)
insujang
authored

Commits on Jul 24, 2024

[FIX BUG] convert env param to int in (#5934)
flymin
authored

Commits on Jul 25, 2024

[Hotfix] Fix ZeRO typo #5936
Edenzzzz
authored

Commits on Jul 26, 2024

Commits on Jul 30, 2024

Commits on Aug 1, 2024

[test] add mixtral for sequence classification

botbw
authored and
ver217
committed
[test] add mixtral transformer test

botbw
authored and
ver217
committed
[moe] fix plugin

botbw
authored and
ver217
committed
[test] mixtra pp shard test

botbw
authored and
ver217
committed
[chore] handle non member group

botbw
authored and
ver217
committed
[zero] solve hang

botbw
authored and
ver217
committed
[test] pass mixtral shardformer test

botbw
authored and
ver217
committed
[moe] implement transit between non moe tp and ep

botbw
authored and
ver217
committed
[zero] solve hang

botbw
authored and
ver217
committed
[misc] solve booster hang by rename the variable

Hz188
authored and
ver217
committed
solve hang when parallel mode = pp + dp

Hz188
authored and
ver217
committed
[moe] implement submesh initialization

botbw
authored and
ver217
committed
[moe] add mixtral dp grad scaling when not all experts are activated

botbw
authored and
ver217
committed
[chore] manually revert unintended commit

botbw
authored and
ver217
committed
[chore] trivial fix

botbw
authored and
ver217
committed
[chore] arg pass & remove drop token

botbw
authored and
ver217
committed
[test] add mixtral modelling test

botbw
authored and
ver217
committed
[moe] implement tp

botbw
authored and
ver217
committed
[moe] test deepseek

botbw
authored and
ver217
committed
[moe] clean legacy code

botbw
authored and
ver217
committed
[Feature] MoE Ulysses Support (#5918)

authored and
ver217
committed
[chore] minor fix

botbw
authored and
ver217
committed
[moe] init moe plugin comm setting with sp

botbw
authored and
ver217
committed
moe sp + ep bug fix

Hz188
authored and
ver217
committed
[moe] finalize test (no pp)

botbw
authored and
ver217
committed
[moe] full test for deepseek and mixtral (pp + sp to fix)

botbw
authored and
ver217
committed
[chore] minor fix after rebase

botbw
authored and
ver217
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
ver217
committed
[chore] solve moe ckpt test failure and some other arg pass failure

botbw
authored and
ver217
committed
[moe] remove ops

botbw
authored and
ver217
committed
[test] fix test: test_zero1_2

botbw
authored and
ver217
committed
[bug] fix: somehow logger hangs the program

botbw
authored and
ver217
committed
[moe] deepseek moe sp support

Hz188
authored and
ver217
committed
[test] add check

botbw
authored and
ver217
committed
[deepseek] replace attn (a workaround for bug in transformers)

botbw
authored and
ver217
committed
[misc] skip redunant test

Hz188
authored and
ver217
committed
[misc] remove debug/print code

Hz188
authored and
ver217
committed
[moe] refactor mesh assignment

botbw
authored and
ver217
committed
Revert "[moe] implement submesh initialization"

botbw
authored and
ver217
committed
[chore] change moe_pg_mesh to private

botbw
authored and
ver217
committed
[misc] remove incompatible test config

Hz188
authored and
ver217
committed
[misc] fix ci failure: change default value to false in moe plugin

Hz188
authored and
ver217
committed
[misc] remove useless condition

Hz188
authored and
ver217
committed
[chore] docstring

botbw
authored and
ver217
committed
[moe] remove force_overlap_comm flag and add warning instead

botbw
authored and
ver217
committed
[doc] add MoeHybridParallelPlugin docstring

botbw
authored and
ver217
committed
[moe] solve dp axis issue

botbw
authored and
ver217
committed
[chore] remove redundant test case, print string & reduce test tokens

botbw
authored and
ver217
committed

Commits on Aug 2, 2024

[feat] Dist Loader for Eval (#5950)

authored
[lora] lora support hybrid parallel plugin (#5956)
wangbluo
authored
fp8 operators for compressed communication

BurkeHulk
authored and
flybird11111
committed
fix scaling algorithm in FP8 casting

BurkeHulk
authored and
flybird11111
committed
support fp8 communication in pipeline parallelism

BurkeHulk
authored and
flybird11111
committed
add fp8_communication flag in the script

BurkeHulk
authored and
flybird11111
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
flybird11111
committed
fix typo

GuangyaoZhang
authored and
flybird11111
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
flybird11111
committed
shardformer fp8

GuangyaoZhang
authored and
flybird11111
committed
fix rebase

GuangyaoZhang
authored and
flybird11111
committed
remove all to all

GuangyaoZhang
authored and
flybird11111
committed
fix shardformer fp8 communication training degradation

GuangyaoZhang
authored and
flybird11111
committed
[fp8] support all-gather flat tensor (#5932)

ver217
authored and
flybird11111
committed
Merge branch 'feature/fp8_comm' into feature/fp8_comm
flybird11111
authored

Commits on Aug 5, 2024

Commits on Aug 6, 2024

Update low_level_optim.py
flybird11111
authored