[zero bubble] support zero#6080

Merged

ver217 merged 167 commits intohpcaitech:feature/zerobubblefrom

flybird11111:zbv-zero

Oct 11, 2024

Commits on Jul 1, 2024

fp8 operators for compressed communication
BurkeHulk
authored

Commits on Jul 4, 2024

Commits on Jul 10, 2024

fix typo
GuangyaoZhang
committed

Commits on Jul 16, 2024

Commits on Jul 17, 2024

Commits on Jul 18, 2024

Commits on Jul 24, 2024

[fp8] support all-gather flat tensor (#5932)
ver217
authored

Commits on Aug 2, 2024

Commits on Aug 5, 2024

[Feature] llama shardformer fp8 support (#5938)
GuangyaoZhang
authored

Commits on Aug 6, 2024

Commits on Aug 7, 2024

Commits on Aug 8, 2024

Commits on Aug 9, 2024

Commits on Aug 12, 2024

[fp8] support hybrid parallel plugin (#5982)
wangbluo
authored

Commits on Aug 13, 2024

[fp8] refactor fp8 linear with compile (#5993)
ver217
authored

Commits on Aug 14, 2024

[fp8] support asynchronous FP8 communication (#5997)

flybird11111
and
pre-commit-ci[bot]
authored

Commits on Aug 15, 2024

Commits on Aug 16, 2024

Commits on Aug 17, 2024

Commits on Aug 19, 2024

Commits on Aug 20, 2024

Commits on Aug 21, 2024

Commits on Aug 26, 2024

Commits on Aug 27, 2024

Commits on Aug 28, 2024

Commits on Aug 29, 2024

[FP8] unsqueeze scale to make it compatible with torch.compile (#6040)
GuangyaoZhang
authored

Commits on Sep 2, 2024

[colossalai/checkpoint_io/...] fix bug in load_state_dict_into_model; format error msg (#6020)

flymin
and
pre-commit-ci[bot]
authored

Commits on Sep 3, 2024

Commits on Sep 9, 2024

[fp8] disable all_to_all_fp8 in intranode (#6045)

BurkeHulk
and
pre-commit-ci[bot]
authored

Commits on Sep 10, 2024

Commits on Sep 11, 2024

Commits on Sep 13, 2024

fix the sp
wangbluo
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot]
committed
fix the attn
wangbluo
committed
fix
wangbluo
committed
fix
wangbluo
committed
fix
wangbluo
committed
fix
wangbluo
committed
[zerobubble]Support ZeroBubble Pipeline (#6034)

duanjunwen
authored and
flybird11111
committed
[fp8] fix missing fp8_comm flag in mixtral (#6057)
botbw
authored
fix
wangbluo
committed
fix
wangbluo
committed
fix
wangbluo
committed

Commits on Sep 16, 2024

fix
wangbluo
committed

Commits on Sep 18, 2024

Commits on Sep 19, 2024

[release] update version (#6062)
ver217
authored

Commits on Sep 29, 2024

Merge branch 'main' of github.com:flybird11111/ColossalAI into main
flybird11111
committed
[feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;

duanjunwen
authored and
flybird11111
committed
[update] update text;

duanjunwen
authored and
flybird11111
committed
[feat] add test run_fwd_bwd automatic scheduling;

duanjunwen
authored and
flybird11111
committed
[feat] fix poc format

duanjunwen
authored and
flybird11111
committed
[fix] fix poc test; add comments in poc;

duanjunwen
authored and
flybird11111
committed
[feat] add optim backward_b_by_grad

duanjunwen
authored and
flybird11111
committed
[feat] fix optimizer bwd b & w; support return accum loss & output

duanjunwen
authored and
flybird11111
committed
[fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;

duanjunwen
authored and
flybird11111
committed
[feat] update test; rm comments;

duanjunwen
authored and
flybird11111
committed
[fix] rm zbv in hybridplugin

duanjunwen
authored and
flybird11111
committed
[fix] fix optim bwd;

duanjunwen
authored and
flybird11111
committed
[fix] fix optim bwd;

duanjunwen
authored and
flybird11111
committed
[fix] rm output.data after send fwd;

duanjunwen
authored and
flybird11111
committed
[fix] fix bwd step if condition; remove useless comments and format info;

duanjunwen
authored and
flybird11111
committed
[fix] fix mem check;

duanjunwen
authored and
flybird11111
committed
[fix] fix mem assertation

duanjunwen
authored and
flybird11111
committed
[fix] fix mem; use a new model shape; only assert mem less and equal than theo;

duanjunwen
authored and
flybird11111
committed
[fix] fix model zoo import;

duanjunwen
authored and
flybird11111
committed
[feat] moehybrid support zerobubble;

duanjunwen
authored and
flybird11111
committed
[fix] fix zerobubble pp for shardformer type input;

duanjunwen
authored and
flybird11111
committed
[fix] fix require_grad & deallocate call;

duanjunwen
authored and
flybird11111
committed
[fix] fix mem assert;

duanjunwen
authored and
flybird11111
committed
[fix] fix fwd branch, fwd pass both micro_batch & internal_inputs'

duanjunwen
authored and
flybird11111
committed
[fix] fix pipeline util func deallocate --> release_tensor_data; fix bwd_b loss bwd branch;

duanjunwen
authored and
flybird11111
committed
[fix] fix zerobubble; support shardformer model type;

duanjunwen
authored and
flybird11111
committed
[fix] fix test_pipeline_utils ci;

duanjunwen
authored and
flybird11111
committed
[plugin] hybrid support zero bubble pipeline (#6060)

committed
[feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;

duanjunwen
authored and
flybird11111
committed
[update] update text;

duanjunwen
authored and
flybird11111
committed
[feat] add test run_fwd_bwd automatic scheduling;

duanjunwen
authored and
flybird11111
committed
[feat] fix poc format

duanjunwen
authored and
flybird11111
committed
[fix] fix poc test; add comments in poc;

duanjunwen
authored and
flybird11111
committed
[feat] add optim backward_b_by_grad

duanjunwen
authored and
flybird11111
committed
[feat] fix optimizer bwd b & w; support return accum loss & output

duanjunwen
authored and
flybird11111
committed
[fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;

duanjunwen
authored and
flybird11111
committed
[feat] update test; rm comments;

duanjunwen
authored and
flybird11111
committed
[fix] fix optim bwd;

duanjunwen
authored and
flybird11111
committed
[fix] fix optim bwd;

duanjunwen
authored and
flybird11111
committed
[fix] rm output.data after send fwd;

duanjunwen
authored and
flybird11111
committed
[fix] fix bwd step if condition; remove useless comments and format info;

duanjunwen
authored and
flybird11111
committed
[fix] fix mem check;

duanjunwen
authored and
flybird11111
committed
[fix] fix mem assertation

duanjunwen
authored and
flybird11111
committed
[fix] fix mem; use a new model shape; only assert mem less and equal than theo;

duanjunwen
authored and
flybird11111
committed
[fix] fix model zoo import;

duanjunwen
authored and
flybird11111
committed
[fix] fix mem assert;

duanjunwen
authored and
flybird11111
committed
[fix] fix fwd branch, fwd pass both micro_batch & internal_inputs'

duanjunwen
authored and
flybird11111
committed
[plugin] hybrid support zero bubble pipeline (#6060)

committed
Merge branch 'feature/zerobubble' into feature/zerobubble
flybird11111
authored
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot]
committed

Commits on Oct 11, 2024

fix
flybird11111
committed