[moe] support hybrid zero strategy. by oahzxl · Pull Request #4877 · hpcaitech/ColossalAI

oahzxl · 2023-10-10T07:18:22Z

Support hybrid zero strategy.
- For non-moe param, zero will be activated for the global dp group.
- For moe param, we will use expert parallel and zero, where ep group * zero group = global dp group. So zero will be activated only for moe zero group here.
add unit test for hybrid zero
enable moe unit test
fix previous unit test

…into bench

github-actions · 2023-10-11T02:52:21Z

The code coverage for the changed files is %.

Click me to view the complete report

Name                                                                 Stmts   Miss  Cover
----------------------------------------------------------------------------------------
colossalai/booster/plugin/hybrid_parallel_plugin.py                    214     13    94%
colossalai/booster/plugin/moe_hybrid_parallel_plugin.py                 92     26    72%
colossalai/context/__init__.py                                           6      0   100%
colossalai/context/random/__init__.py                                    2      0   100%
colossalai/context/random/_helper.py                                    46      9    80%
colossalai/initialize.py                                               180    133    26%
colossalai/kernel/triton/__init__.py                                    14      3    79%
colossalai/kernel/triton/llama_act_combine_kernel.py                    89     67    25%
colossalai/legacy/engine/gradient_handler/__init__.py                    6      0   100%
colossalai/legacy/engine/gradient_handler/_moe_gradient_handler.py      20     20     0%
colossalai/moe/__init__.py                                               6      0   100%
colossalai/moe/_operation.py                                           159     44    72%
colossalai/moe/checkpoint.py                                           133     31    77%
colossalai/moe/experts.py                                               95     14    85%
colossalai/moe/layers.py                                               114     34    70%
colossalai/moe/loss.py                                                  21     21     0%
colossalai/moe/manager.py                                               81      7    91%
colossalai/moe/routers.py                                              176     15    91%
colossalai/moe/utils.py                                                 79     26    67%
colossalai/nn/layer/moe/__init__.py                                     12      0   100%
colossalai/nn/loss/__init__.py                                           0      0   100%
colossalai/tensor/moe_tensor/__init__.py                                 0      0   100%
colossalai/tensor/moe_tensor/api.py                                     24      2    92%
colossalai/tensor/moe_tensor/moe_info.py                                13      0   100%
colossalai/zero/low_level/low_level_optim.py                           431     32    93%
examples/language/openmoe/model/__init__.py                              0      0   100%
examples/language/openmoe/model/modeling_openmoe.py                    462    328    29%
examples/language/openmoe/model/openmoe_policy.py                      248    157    37%
tests/test_infer_ops/triton/test_llama_act_combine.py                   40     23    42%
tests/test_moe/moe_utils.py                                             97      0   100%
tests/test_moe/test_grad_handler.py                                     59      1    98%
tests/test_moe/test_kernel.py                                           57      1    98%
tests/test_moe/test_moe_checkpoint.py                                   69      2    97%
tests/test_moe/test_moe_ep_tp.py                                        48      1    98%
tests/test_moe/test_moe_group.py                                        54      2    96%
tests/test_moe/test_moe_hybrid_zero.py                                  67      2    97%
tests/test_moe/test_moe_local.py                                        49      1    98%
tests/test_moe/test_moe_router.py                                       25      1    96%
tests/test_moe/test_moe_zero_fwd_bwd.py                                 77      3    96%
tests/test_moe/test_moe_zero_optim.py                                   66      9    86%
----------------------------------------------------------------------------------------
TOTAL                                                                 3431   1028    70%

github-actions · 2023-10-11T07:38:03Z

The code coverage for the changed files is %.

Click me to view the complete report

Name                                                                 Stmts   Miss  Cover
----------------------------------------------------------------------------------------
colossalai/booster/plugin/hybrid_parallel_plugin.py                    214     13    94%
colossalai/booster/plugin/moe_hybrid_parallel_plugin.py                 92     26    72%
colossalai/context/__init__.py                                           6      0   100%
colossalai/context/random/__init__.py                                    2      0   100%
colossalai/context/random/_helper.py                                    46     14    70%
colossalai/initialize.py                                               180    134    26%
colossalai/kernel/triton/__init__.py                                    14      3    79%
colossalai/kernel/triton/llama_act_combine_kernel.py                    89     67    25%
colossalai/legacy/engine/gradient_handler/__init__.py                    6      0   100%
colossalai/legacy/engine/gradient_handler/_moe_gradient_handler.py      20     20     0%
colossalai/moe/__init__.py                                               6      0   100%
colossalai/moe/_operation.py                                           159     44    72%
colossalai/moe/checkpoint.py                                           133     31    77%
colossalai/moe/experts.py                                               95     14    85%
colossalai/moe/layers.py                                               114     34    70%
colossalai/moe/loss.py                                                  21     21     0%
colossalai/moe/manager.py                                               81      7    91%
colossalai/moe/routers.py                                              176     15    91%
colossalai/moe/utils.py                                                 79     26    67%
colossalai/nn/layer/moe/__init__.py                                     12      0   100%
colossalai/nn/loss/__init__.py                                           0      0   100%
colossalai/tensor/moe_tensor/__init__.py                                 0      0   100%
colossalai/tensor/moe_tensor/api.py                                     24      2    92%
colossalai/tensor/moe_tensor/moe_info.py                                13      0   100%
colossalai/zero/low_level/low_level_optim.py                           431     32    93%
examples/language/openmoe/model/__init__.py                              0      0   100%
examples/language/openmoe/model/modeling_openmoe.py                    462    328    29%
examples/language/openmoe/model/openmoe_policy.py                      248    157    37%
tests/test_moe/moe_utils.py                                             97      0   100%
tests/test_moe/test_grad_handler.py                                     59      1    98%
tests/test_moe/test_kernel.py                                           57      1    98%
tests/test_moe/test_moe_checkpoint.py                                   69      2    97%
tests/test_moe/test_moe_ep_tp.py                                        48      1    98%
tests/test_moe/test_moe_group.py                                        54      2    96%
tests/test_moe/test_moe_hybrid_zero.py                                  66      2    97%
tests/test_moe/test_moe_local.py                                        49      1    98%
tests/test_moe/test_moe_router.py                                       25      1    96%
tests/test_moe/test_moe_zero_fwd_bwd.py                                 77      3    96%
tests/test_moe/test_moe_zero_optim.py                                   66      9    86%
----------------------------------------------------------------------------------------
TOTAL                                                                 3390   1011    70%

github-actions · 2023-10-11T11:24:38Z

The code coverage for the changed files is 67%.

Click me to view the complete report

Name                                                                 Stmts   Miss  Cover
----------------------------------------------------------------------------------------
colossalai/booster/plugin/hybrid_parallel_plugin.py                    214     13    94%
colossalai/booster/plugin/moe_hybrid_parallel_plugin.py                102     32    69%
colossalai/context/__init__.py                                           6      0   100%
colossalai/context/random/__init__.py                                    2      0   100%
colossalai/context/random/_helper.py                                    46     14    70%
colossalai/initialize.py                                               180    134    26%
colossalai/kernel/triton/__init__.py                                    14      3    79%
colossalai/kernel/triton/llama_act_combine_kernel.py                    89     67    25%
colossalai/legacy/engine/gradient_handler/__init__.py                    6      0   100%
colossalai/legacy/engine/gradient_handler/_moe_gradient_handler.py      20     20     0%
colossalai/moe/__init__.py                                               6      0   100%
colossalai/moe/_operation.py                                           159     44    72%
colossalai/moe/checkpoint.py                                           133     31    77%
colossalai/moe/experts.py                                               95     14    85%
colossalai/moe/layers.py                                               114     34    70%
colossalai/moe/loss.py                                                  21     21     0%
colossalai/moe/manager.py                                               82      7    91%
colossalai/moe/routers.py                                              176     34    81%
colossalai/moe/utils.py                                                 79     26    67%
colossalai/nn/layer/moe/__init__.py                                     12      0   100%
colossalai/nn/loss/__init__.py                                           0      0   100%
colossalai/tensor/moe_tensor/__init__.py                                 0      0   100%
colossalai/tensor/moe_tensor/api.py                                     24      2    92%
colossalai/tensor/moe_tensor/moe_info.py                                13      0   100%
colossalai/zero/low_level/low_level_optim.py                           431    115    73%
examples/language/openmoe/model/__init__.py                              0      0   100%
examples/language/openmoe/model/modeling_openmoe.py                    462    328    29%
examples/language/openmoe/model/openmoe_policy.py                      248    157    37%
tests/test_moe/moe_utils.py                                             97      0   100%
tests/test_moe/test_grad_handler.py                                     59      1    98%
tests/test_moe/test_kernel.py                                           57      1    98%
tests/test_moe/test_moe_checkpoint.py                                   69      2    97%
tests/test_moe/test_moe_ep_tp.py                                        48      1    98%
tests/test_moe/test_moe_group.py                                        54      2    96%
tests/test_moe/test_moe_hybrid_zero.py                                  66      2    97%
tests/test_moe/test_moe_local.py                                        49      1    98%
tests/test_moe/test_moe_zero_fwd_bwd.py                                 77      3    96%
tests/test_moe/test_moe_zero_optim.py                                   66      9    86%
----------------------------------------------------------------------------------------
TOTAL                                                                 3376   1118    67%

* overlap comm * fix typo * update bench script * add option * update script * update bench * param init * support dp zero * fix zero dp * fxi bug * update pg bug * update experts * fix optim bug * update config * kaishen niubi * fix bug * embed * Merge branch 'feature/MoE' of https://github.com/hpcaitech/ColossalAI into bench * update bench * update optim * update doc * update sync * fix test * fix arg * update ckpt * update test * fix * remove print * polish code * update hybrid zero optim * update print

oahzxl added 27 commits October 3, 2023 11:59

overlap comm

8306a33

fix typo

a1dd693

update bench script

a2be80f

add option

28193c4

update script

702fb70

update bench

a46c7d0

param init

34d037e

support dp zero

dac762e

fix zero dp

17fecf8

fxi bug

465412d

update pg bug

93f3850

update experts

ec7b463

fix optim bug

f461d04

update config

61087ce

kaishen niubi

9b9febe

fix bug

444a1bd

embed

d66120f

Merge branch 'feature/MoE' of https://github.com/hpcaitech/ColossalAI …

3332153

…into bench

Merge branch 'feature/MoE' of https://github.com/hpcaitech/ColossalAI …

4d05901

…into bench

update bench

329dfe3

update optim

57c8212

update doc

eca0e30

update sync

2ff6323

fix test

88ab9fb

fix arg

ef569af

update ckpt

bccf7bb

update test

ea01835

ver217 reviewed Oct 11, 2023

View reviewed changes

Comment thread colossalai/zero/low_level/low_level_optim.py Outdated

fix

1266de0

oahzxl added 2 commits October 11, 2023 12:20

remove print

71afab7

polish code

facc99c

ver217 reviewed Oct 11, 2023

View reviewed changes

Comment thread colossalai/booster/plugin/hybrid_parallel_plugin.py Outdated

ver217 reviewed Oct 11, 2023

View reviewed changes

Comment thread colossalai/moe/manager.py Outdated

oahzxl added 2 commits October 11, 2023 17:04

update hybrid zero optim

9931b41

update print

7f4b97e

ver217 approved these changes Oct 11, 2023

View reviewed changes

ver217 merged commit 0d599f6 into hpcaitech:feature/MoE Oct 11, 2023

oahzxl deleted the bench branch October 18, 2023 02:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[moe] support hybrid zero strategy. #4877

[moe] support hybrid zero strategy. #4877
ver217 merged 32 commits intohpcaitech:feature/MoEfrom
oahzxl:bench

oahzxl commented Oct 10, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 11, 2023

Uh oh!

Uh oh!

github-actions Bot commented Oct 11, 2023

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

oahzxl commented Oct 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 11, 2023

Uh oh!

Uh oh!

github-actions Bot commented Oct 11, 2023

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

oahzxl commented Oct 10, 2023 •

edited

Loading