Skip to content

[shardformer] Align bert value#3907

Merged
FrankLeeeee merged 11 commits intohpcaitech:feature/shardformerfrom
FoolPlayer:align_bert
Jun 9, 2023
Merged

[shardformer] Align bert value#3907
FrankLeeeee merged 11 commits intohpcaitech:feature/shardformerfrom
FoolPlayer:align_bert

Conversation

@FoolPlayer
Copy link
Copy Markdown
Contributor

@FoolPlayer FoolPlayer commented Jun 6, 2023 •
edited by FrankLeeeee
Loading

📌 Checklist before creating the PR

  • I have created an issue for this PR for traceability
  • The title follows the standard format: [doc/gemini/tensor/...]: A concise description
  • I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

Fixed #3906

📝 What does this PR do?

Add align test for bert, fix a bug in dist loss when meet ignored labels

💥 Checklist before requesting a review

  • I have linked my PR to an issue (instruction)
  • My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
  • I have performed a self-review of my code
  • I have added thorough tests.
  • I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

  • 🌝 Yes, I do.
  • 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

@FoolPlayer FoolPlayer changed the base branch from main to feature/shardformer June 6, 2023 06:41
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 6, 2023

The code coverage for the changed files is 3%.

Click me to view the complete report 
Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
colossalai/nn/layer/parallel_1d/_operation.py          53     26    51%
colossalai/shardformer/__init__.py                      0      0   100%
colossalai/shardformer/layer/__init__.py                0      0   100%
colossalai/shardformer/layer/_operation.py             54     54     0%
colossalai/shardformer/layer/dist_crossentropy.py      49     49     0%
colossalai/shardformer/layer/dropout.py                32     32     0%
colossalai/shardformer/layer/layers.py                464    464     0%
colossalai/shardformer/model/__init__.py                0      0   100%
colossalai/shardformer/model/modeling_bert.py          20     20     0%
colossalai/shardformer/policies/__init__.py             0      0   100%
colossalai/shardformer/policies/autopolicy.py          16     16     0%
colossalai/shardformer/policies/basepolicy.py          49     49     0%
colossalai/shardformer/policies/bert.py                41     41     0%
colossalai/shardformer/shard/__init__.py                4      4     0%
colossalai/shardformer/shard/shard_config.py           14     14     0%
colossalai/shardformer/shard/sharder.py               109    109     0%
colossalai/shardformer/shard/slicer.py                 35     35     0%
colossalai/shardformer/utils/__init__.py                0      0   100%
colossalai/shardformer/utils/utils.py                  28     28     0%
-----------------------------------------------------------------------
TOTAL                                                 968    941     3%

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 7, 2023

The code coverage for the changed files is 2%.

Click me to view the complete report 
Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
colossalai/nn/layer/parallel_1d/_operation.py          53     35    34%
colossalai/shardformer/__init__.py                      0      0   100%
colossalai/shardformer/layer/__init__.py                0      0   100%
colossalai/shardformer/layer/_operation.py             54     54     0%
colossalai/shardformer/layer/dist_crossentropy.py      49     49     0%
colossalai/shardformer/layer/dropout.py                32     32     0%
colossalai/shardformer/layer/layers.py                464    464     0%
colossalai/shardformer/model/__init__.py                0      0   100%
colossalai/shardformer/model/modeling_bert.py          20     20     0%
colossalai/shardformer/policies/__init__.py             0      0   100%
colossalai/shardformer/policies/autopolicy.py          16     16     0%
colossalai/shardformer/policies/basepolicy.py          49     49     0%
colossalai/shardformer/policies/bert.py                41     41     0%
colossalai/shardformer/shard/__init__.py                4      4     0%
colossalai/shardformer/shard/shard_config.py           14     14     0%
colossalai/shardformer/shard/sharder.py               111    111     0%
colossalai/shardformer/shard/slicer.py                 35     35     0%
colossalai/shardformer/utils/__init__.py                0      0   100%
colossalai/shardformer/utils/utils.py                  28     28     0%
-----------------------------------------------------------------------
TOTAL                                                 970    952     2%

FrankLeeeee
FrankLeeeee reviewed Jun 7, 2023
View reviewed changes
Comment thread colossalai/shardformer/test/module_test.py Outdated
FrankLeeeee
FrankLeeeee reviewed Jun 7, 2023
View reviewed changes
Comment thread colossalai/shardformer/test/align_bert.py Outdated
FrankLeeeee
FrankLeeeee reviewed Jun 7, 2023
View reviewed changes
Comment thread colossalai/shardformer/policies/bert.py
FrankLeeeee
FrankLeeeee reviewed Jun 7, 2023
View reviewed changes
Comment thread colossalai/shardformer/layer/dist_crossentropy.py Outdated
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 7, 2023

The code coverage for the changed files is 56%.

Click me to view the complete report 
Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
colossalai/nn/layer/parallel_1d/_operation.py          53     35    34%
colossalai/shardformer/__init__.py                      0      0   100%
colossalai/shardformer/layer/__init__.py                0      0   100%
colossalai/shardformer/layer/_operation.py             54     27    50%
colossalai/shardformer/layer/dist_crossentropy.py      49     38    22%
colossalai/shardformer/layer/dropout.py                32     32     0%
colossalai/shardformer/layer/layers.py                464    280    40%
colossalai/shardformer/model/__init__.py                0      0   100%
colossalai/shardformer/model/modeling_bert.py          20     11    45%
colossalai/shardformer/policies/__init__.py             0      0   100%
colossalai/shardformer/policies/autopolicy.py          22      1    95%
colossalai/shardformer/policies/basepolicy.py          49      9    82%
colossalai/shardformer/policies/bert.py                40      1    98%
colossalai/shardformer/shard/__init__.py                4      0   100%
colossalai/shardformer/shard/shard_config.py           14      0   100%
colossalai/shardformer/shard/sharder.py               125     18    86%
colossalai/shardformer/shard/slicer.py                 50     12    76%
colossalai/shardformer/utils/__init__.py                0      0   100%
colossalai/shardformer/utils/utils.py                  28      5    82%
tests/test_shardformer/bert/bert_test.py               62      1    98%
-----------------------------------------------------------------------
TOTAL                                                1066    470    56%

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 7, 2023

The code coverage for the changed files is 56%.

Click me to view the complete report 
Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
colossalai/nn/layer/parallel_1d/_operation.py          53     35    34%
colossalai/shardformer/__init__.py                      0      0   100%
colossalai/shardformer/layer/__init__.py                0      0   100%
colossalai/shardformer/layer/_operation.py             54     27    50%
colossalai/shardformer/layer/dist_crossentropy.py      49     38    22%
colossalai/shardformer/layer/dropout.py                32     32     0%
colossalai/shardformer/layer/layers.py                464    280    40%
colossalai/shardformer/model/__init__.py                0      0   100%
colossalai/shardformer/model/modeling_bert.py          20     11    45%
colossalai/shardformer/policies/__init__.py             0      0   100%
colossalai/shardformer/policies/autopolicy.py          22      1    95%
colossalai/shardformer/policies/basepolicy.py          49      9    82%
colossalai/shardformer/policies/bert.py                40      1    98%
colossalai/shardformer/policies/gpt2.py                35     11    69%
colossalai/shardformer/shard/__init__.py                4      0   100%
colossalai/shardformer/shard/shard_config.py            7      0   100%
colossalai/shardformer/shard/sharder.py               125     18    86%
colossalai/shardformer/shard/slicer.py                 50     12    76%
colossalai/shardformer/utils/__init__.py                0      0   100%
colossalai/shardformer/utils/utils.py                  28      5    82%
tests/test_shardformer/bert/bert_test.py               62      1    98%
-----------------------------------------------------------------------
TOTAL                                                1094    481    56%

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 7, 2023

The code coverage for the changed files is %.

Click me to view the complete report 
Name                                                          Stmts   Miss  Cover
---------------------------------------------------------------------------------
colossalai/nn/layer/parallel_1d/_operation.py                    53     35    34%
colossalai/shardformer/__init__.py                                0      0   100%
colossalai/shardformer/layer/__init__.py                          0      0   100%
colossalai/shardformer/layer/_operation.py                       54     27    50%
colossalai/shardformer/layer/dist_crossentropy.py                49      8    84%
colossalai/shardformer/layer/dropout.py                          32     32     0%
colossalai/shardformer/layer/layers.py                          464    280    40%
colossalai/shardformer/model/__init__.py                          0      0   100%
colossalai/shardformer/model/modeling_bert.py                    20     11    45%
colossalai/shardformer/policies/__init__.py                       0      0   100%
colossalai/shardformer/policies/autopolicy.py                    22      1    95%
colossalai/shardformer/policies/basepolicy.py                    49      9    82%
colossalai/shardformer/policies/bert.py                          40      1    98%
colossalai/shardformer/policies/gpt2.py                          35     11    69%
colossalai/shardformer/shard/__init__.py                          4      0   100%
colossalai/shardformer/shard/shard_config.py                      7      0   100%
colossalai/shardformer/shard/sharder.py                         125     18    86%
colossalai/shardformer/shard/slicer.py                           50     12    76%
colossalai/shardformer/utils/__init__.py                          0      0   100%
colossalai/shardformer/utils/utils.py                            28      5    82%
tests/test_shardformer/test_bert/test_bert.py                    62      1    98%
tests/test_shardformer/test_module/test_distcrossentropy.py      27      1    96%
---------------------------------------------------------------------------------
TOTAL                                                          1121    452    60%

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 8, 2023

The code coverage for the changed files is 56%.

Click me to view the complete report 
Name                                                   Stmts   Miss  Cover
--------------------------------------------------------------------------
colossalai/nn/layer/parallel_1d/_operation.py             53     35    34%
colossalai/shardformer/__init__.py                         0      0   100%
colossalai/shardformer/layer/__init__.py                   0      0   100%
colossalai/shardformer/layer/_operation.py                54     27    50%
colossalai/shardformer/layer/dist_crossentropy.py         49     38    22%
colossalai/shardformer/layer/dropout.py                   32     32     0%
colossalai/shardformer/layer/layers.py                   464    280    40%
colossalai/shardformer/model/__init__.py                   0      0   100%
colossalai/shardformer/model/modeling_bert.py             20     11    45%
colossalai/shardformer/policies/__init__.py                0      0   100%
colossalai/shardformer/policies/autopolicy.py             22      1    95%
colossalai/shardformer/policies/basepolicy.py             49      9    82%
colossalai/shardformer/policies/bert.py                   40      1    98%
colossalai/shardformer/policies/gpt2.py                   35     11    69%
colossalai/shardformer/shard/__init__.py                   4      0   100%
colossalai/shardformer/shard/shard_config.py               7      0   100%
colossalai/shardformer/shard/sharder.py                  125     18    86%
colossalai/shardformer/shard/slicer.py                    50     12    76%
colossalai/shardformer/utils/__init__.py                   0      0   100%
colossalai/shardformer/utils/utils.py                     28      5    82%
tests/test_shardformer/test_model/test_shard_bert.py      62      1    98%
--------------------------------------------------------------------------
TOTAL                                                   1094    481    56%

FrankLeeeee
FrankLeeeee reviewed Jun 9, 2023
View reviewed changes
Comment thread colossalai/shardformer/policies/bert.py
FrankLeeeee
FrankLeeeee reviewed Jun 9, 2023
View reviewed changes
Comment thread colossalai/shardformer/README.md Outdated
@FrankLeeeee FrankLeeeee merged commit 158b6cc into hpcaitech:feature/shardformer Jun 9, 2023
@FrankLeeeee FrankLeeeee mentioned this pull request Jun 9, 2023
Closed
@FrankLeeeee FrankLeeeee linked an issue Jun 9, 2023 that may be closed by this pull request
[shardformer] align BERT value #3906
Closed
@FoolPlayer FoolPlayer deleted the align_bert branch June 9, 2023 06:56
FrankLeeeee pushed a commit that referenced this pull request Jun 26, 2023
@FoolPlayer @FrankLeeeee
[shardformer] Align bert value (#3907)
e5bc7e3 
* add bert align test, fix dist loss bug

* forward and backward align

* add ignore index

* add shardformer CI

* add gather_output optional for user in shardconfig

* update readme with optional gather_ouput

* add dist crossentropy loss test, remove unused files

* remove unused file

* remove unused file

* rename the file

* polish code
flybird11111 pushed a commit to flybird11111/ColossalAI that referenced this pull request Jul 3, 2023
@FoolPlayer @flybird11111
[shardformer] Align bert value (hpcaitech#3907)
185ff85 
* add bert align test, fix dist loss bug

* forward and backward align

* add ignore index

* add shardformer CI

* add gather_output optional for user in shardconfig

* update readme with optional gather_ouput

* add dist crossentropy loss test, remove unused files

* remove unused file

* remove unused file

* rename the file

* polish code
FrankLeeeee pushed a commit that referenced this pull request Jul 4, 2023
@FoolPlayer @FrankLeeeee
[shardformer] Align bert value (#3907)
f1cb5ac 
* add bert align test, fix dist loss bug

* forward and backward align

* add ignore index

* add shardformer CI

* add gather_output optional for user in shardconfig

* update readme with optional gather_ouput

* add dist crossentropy loss test, remove unused files

* remove unused file

* remove unused file

* rename the file

* polish code
ver217 pushed a commit to ver217/ColossalAI that referenced this pull request Jul 13, 2023
@FoolPlayer @ver217
[shardformer] Align bert value (hpcaitech#3907)
03b0b65 
* add bert align test, fix dist loss bug

* forward and backward align

* add ignore index

* add shardformer CI

* add gather_output optional for user in shardconfig

* update readme with optional gather_ouput

* add dist crossentropy loss test, remove unused files

* remove unused file

* remove unused file

* rename the file

* polish code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@FrankLeeeee FrankLeeeee FrankLeeeee left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

shardformer

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[shardformer] align BERT value

2 participants

@FoolPlayer @FrankLeeeee