[shardformer] add gpt2 policy and modify shard and slicer to support by FoolPlayer · Pull Request #3883 · hpcaitech/ColossalAI

FoolPlayer · 2023-06-01T09:06:05Z

📌 Checklist before creating the PR

I have created an issue for this PR for traceability
The title follows the standard format: [doc/gemini/tensor/...]: A concise description
I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Add support to GPT2, and modify shard and slicer code to support multihead attention layer

💥 Checklist before requesting a review

I have linked my PR to an issue (instruction)
My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
I have performed a self-review of my code
I have added thorough tests.
I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

🌝 Yes, I do.
🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

github-actions · 2023-06-01T10:50:20Z

The code coverage for the changed files is 3%.

Click me to view the complete report

Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
colossalai/nn/layer/parallel_1d/_operation.py          53     26    51%
colossalai/shardformer/__init__.py                      0      0   100%
colossalai/shardformer/layer/__init__.py                0      0   100%
colossalai/shardformer/layer/_operation.py             54     54     0%
colossalai/shardformer/layer/dist_crossentropy.py      49     49     0%
colossalai/shardformer/layer/dropout.py                32     32     0%
colossalai/shardformer/layer/layers.py                464    464     0%
colossalai/shardformer/model/__init__.py                0      0   100%
colossalai/shardformer/model/modeling_bert.py          20     20     0%
colossalai/shardformer/policies/__init__.py             0      0   100%
colossalai/shardformer/policies/autopolicy.py          22     22     0%
colossalai/shardformer/policies/basepolicy.py          49     49     0%
colossalai/shardformer/policies/bert.py                40     40     0%
colossalai/shardformer/policies/gpt2.py                35     35     0%
colossalai/shardformer/shard/__init__.py                4      4     0%
colossalai/shardformer/shard/shard_config.py           14     14     0%
colossalai/shardformer/shard/sharder.py               124    124     0%
colossalai/shardformer/shard/slicer.py                 50     50     0%
colossalai/shardformer/utils/__init__.py                0      0   100%
colossalai/shardformer/utils/utils.py                  28     28     0%
-----------------------------------------------------------------------
TOTAL                                                1038   1011     3%

github-actions · 2023-06-01T10:54:36Z

The code coverage for the changed files is 2%.

Click me to view the complete report

Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
colossalai/nn/layer/parallel_1d/_operation.py          53     35    34%
colossalai/shardformer/__init__.py                      0      0   100%
colossalai/shardformer/layer/__init__.py                0      0   100%
colossalai/shardformer/layer/_operation.py             54     54     0%
colossalai/shardformer/layer/dist_crossentropy.py      49     49     0%
colossalai/shardformer/layer/dropout.py                32     32     0%
colossalai/shardformer/layer/layers.py                464    464     0%
colossalai/shardformer/model/__init__.py                0      0   100%
colossalai/shardformer/model/modeling_bert.py          20     20     0%
colossalai/shardformer/policies/__init__.py             0      0   100%
colossalai/shardformer/policies/autopolicy.py          22     22     0%
colossalai/shardformer/policies/basepolicy.py          49     49     0%
colossalai/shardformer/policies/bert.py                40     40     0%
colossalai/shardformer/policies/gpt2.py                35     35     0%
colossalai/shardformer/shard/__init__.py                4      4     0%
colossalai/shardformer/shard/shard_config.py           14     14     0%
colossalai/shardformer/shard/sharder.py               124    124     0%
colossalai/shardformer/shard/slicer.py                 50     50     0%
colossalai/shardformer/utils/__init__.py                0      0   100%
colossalai/shardformer/utils/utils.py                  28     28     0%
-----------------------------------------------------------------------
TOTAL                                                1038   1020     2%

github-actions · 2023-06-07T02:57:51Z

The code coverage for the changed files is 3%.

Click me to view the complete report

Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
colossalai/nn/layer/parallel_1d/_operation.py          53     26    51%
colossalai/shardformer/__init__.py                      0      0   100%
colossalai/shardformer/layer/__init__.py                0      0   100%
colossalai/shardformer/layer/_operation.py             54     54     0%
colossalai/shardformer/layer/dist_crossentropy.py      49     49     0%
colossalai/shardformer/layer/dropout.py                32     32     0%
colossalai/shardformer/layer/layers.py                464    464     0%
colossalai/shardformer/model/__init__.py                0      0   100%
colossalai/shardformer/model/modeling_bert.py          20     20     0%
colossalai/shardformer/policies/__init__.py             0      0   100%
colossalai/shardformer/policies/autopolicy.py          22     22     0%
colossalai/shardformer/policies/basepolicy.py          49     49     0%
colossalai/shardformer/policies/bert.py                40     40     0%
colossalai/shardformer/policies/gpt2.py                35     35     0%
colossalai/shardformer/shard/__init__.py                4      4     0%
colossalai/shardformer/shard/shard_config.py           14     14     0%
colossalai/shardformer/shard/sharder.py               123    123     0%
colossalai/shardformer/shard/slicer.py                 50     50     0%
colossalai/shardformer/utils/__init__.py                0      0   100%
colossalai/shardformer/utils/utils.py                  28     28     0%
-----------------------------------------------------------------------
TOTAL                                                1037   1010     3%

…pcaitech#3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

…3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

…pcaitech#3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

…3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

…pcaitech#3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

FoolPlayer added 2 commits June 1, 2023 17:02

add gpt2 policy and modify shard and slicer to support

94a3b4e

remove unused code

3629efd

FoolPlayer changed the title ~~add gpt2 policy and modify shard and slicer to support~~ [shardformer] add gpt2 policy and modify shard and slicer to support Jun 1, 2023

polish code

3246aab

FrankLeeeee approved these changes Jun 7, 2023

View reviewed changes

FrankLeeeee merged commit 69b1a22 into hpcaitech:feature/shardformer Jun 7, 2023

FrankLeeeee pushed a commit to FrankLeeeee/ColossalAI that referenced this pull request Jun 8, 2023

[shardformer] add gpt2 policy and modify shard and slicer to support (h…

7c29259

…pcaitech#3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

FrankLeeeee pushed a commit to FrankLeeeee/ColossalAI that referenced this pull request Jun 8, 2023

[shardformer] add gpt2 policy and modify shard and slicer to support (h…

8b9fa49

…pcaitech#3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

FrankLeeeee pushed a commit to FrankLeeeee/ColossalAI that referenced this pull request Jun 8, 2023

[shardformer] add gpt2 policy and modify shard and slicer to support (h…

ef15377

…pcaitech#3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

FoolPlayer deleted the newpolicy branch June 12, 2023 07:01

FrankLeeeee assigned FoolPlayer Jun 15, 2023

FrankLeeeee added shardformer example example-related issuer or pull request labels Jun 15, 2023

FrankLeeeee pushed a commit that referenced this pull request Jun 26, 2023

[shardformer] add gpt2 policy and modify shard and slicer to support (#…

551fec3

…3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

FrankLeeeee pushed a commit that referenced this pull request Jul 4, 2023

[shardformer] add gpt2 policy and modify shard and slicer to support (#…

79f8d5d

…3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

ver217 pushed a commit to ver217/ColossalAI that referenced this pull request Jul 13, 2023

[shardformer] add gpt2 policy and modify shard and slicer to support (h…

ad1723f

…pcaitech#3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

ver217 pushed a commit to ver217/ColossalAI that referenced this pull request Jul 13, 2023

[shardformer] add gpt2 policy and modify shard and slicer to support (h…

3bf4980

…pcaitech#3883) * add gpt2 policy and modify shard and slicer to support * remove unused code * polish code

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[shardformer] add gpt2 policy and modify shard and slicer to support#3883

[shardformer] add gpt2 policy and modify shard and slicer to support#3883
FrankLeeeee merged 3 commits intohpcaitech:feature/shardformerfrom
FoolPlayer:newpolicy

FoolPlayer commented Jun 1, 2023

Uh oh!

github-actions Bot commented Jun 1, 2023

Uh oh!

github-actions Bot commented Jun 1, 2023

Uh oh!

github-actions Bot commented Jun 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

FoolPlayer commented Jun 1, 2023

📌 Checklist before creating the PR

🚨 Issue number

📝 What does this PR do?

💥 Checklist before requesting a review

⭐️ Do you enjoy contributing to Colossal-AI?

Uh oh!

github-actions Bot commented Jun 1, 2023

Uh oh!

github-actions Bot commented Jun 1, 2023

Uh oh!

github-actions Bot commented Jun 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants