Skip to content

[autoparallel] Patch meta information of torch.nn.LayerNorm#2647

Merged
YuliangLiu0306 merged 58 commits intohpcaitech:mainfrom
Cypher30:feature/layernorm_metainfo
Feb 10, 2023
Merged

[autoparallel] Patch meta information of torch.nn.LayerNorm#2647
YuliangLiu0306 merged 58 commits intohpcaitech:mainfrom
Cypher30:feature/layernorm_metainfo

Conversation

@Cypher30
Copy link
Copy Markdown
Contributor

@Cypher30 Cypher30 commented Feb 8, 2023

📌 Checklist before creating the PR

  • I have created an issue for this PR for traceability
  • The title follows the standard format: [doc/gemini/tensor/...]: A concise description
  • I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

Resolved #2632

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

In this PR, I patch the meta information of torch.nn.LayerNorm, and I add a function in utils.py under the tests/test_auto_parallel/test_tensor_shard/test_metainfo directory for meta information test.

TODO: we could refactor tests under test_metainfo in the future, as we have meta information patches for communication nodes, there is no need to test meta information patches with clumsy SPMD solver codes. I also move all the tests concerning meta information patches of normalization into test_norm_metainfo.py.

The test will be skipped if PyTorch version is lower than 1.12.0, here is a snapshot of test results on PyTorch 1.12.0

Screenshot 2023-02-08 at 21 50 02

💥 Checklist before requesting a review

  • I have linked my PR to an issue (instruction)
  • My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
  • I have performed a self-review of my code
  • I have added thorough tests.
  • I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

  • 🌝 Yes, I do.
  • 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

Cypher30 and others added 30 commits July 14, 2022 16:07
@Cypher30 Cypher30 added the auto-parallel related to the auto-parallel feature label Feb 8, 2023
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 8, 2023

The code coverage for the changed files is 27%.

Click me to view the complete report
Name                                                                               Stmts   Miss  Cover
------------------------------------------------------------------------------------------------------
colossalai/auto_parallel/meta_profiler/meta_registry/norm.py                          57     43    25%
tests/test_auto_parallel/test_tensor_shard/test_metainfo/test_matmul_metainfo.py      48     28    42%
tests/test_auto_parallel/test_tensor_shard/test_metainfo/test_norm_metainfo.py        68     44    35%
tests/test_auto_parallel/test_tensor_shard/test_metainfo/utils.py                     89     75    16%
------------------------------------------------------------------------------------------------------
TOTAL                                                                                262    190    27%

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 9, 2023

The code coverage for the changed files is 29%.

Click me to view the complete report
Name                                                                               Stmts   Miss  Cover
------------------------------------------------------------------------------------------------------
colossalai/auto_parallel/meta_profiler/meta_registry/norm.py                          57     43    25%
colossalai/auto_parallel/tensor_shard/node_handler/layer_norm_handler.py              23     12    48%
tests/test_auto_parallel/test_tensor_shard/test_metainfo/test_matmul_metainfo.py      48     28    42%
tests/test_auto_parallel/test_tensor_shard/test_metainfo/test_norm_metainfo.py        68     44    35%
tests/test_auto_parallel/test_tensor_shard/test_metainfo/utils.py                     89     75    16%
------------------------------------------------------------------------------------------------------
TOTAL                                                                                285    202    29%

@YuliangLiu0306 YuliangLiu0306 merged commit 0385b26 into hpcaitech:main Feb 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-parallel related to the auto-parallel feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Patch meta information of torch.nn.LayerNorm

2 participants