Skip to content

[chat] refactor strategy class with booster api#3987

Merged
ver217 merged 25 commits intohpcaitech:mainfrom
cwher:main
Jun 25, 2023
Merged

[chat] refactor strategy class with booster api#3987
ver217 merged 25 commits intohpcaitech:mainfrom
cwher:main

Conversation

@cwher
Copy link
Copy Markdown
Contributor

@cwher cwher commented Jun 14, 2023

📌 Checklist before creating the PR

  • I have created an issue for this PR for traceability
  • The title follows the standard format: [doc/gemini/tensor/...]: A concise description
  • I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

Close #3978.

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

💥 Checklist before requesting a review

  • I have linked my PR to an issue (instruction)
  • My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
  • I have performed a self-review of my code
  • I have added thorough tests.
  • I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

  • 🌝 Yes, I do.
  • 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

@cwher cwher marked this pull request as ready for review June 14, 2023 13:50
@github-actions
Copy link
Copy Markdown
Contributor

The code coverage for the changed files is 89%.

Click me to view the complete report
Name                                                         Stmts   Miss  Cover
--------------------------------------------------------------------------------
colossalai/booster/booster.py                                   64     10    84%
colossalai/booster/mixed_precision/fp16_torch.py                46      2    96%
colossalai/booster/mixed_precision/mixed_precision_base.py       9      1    89%
colossalai/booster/plugin/gemini_plugin.py                     110      9    92%
colossalai/booster/plugin/low_level_zero_plugin.py              94      8    91%
colossalai/booster/plugin/plugin_base.py                        40     10    75%
colossalai/booster/plugin/torch_ddp_plugin.py                   66      3    95%
colossalai/booster/plugin/torch_fsdp_plugin.py                  97     13    87%
--------------------------------------------------------------------------------
TOTAL                                                          526     56    89%

@cwher cwher requested a review from ver217 June 25, 2023 07:21
Comment thread applications/Chat/examples/train_prompts.py
@cwher cwher requested a review from ver217 June 25, 2023 09:28
@ver217 ver217 merged commit 153b957 into hpcaitech:main Jun 25, 2023
ver217 pushed a commit to ver217/ColossalAI that referenced this pull request Jul 13, 2023
* refactor: adapt boost API in base and naive strategies

* fix: initialize plugin after setup_distributed

* fix: fix save_pretrained fn

* refactor: adapt boost API in DDPStrategy

* to: add _post_init check

* to: fix ddp backward, modify ddp dataloader and unwrap

* feat: adapt boost API in ColossalAIStrategy

* fix: call setup_distributed before use get_current_device

* fix: fix save_model and save_optimizer

* test: remove save_sharded_optimizer test

* style: apply formatter

* fix: fix stage check and add comments

* feat: allow dict type arg in strategy.prepare

* to: temporarily remove lr_scheduler for testing

* style: simplify init of ColossalAIStrategy

* fix: fix lr_scheduler in sft and rm

* style: modify comments

* test: add train_prompts tests

* fix: fix inference only case and use in train_prompts

* test: skip failed tests in ci

* style: fix CodeFactor check

* fix: do not use model.to('cpu') with GeminiPlugin

* test: enable colossalai_gemini tests

* test: set CUDA_VISIBLE_DEVICES in ci

* docs: add note
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[chat] refactor strategy class with booster api

2 participants