[auto-parallel] add auto-offload feature

[zengzh95]

📌 Checklist before creating the PR

• I have created an issue for this PR for traceability
• The title follows the standard format: [doc/gemini/tensor/...]: A concise description
• I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

testing performance

testing solver

💥 Checklist before requesting a review

• I have linked my PR to an issue (instruction)
• My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
• I have performed a self-review of my code
• I have added thorough tests.
• I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

• 🌝 Yes, I do.
• 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

[github-actions]

The code coverage for the changed files is %.

Click me to view the complete report

Name                                                      Stmts   Miss  Cover
-----------------------------------------------------------------------------
colossalai/auto_parallel/offload/__init__.py                  0      0   100%
colossalai/auto_parallel/offload/amp_optimizer.py            99     72    27%
colossalai/auto_parallel/offload/base_offload_module.py      71     47    34%
colossalai/auto_parallel/offload/mem_optimize.py             36     24    33%
colossalai/auto_parallel/offload/region.py                   81     64    21%
colossalai/auto_parallel/offload/region_manager.py          283    262     7%
colossalai/auto_parallel/offload/runtime.py                 146    126    14%
colossalai/auto_parallel/offload/solver.py                  259    209    19%
colossalai/auto_parallel/offload/training_simulator.py      228    191    16%
colossalai/auto_parallel/offload/util.py                     56     25    55%
tests/test_auto_parallel/test_offload/model_utils.py         53     35    34%
tests/test_auto_parallel/test_offload/test_perf.py          109     84    23%
tests/test_auto_parallel/test_offload/test_solver.py         50     34    32%
-----------------------------------------------------------------------------
TOTAL                                                      1471   1173    20%

[github-actions]

The code coverage for the changed files is %.

Click me to view the complete report

Name                                                      Stmts   Miss  Cover
-----------------------------------------------------------------------------
colossalai/auto_parallel/offload/__init__.py                  0      0   100%
colossalai/auto_parallel/offload/amp_optimizer.py            99     72    27%
colossalai/auto_parallel/offload/base_offload_module.py      71     47    34%
colossalai/auto_parallel/offload/mem_optimize.py             36     24    33%
colossalai/auto_parallel/offload/region.py                   81     64    21%
colossalai/auto_parallel/offload/region_manager.py          283    262     7%
colossalai/auto_parallel/offload/runtime.py                 146    126    14%
colossalai/auto_parallel/offload/solver.py                  259    209    19%
colossalai/auto_parallel/offload/training_simulator.py      228    191    16%
colossalai/auto_parallel/offload/util.py                     56     25    55%
tests/test_auto_parallel/test_offload/model_utils.py         53     35    34%
tests/test_auto_parallel/test_offload/test_perf.py          109     84    23%
tests/test_auto_parallel/test_offload/test_solver.py         50     34    32%
-----------------------------------------------------------------------------
TOTAL                                                      1471   1173    20%