[BUG]: gemini 插件微调时，疑似不能正确加载模型权重

[hmzo]

🐛 Describe the bug

环境

------------ Environment ------------
Colossal-AI version: 0.3.4
PyTorch version: 2.0.1
System CUDA version: 11.7
CUDA version required by PyTorch: 11.7

BUG 细节

微调代码修改自：https://github.com/hpcaitech/ColossalAI/blob/main/examples/language/llama2/finetune.py

加载模型：https://github.com/hpcaitech/ColossalAI/blob/main/examples/language/llama2/finetune.py#L237

除了plugin的类型其余变量都保持一致，发现zero2时，loss的表现正常，而使用gemini时，更像是从一个随机初始化的weight进行优化

zero2，loss 正常从比较低的水平开始下降:

gemini，loss 从特别高的水平下降：

Environment

No response

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Title: [BUG]: When fine-tuning the gemini plug-in, it is suspected that the model weights cannot be loaded correctly.

[flybird11111]

感谢您的反馈，我们会尽快看一下这个问题。

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Thanks for your feedback, we'll take a look at this issue as soon as possible.

[luckyyangrun]

When fine-tuning ChatGLM2 / Llama2, I also observed a similar phenomenon, under PyTorch version: 1.13.1. It seems that this phenomenon occurs when using Colossal-AI version >= 0.3.3.