🐛 Describe the bug
At https://github.com/hpcaitech/ColossalAI/blob/main/examples/language/gpt/gemini/train_gpt_demo.py#L253, which is
reduce_bucket_size_in_m=12 * 1024 * 1024, the bucket size of comm is set to 12T(1210241024M), which leads to the bucket can never be full.
Though this example can still run correctly, it would affect the speed.
Environment
No response
🐛 Describe the bug
At https://github.com/hpcaitech/ColossalAI/blob/main/examples/language/gpt/gemini/train_gpt_demo.py#L253, which is
reduce_bucket_size_in_m=12 * 1024 * 1024, the bucket size of comm is set to 12T(1210241024M), which leads to the bucket can never be full.Though this example can still run correctly, it would affect the speed.
Environment
No response