[BUG]: 使用HybridParallelPlugin做Pipeline并行测试GPU利用率不稳定

[imgaojun]

🐛 Describe the bug

使用HybridParallelPlugin做Pipeline并行测试GPU利用率不稳定，经常出现低于50%的情况，而Megatron一般都会维持在90%以上。想问下是Pipeline通信这块还没有优化好吗？

代码是基于 https://github.com/hpcaitech/ColossalAI/blob/main/examples/language/llama2/benchmark.py 修改，仅修改了数据load部分。

我的一些参数
plugin = HybridParallelPlugin(tp_size=1,
pp_size=2,
enable_flash_attention=True,
enable_fused_normalization=True,
enable_jit_fused=True,
microbatch_size=2,
precision='bf16',
zero_stage=1)

batch_size=32
context-length= 4096

速度上也较慢，仅4.8samples/sec

以下是利用率截图

Environment

NCCL version 2.18.1+cuda12.1 ClossalAI3.2 PyTorch2.1

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Title: [BUG]: Using HybridParallelPlugin for Pipeline parallel testing GPU utilization is unstable

[imgaojun]

目前仍使用1F1B策略导致的，更换interleave的策略即可

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

It is still caused by the 1F1B strategy. Just change the interleave strategy.

[xs1997zju]

@imgaojun 如何使用interleave策略？ 看文档目前还不支持？

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@imgaojun How to use interleave strategy? See the documentation and it’s not supported yet?

[imgaojun]

@xs1997zju 目前应该还未支持，后来我发现是p2p的实现方式导致速度不稳定，等官方解决吧

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@xs1997zju It should not be supported yet. Later I found out that the p2p implementation caused the speed to be unstable. Let’s wait for the official solution.