[BUG]: PPO training may be incorrect

[jasper881108]

🐛 Describe the bug

As title, I'm dealing with OpenAI's Paper : Learning to Summarize from Human Feedback. After finishing stage 1, stage 2. I got Model below

1. SFT OPT1.3b with Rogue1 score 0.29
2. RM OPT350m for accuracy of 66.3%.

However, after training with PPO process, the Rogue1 drop to 0.02, which is unusual. I currently doubt Colossal AI's PPO process because its reward is not calculate word by word but whole completion. Maybe trlx's approach is more close to Open AI's ChatGPT
trlx (https://github.com/CarperAI/trlx/tree/main/examples/summarize_rlhf )

Environment

No response

[binmakeswell]

@jasper881108 Thanks for the feedback. @ht-zhou will reply you. Thanks.

[ht-zhou]

Hi @jasper881108 We've discussed this a bit internally, and finally think that both options might make sense, so we plan to support a different approach (think a sentence is a step or a word is a step) in the near future (should be this week).
In addition, the parameters of RL may be difficult to adjust. The default parameters given in our code are not available, and further parameter adjustment is required.

[jasper881108]

@ht-zhou @binmakeswell Thanks for your replies.