Issues encountered in custom multi-turn example

[terrykong]

Discussion of issues encountered in #606

[jialei777]

The current run_multi_turn_rollout method seems not handling multi-turn conversation between policy and simulate user well:

1. The initial user message is wrapped with chat template, so for llama, it will be something like

<|begin_of_text|><|start_header_id|>user<|end_header_id|>This is the initial user message.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

2. Then the policy will generate a message without chat template but eot appended, say

This is the assistant message.<|eot_id|>

3. Finally, the simulated user message is also raw without chat template, say

This is the second user message.

And concatenating all the messages together and query for next policy message is clearly incorrect:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>This is the initial user message.<|eot_id|><|start_header_id|>assistant<|end_header_id|>This is the assistant message.<|eot_id|>This is the second user message.

To address this issue, I wrapped with chat template (and manually remove the start of conversation token <|begin_of_text|>) for the simulated user message in 3, and get

<|start_header_id|>user<|end_header_id|>This is the second user message.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Now the prompt for next policy message seems to be correct:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>This is the initial user message.<|eot_id|><|start_header_id|>assistant<|end_header_id|>This is the assistant message.<|eot_id|><|start_header_id|>user<|end_header_id|>This is the second user message.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

And then repeat 2 and 3 for the whole multi-turn conversation. This seems not elegant and unintuitive, and wondering if there are better design for the issue.