[BUG]: code in ppo.py always read same datas from prompt_dataloader

[lizongzhi]

🐛 Describe the bug

“ prompts = next(iter(self.prompt_dataloader))” always read same data from prompt_dataloader. Is it correct?

And what is meaning of "num_episodes" and "max_timesteps"? How to set these two param?

thank you

Environment

No response

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Title: [BUG]:

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@chengeharrison

[binmakeswell]

Hi @lizongzhi Thanks for your feedback, we will fix it later.

[ws-liutl1]

This will raise a StopIteration exception. How to set num_episodes and max_timesteps ?

[chengeharrison]

The error lies in the __len__ fuction in coati/dataset/prompt_dataset.py in line 47, a quick fix can be:

    def __len__(self):
        return len(self.keyed_prompt["input_ids"])

Don't forget to run pip install . when you make changes to coati.

We will soon fix this in the main branch.

[ws-liutl1]

The error lies in the __len__ fuction in coati/dataset/prompt_dataset.py in line 47, a quick fix can be:

    def __len__(self):
        return len(self.keyed_prompt["input_ids"])

Don't forget to run pip install . when you make changes to coati.

We will soon fix this in the main branch.

it work now. Thank you

[chengeharrison]

🐛 Describe the bug

“ prompts = next(iter(self.prompt_dataloader))” always read same data from prompt_dataloader. Is it correct?

And what is meaning of "num_episodes" and "max_timesteps"? How to set these two param?

thank you

Environment

No response

Hi, @lizongzhi, here are some explanations.

1. During one timestep, experience will be made.
2. When timestep % update_timesteps == 0, the model will learn once.
3. When timestep reaches max_timesteps, one episode is over.

This training process is similar with that of other RL applications such as training an agent to play games.

1. In one episode, the agent will have one life.
2. In each step the agent will choose one action such as attack or something and the system will return a reward (this is "make experience").
3. When timestep % update_timesteps == 0, the agent will learn to play the game accoding to its experience buffer.
4. One episode will end if the agent loses its life or it reaches the max timestep.

To set these parameters, I think you should at least let num_episodes * max_timesteps * experience_batch_size >> prompt dataset size and make sure the model is updated at an appropriate frequency (number of updating: num_episodes * max_timesteps / update_timesteps).