[RLLib ] Better default parameters with training intensity for R2D2

[ArturNiederfahrenhorst]

Why are these changes needed?

This PR deflakes R2D2 learning regression tests.
Since I introduced training intesity to R2D2 in #24923, the relevant parameters had not been tuned.

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[avnishn]

So this all looks fine to me, but can you clarify:

what are the "necessary parameters that you needed to turn on"?
It doesn't look like you turned on any parameters in this PR.

[ArturNiederfahrenhorst]

There are no "necessary parameters to turn on" but "relevant parameters to be tuned"! 😄

The relevant parameters I was referring to are target_network_update_freq and training_intensity. Both are not part of the R2D2 paper and I've therefore ventured to tune them a little for the only learning regression test that we have for R2D2.
The relevant parameters should have been tuned when I introduced training intensity to R2D2, which was also when I introduced training_intensity.

[gjoliver]

how long does it take to train this on your laptop after the change?

[ArturNiederfahrenhorst]

Hey @gjoliver ,
Before the changes: 150s
After the changes: 50s