Merged
Conversation
Edilmo
commented
Jan 29, 2021
|
|
||
| thread_local = threading.local() | ||
|
|
||
| UNION_MAX_PULL = 1000 |
Author
There was a problem hiding this comment.
Before this value hardcoded and not accesible, with a value of 100.
The value controls how many pull operations are going to be tried before stoping (StopIteration) an UNION of PARALLEL-ITERATORS.
Also, before, all pull reties over an UNION of PARALLEL-ITERATORS were made one after the other without any waiting time. This was a too optimistic approach that could lead to a lot of false-negative when the source of one of the iterators in the UNION is slow (or even worst if all iterators are slow).
Here we are not only increasing the amount of retries to 1000 but also introducing an exponential backoff logic in these retries operations.
There are some trade-off with this approach though:
- FAST sources: the exponential backoff logic could slow down a little the pipeline when some sort of intermittent failure or delays.
- SLOW sources: the default values could be too small if the sources are extremely slow.
Edilmo
added a commit
that referenced
this pull request
Feb 4, 2021
* fixing broken generators * improving fix for broken generators * Remove npm authenticate * Disable Lint Job * Testing parallel iterators in finite sequences explicitly * Tuning some tests * Remove logging * Tuning timeouts * More fine tuning * Improving Parallel Iterators tests * Increasing time * Increasing time # Conflicts: # ci/azure_pipelines/templates/gcs-python.yml # ci/azure_pipelines/templates/multi-node.yml # ci/azure_pipelines/templates/rlib-regression-tf-2.yml # python/ray/tests/test_memory_scheduling.py # python/ray/util/iter.py # python/ray/util/sgd/tests/test_torch.py # rllib/execution/concurrency_ops.py # rllib/execution/replay_buffer.py # rllib/execution/replay_ops.py # rllib/tests/test_execution.py
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
There are some generators swallowing StopIteration exceptions in SAC execution plan.
The fix take advantage of PEP 479 to address the problem.
For more details about this PEP, here is the CPython PR.
Related issue number
None
Checks
scripts/format.shto lint the changes in this PR.