Skip to content

Conversation

@gjoseph92
Copy link
Collaborator

I was reading through the scheduling policy docs and noticed some things were outdated or missing important details. I think it could still benefit from a bigger revamp, but makes it a bit more accurate.

cc @fjetter @crusaderky

  • Closes #xxxx
  • Passes pre-commit run --all-files

I was reading through the [scheduling policy docs](http://distributed.dask.org/en/latest/scheduling-policies.html) and noticed some things were outdated or missing important details. I think it could still benefit from a bigger revamp, but makes it a bit more accurate.
@github-actions
Copy link
Contributor

github-actions bot commented Mar 8, 2022

Unit Test Results

       12 files  ±       0         12 suites  ±0   5h 50m 30s ⏱️ - 1h 29m 1s
  2 653 tests +     30    2 570 ✔️ +     30    80 💤 ±    0  3 ±0 
13 021 runs   - 2 641  12 380 ✔️  - 2 417  637 💤  - 225  4 +1 

For more details on these failures, see this check.

Results for commit 641f3b7. ± Comparison against base commit 39c5e88.

♻️ This comment has been updated with latest results.

before starting new work. This often conflicts with the
first-come-first-served objective but often results in shorter total runtimes
first-come-first-served objective, but often results in shorter total runtimes
and significantly reduced memory footprints.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

short of avoiding disk spillage, why would it result in a shorter runtime?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't write this, really not sure. Maybe just that depth-first traversal is faster than breadth-first traversal (simply because it has a reduced memory footprint)? I feel like it might be referring to something about when multiple clients are submitting work, completing the work of one, rather than fully interleaving them, results in better overall runtimes (again because of reduced memory footprint)?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It says that it does in fact go against first-come-first-served expectations.
Maybe change to:

but often results in significantly reduced memory footprints and, due to avoiding data spillage to disk, often in better overall runtimes.

@gjoseph92 gjoseph92 self-assigned this Mar 8, 2022
@gjoseph92 gjoseph92 requested a review from crusaderky March 10, 2022 05:29
gjoseph92 and others added 2 commits March 16, 2022 20:49
Co-authored-by: crusaderky <crusaderky@gmail.com>
@gjoseph92
Copy link
Collaborator Author

@crusaderky I believe this is ready

@crusaderky crusaderky merged commit 8957c14 into dask:main Mar 17, 2022
@crusaderky
Copy link
Collaborator

Thank you!

@gjoseph92 gjoseph92 deleted the scheduling-policies-updates branch March 17, 2022 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants