Skip to content

Refactor SchedulerState from Scheduler #4365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mrocklin merged 44 commits into from
Jan 22, 2021
Merged

Refactor SchedulerState from Scheduler #4365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
6dbb36f
Move `worker_send` into transition functions
jakirkham Dec 16, 2020
1bf1faa
Refactor `_task_to_msg` from `send_task_to_worker`
jakirkham Dec 16, 2020
9f44f06
Move `report` out of `_add_to_memory`
jakirkham Dec 16, 2020
68cf243
Refactor out `_client_releases_keys`
jakirkham Dec 16, 2020
3c9d3ac
Collect client recs in `_add_to_memory`
jakirkham Dec 16, 2020
1332985
Use `_client_releases_keys` in transitions
jakirkham Dec 16, 2020
fe31d61
Refactor out `_task_to_report_msg`
jakirkham Dec 16, 2020
6b533a1
Collect and send worker messages from transitions
jakirkham Dec 16, 2020
2abe46c
Handle `report` in `transition`
jakirkham Dec 16, 2020
bc13fbe
Add method to send a message to a specific client
jakirkham Dec 16, 2020
0138c53
Add `_task_to_client_msgs`
jakirkham Dec 16, 2020
b0ffcf2
Replace `report_msg` with `client_msgs`
jakirkham Dec 16, 2020
fe33d4a
Create empty `SchedulerState` class
jakirkham Dec 16, 2020
5960577
Move `transition*` methods into `SchedulerState`
jakirkham Dec 16, 2020
3a02cea
Add attributes for `SchedulerState`
jakirkham Dec 16, 2020
ac4fe97
Initialize attributes in `SchedulerState`
jakirkham Dec 16, 2020
f39caef
Pass arguments to `super` class
jakirkham Dec 16, 2020
6379cd1
Use `SchedulerState` attributes
jakirkham Dec 16, 2020
980fd67
Use `cast` to access parent class attributes
jakirkham Dec 16, 2020
8dc806d
Drop no longer needed `cast`s & local assignments
jakirkham Dec 16, 2020
f3b07f2
Use `dict` views onto `SortedDict` where possible
jakirkham Dec 16, 2020
38c1c35
Add `@property`s for `SchedulerState` attributes
jakirkham Dec 16, 2020
8fe8388
Take `worker_msgs` arg in `_propagate_forgotten`
jakirkham Dec 16, 2020
7b5dec3
Use `parent` for private methods
jakirkham Dec 18, 2020
8a9a1de
Annotate function arguments and return types
jakirkham Dec 16, 2020
2114ea8
Use `cfunc` on private `SchedulerState` methods
jakirkham Dec 18, 2020
096b661
Decorate `SchedulerState` methods with `@ccall`
jakirkham Dec 16, 2020
bbf156c
Add optional args to `transition_waiting_memory`
jakirkham Jan 8, 2021
5fdc871
Drop `**kwargs` from `_add_to_memory`
jakirkham Jan 8, 2021
2c76c6a
Use `cfunc` to decorate `_add_to_memory`
jakirkham Jan 8, 2021
7fb5228
Make `SchedulerState` private methods functions
jakirkham Jan 8, 2021
c7f2078
Collect `@property`s in `__pdict__`
jakirkham Jan 9, 2021
2212edc
Use `__pdict__` with `__dict__` in dashboard
jakirkham Jan 9, 2021
cd99140
Merge dask/master into jakirkham/ref_trans2
jakirkham Jan 14, 2021
460f12f
Merge dask/master into jakirkham/ref_trans2
jakirkham Jan 20, 2021
bef5240
Refactor `consume_resources` & `release_resources`
jakirkham Jan 20, 2021
caf3c1a
Refactor `aliases` into `SchedulerState`
jakirkham Jan 20, 2021
ab5beca
Refactor `coerce_hostname` into `SchedulerState`
jakirkham Jan 20, 2021
f93ecad
Refactor `task_metadata` into `SchedulerState`
jakirkham Jan 20, 2021
9275830
Refactor `remove_key` into `SchedulerState`
jakirkham Jan 20, 2021
19174d2
Merge dask/master into jakirkham/ref_trans2
jakirkham Jan 21, 2021
a2f00e2
Merge dask/master into jakirkham/ref_trans2
jakirkham Jan 21, 2021
db04403
Merge dask/master into jakirkham/ref_trans2
jakirkham Jan 22, 2021
4c0720e
Merge dask/master into jakirkham/ref_trans2
jakirkham Jan 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions distributed/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ def __init__(
connection_args=None,
timeout=None,
io_loop=None,
**kwargs,
):
self.handlers = {
"identity": self.identity,
Expand Down Expand Up @@ -236,6 +237,8 @@ def set_thread_ident():

self.__stopped = False

super().__init__(**kwargs)
Copy link
Contributor

@madsbk madsbk Jan 14, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit confused, why do we need kwargs? And call to super?

Copy link
Member Author

@jakirkham jakirkham Jan 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's fair. It's a little confusing.

The call to super is needed to call the parent constructors of Scheduler. These are ServerNode and SchedulerState. We also want to affect when these get called relative to other things in this constructor (as this constructor needs some of the attributes of the parent classes to be setup in later steps).

As Scheduler inherits from two base classes, we need to make sure that arguments for those constructors get passed through and picked up by the right parent class. There are some more details in this SO answer that may help clarify this further.


@property
def status(self):
return self._status
Expand Down
24 changes: 21 additions & 3 deletions distributed/http/scheduler/info.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,13 @@ def get(self):
"workers.html",
title="Workers",
scheduler=self.server,
**merge(self.server.__dict__, ns, self.extra, rel_path_statics),
**merge(
self.server.__dict__,
self.server.__pdict__,
ns,
self.extra,
rel_path_statics,
),
)


Expand All @@ -49,7 +55,13 @@ def get(self, worker):
title="Worker: " + worker,
scheduler=self.server,
Worker=worker,
**merge(self.server.__dict__, ns, self.extra, rel_path_statics),
**merge(
self.server.__dict__,
self.server.__pdict__,
ns,
self.extra,
rel_path_statics,
),
)


Expand All @@ -65,7 +77,13 @@ def get(self, task):
title="Task: " + task,
Task=task,
scheduler=self.server,
**merge(self.server.__dict__, ns, self.extra, rel_path_statics),
**merge(
self.server.__dict__,
self.server.__pdict__,
ns,
self.extra,
rel_path_statics,
),
)


Expand Down
Loading