Skip to content

[Ballista] Refactor scheduler state #1910

@yahoNanJing

Description

@yahoNanJing

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently the ballista scheduler state mod messes many things up in the mod file, which is not good for code reviewing.

Describe the solution you'd like

It's better to reorganize the scheduler contents as following:

  • backend, for persisting states in some storage
  • in_memory_state, for managing part of the scheduler state which is volatile and only be stored in memory
  • persistent_state, for managing part of the scheduler state which is relatively stable and be stored in one of the backend storage and cached in memory for fast reading.
  • scheduler_state, for providing a unified interface for scheduler state
  • task_scheduler, for fetching tasks to be scheduled
  • query_stage_scheduler, for dealing with stage status changes (TODO)

Additional context
#1704

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions