Skip to content

feat: scheduler (8/): add more scheduler framework logic #389

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
michaelawyu wants to merge 5 commits into from
Closed

feat: scheduler (8/): add more scheduler framework logic #389

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9a1de88
Added more scheduler logic
michaelawyu Jun 16, 2023
aa13324
Minor fixes
michaelawyu Jun 19, 2023
0c7d0bf
Reflect changes in design refresh
michaelawyu Jun 21, 2023
90d2ec1
Minor fixes
michaelawyu Jun 21, 2023
d3aebd1
Merge branch 'main' into scheduler-framework-logic-2
ryanzhang-oss Jun 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions apis/placement/v1beta1/binding_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,49 @@ import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

const (
// DispatcherFinalizer is added by the dispatcher to make sure that a binding can only be deleted if the dispatcher
// has removed all selected resources from the bound cluster.
DispatcherFinalizer = fleetPrefix + "dispatcher-cleanup"

// SchedulerFinalizer is added by the scheduler to make sure that a binding can only be deleted if the scheduler
// has relieved it from scheduling consideration.
SchedulerFinalizer = fleetPrefix + "scheduler-cleanup"
Copy link
Contributor Author

@michaelawyu michaelawyu Jun 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically speaking, since at this moment user no longer has the permission to delete bindings manually, we do not need this finalizer any more (it was originally designed for making sure that the scheduler will always get notified when a binding is deleted so that it can creates a new one in replacement).

However, in this PR I preserved this finalizer just for the case where a binding still needs to be manually deleted, in case of, say, troubleshooting, emergency fix, or future support for individual eviction -> it might not be optimal if the only way for a customer to remove resources from a cluster is to trigger a rescheduling via policy update; though at this stage it might not be of too much concern.


// ActiveBindingLabel is added by the update controller to mark that a binding is active, i.e., the dispatcher
// should place resources to it.
//
// Note that an active binding may not be associated with the latest scheduling policy snapshot or the latest
// resource snapshot. It may be up to another controller, e.g., the rolling update controller, to modify the
// association (if applicable). In certain cases (e.g., not enough fitting clusters), the binding may not even
// has a target cluster.
//
// Note also that it is not the scheduler's responsibility to add this label, even though it does
// reads this label to inform the scheduling cycle..
ActiveBindingLabel = fleetPrefix + "is-active-binding"
Copy link
Contributor Author

@michaelawyu michaelawyu Jun 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The states are kept as labels for easier queries; a field would also work though. Anti-tempering is needed to make sure that the system functions correctly.


// CreatingBindingLabel is added by the scheduler to mark that a binding is being created. Any binding in
// this state should not be picked up by the dispatcher.
//
// Note that the scheduler **always** produces enough number of bindings, as user specified, after a scheduling run,
// even if there might not be enough number of fitting clusters.
//
// Note also that it is up to another controller, e.g., the rolling update controller, to mark a creating
// binding as active.
CreatingBindingLabel = fleetPrefix + "is-creating-binding"

// ObsoleteBindingLabel is added by the scheduler to mark that a binding is no longer needed, i.e., its
// associated cluster (if any) no longer fits the current (as seen by the scheduler) scheduling policy.
//
// Note that it is up to another controller, e.g, the rolling update controller, to actually delete the
// binding.
ObsoleteBindingLabel = fleetPrefix + "is-obsolete-binding"

// NoTargetClusterBindingLabel is added by the scheduler to mark that a binding does not have a target cluster.
// This usually happens when there is not enough number of fitting clusters in the system.
NoTargetClusterBindingLabel = fleetPrefix + "no-target-cluster-binding"
)

// +kubebuilder:object:root=true
// +kubebuilder:resource:scope=Cluster,categories={fleet},shortName=rb
// +kubebuilder:subresource:status
Expand Down
22 changes: 11 additions & 11 deletions ...placement/v1beta1/policysnapshot_types.go → ...ement/v1beta1/schedulingpolicysnapshot.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ const (
// PolicyIndexLabel is the label that indicate the policy snapshot index of a cluster policy.
PolicyIndexLabel = fleetPrefix + "policyIndex"

// PolicySnapshotNameFmt is clusterPolicySnapshot name format: {CRPName}-{PolicySnapshotIndex}.
// PolicySnapshotNameFmt is schedulingPolicySnapshot name format: {CRPName}-{PolicySnapshotIndex}.
PolicySnapshotNameFmt = "%s-%d"

// NumberOfClustersAnnotation is the annotation that indicates how many clusters should be selected for selectN placement type.
Expand All @@ -30,15 +30,15 @@ const (
// +kubebuilder:printcolumn:JSONPath=`.metadata.creationTimestamp`,name="Age",type=date
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// ClusterPolicySnapshot is used to store a snapshot of cluster placement policy.
// SchedulingPolicySnapshot is used to store a snapshot of cluster placement policy.
// Its spec is immutable.
// The naming convention of a ClusterPolicySnapshot is {CRPName}-{PolicySnapshotIndex}.
// The naming convention of a SchedulingPolicySnapshot is {CRPName}-{PolicySnapshotIndex}.
// PolicySnapshotIndex will begin with 0.
// Each snapshot must have the following labels:
// - `CRPTrackingLabel` which points to its owner CRP.
// - `PolicyIndexLabel` which is the index of the policy snapshot.
// - `IsLatestSnapshotLabel` which indicates whether the snapshot is the latest one.
type ClusterPolicySnapshot struct {
type SchedulingPolicySnapshot struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`

Expand Down Expand Up @@ -131,27 +131,27 @@ type ClusterScore struct {
TopologySpreadScore *int32 `json:"priorityScore,omitempty"`
}

// ClusterPolicySnapshotList contains a list of ClusterPolicySnapshot.
// SchedulingPolicySnapshotList contains a list of SchedulingPolicySnapshot.
// +kubebuilder:resource:scope="Cluster"
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type ClusterPolicySnapshotList struct {
type SchedulingPolicySnapshotList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []ClusterPolicySnapshot `json:"items"`
Items []SchedulingPolicySnapshot `json:"items"`
}

// SetConditions sets the given conditions on the ClusterPolicySnapshot.
func (m *ClusterPolicySnapshot) SetConditions(conditions ...metav1.Condition) {
// SetConditions sets the given conditions on the SchedulingPolicySnapshot.
func (m *SchedulingPolicySnapshot) SetConditions(conditions ...metav1.Condition) {
for _, c := range conditions {
meta.SetStatusCondition(&m.Status.Conditions, c)
}
}

// GetCondition returns the condition of the given type if exists.
func (m *ClusterPolicySnapshot) GetCondition(conditionType string) *metav1.Condition {
func (m *SchedulingPolicySnapshot) GetCondition(conditionType string) *metav1.Condition {
return meta.FindStatusCondition(m.Status.Conditions, conditionType)
}

func init() {
SchemeBuilder.Register(&ClusterPolicySnapshot{}, &ClusterPolicySnapshotList{})
SchemeBuilder.Register(&SchedulingPolicySnapshot{}, &SchedulingPolicySnapshotList{})
}
118 changes: 59 additions & 59 deletions apis/placement/v1beta1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 7 additions & 7 deletions ...nt.karavel.io_clusterpolicysnapshots.yaml → ...karavel.io_schedulingpolicysnapshots.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,18 @@ kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.11.4
name: clusterpolicysnapshots.placement.karavel.io
name: schedulingpolicysnapshots.placement.karavel.io
spec:
group: placement.karavel.io
names:
categories:
- fleet-workload
kind: ClusterPolicySnapshot
listKind: ClusterPolicySnapshotList
plural: clusterpolicysnapshots
kind: SchedulingPolicySnapshot
listKind: SchedulingPolicySnapshotList
plural: schedulingpolicysnapshots
shortNames:
- pss
singular: clusterpolicysnapshot
singular: schedulingpolicysnapshot
scope: Cluster
versions:
- additionalPrinterColumns:
Expand All @@ -28,8 +28,8 @@ spec:
name: v1beta1
schema:
openAPIV3Schema:
description: 'ClusterPolicySnapshot is used to store a snapshot of cluster
placement policy. Its spec is immutable. The naming convention of a ClusterPolicySnapshot
description: 'SchedulingPolicySnapshot is used to store a snapshot of cluster
placement policy. Its spec is immutable. The naming convention of a SchedulingPolicySnapshot
is {CRPName}-{PolicySnapshotIndex}. PolicySnapshotIndex will begin with
0. Each snapshot must have the following labels: - `CRPTrackingLabel` which
points to its owner CRP. - `PolicyIndexLabel` which is the index of the
Expand Down
Loading