[SPARK-54789] Support label selector based filter on resources to reconcile #439

jiangzho · 2025-12-20T02:07:15Z

What changes were proposed in this pull request?

This PR adds a new property spark.kubernetes.operator.reconciler.labelSelector. If set to non empty value, operator would only reconcile custom resources that match the given selector filter. By default, operator would still reconcile all resources.

Why are the changes needed?

Introducing optional label based selectors allows operator to reconcile only a targeted subset of custom resources when configured explicitly.

It's a common case for user to leverage label for isolation on resources. This can be a result of various reason, including setting-up multiple operator per environment / group in shared clusters, or for progressive adoption upon new versions. This feature would make operator fits better for prod level multi-tenant environments.

Does this PR introduce any user-facing change?

New property becomes available while behavior is the same as before by default.

How was this patch tested?

Existing e2e validates the default behavior. Added new scenario to test selector functionality.

Was this patch authored or co-authored using generative AI tooling?

No

jiangzho · 2025-12-20T02:08:00Z

cc @peter-toth for review - many thanks in advance!

dongjoon-hyun · 2025-12-20T07:22:35Z

docs/config_properties.md

 | spark.kubernetes.operator.reconciler.retry.maxIntervalSeconds | Integer | -1 | false | Max interval(in seconds) of retries on unhandled controller errors. Set to non-positive for unlimited. | 
 | spark.kubernetes.operator.reconciler.terminationTimeoutSeconds | Integer | 30 | false | Grace period for operator shutdown before reconciliation threads are killed. | 
 | spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled | Boolean | true | true | When enabled, operator would trim state transition history when a new attempt starts, keeping previous attempt summary only. | 
+ | spark.kubernetes.operator.reconciliationSelectors | String |  | false | Comma-separated label selector(s) that the operator would be reconciling for Spark resources. If not set or set to empty string, operator would reconcile all watched Spark Apps and Clusters. | 


Can we reuse the existing namespace, spark.kubernetes.operator.reconciler, like this? And, selector might be better because we use spark-app-selector and spark.kubernetes.node.selector.* before, @jiangzho .

- spark.kubernetes.operator.reconciliationSelectors + spark.kubernetes.operator.reconciler.selector

that's a good idea! I've renamed that as spark.kubernetes.operator.reconciler.selectors , plural just to indicate the value can include multiple selectors

dongjoon-hyun · 2025-12-20T07:24:15Z

tests/e2e/resource-selector/chainsaw-test.yaml

+        - name: JAVA_VERSION
+          value: "17"
+        - name: IMAGE
+          value: "apache/spark:4.1.0-scala-java17"


Please use apache/spark:4.1.0-scala because we are currently advertising Java 21 over 17.

updated, thanks!

dongjoon-hyun · 2025-12-20T07:25:19Z

tests/e2e/resource-selector/chainsaw-test.yaml

+        - name: SPARK_VERSION
+          value: "4.1.0"
+        - name: JAVA_VERSION
+          value: "17"


Ditto. Java 21 please.

dongjoon-hyun · 2025-12-20T07:26:19Z

tests/e2e/resource-selector/spark-example.yaml

+    spark.kubernetes.executor.request.cores: "0.5"
+  runtimeVersions:
+    sparkVersion: ($V_SPARK_VERSION)
+    jdkVersion: ($V_JAVA_VERSION)


Shall we remove this because technically jdkVersion didn't provide a meaningful effect in the community?

actually yes - we shall not care more about it

dongjoon-hyun

Do you have any reference for your definition of selectors, @jiangzho ?

that's a good idea! I've renamed that as spark.kubernetes.operator.reconciler.selectors , plural just to indicate the value can include multiple selectors

AFAIK, in K8s, it's still a single selector to match all targets. For example, everything is a single selector, fieldSelector, labelSelector, nodeSelector, and selector. We don't use in a plural form, selectors, to specific a single set of target.

https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements

selector:
  matchLabels:
    component: redis
  matchExpressions:
    - { key: tier, operator: In, values: [cache] }
    - { key: environment, operator: NotIn, values: [dev] }

If you really want to use a plural form, I'd like to recommend to use matchLabels for the config name.

- spark.kubernetes.operator.reconciler.selectors
+ spark.kubernetes.operator.reconciler.matchLabels

dongjoon-hyun · 2025-12-23T02:27:19Z

tests/e2e/helm/selector-config-values.yaml

+
+operatorConfiguration:
+  spark-operator.properties: |+
+    spark.kubernetes.operator.reconciler.selectors=foo=bar


We need more test coverage to specify multiple labels.

ack - updated the test case such that

Operator is configured with comma-separated key values

Validate resources is not being reconciled if

no label matches the given selector string

resource has only one matching label, but not all

resource has given label, but the value does not match

Resource is reconciled only when matching all required label selectors

dongjoon-hyun · 2025-12-23T02:29:16Z

tests/e2e/resource-selector/spark-example.yaml

+    spark.kubernetes.container.image: ($IMAGE)
+    spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
+    spark.kubernetes.driver.request.cores: "0.5"
+    spark.kubernetes.executor.request.cores: "0.5"


Please remove the above two lines because those are irrelevant to this PR.

dongjoon-hyun · 2025-12-23T05:45:04Z

It seems that the following is not addressed yet.

[SPARK-54789] Support label selector based filter on resources to reconcile #439 (review)

In addition, please rebase once more in order to resolve the conflicts.

…oncile ### What changes were proposed in this pull request? This PR adds a new property `spark.kubernetes.operator.reconciliationSelectors`. If set to non empty value, operator would only reconcile custom resources that match the given selector filter. By default, operator would still reconcile all resources. ### Why are the changes needed? Introducing optional label based selectors allows operator to reconcile only a targeted subset of custom resources when configured explicitly. It's a common case for user to leverage label for isolation on resources. This can be a result of various reason, including setting-up multiple operator per environment / group in shared clusters, or for progressive adoption upon new versions. This feature would make operator fits better for prod level multi-tenant environments. ### Does this PR introduce any user-facing change? New property becomes available while behavior is the same as before by default. ### How was this patch tested? Existing e2e validates the default behavior. Added new scenario to test selector functionality. ### Was this patch authored or co-authored using generative AI tooling? No

…nf, and update test case

jiangzho · 2025-12-23T07:30:06Z

Sorry overlooked previous comment - agreed that one selector shall applied to all resources given

Via a label selector, the client/user can identify a set of objects. The label selector is the core grouping primitive in Kubernetes.

A label selector can be made of multiple requirements which are comma-separated.

from https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors

I've updated the naming back to singular labelSelector to better indicate the nature of this feature. How does this sound ?

dongjoon-hyun · 2025-12-23T21:14:29Z

Please update the PR description accordingly.

This PR adds a new property spark.kubernetes.operator.reconciliationSelectors.

dongjoon-hyun · 2025-12-23T21:15:56Z

docs/config_properties.md

 | spark.kubernetes.operator.reconciler.clusterStatusListenerClassNames | String |  | false | Comma-separated names of SparkClusterStatusListener class implementations | 
 | spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds | Long | 60 | true | Timeout (in seconds) for requests made to API server. This applies only to foreground requests. | 
 | spark.kubernetes.operator.reconciler.intervalSeconds | Long | 120 | true | Interval (in seconds, non-negative) to reconcile Spark applications. Note that reconciliation is always expected to be triggered when app spec / status is updated. This interval controls the reconcile behavior of operator reconciliation even when there's no update on SparkApplication, e.g. to determine whether a hanging app needs to be proactively terminated. Thus this is recommended to set to above 2 minutes to avoid unnecessary no-op reconciliation. | 
+ | spark.kubernetes.operator.reconciler.labelSelector | String |  | false | Comma-separated label selector(s) that the operator would be reconciling for Spark resources. If not set or set to empty string, operator would reconcile all watched Spark Apps and Clusters. | 


Comma-separated label selector(s) sounds a little weird. Could you revise the sentence.

dongjoon-hyun · 2025-12-23T21:16:37Z

spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java

+          .key("spark.kubernetes.operator.reconciler.labelSelector")
+          .enableDynamicOverride(false)
+          .description(
+              "Comma-separated label selector(s) that the operator would be reconciling for " +


ditto. We need to revise the phrase, Comma-separated label selector(s).

csviri

LGTM

jiangzho · 2025-12-23T23:08:46Z

Checked CI failures for previous commit, but seems they are not directly related to this PR

jiangzho · 2026-01-02T23:24:12Z

cc @dongjoon-hyun - May I get another round of review for this please ? Thanks a lot

dongjoon-hyun

+1, LGTM. Thank you, @jiangzho and all.

Merged to main.

github-actions bot added OPERATOR INFRA labels Dec 20, 2025

dongjoon-hyun reviewed Dec 20, 2025

View reviewed changes

dongjoon-hyun requested changes Dec 23, 2025

View reviewed changes

dongjoon-hyun reviewed Dec 23, 2025

View reviewed changes

jiangzho added 4 commits December 23, 2025 15:21

Update property name and test image

48cfdaf

Ensure consistent variable name for conf prop, remove unnececssary co…

439b09c

…nf, and update test case

Rename to 'labelSelector' for better alignment

1f159ea

jiangzho force-pushed the selector branch from 0e24b71 to 1f159ea Compare December 23, 2025 07:22

dongjoon-hyun reviewed Dec 23, 2025

View reviewed changes

csviri approved these changes Dec 23, 2025

View reviewed changes

Update the description for new property

7dc71a7

dongjoon-hyun approved these changes Jan 4, 2026

View reviewed changes

dongjoon-hyun closed this in c2b9a0a Jan 4, 2026

[SPARK-54789] Support label selector based filter on resources to reconcile #439

[SPARK-54789] Support label selector based filter on resources to reconcile #439

Uh oh!

Conversation

jiangzho commented Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

jiangzho commented Dec 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiangzho commented Dec 23, 2025

Uh oh!

dongjoon-hyun commented Dec 23, 2025

Uh oh!

dongjoon-hyun Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csviri left a comment

Choose a reason for hiding this comment

Uh oh!

jiangzho commented Dec 23, 2025

Uh oh!

jiangzho commented Jan 2, 2026

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jiangzho commented Dec 20, 2025 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Dec 23, 2025 •

edited

Loading

dongjoon-hyun Dec 23, 2025 •

edited

Loading