Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Jan 6, 2026

What changes were proposed in this pull request?

This PR aims to enable volcano profile by default for Apache Spark 4.2.0.

Why are the changes needed?

Volcano scheduler has been supported experimentally since Apache Spark 3.3.0 and stably since 3.4.0.

This helps volcano users.

Does this PR introduce any user-facing change?

No behavior change.

How was this patch tested?

Pass the CIs and manually build.

$ dev/make-distribution.sh -Pkubernetes

$ jar tvf dist/jars/spark-kubernetes_2.13-4.2.0-SNAPSHOT.jar | grep Volcano
  1059 Sat Mar 15 11:45:26 KST 2025 org/apache/spark/deploy/k8s/features/VolcanoFeatureStep$.class
 19891 Sat Mar 15 11:45:26 KST 2025 org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.class

$ ls -al dist/jars/volcano-*
-rw-r--r--@ 1 dongjoon  staff   10108 Jan  6 20:51 dist/jars/volcano-client-7.4.0.jar
-rw-r--r--@ 1 dongjoon  staff  547522 Jan  6 20:51 dist/jars/volcano-model-7.4.0.jar

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions
Copy link

github-actions bot commented Jan 6, 2026

JIRA Issue Information

=== Sub-task SPARK-54916 ===
Summary: Enable volcano profile by default
Assignee: Dongjoon Hyun
Status: Open
Affected: ["4.2.0"]


This comment was automatically generated by GitHub Actions

@dongjoon-hyun dongjoon-hyun marked this pull request as draft January 6, 2026 11:34
@dongjoon-hyun dongjoon-hyun marked this pull request as ready for review January 6, 2026 11:53
Copy link
Member

@pan3793 pan3793 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Volcano has been running in our production for many years.

@dongjoon-hyun
Copy link
Member Author

Thank you, @pan3793 !

@dongjoon-hyun
Copy link
Member Author

Thank you, @HyukjinKwon .

Merged to master for Apache Spark 4.2.0.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-54916 branch January 6, 2026 23:28
dongjoon-hyun added a commit to apache/spark-kubernetes-operator that referenced this pull request Jan 7, 2026
### What changes were proposed in this pull request?

This PR aims to add `Volcano` example.

### Why are the changes needed?

To align with Apache Spark 4.2.0 (main repository) in order to support `Volcano` users.

- apache/spark#53693

### Does this PR introduce _any_ user-facing change?

No behavior change.

### How was this patch tested?

Manual review.

### Was this patch authored or co-authored using generative AI tooling?

Yes (`Gemini 3 Pro` on `Antigravity`)

Closes #447 from dongjoon-hyun/SPARK-54927.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants