Skip to content

[GLUTEN-8653][CORE] Bump Spark to 3.5.4#8597

Closed
jackylee-ch wants to merge 3 commits intoapache:mainfrom
jackylee-ch:bump_spark_3.5.4
Closed

[GLUTEN-8653][CORE] Bump Spark to 3.5.4#8597
jackylee-ch wants to merge 3 commits intoapache:mainfrom
jackylee-ch:bump_spark_3.5.4

Conversation

@jackylee-ch
Copy link
Copy Markdown
Contributor

@jackylee-ch jackylee-ch commented Jan 23, 2025

What changes were proposed in this pull request?

Support Spark 3.5.4

How was this patch tested?

GA

@jackylee-ch jackylee-ch marked this pull request as draft January 23, 2025 07:31
@github-actions github-actions bot added CORE works for Gluten Core INFRA TOOLS DOCS labels Jan 23, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 23, 2025

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 23, 2025

Run Gluten Clickhouse CI on x86

@zhouyuan
Copy link
Copy Markdown
Member

zhouyuan commented Jan 26, 2025

we may need to wait for a new release with this commit:
apache/iceberg@dbfefb0

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 30, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 1, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 2, 2025

Run Gluten Clickhouse CI on x86

@github-actions github-actions bot added the VELOX label Feb 2, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 2, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 2, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 3, 2025

Run Gluten Clickhouse CI on x86

@jackylee-ch jackylee-ch changed the title [DNM] Bump Spark to 3.5.4 [GLUTEN-8653][CORE] Bump Spark to 3.5.4 Feb 3, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 3, 2025

#8653

FelixYBW
FelixYBW reviewed Feb 5, 2025
View reviewed changes
Comment thread pom.xml Outdated
<sparkshim.artifactId>spark-sql-columnar-shims-spark35</sparkshim.artifactId>
<spark.version>3.5.2</spark.version>
<spark.version>3.5.4</spark.version>
<iceberg.version>1.5.0</iceberg.version>
Copy link
Copy Markdown
Contributor

@FelixYBW FelixYBW Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change the iceberg version here?

Copy link
Copy Markdown
Contributor Author

@jackylee-ch jackylee-ch Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. However, iceberg has not released version 1.7.2 yet. Iceberg 1.7.2 have a bugfix related to Spark 3.5.4, this PR needs to wait until iceberg 1.7.2 is released.

@zhouyuan
Copy link
Copy Markdown
Member

zhouyuan commented Feb 14, 2025

@jackylee-ch FYI. New iceberg is ready:
https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.8.0

@jackylee-ch
Copy link
Copy Markdown
Contributor Author

jackylee-ch commented Feb 14, 2025

@jackylee-ch FYI. New iceberg is ready: https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.8.0

It seems that 1.8.0 is not compatible with previous versions
https://lists.apache.org/thread/w403yzmpy9n9swcpwl2dcj2skdc0dskg

@github-actions github-actions bot removed the VELOX label Feb 17, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 17, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 17, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 17, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 17, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 17, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 17, 2025

Run Gluten Clickhouse CI on x86

@jackylee-ch
Copy link
Copy Markdown
Contributor Author

jackylee-ch commented Feb 24, 2025

@baibaichen I am not familiar with the compilation process of the ClickHouse backend, but upgrading to version 3.5.4 requires JDK17 for building, as Iceberg 1.8.0 mandates JDK11 or higher. Could you help verify the compilation compatibility of the ClickHouse backend under JDK17?

@zhouyuan
Copy link
Copy Markdown
Member

zhouyuan commented Feb 26, 2025

Cc @lwz9103

@zhouyuan zhouyuan mentioned this pull request Feb 27, 2025
Merged
@zhouyuan
Copy link
Copy Markdown
Member

zhouyuan commented Mar 3, 2025

note: Spark-355 is released on 27-Feb
https://spark.apache.org/releases/spark-release-3-5-5.html

@FelixYBW
Copy link
Copy Markdown
Contributor

FelixYBW commented Mar 3, 2025

note: Spark-355 is released on 27-Feb spark.apache.org/releases/spark-release-3-5-5.html

should we skip 354? does Iceberg support 355?

@jackylee-ch
Copy link
Copy Markdown
Contributor Author

jackylee-ch commented Mar 4, 2025

should we skip 354? does Iceberg support 355?

There seem to be no conflicts for upgrading Iceberg with Spark 3.5.5, I will create a new PR for upgradding Spark 3.5.5.

@FelixYBW
Copy link
Copy Markdown
Contributor

FelixYBW commented Mar 4, 2025

should we skip 354? does Iceberg support 355?

There seem to be no conflicts for upgrading Iceberg with Spark 3.5.5, I will create a new PR for upgradding Spark 3.5.5.

Sounds good. Let's skip 354 then.

@FelixYBW
Copy link
Copy Markdown
Contributor

FelixYBW commented Mar 4, 2025

Let's close this one since #8890

@FelixYBW FelixYBW closed this Mar 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FelixYBW FelixYBW FelixYBW left review comments

Assignees

No one assigned

Labels

CORE works for Gluten Core DOCS INFRA TOOLS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jackylee-ch @zhouyuan @FelixYBW