[GLUTEN-6876] Support Spark-352 by zhouyuan · Pull Request #7138 · apache/gluten

zhouyuan · 2024-09-06T00:05:39Z

What changes were proposed in this pull request?

Support Spark 352

fixes #6876

How was this patch tested?

Pass GHA

github-actions · 2024-09-06T00:05:56Z

#6876

github-actions · 2024-09-06T00:06:11Z

Run Gluten Clickhouse CI

github-actions · 2024-09-06T00:50:08Z

Run Gluten Clickhouse CI

wForget · 2024-09-06T02:21:36Z

          shuffleExecutorComponents)
      case other: BaseShuffleHandle[K @unchecked, V @unchecked, _] =>
-        new SortShuffleWriter(other, mapId, context, shuffleExecutorComponents)
+        new SortShuffleWriter(other, mapId, context, _, shuffleExecutorComponents)


Does this mean that gluten will not guarantee spark minor version compatibility?

Hi @wForget,
Not guaranteed for all minor versions indeed. Actually it's the same for Spark 32 & 3.3 & 3.4. It's only tested with the latest minor release.
The issue is mostly due to the shim layer design, and the CI/CD resources are also not enough for so many small versions.

I see, thank you.

github-actions · 2024-09-06T06:19:55Z

Run Gluten Clickhouse CI

xhumanoid · 2024-09-06T06:42:40Z

+      p: SparkPlan,
+      reason: String,
+      fallbackNodeToReason: mutable.HashMap[String, String]): Unit = {
+    p.getTagValue(QueryPlan.OP_ID_TAG).foreach {


https://issues.apache.org/jira/browse/SPARK-48610

QueryPlan.OP_ID_TAG doesn't exist in spark 3.5.2+ / 4.0.0

Thanks, there's also a minor change on sortshufflewriter, will move these changes to shim layer

github-actions · 2024-09-06T06:46:14Z

Run Gluten Clickhouse CI

github-actions · 2024-09-06T06:47:23Z

Run Gluten Clickhouse CI

github-actions · 2024-09-06T09:28:07Z

Run Gluten Clickhouse CI

github-actions · 2024-09-06T09:30:55Z

Run Gluten Clickhouse CI

github-actions · 2024-09-09T09:43:13Z

Run Gluten Clickhouse CI

github-actions · 2024-10-04T14:55:47Z

Run Gluten Clickhouse CI

github-actions · 2024-10-09T02:59:12Z

Run Gluten Clickhouse CI

github-actions · 2024-10-09T03:46:01Z

Run Gluten Clickhouse CI

github-actions · 2024-10-09T05:35:25Z

Run Gluten Clickhouse CI

leoluan2009 · 2024-10-09T06:22:16Z

Spark 3.5.3 is out https://spark.apache.org/news/spark-3-5-3-released.html, should we upgrade to 3.5.3?

zhouyuan · 2024-10-09T06:49:31Z

Spark 3.5.3 is out https://spark.apache.org/news/spark-3-5-3-released.html, should we upgrade to 3.5.3?

Yes, plan to submit one new patch for Spark-353 after this - actually there's not much change from 352 to 353
here's the discussion before the PRC holiday:
#7336

thanks, -yuan

github-actions · 2024-10-09T08:42:16Z

Run Gluten Clickhouse CI

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

github-actions · 2024-10-14T13:02:09Z

Run Gluten Clickhouse CI

baibaichen

LGTM

This reverts commit 32cd1dc.

This reverts commit 32cd1dc

This reverts commit 8e46b5c.

This reverts commit 32cd1dc

github-actions bot added CORE works for Gluten Core INFRA TOOLS labels Sep 6, 2024

zhouyuan force-pushed the wip_spark352 branch from e844f5d to 2497695 Compare September 6, 2024 00:49

wForget reviewed Sep 6, 2024

View reviewed changes

zhouyuan marked this pull request as draft September 6, 2024 06:11

xhumanoid reviewed Sep 6, 2024

View reviewed changes

zhouyuan force-pushed the wip_spark352 branch from 2f439e6 to 7ce863d Compare September 6, 2024 06:46

zhouyuan mentioned this pull request Sep 28, 2024

[VL] Bump Spark to 3.5.3 #7336

Closed

zhouyuan force-pushed the wip_spark352 branch from ecd2cae to 4775901 Compare October 9, 2024 05:34

github-actions bot added the DATA_LAKE label Oct 9, 2024

zhouyuan marked this pull request as ready for review October 9, 2024 05:35

zhztheplayer mentioned this pull request Oct 9, 2024

[VL] Prepare shim API for breaking change in SPARK-48610 #7445

Merged

zhztheplayer force-pushed the wip_spark352 branch from df34d44 to a988a6b Compare October 9, 2024 09:01

zhouyuan and others added 19 commits October 14, 2024 21:01

fix sort and op tag

5232db3

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

fix explain utils

8108980

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

fix format

9450f40

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

fix version

aff0e79

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

revert big shim layer

c344779

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

fix Spark 352 build

ed564d0

fix shim layer

bc59b55

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

fix API change

f2bf361

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

fix package

6be621d

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

fix data lake connectors

98aeb3f

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

exclude csv varcha test

ff6f3a3

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

simplify shim 1

1bf13f4

simplify shim 2

88b257a

fix shim layer code style

b7d0fdd

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

fix spark-352 source

eec45b2

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

ignore csv varchar test

0fe5663

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

ignore base64 unit tests

a303a1b

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

disable csv varchar ut in v1 suite

ef3bc44

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

ignore more unit tests

6125e92

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

baibaichen force-pushed the wip_spark352 branch from 082196c to 6125e92 Compare October 14, 2024 13:01

baibaichen approved these changes Oct 15, 2024

View reviewed changes

zhouyuan merged commit 32cd1dc into apache:main Oct 15, 2024

FelixYBW mentioned this pull request Jan 15, 2025

[GLUTEN-6876][VL] update spark 3.5.2 in doc #8543

Merged

yjshen added a commit to yjshen/incubator-gluten that referenced this pull request Feb 8, 2025

Revert "[GLUTEN-6876][CORE] Support Spark-352 (apache#7138)"

8e46b5c

This reverts commit 32cd1dc.

smatytsin pushed a commit to smatytsin/incubator-gluten that referenced this pull request May 27, 2025

Revert "[GLUTEN-6876][CORE] Support Spark-352 (apache#7138)"

f27ffaa

This reverts commit 32cd1dc

smatytsin pushed a commit to smatytsin/incubator-gluten that referenced this pull request May 28, 2025

Revert "[GLUTEN-6876][CORE] Support Spark-352 (apache#7138)"

fdef7e9

This reverts commit 32cd1dc

yjshen pushed a commit to yjshen/incubator-gluten that referenced this pull request Jun 2, 2025

Revert "Revert "[GLUTEN-6876][CORE] Support Spark-352 (apache#7138)""

32afc8b

This reverts commit 8e46b5c.

smatytsin pushed a commit to smatytsin/incubator-gluten that referenced this pull request Aug 13, 2025

Revert "[GLUTEN-6876][CORE] Support Spark-352 (apache#7138)"

90257ad

This reverts commit 32cd1dc

smatytsin pushed a commit to smatytsin/incubator-gluten that referenced this pull request Dec 6, 2025

Revert "[GLUTEN-6876][CORE] Support Spark-352 (apache#7138)"

1cf1b65

This reverts commit 32cd1dc

Conversation

zhouyuan commented Sep 6, 2024

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

wForget Sep 6, 2024

Choose a reason for hiding this comment

Uh oh!

zhouyuan Sep 6, 2024

Choose a reason for hiding this comment

Uh oh!

wForget Sep 9, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

xhumanoid Sep 6, 2024

Choose a reason for hiding this comment

Uh oh!

zhouyuan Sep 6, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

github-actions bot commented Sep 6, 2024

Uh oh!

github-actions bot commented Sep 9, 2024

Uh oh!

github-actions bot commented Oct 4, 2024

Uh oh!

github-actions bot commented Oct 9, 2024

Uh oh!

github-actions bot commented Oct 9, 2024

Uh oh!

github-actions bot commented Oct 9, 2024

Uh oh!

leoluan2009 commented Oct 9, 2024

Uh oh!

zhouyuan commented Oct 9, 2024

Uh oh!

github-actions bot commented Oct 9, 2024

Uh oh!

github-actions bot commented Oct 14, 2024

Uh oh!

baibaichen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants