[GLUTEN-7706][CORE] Support Spark-344 + JDK17#7789
Conversation
|
Run Gluten Clickhouse CI |
This comment was marked as duplicate.
This comment was marked as duplicate.
zhouyuan
force-pushed
the
wip_spark344
branch
from
e269bae to
af0f113
Compare
|
Run Gluten Clickhouse CI |
|
what's the specifc reason here to switch to 3.4.4 ? |
|
Run Gluten Clickhouse CI on x86 |
4 similar comments
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
2 similar comments
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI on x86 |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI on x86 |
This reverts commit f2f14ab.
|
Run Gluten Clickhouse CI on x86 |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
zhouyuan
force-pushed
the
wip_spark344
branch
from
7b9fff7 to
396e63f
Compare
|
Run Gluten Clickhouse CI on x86 |
This reverts commit 1e504a3.
|
Run Gluten Clickhouse CI on x86 |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI on x86 |
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
|
Run Gluten Clickhouse CI on x86 |
zhouyuan
changed the title
|
Run Gluten Clickhouse CI on x86 |
|
Thank you, Yuan |
| export JAVA_HOME=/usr/lib/jvm/java-17-openjdk | ||
| export SPARK_HOME=/opt/shims/spark34/spark_home/ | ||
| ls -l /opt/shims/spark34/spark_home/ | ||
| $MVN_CMD clean test -Pspark-3.4 -Pjava-17 -Pbackends-velox -Pceleborn -Piceberg -Pdelta -Phudi -Pspark-ut \ |
There was a problem hiding this comment.
seems Spark 344 UT will only run with jdk17, will we drop UT+jdk8 support in the future?
There was a problem hiding this comment.
seems Spark 344 UT will only run with jdk17, will we drop UT+jdk8 support in the future?
Hi @Yohahaha,
In general we are trying to bump the JDK support in 1.4 release, due to many dependent libs dropped JDK8 support.
- The main reason for this bump is due to iceberg dropped JDK8 support starts from 1.7.0(https://lists.apache.org/thread/xd4sjd0cx010qcv9qmlt89zdx4s1cyp4). JDK8 should be working if
-Picebergis not activated - based on my local tests, JDK11 is also working, the current latest LTS JDK17 seems more preferred based on what I collected so far
- If JDK8 requirements are strong in the community, we could add back the JDK8 based unit tests CI(without
-Piceberg)
Cc: @weiting-chen
Thanks, -yuan
There was a problem hiding this comment.
thank you for the explanation! @zhouyuan
based on my local tests, JDK11 is also working, the current latest LTS JDK17 seems more preferred based on what I collected so far
yes, spark 4.0 requires JDK17+
Support Spark-344 + JDK17
4 participants
What changes were proposed in this pull request?
Support Spark-344 + JDK17
How was this patch tested?
pass GHA tests