[WIP][SPARK-31168][BUILD][test-maven] Upgrade Scala to 2.12.11 by wangyum · Pull Request #27929 · apache/spark

wangyum · 2020-03-16T20:10:13Z

What changes were proposed in this pull request?

This PR(SPARK-31168) upgrade Scala to 2.12.11.

Scala 2.12.11 release notes:
Highlights

Performance improvements in the collections library
Performance improvements in the compiler
Improvements to -Yrepl-class-based, an alternative internal REPL encoding that avoids deadlocks
A new -Yrepl-use-magic-imports flag that avoids deep class nesting in the REPL, which can lead to deteriorating performance in long sessions
Fix some toX methods that could expose the underlying mutability of a ListBuffer-generated collection

JDK 9+ support

ASM was upgraded to 7.3.1, allowing the optimizer to run on JDK 13+
:javap in the REPL now works on JDK 9+

Why are the changes needed?

Improvement performance.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Exist test.

SparkQA · 2020-03-16T20:23:37Z

Test build #119892 has finished for PR 27929 at commit 7811091.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

srowen

I think the important thing will be to make sure the items tested at #27681 work

srowen · 2020-03-16T20:37:30Z

 SPARK_VERSION_SHORT: 3.1.0
 SCALA_BINARY_VERSION: "2.12"
-SCALA_VERSION: "2.12.10"
+SCALA_VERSION: "2.12.1"


dongjoon-hyun · 2020-03-17T00:58:21Z

Retest this please.

dongjoon-hyun · 2020-03-17T01:03:28Z

It seems that we need to make a PR to genjavadoc.

https://github.com/lightbend/genjavadoc/blob/master/build.sbt#L51

SparkQA · 2020-03-17T07:05:01Z

Test build #119899 has finished for PR 27929 at commit 7811091.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-17T07:05:02Z

Test build #119900 has finished for PR 27929 at commit fa21f57.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-18T21:15:30Z

Test build #119994 has finished for PR 27929 at commit 678e5d6.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-03-18T22:21:58Z

retest this please

SparkQA · 2020-03-19T05:11:53Z

Test build #120010 has finished for PR 27929 at commit 678e5d6.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-03-19T08:26:51Z

retest this please

SparkQA · 2020-03-19T15:18:38Z

Test build #120042 has finished for PR 27929 at commit 678e5d6.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-03-19T15:28:47Z

retest this please

maropu · 2020-03-19T15:50:09Z

(Does this upgrade mak the testing time much longer..?)

SparkQA · 2020-03-19T22:15:45Z

Test build #120064 has finished for PR 27929 at commit 678e5d6.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-03-19T22:38:51Z

Retest this please.

SparkQA · 2020-03-20T05:26:58Z

Test build #120073 has finished for PR 27929 at commit 678e5d6.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-20T14:34:34Z

Test build #120084 has finished for PR 27929 at commit f1d8e38.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2020-03-20T15:09:03Z

Retest this please.

SparkQA · 2020-03-20T22:22:26Z

Test build #120105 has finished for PR 27929 at commit f1d8e38.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-03-20T22:59:34Z

retest this please

SparkQA · 2020-03-21T05:36:18Z

Test build #120118 has finished for PR 27929 at commit f1d8e38.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2020-03-21T11:56:46Z

It seems 2.12.11 breaks generate Java doc: scala/bug#11917

wangyum · 2020-03-27T11:14:31Z

Close it due to this issue: scala/bug#11917

dongjoon-hyun · 2020-03-27T16:11:21Z

Thank you, @wangyum .
I updated SPARK-31168 from Upgrade Scala to 2.12.11 to Upgrade Scala to 2.12.12.

…to 2.7.0 ### What changes were proposed in this pull request? This PR is the 3rd try to upgrade Scala 2.12.x in order to see the feasibility. - #27929 (Upgrade Scala to 2.12.11, wangyum ) - #30940 (Upgrade Scala to 2.12.12, viirya ) `silencer` library is updated accordingly. And, Kafka version upgrade is required because it fails like the following. ``` [info] KafkaDataConsumerSuite: [info] org.apache.spark.streaming.kafka010.KafkaDataConsumerSuite *** ABORTED *** (1 second, 580 milliseconds) [info] java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7 [info] at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45) ``` ### Why are the changes needed? Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11 and 2.12.12. This will bring all the bug fixes. - https://github.com/scala/scala/releases/tag/v2.12.13 - https://github.com/scala/scala/releases/tag/v2.12.12 - https://github.com/scala/scala/releases/tag/v2.12.11 ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug-fixed version. ### How was this patch tested? Pass the CIs. Closes #31223 from dongjoon-hyun/SPARK-31168. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…to 2.7.0 ### What changes were proposed in this pull request? This PR is the 3rd try to upgrade Scala 2.12.x in order to see the feasibility. - apache#27929 (Upgrade Scala to 2.12.11, wangyum ) - apache#30940 (Upgrade Scala to 2.12.12, viirya ) `silencer` library is updated accordingly. And, Kafka version upgrade is required because it fails like the following. ``` [info] KafkaDataConsumerSuite: [info] org.apache.spark.streaming.kafka010.KafkaDataConsumerSuite *** ABORTED *** (1 second, 580 milliseconds) [info] java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7 [info] at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45) ``` ### Why are the changes needed? Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11 and 2.12.12. This will bring all the bug fixes. - https://github.com/scala/scala/releases/tag/v2.12.13 - https://github.com/scala/scala/releases/tag/v2.12.12 - https://github.com/scala/scala/releases/tag/v2.12.11 ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug-fixed version. ### How was this patch tested? Pass the CIs. Closes apache#31223 from dongjoon-hyun/SPARK-31168. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility. - #27929 (Upgrade Scala to 2.12.11, wangyum ) - #30940 (Upgrade Scala to 2.12.12, viirya ) - #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun ) Note that Scala 2.12.14 has the following fix for Apache Spark community. - Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13 REQUIREMENTS: - [x] `silencer` library is released via ghik/silencer#66 - [x] `genjavadoc` library is released via lightbend/genjavadoc#282 ### Why are the changes needed? Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes. - https://github.com/scala/scala/releases/tag/v2.12.14 - https://github.com/scala/scala/releases/tag/v2.12.13 - https://github.com/scala/scala/releases/tag/v2.12.12 - https://github.com/scala/scala/releases/tag/v2.12.11 ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug-fixed version. ### How was this patch tested? Pass the CIs. Closes #32697 from dongjoon-hyun/SPARK-31168. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

* [SPARK-31168][BUILD] Upgrade Scala to 2.12.14 ### What changes were proposed in this pull request? This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility. - #27929 (Upgrade Scala to 2.12.11, wangyum ) - #30940 (Upgrade Scala to 2.12.12, viirya ) - #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun ) Note that Scala 2.12.14 has the following fix for Apache Spark community. - Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13 REQUIREMENTS: - [x] `silencer` library is released via ghik/silencer#66 - [x] `genjavadoc` library is released via lightbend/genjavadoc#282 ### Why are the changes needed? Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes. - https://github.com/scala/scala/releases/tag/v2.12.14 - https://github.com/scala/scala/releases/tag/v2.12.13 - https://github.com/scala/scala/releases/tag/v2.12.12 - https://github.com/scala/scala/releases/tag/v2.12.11 ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug-fixed version. ### How was this patch tested? Pass the CIs. Closes #32697 from dongjoon-hyun/SPARK-31168. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 6c4b60f) * [SPARK-31168][BUILD] Upgrade Scala to 2.12.14 ### What changes were proposed in this pull request? This PR is the 4th try to upgrade Scala 2.12.x in order to see the feasibility. - #27929 (Upgrade Scala to 2.12.11, wangyum ) - #30940 (Upgrade Scala to 2.12.12, viirya ) - #31223 (Upgrade Scala to 2.12.13, dongjoon-hyun ) Note that Scala 2.12.14 has the following fix for Apache Spark community. - Fix cyclic error in runtime reflection (protobuf), a regression that prevented Spark upgrading to 2.12.13 REQUIREMENTS: - [x] `silencer` library is released via ghik/silencer#66 - [x] `genjavadoc` library is released via lightbend/genjavadoc#282 ### Why are the changes needed? Apache Spark was stuck to 2.12.10 due to the regression in Scala 2.12.11/2.12.12/2.12.13. This will bring all the bug fixes. - https://github.com/scala/scala/releases/tag/v2.12.14 - https://github.com/scala/scala/releases/tag/v2.12.13 - https://github.com/scala/scala/releases/tag/v2.12.12 - https://github.com/scala/scala/releases/tag/v2.12.11 ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug-fixed version. ### How was this patch tested? Pass the CIs. Closes #32697 from dongjoon-hyun/SPARK-31168. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 6c4b60f) * [SPARK-36759][BUILD] Upgrade Scala to 2.12.15 ### What changes were proposed in this pull request? This PR aims to upgrade Scala to 2.12.15 to support Java 17/18 better. ### Why are the changes needed? Scala 2.12.15 improves compatibility with JDK 17 and 18: https://github.com/scala/scala/releases/tag/v2.12.15 - Avoids IllegalArgumentException in JDK 17+ for lambda deserialization - Upgrades to ASM 9.2, for JDK 18 support in optimizer ### Does this PR introduce _any_ user-facing change? Yes, this is a Scala version change. ### How was this patch tested? Pass the CIs Closes #33999 from dongjoon-hyun/SPARK-36759. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 16f1f71) * [SPARK-36759][BUILD][FOLLOWUP] Update version in scala-2.12 profile and doc ### What changes were proposed in this pull request? This is a follow-up to fix the leftover during switching the Scala version. ### Why are the changes needed? This should be consistent. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This is not tested by UT. We need to check manually. There is no more `2.12.14`. ``` $ git grep 2.12.14 R/pkg/tests/fulltests/test_sparkSQL.R: c(as.Date("2012-12-14"), as.Date("2013-12-15"), as.Date("2014-12-16"))) data/mllib/ridge-data/lpsa.data:3.5307626,0.987291634724086 -0.36279314978779 -0.922212414640967 0.232904453212813 -0.522940888712441 1.79270085261407 0.342627053981254 1.26288870310799 sql/hive/src/test/resources/data/files/over10k:-3|454|65705|4294967468|62.12|14.32|true|mike white|2013-03-01 09:11:58.703087|40.18|joggying ``` Closes #34020 from dongjoon-hyun/SPARK-36759-2. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit adbea25) * [SPARK-39414][BUILD] Upgrade Scala to 2.12.16 ### What changes were proposed in this pull request? This PR aims to upgrade Scala to 2.12.16 ### Why are the changes needed? This version bring some bug fix and start to try to support Java 19 scala/scala@v2.12.15...v2.12.16 - [Upgrade to asm 9.3, for JDK19 support](scala/scala#10000) - [Fix codegen for MH.invoke etc under JDK 17 -release](scala/scala#9930) - [Deprecation related SecurityManager on JDK 17 ](scala/scala#9775) ### Does this PR introduce _any_ user-facing change? Yes, this is a Scala version change. ### How was this patch tested? Pass Github Actions Closes #36807 from LuciferYang/SPARK-39414. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit ed875a8) * fix * fix * fix * fix �[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/core/src/main/scala/org/apache/spark/scheduler/SpillableTaskResultGetter.scala:36: non-variable type argument org.apache.spark.scheduler.DirectTaskResult[_] in type pattern scala.runtime.NonLocalReturnControl[org.apache.spark.scheduler.DirectTaskResult[_]] is unchecked since it is eliminated by erasure�[0m �[0m[�[31merror�[0m] �[0m[warn] private[spark] class SpillableTaskResultGetter(sparkEnv: SparkEnv, scheduler: TaskSchedulerImpl)�[0m �[0m[�[31merror�[0m] �[0m[warn] �[0m * fix �[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:287: match may not be exhaustive.�[0m �[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "^")), _)�[0m �[0m[�[31merror�[0m] �[0m[warn] private val pow: Parser[Term] = term ~ "^" ~ "^[1-9]\\d*".r ^^ {�[0m �[0m[�[31merror�[0m] �[0m[warn] �[0m �[0m[�[31merror�[0m] �[0m[warn] /home/jenkins/workspace/spark-sql-catalyst-3.0/mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:301: match may not be exhaustive.�[0m �[0m[�[31merror�[0m] �[0mIt would fail on the following input: ~(~(_, (x: String forSome x not in "~")), _)�[0m �[0m[�[31merror�[0m] �[0m[warn] (label ~ "~" ~ expr) ^^ { case r ~ "~" ~ t => ParsedRFormula(r, t.asTerms.terms) }�[0m �[0m[�[31merror�[0m] �[0m[warn] �[0m * fix Co-authored-by: Dongjoon Hyun <dhyun@apple.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by: yangjie01 <yangjie01@baidu.com>

Upgrade Scala to 2.12.11

7811091

srowen reviewed Mar 16, 2020

View reviewed changes

dongjoon-hyun added the BUILD label Mar 16, 2020

dongjoon-hyun changed the title ~~[WIP][SPARK-31168][BUILD] Upgrade Scala to 2.12.11~~ [WIP][SPARK-31168][BUILD][test-maven] Upgrade Scala to 2.12.11 Mar 17, 2020

Fix typo

fa21f57

git commit --allow-empty -m "Trigger GithubAction"

678e5d6

fail-fast: false && tests_timeout = "600m"

f1d8e38

wangyum closed this Mar 27, 2020

wangyum deleted the SPARK-31168 branch March 28, 2020 01:49

dongjoon-hyun mentioned this pull request Jan 18, 2021

[SPARK-31168][SPARK-33913][BUILD] Upgrade Scala to 2.12.13 and Kafka to 2.7.0 #31223

Closed

dongjoon-hyun mentioned this pull request May 28, 2021

[SPARK-31168][BUILD] Upgrade Scala to 2.12.14 #32697

Closed

2 tasks

Conversation

wangyum commented Mar 16, 2020 • edited by dongjoon-hyun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Mar 16, 2020

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

srowen Mar 16, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Mar 17, 2020

Uh oh!

dongjoon-hyun commented Mar 17, 2020

Uh oh!

SparkQA commented Mar 17, 2020

Uh oh!

SparkQA commented Mar 17, 2020

Uh oh!

SparkQA commented Mar 18, 2020

Uh oh!

maropu commented Mar 18, 2020

Uh oh!

SparkQA commented Mar 19, 2020

Uh oh!

HyukjinKwon commented Mar 19, 2020

Uh oh!

SparkQA commented Mar 19, 2020

Uh oh!

HyukjinKwon commented Mar 19, 2020

Uh oh!

maropu commented Mar 19, 2020

Uh oh!

SparkQA commented Mar 19, 2020

Uh oh!

dongjoon-hyun commented Mar 19, 2020

Uh oh!

SparkQA commented Mar 20, 2020

Uh oh!

SparkQA commented Mar 20, 2020

Uh oh!

wangyum commented Mar 20, 2020

Uh oh!

SparkQA commented Mar 20, 2020

Uh oh!

maropu commented Mar 20, 2020

Uh oh!

SparkQA commented Mar 21, 2020

Uh oh!

wangyum commented Mar 21, 2020

Uh oh!

wangyum commented Mar 27, 2020

Uh oh!

dongjoon-hyun commented Mar 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wangyum commented Mar 16, 2020 •

edited by dongjoon-hyun

Loading