[SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compile Spark REPL for Scala 2.12 + other 2.12 fixes #19307

srowen · 2017-09-21T11:00:09Z

What changes were proposed in this pull request?

Enable Scala 2.12 REPL. Fix most remaining issues with 2.12 compilation and warnings, including:

Selecting Kafka 0.10.1+ for Scala 2.12 and patching over a minor API difference
Fixing lots of "eta expansion of zero arg method deprecated" warnings
Resolving the SparkContext.sequenceFile implicits compile problem
Fixing an odd but valid jetty-server missing dependency in hive-thriftserver

How was this patch tested?

Existing tests

srowen · 2017-09-21T11:01:15Z

core/src/main/scala/org/apache/spark/SparkContext.scala

@@ -2826,33 +2826,33 @@ object WritableConverter {
  // them automatically. However, we still keep the old functions in SparkContext for backward
  // compatibility and forward to the following functions directly.

-  implicit def intWritableConverter(): WritableConverter[Int] =
-    simpleWritableConverter[Int, IntWritable](_.get)
+  implicit val intWritableConverter: () => WritableConverter[Int] =


These changes were necessary to make the implicits work in 2.12 now that eta-expansion of zero-arg methods is deprecated and apparently doesn't work for implicit resolution. It passes MiMa, but we could be conservative and retain the existing methods, and add function vals instead.

srowen · 2017-09-21T11:01:29Z

core/src/main/scala/org/apache/spark/SparkContext.scala


-  implicit def writableWritableConverter[T <: Writable](): WritableConverter[T] =
-    new WritableConverter[T](_.runtimeClass.asInstanceOf[Class[T]], _.asInstanceOf[T])
+  implicit def writableWritableConverter[T <: Writable : ClassTag]: () => WritableConverter[T] =


ClassTag was required here, for reasons I don't fully get

srowen · 2017-09-21T11:02:02Z

core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala

@@ -218,11 +218,13 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
    if (!conf.contains("spark.testing")) {
      // A task that periodically checks for event log updates on disk.
      logDebug(s"Scheduling update thread every $UPDATE_INTERVAL_S seconds")
-      pool.scheduleWithFixedDelay(getRunner(checkForLogs), 0, UPDATE_INTERVAL_S, TimeUnit.SECONDS)
+      pool.scheduleWithFixedDelay(
+        getRunner(() => checkForLogs()), 0, UPDATE_INTERVAL_S, TimeUnit.SECONDS)


These changes avoid warnings about eta-expansion of zero-arg methods. It works fine in 2.11 as well; just not relying on syntactic sugar for the same.

srowen · 2017-09-21T11:02:41Z

core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala

-        case e: Throwable =>
-          logError("App dir cleanup failed: " + e.getMessage, e)
-      }(cleanupThreadExecutor)
+      cleanupFuture.failed.foreach(e =>


onFailure, onSuccess are deprecated in 2.12. This should be equivalent in 2.11+2.12

srowen · 2017-09-21T11:03:18Z

core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala

@@ -396,12 +396,12 @@ class DAGScheduler(

  /** Find ancestor shuffle dependencies that are not registered in shuffleToMapStage yet */
  private def getMissingAncestorShuffleDependencies(
-      rdd: RDD[_]): Stack[ShuffleDependency[_, _, _]] = {
-    val ancestors = new Stack[ShuffleDependency[_, _, _]]
+      rdd: RDD[_]): ArrayStack[ShuffleDependency[_, _, _]] = {


Stack is deprecated in 2.12 for poor performance; ArrayStack should work the same and be faster, in 2.11 too

srowen · 2017-09-21T11:06:04Z

external/kafka-0-10-sql/pom.xml

+    <profile>
+      <id>scala-2.12</id>
+      <properties>
+        <kafka.version>0.10.1.1</kafka.version>


Only 0.10.1+ supports Scala 2.12. By the time a 2.12 build is actually supported we may be on to 0.10.2. Not sure. This at least makes it work

srowen · 2017-09-21T11:06:57Z

repl/scala-2.12/src/main/scala/org/apache/spark/repl/SparkILoop.scala

+   * sees any files, so that the Spark context is visible in those files. This is a bit of a
+   * hack, but there isn't another hook available to us at this point.
+   */
+  override def createInterpreter(): Unit = {


This is the only meaningful difference from the 2.11 REPL, as it has to hook into a different place. All other REPL-related code isn't specific to 2.11/2.12 and was moved out into the common src directory in the repl module

srowen · 2017-09-21T11:07:37Z

sql/core/src/test/scala/org/apache/spark/sql/DataFrameStatSuite.scala

@@ -230,19 +230,16 @@ class DataFrameStatSuite extends QueryTest with SharedSQLContext {

    val resNaN1 = dfNaN.stat.approxQuantile("input1", Array(q1, q2), epsilon)
    assert(resNaN1.count(_.isNaN) === 0)
-    assert(resNaN1.count(_ == null) === 0)


resNaN1 is an Array[Double] so can never contain null. This is always true and generated a warning

srowen · 2017-09-21T11:07:57Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/EnsureStatefulOpPartitioningSuite.scala

@@ -20,6 +20,7 @@ package org.apache.spark.sql.streaming
 import java.util.UUID

 import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.DataFrame


This whole file fix is already going into master separately as a hotfix

srowen · 2017-09-21T11:10:15Z

sql/hive-thriftserver/pom.xml

@@ -63,6 +63,16 @@
      <groupId>${hive.group}</groupId>
      <artifactId>hive-beeline</artifactId>
    </dependency>
+    <dependency>


For some reason, hive-thriftserver wouldn't compile, unable to find jetty.server classes. It has a point, these should explicit, even if it somehow worked before.

SparkQA · 2017-09-21T13:17:34Z

Test build #82033 has finished for PR 19307 at commit 99f082a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-09-21T14:07:13Z

retest this please

SparkQA · 2017-09-21T17:24:49Z

Test build #82036 has finished for PR 19307 at commit 99f082a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2017-09-21T17:34:58Z

Good to see this passes, as I expected, because it means we can also commit this chunk of progress towards 2.12 without any downside for 2.11.

Right now this means that the 2.12 build does compile, but I know it doesn't quite pass all tests (some weird paranamer problem, to start with)

The only part I was slightly uncertain about was the change to sequenceFile implicits, but MiMa hasn't complained.

Worth a look at the changes, but I think this is worth committing before too long.

…ompatibility

SparkQA · 2017-09-22T13:43:38Z

Test build #82076 has finished for PR 19307 at commit 05e4891.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

Up to my little knowledge, it looks good to me in general. What I checked was, to double check the comments left here and the same instances.

srowen · 2017-09-24T08:41:12Z

Merged to master

srowen commented Sep 21, 2017

View reviewed changes

srowen added 2 commits September 22, 2017 11:23

Initial Scala 2.12 deprecation fixes and compilation fixes

9a73e7a

Add back previous SparkContext sequenceFile implicits for backwards c…

05e4891

…ompatibility

srowen force-pushed the Scala212 branch from 99f082a to 05e4891 Compare September 22, 2017 10:32

HyukjinKwon reviewed Sep 22, 2017

View reviewed changes

asfgit closed this in 576c43f Sep 24, 2017

srowen deleted the Scala212 branch September 24, 2017 16:35

srowen mentioned this pull request Oct 30, 2017

[SPARK-14650][REPL][BUILD] Compile Spark REPL for Scala 2.12 #19612

Closed

[SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compile Spark REPL for Scala 2.12 + other 2.12 fixes #19307

[SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compile Spark REPL for Scala 2.12 + other 2.12 fixes #19307

Uh oh!

Conversation

srowen commented Sep 21, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 21, 2017

Uh oh!

HyukjinKwon commented Sep 21, 2017

Uh oh!

SparkQA commented Sep 21, 2017

Uh oh!

srowen commented Sep 21, 2017

Uh oh!

SparkQA commented Sep 22, 2017

Uh oh!

HyukjinKwon left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srowen commented Sep 24, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HyukjinKwon left a comment •

edited

Loading