-
Notifications
You must be signed in to change notification settings - Fork 3k
Closed
Description
It has been proposed to build Spark 3 support -- at least for Spark 3.2 -- against both Scala 2.12 and 2.13.
The code in spark/v3.2 currently does not compile against Scala 2.13.
Some of the sources of source incompatibility are:
scala.Predef.conforms(deprecated since 2.11) has been removed in 2.13.scala.Seq[+A]in 2.13 is an alias forscala.collection.immutable.Seq[A]instead ofscala.collection.Seq[A]. Some Spark APIs that Iceberg call takescala.Seqparameters, instead of parameters explicitly declared asscala.collection.Seq. (The problem is that the APIs are called from Java code, not Scala code. Java lacks the type alias, so the Java code currently usescala.collection.Seq, but that cannot be used in 2.13 where ascala.Seqis required.) Another manifestation of this issue isscala.collection.mutable.Buffercannot be used where ascala.Seqis required.scala.collection.JavaConversionshas been removed in 2.13 (it is used in some tests).
Incidentally, scala.collection.JavaConverters is still in 2.13 but is deprecated and use of scala.jdk.CollectionConverters is recommended instead (but that is only in 2.13, not in 2.12).
Metadata
Metadata
Assignees
Labels
No labels