Upgrade to Spark 3.5.0

[gortiz]

This issue is created to analyze whether we can upgrade our Spark 3.x dependencies to 3.5.0.

Apache Pinot connects to Apache Spark in two different types of modules:

• batch ingestion plugins (modules pinot-batch-ingestion-spark-3.2 and pinot-batch-ingestion-spark-2.4), used to ingest data from Spark.
• connectors (modules pinot-spark-3-connector and pinot-spark-2-connector), used to use Spark to query Pinot.

Note that we need to support these two major Spark versions for compatibility reasons.

As explained in #11656, one of the reasons we cannot officially support Java 21 is that Spark 3.x do not support Java 21 until version 3.5. It seems there is no issues with Spark 2.x versions.

It is not clear to me exactly which Spark version we are using. As expected, pinot-batch-ingestion-spark-3.2 uses spark 3.2.x (specifically 3.2.1) while pinot-spark-3-connector uses 3.4.0. It also seems that Pinot distribution does not includes the connectors (which makes sense), and the final jars will contain the 3.2.1 version.

Spark versioning semantics are defined here. Notice that they do not follow semantic versioning. Specifically:

An API is any public class or interface exposed in Spark that is not marked as “developer API” or “experimental”. Release A is API compatible with release B if code compiled against release A compiles cleanly against B. Currently, does not guarantee that a compiled application that is linked against version A will link cleanly against version B without re-compiling. Link-level compatibility is something we’ll try to guarantee in future releases.

There is also a document where they specify the changes from version to version. We would need to understand the changes from 3.2 to 3.3, 3.3 to 3.4 and finally 3.4 to 3.5.

I've done some checks, modifying the dependency to 3.5.0 and:

• Tests run correctly
• We also need to change our dependency from slf4j from 1.7.x to 2.x.x. This should be a backward compatible change (see https://www.slf4j.org/faq.html#changesInVersion200)

I've opened #11702 to test a blind upgrade from current 3.2.4 to 3.5.0

[gortiz]

I've modified #11702 as ready to review.

I was able to run all our tests without issues with Spark 3.5 instead of 3.2, but I didn't tried my branch in an actual scenario where spark is being used. Reading Spark documentation, it seems that the changes are backward compatible.

It would be nice to have a review from contributors with experience in Spark.

[gortiz]

We can consider this done

[jadami10]

Mostly curious because don't actually use the connector. But why don't we split the connector modules into spark3.x versions?

[gortiz]

You mean having one 3.2 and one 3.5 project? We don't usually do that. I mean, imagine we needed to create one single plugin for each technology and each version. It would be very difficult to maintain!

For some wildly used technologies like Kafka and Spark we maintain several backward incompatible projects. For example we have one plugin for kafka 0.9 and another for kafka 2.0. We also had a spark 2.4 and 3.2, so users that use the incompatible 2.x and 3.x versions can actually deploy the plugin they need.

But Spark 3.5 should be backward compatible with 3.2, so we don't see a reason to have to maintain 3 different spark plugins (2.4, 3.2 and 3.5). Therefore we decided to rename 3.2 as 3.x and upgrade spark client version from 3.2 to 3.5.

[jadami10]

But Spark 3.5 should be backward compatible with 3.2

ah my bad. I should have read the spark docs first. It does seem they expect all the same major versions to be compatible.

[indreshverma]

While upgrading Spark 3.3.0 to 3.5.0 facing errors. { using java 17 , scala 2.13} for building spark 3.5.0
step to produce error..

1. clone spark 3.5.0 from apache spark github
2. Go to spark directory and run the command. ----> ./build/mvn -DskipTests clean package

Tried with diff versions of scala also same issue.

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.8.0:compile (scala-compile-first) on project spark-core_2.13: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.8.0:compile failed: org.apache.commons.exec.ExecuteException.

Please help me on this, i'm stuck here, did research , did googling all but could not solve it .

[gortiz]

While upgrading Spark 3.3.0 to 3.5.0 facing errors. { using java 17 , scala 2.13} for building spark 3.5.0
step to produce error..

clone spark 3.5.0 from apache spark github
Go to spark directory and run the command. ----> ./build/mvn -DskipTests clean package

I don't think we compile Spark our self. You may need to go to the Spark github repo to look for support compiling it.