Prepare for Java 21

[gortiz]

Java 21 is already here and it includes tons of performance updates we can get for free by just changing the JVM at runtime. Remember that issues that prevent Pinot to run with Java > 15 were solved before summer (see #10528). The objective of this issue is to discuss about the changes we want to make in order to support Java 21 in Apache Pinot.

To be clear, Apache Pinot should work with Java 21 as it is right now, but we don't run any test in Java 21 and we don't provide any docker image that uses Java 21.

Current state

The current state of Apache Pinot in relation to Java version is:

• Most of the source code must be compilable with Java 11. There are some Maven modules that must be compilable with Java 8 (mainly clients and their dependencies)
• We provide docker images running with Java 11 and 17.
• Our pipelines compile the source code using Java 11, 17 and 20.
• Our pipelines run tests with Java 11, 17 and 20.

Java 21

Being close to their release, we tried to add Java 21 to Apache Pinot pipelines during the changes applied early June, but we found out that there were some Maven plugins that didn't support that. Therefore we decided to test with Java 20 instead. It would be nice to be able to at least run the tests with Java 21, even if the code is compiled with an older version of Java.

If you try to compile the project with Java 21 as it is today, you will find issues with, at least:

• Spotless: Which fails in Java 21 even using -Dspotless.skip. It seems they are working on supporting it in update palantirJavaFormat for Java 21 diffplug/spotless#1822.
  • Fixed by disabling Spotless when compiling with Java 21 (see Do not execute spotless in Java 21 #11670)
• Scala: Scala 2.12.18 should be compatible with Java 21 (according to their official jdk compatibility matrix)
  • Fixed by updating scala to 2.12.18 (see Move scala dependencies to root pom #11671)
• Spark 3.2 does not support Java 21. Options:
  • Do not include Spark 3.2 in Java 21
  • Use Spark 3.5. This requires changes in the code because it uses slf4j 2. We are using 1.7. In theory slf4j could be upgraded without code changes (see https://www.slf4j.org/faq.html#changesInVersion200)
  • See Upgrade to Spark 3.5.0 #11701

It is also important to know that although Java 21 was released in Sept 19th, as today the only Java distribution we use that provide docker images for Java 21 is Amazon Corretto. The other two (Temurin and Microsoft) do not.

Plan

At short term

Given that it seems we cannot run all our pipeline with Java 21, we need to:

• Register all issues we have using Java 21 in our current pipeline.
• (Optional) Compile with Java 11 and test with Java 21.

At long term

• We should remove Java 20 from our pipelines and replace it with Java 21. All maven phases should be supported with Java 21.
• Optional: We should also run all tests using UnsafePinotBufferFactory.

[gortiz]

All issues have been fixed in #11672, including the optional ones.

Right now that PR multiplies the executions by two because they run all tests with and without the alternative buffer library. One execution always use the alternative library and the other uses bytebuffers in case the requested buffer is smaller than 2GBs.

In the current code, there are some tests that fail in scenarios where Java 11 is used and bytebuffers are skipped. This is the only scenario where LArray is used. That means that LArray doesn't behaves as expected. We should fix that, but it is not related to Java 21 migration. The good news is that the Unsafe buffer library used in Java 17 and 21 pass all the tests.

I consider this is ready to review. We should decide what to do with Spark (see #11701) and also which combinations do we want to test in our pipelines.

[vmarchaud]

I couldn't find any comment on this on the different PR but does the pinot team advise upgrading to java 21 or stay on 11 for now ? thanks

[gortiz]

There is no official statement related to the upgrade but the main docker image is still using Java 11. In StarTree we have been trying Java 21 internally and the performance gains are considerable, so we are planning to use Java 21 as our default installation in our next release.

[vmarchaud]

@gortiz Thanks for your feedback on this, i will try to upgrade it on my env as well then