[Java] Remove Java 8 support in Arrow v18

[danepitkin]

Describe the enhancement requested

1. Java 8 is holding back development of newer Java features. For example, the Java Platform Module System (JPMS)[1], which was introduced in Java 9.
2. Java 8 is preventing Arrow from using latest packages/dependencies in some places. See examples[2][3][4].
3. Arrow Java is quite stable, so Java 8 users can probably be fine pinning the Arrow dependency if they aren't interested in upgrading Java versions.
4. Java 8 is on the decline, and is not the most used Java version in 2023[5].

[1]https://en.wikipedia.org/wiki/Java_Platform_Module_System
[2]https://github.com/apache/arrow/blob/main/dev/release/verify-release-candidate.sh#L571
[3]#37723 (comment)
[4]#13072 (comment)
[5]https://newrelic.com/sites/default/files/2023-04/new-relic-2023-state-of-the-java-ecosystem-2023-04-20.pdf

Post-upgrade tasks

• Bump ErrorProne (MINOR: [Java] Bump com.google.errorprone:error_prone_core from 2.4.0 to 2.24.0 in /java #39409)
• Bump Mockito (MINOR: [Java] Bump org.mockito:mockito-core from 4.11.0 to 5.8.0 in /java #39410)
• Bump mockito-junit-jupiter (MINOR: [Java] Bump org.mockito:mockito-junit-jupiter from 2.25.1 to 5.12.0 in /java #43308)
• Bump Derby (MINOR: [Java] Bump org.apache.derby:derby from 10.14.2.0 to 10.15.2.0 in /java #43309)
• Bump Checkstyle (MINOR: [Java] Bump com.puppycrawl.tools:checkstyle from 8.29 to 10.17.0 in /java #43311)
• Bump logback (MINOR: [Java] Bump ch.qos.logback:logback-classic from 1.3.14 to 1.4.14 in /java #43310)
• Remove the maven plugin (MINOR: [Java] Bump maven.version from 3.8.7 to 3.9.7 in /java #42074)

Component(s)

Java

[kou]

The discussion thread on dev@arrow.apache.org: https://lists.apache.org/thread/s07jx58yw4mkl54t3bkggnyg0sftcrr8

[davisusanibar]

In addition, the following dependencies are pinned for JDK8:

• In order to enable support for JDK21, this draft PR will force us to pin Mockito dependency version needed for JDK8 and another version for JDK11+.

• Error prone 2.11+ requires JDK11 minimum supported version, thus it was necessary to pin JDK8 compatibility.

[danepitkin]

Apache Spark has dropped support for Java 8 and 11 on the main branch (targeting a 4.0 release) apache/spark#43005

Edit: Spark 4.0 release timeframe is 2024-06[1]

[1]https://lists.apache.org/thread/xhkgj60j361gdpywoxxz7qspp2w80ry6

[danepitkin]

Netty 5.0 will remove support for Java 8 netty/netty#10650

[danepitkin]

The current consensus on the Arrow mailing list[1] is to postpone Java 8 deprecation and to revisit it when Spark releases 4.0, which deprecates Java 8 (~2024-06).

[1] https://lists.apache.org/thread/kml53f81z1oskcf00xl7wlbcjssmn91g

[danepitkin]

Apache Derby continuously drops support for older JDK versions #38813