ARROW-16327: [Java][CI] Add Java 17 to CI matrix for java workflows #13021

raulcd · 2022-04-28T07:07:20Z

This PR aims to add support for java 17 on CI as required on #12941
We probably should cherrypick the commit here on that PR.

github-actions · 2022-04-28T07:07:42Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

raulcd · 2022-04-28T07:10:30Z

@github-actions crossbow submit java

github-actions · 2022-04-28T07:11:49Z

Revision: fe4354fcff315f59ef52a07b5107d1adcdb59877

Submitted crossbow builds: ursacomputing/crossbow @ actions-1984

Task	Status
java-jars
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-18.04-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

github-actions · 2022-04-28T07:21:23Z

https://issues.apache.org/jira/browse/ARROW-16327

lidavidm

Thanks for setting this up!

kou · 2022-04-28T20:27:07Z

It seems that maven:3.8.5-openjdk-17 is based on Oracle Linux 8 not Debian:

$ docker run --rm maven:3.8.5-openjdk-17 cat /etc/system-release
Oracle Linux Server release 8.5

So existing ci/docker/debian-9-java.dockerfile filename isn't suitable for OpenJDK 17.

How about adding ci/docker/oracle-8-java.dockerfile, adding oracle-java to docker-compose.yml and removing jdk_name from .env?

BTW, it seems that nobody uses ARROW_JAVA_SHADE_FLATBUFS defined in ci/docker/debian-9-java.dockerfile. See also: 7bc2b0f

We may be able to use maven:* images directly instead of defining our (thin) wrapper image. For example, we use existing images directly for python-wheel-manylinux-test-imports.

kou · 2022-04-28T20:30:29Z

Could you also confirm CI failure? https://github.com/apache/arrow/runs/6218109762?check_suite_focus=true

Error:  Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.2:compile (default-compile) on project arrow-format: Compilation failure -> [Help 1]
Error:  Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.2:compile (default-compile) on project arrow-plasma: Compilation failure -> [Help 1]
Error:  Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.2:compile (default-compile) on project arrow-memory-core: Compilation failure -> [Help 1]

The following log may be related:

[INFO] -----------------< org.apache.arrow:arrow-memory-core >-----------------
[INFO] Building Arrow Memory - Core 8.0.0-SNAPSHOT                       [6/19]
[INFO] --------------------------------[ jar ]---------------------------------
Warning:  *****************************************************************
Warning:  * Your build is requesting parallel execution, but project      *
Warning:  * contains the following plugin(s) that have goals not marked   *
Warning:  * as @threadSafe to support parallel building.                  *
Warning:  * While this /may/ work fine, please look for plugin updates    *
Warning:  * and/or request plugins be made thread-safe.                   *
Warning:  * If reporting an issue, report it against the plugin in        *
Warning:  * question, not against maven-core                              *
Warning:  *****************************************************************
Warning:  The following plugins are not marked @threadSafe in Arrow Memory - Core:
Warning:  org.apache.rat:apache-rat-plugin:0.13
Warning:  Enable debug to see more precisely which goals are not marked @threadSafe.
Warning:  *****************************************************************

raulcd · 2022-05-02T10:31:31Z

Thanks for the review @kou ! I have added a new oracle-java entry on the docker-compose.yml and I have updated the debian-java to use the maven:* images directly removing the wrapper image ci/docker/debian-9-java.dockerfile, as you said I can't find anywhere where the ARROW_JAVA_SHADE_FLATBUFS were being used. I have also removed the JDK_NAME and have added a JDK 8 CI test (as it seems to be the default defined on our .env).

As per the failure on JDK 17 it is expected until we merge #12941, the idea is to rebase once that PR is merged and validate the JDK 17 image is passing at that point!

Update: The PR to add support has been merged and I have rebased master. All the Java jobs are passing now.

ARROW-16327: [CI][Java] Remove minor version from maven in order to be able to find the docker image

…essary debian-9-java dockerfile

pitrou · 2022-05-02T13:18:48Z

docker-compose.yml

        /arrow/ci/scripts/java_build.sh /arrow /build &&
        /arrow/ci/scripts/java_test.sh /arrow /build"

+  oracle-java:


How is this different from debian-java? It seems to be based on the same base image.

the maven image for openjdk 17 is based on Oracle Linux instead of debian maven:3.8.5-openjdk-17:

$ docker run --rm maven:3.8.5-openjdk-17 cat /etc/system-release Oracle Linux Server release 8.5

kou · 2022-05-03T21:14:59Z

@github-actions crossbow submit java-jars

github-actions · 2022-05-03T21:16:14Z

Revision: c2238ba

Submitted crossbow builds: ursacomputing/crossbow @ actions-2006

Task	Status
java-jars

raulcd · 2022-05-05T09:54:22Z

@kou @pitrou is there anything else you think we should do on this PR in order to get it merged?
About the oracle-java vs debian-java I am not sure how we could manage the inconsistency from the maven images repository better on our side.I did add the params to the usage on the docker-compose and the openjdk vs jdk naming convention for the image:

    # Usage:
    #   docker-compose build oracle-java
    #   docker-compose run oracle-java
    # Parameters:
    #   MAVEN: 3.8.5
    #   JDK: 17
    image: ${ARCH}/maven:${MAVEN}-openjdk-${JDK}

vs

    # Usage:
    #   docker-compose build debian-java
    #   docker-compose run debian-java
    # Parameters:
    #   MAVEN: 3.5.4, 3.6.2
    #   JDK: 8, 11
    image: ${ARCH}/maven:${MAVEN}-jdk-${JDK}

let me know if you have a better idea.

pitrou · 2022-05-05T12:36:10Z

@raulcd I think this is good to go, I'll let @kou comment if he wants to.

kou

+1

Thanks!

ursabot · 2022-05-09T14:22:18Z

Benchmark runs are scheduled for baseline = 4ea776a and contender = 1fc251f. 1fc251f is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.47% ⬆️0.12%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.63% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 1fc251f1 ec2-t3-xlarge-us-east-2
[Finished] 1fc251f1 test-mac-arm
[Finished] 1fc251f1 ursa-i9-9960x
[Finished] 1fc251f1 ursa-thinkcentre-m75q
[Finished] 4ea776a0 ec2-t3-xlarge-us-east-2
[Finished] 4ea776a0 test-mac-arm
[Finished] 4ea776a0 ursa-i9-9960x
[Finished] 4ea776a0 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

### What changes were proposed in this pull request? This PR aims to upgrade Apache Arrow to 9.0.0 ### Why are the changes needed? This version bring some bug fix and start to [validating Java 17](apache/arrow#13021) through CI. (The release notes have not been released yet, will update later https://arrow.apache.org/release/) ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions Closes #37331 from LuciferYang/SPARK-39913. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

raulcd marked this pull request as draft April 28, 2022 07:17

raulcd changed the title ~~[WIP] ARROW-16327: [Java][CI] Add Java 17 to CI matrix for java workflows~~ ARROW-16327: [Java][CI] Add Java 17 to CI matrix for java workflows Apr 28, 2022

raulcd force-pushed the ARROW-16327 branch 4 times, most recently from 36cac69 to 128f18f Compare April 28, 2022 09:16

raulcd marked this pull request as ready for review April 28, 2022 09:28

raulcd mentioned this pull request Apr 28, 2022

ARROW-15755: [Java] Support Java 17 #12941

Closed

lidavidm approved these changes Apr 28, 2022

View reviewed changes

raulcd added 6 commits May 2, 2022 14:29

ARROW-16327: [Java][CI] Add Java 17 to CI matrix for java workflows

17edb0e

ARROW-16327: [CI][Java] Remove minor version from maven in order to be able to find the docker image

ARROW-16327: [Java][CI] Add Oracle java docker image and remove unnec…

ba243a0

…essary debian-9-java dockerfile

ARROW-16327: Fix volumes and command for oracle-java

f1eca80

ARROW-16327: Add JDK 8 to CI, comments and fix x-hierarchy

81fe8ec

ARROW-16327: Add missing args on linux-apt-jni dockerfile

44d8229

ARROW-16327: Update java docs to reflect new CI tests

c2238ba

raulcd force-pushed the ARROW-16327 branch from 52b72ac to c2238ba Compare May 2, 2022 12:32

pitrou reviewed May 2, 2022

View reviewed changes

kou approved these changes May 5, 2022

View reviewed changes

kou closed this in 1fc251f May 5, 2022

LuciferYang mentioned this pull request Aug 4, 2022

[SPARK-39913][BUILD] Upgrade to Arrow 9.0.0 apache/spark#37331

Closed

asfimport mentioned this pull request May 25, 2022

[Java][CI]: Add support for Java 17 CI process #31712

Closed

ARROW-16327: [Java][CI] Add Java 17 to CI matrix for java workflows #13021

ARROW-16327: [Java][CI] Add Java 17 to CI matrix for java workflows #13021

Uh oh!

Conversation

raulcd commented Apr 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 28, 2022

Uh oh!

raulcd commented Apr 28, 2022

Uh oh!

github-actions bot commented Apr 28, 2022

Uh oh!

github-actions bot commented Apr 28, 2022

Uh oh!

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

kou commented Apr 28, 2022

Uh oh!

kou commented Apr 28, 2022

Uh oh!

raulcd commented May 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou May 2, 2022

Choose a reason for hiding this comment

Uh oh!

raulcd May 2, 2022

Choose a reason for hiding this comment

Uh oh!

kou commented May 3, 2022

Uh oh!

github-actions bot commented May 3, 2022

Uh oh!

raulcd commented May 5, 2022

Uh oh!

pitrou commented May 5, 2022

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

ursabot commented May 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

raulcd commented Apr 28, 2022 •

edited

Loading

raulcd commented May 2, 2022 •

edited

Loading