[VL] Caching java dependencies in testing docker

[zhouyuan]

Description

In CI jobs Gluten is using clean docker image, all Java dependencies need to be downloaded from maven repo. It's better to pre-install these dependencies in CI docker. For now below docker images are used

• Centos-7
• Centos-8
• Centos-9
• Ubuntu-20.04
• Ubuntu-22.04

related: #11469

Gluten version

None

[sh-shamsan]

Hi @AshharAhmadKhan I started on this but i think there was an issue with my comment "take" wasn't added here to assign the issue, let me know if you’d like to work on it together

[sh-shamsan]

Take

[sh-shamsan]

Im running this set to install the dependencies:

          $MVN_CMD -Pjava-17,spark-3.5,backends-velox,hadoop-3.3,spark-ut \
            -Piceberg,iceberg-test,delta,paimon \
            clean test-compile

          $MVN_CMD -Pjava-17,spark-3.4,backends-velox,hadoop-3.3,spark-ut \
            -Piceberg,iceberg-test,delta,paimon \
            clean test-compile

          $MVN_CMD -Pjava-17,spark-3.3,backends-velox,hadoop-3.3,spark-ut \
            -Piceberg,delta,paimon \
            clean test-compile


the build fails at gluten-arrow because the arrow-15.0.0-gluten artifacts are not available in Maven Central.

• I tried two approaches:
  1. Downloading Arrow jars via actions/download-artifact using ${github.sha} from main, but the artifacts were not found (likely expired or not produced for that commit).
  2. Building Arrow locally in the workflow before running Maven, but that has not succeeded yet due to build issues

At this point, I’m not sure what the recommended approach is for the cache job:

• Continue to try build Arrow locally as part of the cache workflow?
• Or skip gluten-arrow for the purpose of the unified .m2 cache?
  @zhouyuan

[zhouyuan]

@sh-shamsan thanks for working on this, the arrow-15 jar will be fetched from other native lib build:
https://github.com/apache/incubator-gluten/blob/main/.github/workflows/velox_backend_x86.yml#L98
https://github.com/apache/incubator-gluten/blob/main/.github/workflows/velox_backend_x86.yml#L152

For the dependencies caching stuff, below cmd may help
./build/mvn dependency:go-offline -Pxxx -DskipTests || true