Upgrade to hive 4.0.1

[Abacn]

Fix #32189

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[Abacn]

it indeed eliminated the calcite vulneribility, as Hive 4 has dropped calcite dependency

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @m-trieu for label java.
R: @damccorm for label build.
R: @damondouglas for label io.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[damccorm]

Thanks!

[Abacn]

from https://hive.apache.org/general/downloads/ Hive 4 no longer support Hadoop 2, which indeed causes Hadoop compat tests to fail

[Abacn]

Now hit apache/iceberg#10429

in particular

2025-01-27T20:35:16.4163752Z org.apache.beam.sdk.io.iceberg.catalog.HiveCatalogIT > testStreamToPartitionedDynamicDestinations FAILED
2025-01-27T20:35:16.4165463Z     java.lang.NoSuchFieldError: METASTOREURIS
2025-01-27T20:35:16.4166824Z         at org.apache.iceberg.hive.CachedClientPool.extractKey(CachedClientPool.java:135)
2025-01-27T20:35:16.4168686Z         at org.apache.iceberg.hive.CachedClientPool.<init>(CachedClientPool.java:90)
2025-01-27T20:35:16.4170296Z         at org.apache.iceberg.hive.HiveCatalog.initialize(HiveCatalog.java:113)
2025-01-27T20:35:16.4171820Z         at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:256)
2025-01-27T20:35:16.4173671Z         at org.apache.beam.sdk.io.iceberg.catalog.HiveCatalogIT.createCatalog(HiveCatalogIT.java:76)

[Abacn]

it's become challenging.

For Hive 4.x, Iceberg Hive catalog is released by hive (org.apache.hive:hive-iceberg-handler), not iceberg (org.apache.iceberg:iceberg-hive-metastore). Moreover, hive-iceberg-handler shaded old iceberg versions inside,

This gives runtime errors like

java.lang.NoSuchMethodError: 'long org.apache.iceberg.io.PositionOutputStream.storedLength()'
	at org.apache.iceberg.avro.AvroFileAppender.length(AvroFileAppender.java:83)

checking the class path resolution of PositionOutputStream indicates it came from hive-iceberg-handler-4.0.1.jar

[Abacn]

superceded by #33826

[Abacn]

it's become challenging.

For Hive 4.x, Iceberg Hive catalog is released by hive (org.apache.hive:hive-iceberg-handler), not iceberg (org.apache.iceberg:iceberg-hive-metastore). Moreover, hive-iceberg-handler shaded old iceberg versions inside,

This gives runtime errors like

java.lang.NoSuchMethodError: 'long org.apache.iceberg.io.PositionOutputStream.storedLength()'
	at org.apache.iceberg.avro.AvroFileAppender.length(AvroFileAppender.java:83)

checking the class path resolution of PositionOutputStream indicates it came from hive-iceberg-handler-4.0.1.jar

created https://issues.apache.org/jira/browse/HIVE-28738

[github-actions]

Reminder, please take a look at this pr: @m-trieu @damccorm @damondouglas

[damccorm]

stop reviewer notifications

[github-actions]

Stopping reviewer notifications for this pull request: requested by reviewer. If you'd like to restart, comment assign set of reviewers

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.