HIVE-28658: Iceberg: Implement REST Catalog HMS Client #5995

difin · 2025-07-25T19:36:25Z

What changes were proposed in this pull request?

Hive Iceberg RESTCatalog client adaptor.
HMSClient can be backed by this adaptor and this allows Hive to work directly with Iceberg REST catalogs.

Why are the changes needed?

To enable HMSClient work with Iceberg RESTCatalog without HMS.

Does this PR introduce any user-facing change?

Yes, users can make Hive work directly with Iceberg REST catalogs.

How was this patch tested?

New unit and integration tests.

Note: This PR builds on the work originally started by @zratkai in PR #5628. The goal of this PR is to bring that work to completion by fixing HMS REST Client implementation, handling all open issues brought up in code reviews, adapting the implementation to the latest hive changes, and development of Java-based integration tests and q-tests.

zratkai · 2025-07-28T13:39:32Z

Hi @difin, thanks for making this ticket forward. However this PR is a copy of my original PR (#5628) but I can not see my name mentioned in the description as an author.

common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

data/conf/iceberg/llap/hive-site.xml

iceberg/iceberg-catalog/pom.xml

...eberg-catalog/src/main/java/org/apache/iceberg/hive/HiveIcebergRESTCatalogClientAdapter.java

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/Catalogs.java

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java

pom.xml

...rc/main/java/org/apache/hadoop/hive/metastore/client/builder/HiveMetaStoreClientBuilder.java

...ava/org/apache/hadoop/hive/metastore/client/builder/HiveMetaStoreClientBuilderException.java

...tore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java

iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HMSTablePropertyHelper.java

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/Catalogs.java

deniskuzZ · 2025-09-04T13:42:39Z

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java

+    if (StringUtils.isEmpty(catalogType) || CatalogUtil.ICEBERG_CATALOG_TYPE_HIVE.equals(catalogType)) {
+      return new HiveIcebergMetaHook(conf);
+    } else {
+      conf.set(ConfigProperties.LOCK_HIVE_ENABLED, "false");


probably not part of this PR, however, I am not sure if we are not breaking transaction isolation for non-native catalogs.

HiveTableOperations.doCommit() is doing an atomic METADATA_LOCATION_PROP check & update in HMS backend DB. If location was changed by concurrent transaction - we rollback.
How is that supposed to work in case of Rest or Hadoop catalogs?

iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/CatalogUtils.java

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/BaseHiveIcebergMetaHook.java

iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/CatalogUtils.java

deniskuzZ · 2025-09-05T19:27:46Z

+1
@difin, please merge with the green build

sonarqubecloud · 2025-09-05T20:03:43Z

Quality Gate passed

Issues
460 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.5% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added tests pending tests failed and removed tests pending labels Jul 25, 2025

difin force-pushed the iceberg-rest-catalog-client branch from ade3343 to 35b7879 Compare July 25, 2025 19:46

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Jul 26, 2025