HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1 by kasakrisz · Pull Request #5846 · apache/hive

kasakrisz · 2025-06-04T13:37:08Z

What changes were proposed in this pull request?

Backport patches from iceberg:

mr
+ ec2c2e978e40a4bf7bf6a3a0bccbe7e3a68a0cc1 mr：Fix ugi not correct in WORKER_POOL (#10661)
+ a5c8f9cd4557639d39eed716d499e9837d13b88e Build: Remove unused variables, fields and parameters (#11101)
+ 168a9839425f9d3e2518ed345b7847ecb7cabb89 ThreadPools introduce newExitingWorkerPool and newFixedThreadPool for clearer semantics (#11073)
+ 5e279c868f8e2087f88ae779d8cc8474768bc5e2 Build, Spark, Flink: Bump junit from 5.10.1 to 5.11.1 (#11262)
+ 0280885ac95bdf763556a84bb9d7c6fd9c8c5e2a Pig: Remove iceberg-pig (#11380)
+ b38951db6a7061a595605229c21c1a1912a3a4c1 Data, Flink, MR, Spark: Test deletes with format-version=3 (#11538)
+ da53495bc1bb52db37cdd1ced5c2377001c9d482 Core, Flink, Spark, KafkaConnect: Remove usage of deprecated path API (#11744)
+ b9b61b1d72ebb192d5e90453ff7030ece73d2603 Avro: Support default values for generic data (#11786)

hive-metastore
+ d17a7f189afa25c6be37df1415f4e2f8594effbe Core: Remove deprecated APIs for 1.7.0 (#10818)
+ cf02ffac4329141b30bca265cafb9987f64f6cc4 AWS, Core, Hive: Extract FileIO closing into separate FileIOTracker class (#10893)
+ e449d3405cfdb304c94835845bd8f34a73b4a517 Hive: Add View support for HIVE catalog (#9852)
+ 6a5ae1ae6a01f1395ee70e046537cd87b990c4ae Core: Switch usage to DataFileSet / DeleteFileSet (#11158)
+ 6e9e07aa0f35197bb23b218d23716e928d1c2814 Hive: Bugfix for incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError (#11576)
+ a95943e5561c78c18062852e7f8027a191562e08 Core: Propagate custom metrics reporter when table is created/replaced through Transaction (#11671)
+ da53495bc1bb52db37cdd1ced5c2377001c9d482 Core, Flink, Spark, KafkaConnect: Remove usage of deprecated path API (#11744)
+ a3dcfd19fd1b2a709f7bdf013b83836953d49c6f Hive: Optimize tableExists API in hive catalog (#11597)
+ e1d2271ad911d4224ad53ac2e0142b28984e5f0b Hive: Optimize viewExists API in hive catalog (#11813)
+ f129588461ad02c2fa2021af30b4e9bca70eee93 Core: Add support for view-default property in catalog (#11064)
- a05b2b53b792a6215732aa8b1118adfba0cf3317 Hive: Use correct classloader to load SQL script (#12140)
+ c02ebe4740b22d6f5a78b636aea2d918037b2751 Core: Set missing table-default property in RESTSessionCatalog (#11646)
+ d35cf23eb63cc88d4fdb6ae1db8e63630b073d13 Core: Add missing table-override property to REST catalog (#12548)
+ 8f6ebb5b36a0263edfcb04e0c104b26225f95b07 Core: Add `view-override` catalog property (#12534)
+ cbf34a6ab980ab9b7bb01e45c9e2bcfad8e2986b Build: Enforce error message check on Exception assertions (#12624)
+ c661a71091e496393c743ddd879d9e1a0f2747b2 Core, Hive: Double check commit status in case of commit conflict for NoLock (#12637)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Run existing and backported tests

deniskuzZ · 2025-06-04T16:25:39Z

@kasakrisz, please drop the following

from iceberg-handler
org.apache.iceberg.data.PartitionStatsHandler
org.apache.iceberg.data.TestPartitionStatsHandler
from patched-iceberg-core
org.apache.iceberg.avro.*
org.apache.iceberg.BaseScan.java
org.apache.iceberg.Partitioning.java
org.apache.iceberg.PartitionStats.java
org.apache.iceberg.PartitionStatsUtil.java
cleanup excludes in patched-iceberg-core pom (remove the dropped classes)

<excludes>
    **/HadoopInputFile.class
    **/HadoopTableOperations.class
    **/StructLikeMap.class
    **/StructLikeWrapper.class
    org.apache.iceberg.PartitionsTable.class
</excludes>

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java

deniskuzZ

LGTM, pending tests

okumin · 2025-06-05T12:48:36Z

iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveCatalog.java

 import org.slf4j.LoggerFactory;

-public class HiveCatalog extends BaseMetastoreCatalog implements SupportsNamespaces, Configurable {
+public class HiveCatalog extends BaseMetastoreViewCatalog


This is so nice. The lack of view support is one of the bottlenecks when I try HIVE-28059 with other query engines.

@okumin, are you using RestCatalog locally? if yes, please suggest the default configs for the docker image: https://github.com/apache/hive/pull/5834/files#diff-cfa8481579367e1e9127939f845eba4d2ee8c796d6858bcd0ca0a4b5fbfb8019R35-R42

Yes, but I'm still halfway and I don't have very concrete suggestions. I have not successfully integrated it with other query engines.

Aggarwal-Raghav · 2025-06-09T16:08:50Z

Should we upgrade parquet to 1.15.2 just to be in sync with iceberg 1.9.1? Let me know I can raise the PR for the same.

kasakrisz · 2025-06-10T13:11:10Z

Should we upgrade parquet to 1.15.2 just to be in sync with iceberg 1.9.1? Let me know I can raise the PR for the same.

@Aggarwal-Raghav
Yes please go ahead.

kasakrisz · 2025-06-13T08:24:31Z

...rg-handler/src/test/results/positive/llap/iceberg_minor_compaction_partition_evolution.q.out

 	numFiles            	2                   
 	numPartitions       	2                   
-	numRows             	6                   
+	numRows             	4                   


Caused by apache/iceberg#11131 (comment)
Created: https://issues.apache.org/jira/browse/HIVE-29014

kasakrisz · 2025-06-13T08:26:06Z

.../iceberg-handler/src/test/results/positive/llap/iceberg_minor_compaction_unpartitioned.q.out

 #### A masked pattern was here ####
 	numFiles            	1                   
-	numRows             	7                   
+	numRows             	6                   


Caused by apache/iceberg#11131 (comment)
Created: https://issues.apache.org/jira/browse/HIVE-29014

sonarqubecloud · 2025-06-14T09:48:23Z

Quality Gate passed

Issues
88 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

kasakrisz marked this pull request as draft June 4, 2025 13:37

asf-ci-hive added the tests pending label Jun 4, 2025

deniskuzZ reviewed Jun 4, 2025

View reviewed changes

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java Show resolved Hide resolved

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Jun 4, 2025

kasakrisz force-pushed the HIVE-28665-master-iceberg-191 branch from fa9c656 to 16da407 Compare June 5, 2025 10:18

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Jun 5, 2025

deniskuzZ approved these changes Jun 5, 2025

View reviewed changes

kasakrisz marked this pull request as ready for review June 5, 2025 11:44

okumin reviewed Jun 5, 2025

View reviewed changes

asf-ci-hive added tests failed and removed tests pending labels Jun 6, 2025

kasakrisz force-pushed the HIVE-28665-master-iceberg-191 branch from 16da407 to a7a1cbf Compare June 6, 2025 09:16

asf-ci-hive added tests pending tests failed tests unstable and removed tests failed tests pending labels Jun 6, 2025

kasakrisz force-pushed the HIVE-28665-master-iceberg-191 branch from a7a1cbf to 931bdec Compare June 10, 2025 13:40

asf-ci-hive added tests pending and removed tests failed labels Jun 13, 2025

kasakrisz commented Jun 13, 2025

View reviewed changes

asf-ci-hive added tests unstable tests pending and removed tests pending tests unstable labels Jun 13, 2025

deniskuzZ force-pushed the HIVE-28665-master-iceberg-191 branch from 127743a to 85f7ca9 Compare June 13, 2025 17:34

asf-ci-hive added tests failed tests pending tests unstable and removed tests pending tests failed labels Jun 13, 2025

deniskuzZ force-pushed the HIVE-28665-master-iceberg-191 branch from 85f7ca9 to 8140ed1 Compare June 14, 2025 07:47

asf-ci-hive added tests pending and removed tests unstable labels Jun 14, 2025

fixed masking and added extra meta columns

0ed29d2

deniskuzZ force-pushed the HIVE-28665-master-iceberg-191 branch from 8140ed1 to 0ed29d2 Compare June 14, 2025 08:38

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Jun 14, 2025

asf-ci-hive added tests passed and removed tests pending labels Jun 14, 2025

deniskuzZ merged commit 62402e9 into apache:master Jun 14, 2025
4 checks passed

basapuram-kumar pushed a commit to acceldata-io/hive that referenced this pull request Jul 7, 2025

HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1 (apache#5846)

25ed32d

basapuram-kumar pushed a commit to acceldata-io/hive that referenced this pull request Jul 8, 2025

HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1 (apache#5846)

13e1bc3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1#5846

HIVE-28665: Iceberg: Upgrade iceberg version to 1.9.1#5846
deniskuzZ merged 30 commits intoapache:masterfrom
kasakrisz:HIVE-28665-master-iceberg-191

kasakrisz commented Jun 4, 2025

Uh oh!

deniskuzZ commented Jun 4, 2025 •

edited

Loading

Uh oh!

Uh oh!

deniskuzZ left a comment

Uh oh!

okumin Jun 5, 2025

Uh oh!

deniskuzZ Jun 5, 2025 •

edited

Loading

Uh oh!

okumin Jun 5, 2025

Uh oh!

Aggarwal-Raghav commented Jun 9, 2025

Uh oh!

kasakrisz commented Jun 10, 2025 •

edited

Loading

Uh oh!

kasakrisz Jun 13, 2025 •

edited

Loading

Uh oh!

kasakrisz Jun 13, 2025

Uh oh!

sonarqubecloud bot commented Jun 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

kasakrisz commented Jun 4, 2025

What changes were proposed in this pull request?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

deniskuzZ commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

deniskuzZ left a comment

Choose a reason for hiding this comment

Uh oh!

okumin Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

okumin Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Jun 9, 2025

Uh oh!

kasakrisz commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kasakrisz Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kasakrisz Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jun 14, 2025

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

deniskuzZ commented Jun 4, 2025 •

edited

Loading

deniskuzZ Jun 5, 2025 •

edited

Loading

kasakrisz commented Jun 10, 2025 •

edited

Loading

kasakrisz Jun 13, 2025 •

edited

Loading