Hive: Add View support for HIVE catalog #9852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

nastra merged 17 commits into apache:main from nk1506:hive_view_new

Sep 13, 2024

Contributor

nk1506 commented Mar 2, 2024

This changes include:

Introduction of common metadata interface(BaseMetadata) for table and view.
Refactor for HiveTableOperation to have common code for table and view commits.

Ref: #9682 , #9461 , #8907

github-actions bot added API core hive labels

nk1506 mentioned this pull request

Core: Common metadata for TableMetadata and ViewMetadata #9682

Closed

nk1506 marked this pull request as ready for review

March 4, 2024 18:45

szehon-ho reviewed

View reviewed changes

Member

szehon-ho left a comment

Thanks for combining the earlier prs. It's clearer now.

Some preliminary comments.

@nastra please take a look as well to see if there are any major concerns.

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java Outdated Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseMetastoreOperations.java Outdated Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseMetastoreOperations.java Outdated Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java Outdated Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java Outdated Show resolved Hide resolved

core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveViewOperations.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java Outdated Show resolved Hide resolved

szehon-ho reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java Outdated Show resolved Hide resolved

nk1506 force-pushed the hive_view_new branch from 8969987 to e9d1660 Compare

March 5, 2024 10:53

szehon-ho reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Show resolved Hide resolved

szehon-ho reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nk1506 force-pushed the hive_view_new branch from e9d1660 to bb2efa4 Compare

March 6, 2024 05:56

nastra reviewed

View reviewed changes

.palantir/revapi.yml Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/CatalogUtil.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/CatalogUtil.java Show resolved Hide resolved

nastra reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/MetadataTableUtils.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

.palantir/revapi.yml Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseMetadata.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseMetastoreOperations.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java Outdated Show resolved Hide resolved


          Addressed comments: Fix incorrect tests from TestHiveViewCommits.

b8c0446

nastra reviewed

View reviewed changes

hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java Outdated Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java Show resolved Hide resolved

nastra reviewed

View reviewed changes

hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveViewCommits.java Show resolved Hide resolved

nk1506 force-pushed the hive_view_new branch from dd82f4b to 7da5812 Compare

September 13, 2024 13:25


          Addressed comments: added metadata file counts for tests of TestHiveV…

2b88e13

…iewCommits

nk1506 force-pushed the hive_view_new branch from 7da5812 to 2b88e13 Compare

September 13, 2024 14:02

Contributor Author

nk1506 commented Sep 13, 2024

Build is failing because of flaky issue #11046

nastra approved these changes

View reviewed changes

Contributor

nastra left a comment

will wait a bit with merging in case @danielcweeks has some additional comments

nastra closed this

nastra reopened this

danielcweeks approved these changes

View reviewed changes

Contributor

nastra commented Sep 13, 2024

thanks @nk1506 for your patience here. Also thanks for everyone that helped out with reviews

nastra merged commit e449d34 into apache:main

Contributor Author

nk1506 commented Sep 13, 2024

Thanks @nastra and @danielcweeks for your reviews and feedback.

nqvuong1998 commented Sep 17, 2024

Hi @nastra @danielcweeks , Is this pull request scheduled for inclusion in the v1.7 release?

Contributor

nastra commented Sep 17, 2024

@nqvuong1998 yes that will be shipped with 1.7.0

ajantha-bhat mentioned this pull request

Add view support for Hive catalog #8698

Closed

nqvuong1998 mentioned this pull request

Add Iceberg view support for Hive catalog in Iceberg connector trinodb/trino#24093

Open

zachdisc pushed a commit to zachdisc/iceberg that referenced this pull request


          Hive: Add View support for HIVE catalog (apache#9852)

100faa5

ajantha-bhat mentioned this pull request

Add Support for Views in the HIVE Catalog #12118

Closed

3 tasks

FANNG1 mentioned this pull request

Iceberg Rest Unable to list tables for Hive Metastore with mixed Hive and Iceberg apache/gravitino#6382

Open

hsiang-c mentioned this pull request

Core: JDBCCatalog's dropView() should purge metadata files if GC is enabled #12511

Merged

monologuist mentioned this pull request

Spark: Support creating views via SQL #9423

Merged

nastra mentioned this pull request

Issues caused by adding view functionality to SparkSessionCatalog #14557

Open

3 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

pvary pvary left review comments

szehon-ho szehon-ho left review comments

nastra nastra approved these changes

danielcweeks danielcweeks approved these changes

+1 more reviewer

vteja11 vteja11 left review comments

Labels