Skip to content

Conversation

@InvisibleProgrammer
Copy link
Contributor

@InvisibleProgrammer InvisibleProgrammer commented Jan 20, 2023

Iceberg released 1.1.0. Hive currently uses 1.0.0

What changes were proposed in this pull request?

Upgrading the iceberg/pom.xml maven dependency and porting/fixing necessary changes to be able to make all tests green.
Some comments about the changes:

  • There were some qfile changes as the output contains more detailed information when describing tables.
  • There were issues about altering partition specs. Had to patch BaseUpdatePartitionSpec.java. After the iceberg fix is merged and released, we can remove that patch: API: Fix Transform backward compatibility in PartitionSpec iceberg#6653
  • Had to port some old changes that haven't ported yet. See the commits.

Why are the changes needed?

To make sure the code compiles and all tests are green.

Does this PR introduce any user-facing change?

There will be some changes in describe metadata tables. See c13294b

How was this patch tested?

Ran all the existing tests.

@InvisibleProgrammer InvisibleProgrammer changed the title HIVE-26808 - Iceberg 1.1.0 HIVE-26990 - Iceberg 1.1.0 Jan 26, 2023
@InvisibleProgrammer InvisibleProgrammer changed the title HIVE-26990 - Iceberg 1.1.0 HIVE-26990 - Upgrade Iceberg from 1.0.0 to 1.1.0 Jan 26, 2023
@InvisibleProgrammer InvisibleProgrammer marked this pull request as ready for review January 26, 2023 10:04
Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes Looks good.
We have patched HadoopInputFile as well in Hive. We should sync that as well with the 1.1.0 code.
Guess this commit is the only extra one:
apache/iceberg@1bdc699

Can you check if that can be included?

@deniskuzZ
Copy link
Member

Looks like apache/iceberg#5670 was included only downstream, I guess we can disregard it for now, but it might bite us eventually.

@sonarqubecloud
Copy link

sonarqubecloud bot commented Feb 8, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ayushtkn ayushtkn merged commit 245c9f3 into apache:master Feb 10, 2023
yeahyung pushed a commit to yeahyung/hive that referenced this pull request Jul 20, 2023
…t Miskolczi, reviewed by Ayush Saxena)

* Update iceberg version

* Core: Use CharSequenceSet instead of Set<CharSequence>

* Flink: Add SupportsRowPosition to Avro reader to fix position deletes

* Core: Deprecate functions in DeleteWriters (apache#5771)

* API: Fix Transform backward compatibility in PartitionSpec

* Core: Avoid extra getFileStatus call in HadoopInputFile
tarak271 pushed a commit to tarak271/hive-1 that referenced this pull request Dec 19, 2023
…t Miskolczi, reviewed by Ayush Saxena)

* Update iceberg version

* Core: Use CharSequenceSet instead of Set<CharSequence>

* Flink: Add SupportsRowPosition to Avro reader to fix position deletes

* Core: Deprecate functions in DeleteWriters (apache#5771)

* API: Fix Transform backward compatibility in PartitionSpec

* Core: Avoid extra getFileStatus call in HadoopInputFile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants