fix: [iceberg] more fixes for Iceberg integration APIs. by parthchandra · Pull Request #2078 · apache/datafusion-comet

parthchandra · 2025-08-06T19:28:23Z

Which issue does this PR close?

When running iceberg unit and integration tests with Comet, we still encounter some issues due to Parquet shading. This PR addresses those issues.
The main changes are: For all classes in Comet that are used in Iceberg integration, make all constructors and methods package private if they have a Parquet class in the signature. Provide equivalent methods that use Parquet encapsulation classes.
Refactor the FileReader API to use an InputStream (instead of a file path) to allow implementations of Iceberg's InputFile. The refactoring introduces some more encapsulation classes that use reflection to call back into Iceberg (to avoid adding a circular dependency on Iceberg)
This PR may break the complementary (draft) PR in Iceberg: apache/iceberg#13378

codecov-commenter · 2025-08-06T19:54:21Z

Codecov Report

❌ Patch coverage is 0% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.35%. Comparing base (f09f8af) to head (0ae3c34).
⚠️ Report is 1153 commits behind head on main.

Files with missing lines	Patch %	Lines
.../src/main/java/org/apache/comet/parquet/Utils.java	0.00%	35 Missing ⚠️
...ava/org/apache/comet/parquet/WrappedInputFile.java	0.00%	15 Missing ⚠️
...ache/comet/parquet/WrappedSeekableInputStream.java	0.00%	15 Missing ⚠️
...org/apache/comet/parquet/ConstantColumnReader.java	0.00%	3 Missing ⚠️
...org/apache/comet/parquet/MetadataColumnReader.java	0.00%	3 Missing ⚠️
...main/java/org/apache/comet/parquet/FileReader.java	0.00%	2 Missing ⚠️
...va/org/apache/comet/parquet/NativeBatchReader.java	0.00%	1 Missing ⚠️
...c/main/java/org/apache/comet/parquet/TypeUtil.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2078      +/-   ##
============================================
+ Coverage     56.12%   58.35%   +2.22%     
- Complexity      976     1252     +276     
============================================
  Files           119      140      +21     
  Lines         11743    13322    +1579     
  Branches       2251     2370     +119     
============================================
+ Hits           6591     7774    +1183     
- Misses         4012     4317     +305     
- Partials       1140     1231      +91

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

parthchandra · 2025-08-06T22:48:15Z

Reverted some of the changes to allow ci to pass.
#2079 to track the methods that need to be deprecated once everything is in sync

huaxingao · 2025-08-07T06:08:16Z

Refactor the FileReader API to use an InputStream (instead of a file path)... — Does using a file path still have shading issues?

andygrove

LGTM. Thanks @parthchandra

hsiang-c · 2025-08-07T16:08:59Z

+
+/**
+ * Wraps an InputStream that possibly implements the methods of a Parquet SeekableInputStream (but
+ * is not a Parquet SeekableInputStream). Such an InputStream exists, fir instance, in Iceberg's


(nit) fir -> for.

hsiang-c · 2025-08-07T16:11:32Z

+
+/**
+ * A Parquet {@link InputFile} implementation that's similar to {@link
+ * org.apache.parquet.hadoop.util.HadoopInputFile}, but with optimizations introduced in Hadoop 3.x,


Is the Hadoop 3.x part still valid b/c this class is not a CometInputFile

Oops. Updated the comment.

hsiang-c

LGTM

parthchandra · 2025-08-07T18:02:42Z

Refactor the FileReader API to use an InputStream (instead of a file path)... — Does using a file path still have shading issues?

No, but the input to CometVectorizedParquetReader.newCometReader is an InputFile that need not be a HadoopInputFile and need not be backed by an actual file. In that case, the file path is invalid and the Comet FileReader fails. A bunch of Iceberg unit tests failed because of that.

The WrappedInputFile implementation exists to work around that. The underlying object in WrappedInputFile is an org.apache.iceberg.io.InputFile which is accessed via reflection to avoid creating a dependency on Iceberg. Similarly the org.apache.iceberg.io.SeekableInputStream that is returned by org.apache.iceberg.io.InputFile is wrapped in WrappedSeekableInputStream to avoid dependencies on Iceberg.

huaxingao · 2025-08-07T21:29:40Z

+    }
+
+    return new ParquetColumnSpec(
+        1, // ToDo: pass in the correct id


ColumnDescriptor doesn't expose fieldId, but we need a placeholder here. I guess put -1 for unknown id?

Apparently one can but id may not be set. I've changed this to set the id from the descriptor else to -1

@huaxingao if this change looks good to you, I will merge this PR. Please take a look.

LGTM. Thanks for the fix!

parthchandra · 2025-08-08T15:14:24Z

Merged. Thank you for the review @andygrove @huaxingao @hsiang-c

* fix: [iceberg] more fixes for Iceberg intergation APIs.

fix: [iceberg] more fixes for Iceberg intergation APIs.

151063a

fic ci failure

fd9b696

spotless

b1de7c8

parthchandra requested review from andygrove and huaxingao August 7, 2025 01:30

andygrove approved these changes Aug 7, 2025

View reviewed changes

hsiang-c reviewed Aug 7, 2025

View reviewed changes

hsiang-c approved these changes Aug 7, 2025

View reviewed changes

fix comments

4171a9e

huaxingao reviewed Aug 7, 2025

View reviewed changes

huaxingao approved these changes Aug 7, 2025

View reviewed changes

parthchandra mentioned this pull request Aug 7, 2025

[iceberg] Tracking PR to update Iceberg for Parquet shading issues #2093

Open

fix for id

0ae3c34

parthchandra merged commit bdd5c43 into apache:main Aug 8, 2025
92 checks passed

parthchandra mentioned this pull request Aug 11, 2025

Spark: Encapsulate parquet objects for Comet apache/iceberg#13786

Closed

coderfender pushed a commit to coderfender/datafusion-comet that referenced this pull request Dec 13, 2025

fix: [iceberg] more fixes for Iceberg integration APIs. (apache#2078)

43b4aab

* fix: [iceberg] more fixes for Iceberg intergation APIs.

parthchandra deleted the iceberg-shading branch January 14, 2026 22:02

Conversation

parthchandra commented Aug 6, 2025

Which issue does this PR close?

Uh oh!

codecov-commenter commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

parthchandra commented Aug 6, 2025

Uh oh!

huaxingao commented Aug 7, 2025

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsiang-c left a comment

Choose a reason for hiding this comment

Uh oh!

parthchandra commented Aug 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

parthchandra commented Aug 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Aug 6, 2025 •

edited

Loading