Skip to content

feat: Support hadoop s3a config in native_iceberg_compat#1925

Merged
parthchandra merged 4 commits intoapache:mainfrom
parthchandra:iceberg_compat_s3_config
Jun 25, 2025
Merged

feat: Support hadoop s3a config in native_iceberg_compat#1925
parthchandra merged 4 commits intoapache:mainfrom
parthchandra:iceberg_compat_s3_config

Conversation

@parthchandra
Copy link
Copy Markdown
Contributor

#1817 introduced S3A configuration for the native_datafusion reader. This PR does the same for native_iceberg_compat

How are these changes tested?

Existing unit test

@parthchandra parthchandra marked this pull request as draft June 23, 2025 23:41
@parthchandra
Copy link
Copy Markdown
Contributor Author

@Kontinuation please review if you can.
(This PR is draft because I haven't been able to test it with S3 yet. The unit test passes, though).

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Jun 24, 2025

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 58.85%. Comparing base (f09f8af) to head (1a7362d).
Report is 298 commits behind head on main.

Files with missing lines Patch % Lines
...va/org/apache/comet/parquet/NativeBatchReader.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1925      +/-   ##
============================================
+ Coverage     56.12%   58.85%   +2.72%     
- Complexity      976     1141     +165     
============================================
  Files           119      130      +11     
  Lines         11743    12867    +1124     
  Branches       2251     2420     +169     
============================================
+ Hits           6591     7573     +982     
- Misses         4012     4075      +63     
- Partials       1140     1219      +79     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Member

@Kontinuation Kontinuation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested it with AWS S3 and it worked fine

@parthchandra
Copy link
Copy Markdown
Contributor Author

I have tested it with AWS S3 and it worked fine

Thank you! I managed to test it with S3 as well.

@parthchandra parthchandra marked this pull request as ready for review June 24, 2025 16:26
Comment thread native/core/src/parquet/mod.rs Outdated
let map = JMap::from_env(env, &map_object)?;
// Convert to a HashMap
let mut collected_map = HashMap::new();
// let mut iter = map.iter(&mut env)?;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this commented out code be removed?

Copy link
Copy Markdown
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @parthchandra

Copy link
Copy Markdown
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks @parthchandra

@parthchandra parthchandra merged commit 439944b into apache:main Jun 25, 2025
82 checks passed
coderfender pushed a commit to coderfender/datafusion-comet that referenced this pull request Dec 13, 2025
* Support hadoop s3a config in native_iceberg_compat
@parthchandra parthchandra deleted the iceberg_compat_s3_config branch January 14, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants