Skip to content

feat[datafusion]: Support DataFusion PartitionedFile extensions for external index#5556

Merged
connortsui20 merged 2 commits intovortex-data:developfrom
haohuaijin:support-partionfile-ext
Dec 16, 2025
Merged

feat[datafusion]: Support DataFusion PartitionedFile extensions for external index#5556
connortsui20 merged 2 commits intovortex-data:developfrom
haohuaijin:support-partionfile-ext

Conversation

@haohuaijin
Copy link
Contributor

@haohuaijin haohuaijin commented Nov 27, 2025 •
edited
Loading

related impl in datafusion for parquet format apache/datafusion#10813,
see discussion in #5481

Signed-off-by: Huaijin haohuaijin@gmail.com

@haohuaijin haohuaijin force-pushed the support-partionfile-ext branch from a7945a1 to 9f4cacd Compare November 27, 2025 06:49
joseph-isaacs
joseph-isaacs reviewed Nov 27, 2025
View reviewed changes
@codecov
Copy link

codecov bot commented Nov 27, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.30%. Comparing base (9a76489) to head (215f903).
⚠️ Report is 18 commits behind head on develop.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@connortsui20 connortsui20 self-requested a review December 15, 2025 14:38
@connortsui20 connortsui20 added the changelog/feature A new feature label Dec 15, 2025
@connortsui20 connortsui20 force-pushed the support-partionfile-ext branch from f26674e to a13deae Compare December 15, 2025 15:41
Copilot AI review requested due to automatic review settings December 15, 2025 15:41
Copilot AI reviewed Dec 15, 2025
View reviewed changes
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

connortsui20
connortsui20 requested changes Dec 15, 2025
View reviewed changes
Copy link
Contributor

@connortsui20 connortsui20 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the very long wait! We have been doing a bunch of stuff recently, next year we will do our best to be more on top of issues.

I took the liberty of rebasing and squashing to take care of the merge conflicts, everything is now just 1 commit on top of a more recent develop commit. Please make sure you force pull / hard reset!

Just a few nits, and then it should be ready to go.

@haohuaijin @connortsui20
feat: enhance VortexOpener with selection support in ScanBuilder
2907f45 
Signed-off-by: Huaijin <haohuaijin@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 force-pushed the support-partionfile-ext branch from a13deae to 2907f45 Compare December 15, 2025 15:49
@haohuaijin
apply suggestion
215f903 
Signed-off-by: Huaijin <haohuaijin@gmail.com>
@haohuaijin
Copy link
Contributor Author

haohuaijin commented Dec 16, 2025

Thanks for reviews @joseph-isaacs @connortsui20 @hengfeiyang , i already resolve the comments.

connortsui20
connortsui20 approved these changes Dec 16, 2025
View reviewed changes
Copy link
Contributor

@connortsui20 connortsui20 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the PR!

@connortsui20 connortsui20 enabled auto-merge (squash) December 16, 2025 21:12
@connortsui20 connortsui20 merged commit dcaa135 into vortex-data:develop Dec 16, 2025
47 checks passed
paultiq pushed a commit to paultiq/vortex that referenced this pull request Dec 17, 2025
@haohuaijin @paultiq
feat[datafusion]: Support DataFusion PartitionedFile extensions for…
1d34287 
… external index (vortex-data#5556)

related impl in datafusion for parquet format
apache/datafusion#10813,
see discussion in vortex-data#5481

Signed-off-by: Huaijin <haohuaijin@gmail.com>

---------

Signed-off-by: Huaijin <haohuaijin@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@haohuaijin haohuaijin deleted the support-partionfile-ext branch December 17, 2025 01:54
@haohuaijin
Copy link
Contributor Author

haohuaijin commented Dec 17, 2025

Thank you again @connortsui20

gatesn
gatesn reviewed Dec 17, 2025
View reviewed changes
vortex-datafusion/src/persistent/opener.rs
fn get_selection_from_extensions(
extensions: Option<Arc<dyn std::any::Any + Send + Sync>>,
) -> Option<Selection> {
extensions?.downcast_ref::<Selection>().cloned()
Copy link
Contributor

@gatesn gatesn Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we wrap this up in VortexExtensions { selection: Option<Selection> }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gatesn gatesn gatesn left review comments

Copilot code review Copilot Copilot left review comments

@connortsui20 connortsui20 connortsui20 approved these changes

@joseph-isaacs joseph-isaacs Awaiting requested review from joseph-isaacs

@hengfeiyang hengfeiyang Awaiting requested review from hengfeiyang

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@haohuaijin @hengfeiyang @gatesn @joseph-isaacs @connortsui20