fix: avoid creating empty encoding task and part for PrimitiveFieldEncoder#3607
Merged
westonpace merged 3 commits intolance-format:mainfrom Mar 27, 2025
Merged
fix: avoid creating empty encoding task and part for PrimitiveFieldEncoder#3607westonpace merged 3 commits intolance-format:mainfrom
westonpace merged 3 commits intolance-format:mainfrom
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3607 +/- ##
==========================================
+ Coverage 78.68% 78.70% +0.02%
==========================================
Files 258 258
Lines 96897 96900 +3
Branches 96897 96900 +3
==========================================
+ Hits 76241 76268 +27
+ Misses 17588 17561 -27
- Partials 3068 3071 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
… when max page size is small.
niyue
force-pushed
the
bugfix/empty-part
branch
from
105d616 to
4b35d01
Compare
westonpace
approved these changes
Mar 27, 2025
Member
westonpace
left a comment
westonpace
left a comment
There was a problem hiding this comment.
I had noticed this in some files with large data (which I guess is similar to small pages) and never really tracked down why it happened. I think I see now how it can happen. We should probably have made the for _ in 0..num_parts loop a while offset < array.len() loop.
This fix is fine too.
Member
|
Thanks for figuring this out! |
Jay-ju
pushed a commit
to Jay-ju/lance
that referenced
this pull request
Mar 28, 2025
…coder (lance-format#3607) `PrimitiveFieldEncoder` may generate empty `part`s and their corresponding encoding tasks, especially when `max_page_size` is small. This is unnecessary and can be confusing, as some empty part information gets recorded at the end, and redundant encoding tasks are processed needlessly. This PR fixes the issue by exiting the loop early when there is no data to process. Co-authored-by: LuQQiu <luqiujob@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PrimitiveFieldEncodermay generate emptyparts and their corresponding encoding tasks, especially whenmax_page_sizeis small. This is unnecessary and can be confusing, as some empty part information gets recorded at the end, and redundant encoding tasks are processed needlessly. This PR fixes the issue by exiting the loop early when there is no data to process.