Add index-based chunk skip support in ChunkedDecoder #400
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
|
@xiaoxmeng has exported this pull request. If you are a Meta employee, you can view the originating Diff in D89999516. |
xiaoxmeng
added a commit
to xiaoxmeng/nimble
that referenced
this pull request
Jan 2, 2026
…or#400) Summary: This diff adds index-based chunk skip optimization to `ChunkedDecoder`, enabling efficient skip operations that can jump directly to target chunks using the stream index instead of sequentially scanning through all intermediate chunks. 1. **ChunkedDecoder enhancements**: - Added `StreamIndex` support via constructor parameter for index-accelerated skip operations - Added `endRow_` member to track stream end row from index, enabling bounds validation - Added `currentRow_` member to track absolute row position in the stream - Implemented `skipWithIndex()` for index-based skipping that uses chunk lookup - Implemented `skipWithoutIndex()` for legacy sequential skipping - Added `seekToChunk()` to seek input stream to specific chunk offset - Added `advancePosition()` helper to update both `remainingValues_` and `currentRow_` - Improved error handling with `NIMBLE_CHECK` macros instead of `VELOX_CHECK` 2. **Skip optimization logic**: - Within-chunk skips use encoding's skip directly (no index lookup needed) - Cross-chunk skips use `StreamIndex::lookupChunk()` to find target chunk - Skip-to-end optimization resets decoder state without chunk lookup - Proper bounds validation using `endRow_` before attempting skip Differential Revision: D89999516
xiaoxmeng
force-pushed
the
export-D89999516
branch
from
0be6fea to
7621efd
Compare
xiaoxmeng
added a commit
to xiaoxmeng/nimble
that referenced
this pull request
Jan 2, 2026
…or#400) Summary: This diff adds index-based chunk skip optimization to `ChunkedDecoder`, enabling efficient skip operations that can jump directly to target chunks using the stream index instead of sequentially scanning through all intermediate chunks. 1. **ChunkedDecoder enhancements**: - Added `StreamIndex` support via constructor parameter for index-accelerated skip operations - Added `endRow_` member to track stream end row from index, enabling bounds validation - Added `currentRow_` member to track absolute row position in the stream - Implemented `skipWithIndex()` for index-based skipping that uses chunk lookup - Implemented `skipWithoutIndex()` for legacy sequential skipping - Added `seekToChunk()` to seek input stream to specific chunk offset - Added `advancePosition()` helper to update both `remainingValues_` and `currentRow_` - Improved error handling with `NIMBLE_CHECK` macros instead of `VELOX_CHECK` 2. **Skip optimization logic**: - Within-chunk skips use encoding's skip directly (no index lookup needed) - Cross-chunk skips use `StreamIndex::lookupChunk()` to find target chunk - Skip-to-end optimization resets decoder state without chunk lookup - Proper bounds validation using `endRow_` before attempting skip Differential Revision: D89999516
xiaoxmeng
force-pushed
the
export-D89999516
branch
2 times, most recently
from
02d13dd to
21f5454
Compare
xiaoxmeng
added a commit
to xiaoxmeng/nimble
that referenced
this pull request
Jan 2, 2026
…ader (facebookincubator#400) Summary: This diff adds index-based chunk skip optimization to `ChunkedDecoder`, enabling efficient skip operations that can jump directly to target chunks using the stream index instead of sequentially scanning through all intermediate chunks. 1. **ChunkedDecoder enhancements**: - Added `StreamIndex` support via constructor parameter for index-accelerated skip operations - Added `endRow_` member to track stream end row from index, enabling bounds validation - Added `currentRow_` member to track absolute row position in the stream - Implemented `skipWithIndex()` for index-based skipping that uses chunk lookup - Implemented `skipWithoutIndex()` for legacy sequential skipping - Added `seekToChunk()` to seek input stream to specific chunk offset - Added `advancePosition()` helper to update both `remainingValues_` and `currentRow_` - Improved error handling with `NIMBLE_CHECK` macros instead of `VELOX_CHECK` 2. **Skip optimization logic**: - Within-chunk skips use encoding's skip directly (no index lookup needed) - Cross-chunk skips use `StreamIndex::lookupChunk()` to find target chunk - Skip-to-end optimization resets decoder state without chunk lookup - Proper bounds validation using `endRow_` before attempting skip Differential Revision: D89999516
…ader (facebookincubator#400) Summary: This diff adds index-based chunk skip optimization to `ChunkedDecoder`, enabling efficient skip operations that can jump directly to target chunks using the stream index instead of sequentially scanning through all intermediate chunks. 1. **ChunkedDecoder enhancements**: - Added `StreamIndex` support via constructor parameter for index-accelerated skip operations - Added `endRow_` member to track stream end row from index, enabling bounds validation - Added `currentRow_` member to track absolute row position in the stream - Implemented `skipWithIndex()` for index-based skipping that uses chunk lookup - Implemented `skipWithoutIndex()` for legacy sequential skipping - Added `seekToChunk()` to seek input stream to specific chunk offset - Added `advancePosition()` helper to update both `remainingValues_` and `currentRow_` - Improved error handling with `NIMBLE_CHECK` macros instead of `VELOX_CHECK` 2. **Skip optimization logic**: - Within-chunk skips use encoding's skip directly (no index lookup needed) - Cross-chunk skips use `StreamIndex::lookupChunk()` to find target chunk - Skip-to-end optimization resets decoder state without chunk lookup - Proper bounds validation using `endRow_` before attempting skip Reviewed By: zzhao0 Differential Revision: D89999516
xiaoxmeng
force-pushed
the
export-D89999516
branch
from
21f5454 to
baa1fae
Compare
|
This pull request has been merged in f658828. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This diff adds index-based chunk skip optimization to
ChunkedDecoder, enabling efficient skip operations that can jump directly to target chunks using the stream index instead of sequentially scanning through all intermediate chunks.ChunkedDecoder enhancements:
StreamIndexsupport via constructor parameter for index-accelerated skip operationsendRow_member to track stream end row from index, enabling bounds validationcurrentRow_member to track absolute row position in the streamskipWithIndex()for index-based skipping that uses chunk lookupskipWithoutIndex()for legacy sequential skippingseekToChunk()to seek input stream to specific chunk offsetadvancePosition()helper to update bothremainingValues_andcurrentRow_NIMBLE_CHECKmacros instead ofVELOX_CHECKSkip optimization logic:
StreamIndex::lookupChunk()to find target chunkendRow_before attempting skipDifferential Revision: D89999516