ARROW-6883: [C++][Python] Allow writing dictionary deltas #8811
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
Acknowledged, will review when I can |
pitrou
force-pushed
the
ARROW-6883-ipc-write-deltas
branch
from
47a1e91 to
3e11e6b
Compare
* Add an ipc::IpcWriteOptions member to govern emission of dictionary deltas. If the option is enabled, deltas are detected by checking whether the new dictionary starts with the last emitted one for the same field. However, for nested dictionaries, deltas are not emitted for the outer dictionary, as the read path doesn't support it. * Add a stats() method to ipc::StreamDecoder * Expose the IPC statistics in Python, and add tests
pitrou
force-pushed
the
ARROW-6883-ipc-write-deltas
branch
from
3e11e6b to
16beb03
Compare
pitrou
force-pushed
the
ARROW-6883-ipc-write-deltas
branch
from
16beb03 to
be9d921
Compare
Member
Author
|
cc @lidavidm |
Member
|
The Flight changes look good to me (though there's still ARROW-10787). |
wesm
approved these changes
Dec 15, 2020
Member
wesm
left a comment
wesm
left a comment
There was a problem hiding this comment.
+1 here. Sorry for the long delay
cpp/src/arrow/ipc/options.h
Outdated
Member
There was a problem hiding this comment.
Might help to be explicit to say that otherwise the order of dictionaries may be altered by a passage from sender to receiver (I assume that's what you mean here by "stream compatibility" but others might be scratching their heads)
Member
Author
There was a problem hiding this comment.
No, I simply meant that it's off for compatibility with implementations that don't support delta dictionaries.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add an ipc::IpcWriteOptions member to govern emission of dictionary deltas.
If the option is enabled, deltas are detected by checking whether the new
dictionary starts with the last emitted one for the same field.
However, for nested dictionaries, deltas are not emitted for the outer dictionary,
as the read path doesn't support it.
Add a stats() method to ipc::StreamDecoder
Expose the IPC statistics in Python, and add tests