Implement HashJoinP2P for merge #272
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rjzamora
changed the title
rjzamora
approved these changes
Aug 21, 2023
| from dask_expr._shuffle import Shuffle, _contains_index_name | ||
| from dask_expr._shuffle import AssignPartitioningIndex, Shuffle, _contains_index_name | ||
|
|
||
| _HASH_COLUMN_NAME = "__hash_partition" |
Member
There was a problem hiding this comment.
I take it this is a "magic" string used by distributed's p2p merge?
| self.how, | ||
| self.left_on, | ||
| self.right_on, | ||
| self.left._meta.drop(columns=_HASH_COLUMN_NAME), |
Member
There was a problem hiding this comment.
I don't love how the p2p merge depends on _HASH_COLUMN_NAME and drops this column, but isn't responsible for generating this column. Unfortunately, the main issue is that merge_unpack drops the shuffle column, so there is not much we can do here to make the algorithm more intuitive.
To clarify, it would be much clearer if we could just lower the merge to "add partitioning columns" + "p2p merge" + "drop partitioning column"
Collaborator
Author
There was a problem hiding this comment.
Yeah I mostly agree but I fixed something else there today. This is on my medium term todo list
Collaborator
Author
|
Merging this |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The memory footprint of the blockwise version was terrible