[opt](nereids) recover adoptive bucket shuffle #36784
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
xzj7019
force-pushed
the
recover_adoptive_bucket_shuffle
branch
from
917e3ba to
7cb2b14
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40420 ms |
TPC-DS: Total hot run time: 174668 ms |
ClickBench: Total hot run time: 30.26 s |
xzj7019
force-pushed
the
recover_adoptive_bucket_shuffle
branch
from
c814ff0 to
f1b0fe6
Compare
xzj7019
force-pushed
the
recover_adoptive_bucket_shuffle
branch
from
f1b0fe6 to
c1cd82e
Compare
Contributor
Author
|
run buildall |
morrySnow
reviewed
Jun 26, 2024
| // - base table and tablets' number is small enough (< paraInstanceNum) | ||
| // otherSide: | ||
| // - ShuffleType.EXECUTION_BUCKETED | ||
| boolean isBucketShuffleDownGrade = ConnectContext.get().getSessionVariable().isEnableBucketShuffleJoin(); |
Contributor
There was a problem hiding this comment.
when enableBucketShuffleJoin = false, we should never do bucket shuffle at all. so when isEnableBucketShuffleJoin = false, function isBucketShuffleDownGrade should always return true?
Contributor
Author
There was a problem hiding this comment.
forget to change the name, isBucketShuffleDownGrade should be isEnableBucketShuffle.
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
Show resolved
Hide resolved
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39526 ms |
TPC-DS: Total hot run time: 173632 ms |
ClickBench: Total hot run time: 30.81 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40326 ms |
TPC-DS: Total hot run time: 173538 ms |
ClickBench: Total hot run time: 30.67 s |
Contributor
Author
|
run buildall |
xzj7019
force-pushed
the
recover_adoptive_bucket_shuffle
branch
from
75daeb8 to
21b461a
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39733 ms |
TPC-DS: Total hot run time: 174220 ms |
ClickBench: Total hot run time: 31 s |
Contributor
Author
|
run buildall |
xzj7019
force-pushed
the
recover_adoptive_bucket_shuffle
branch
from
ae9d72a to
0adcecb
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40099 ms |
xzj7019
force-pushed
the
recover_adoptive_bucket_shuffle
branch
from
4303423 to
450f4c6
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40177 ms |
TPC-DS: Total hot run time: 172242 ms |
ClickBench: Total hot run time: 30.18 s |
Contributor
Author
|
run feut |
morrySnow
approved these changes
Jul 3, 2024
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
keanji-x
approved these changes
Jul 3, 2024
dataroaring
pushed a commit
that referenced
this pull request
Jul 17, 2024
1. Recover adoptive bucket shuffle and re-using the enable_bucket_shuffle_join to control whether to enable it, default remains true. 2. Remove enable_bucket_shuffle_downgrade option. Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
Closed
yiguolei
pushed a commit
that referenced
this pull request
Aug 21, 2024
## Proposed changes pick from #36784 Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
Contributor
|
need merge to 2.1 because some USER need it |
16 tasks
morrySnow
added a commit
that referenced
this pull request
Jan 13, 2026
…le side (#59506) Related PR: #36784 This pull request refactors the logic in the `visitPhysicalHashJoin` method to improve the handling and enforcement of bucket shuffle join strategies in the Nereids planner. It introduces new checks and flags to more accurately determine when bucket shuffle downgrades are needed, and updates the plan output to reflect the use of `bucketShuffle` joins in various TPC-DS queries. Key changes include: ### Planner logic improvements * Added `shouldCheckLeftBucketDownGrade` and `shouldCheckrightBucketDownGrade` flags to control when to check for bucket shuffle downgrades, leading to more precise enforcement of shuffle strategies. * Refactored conditional logic to use these new flags, replacing previous direct downgrade checks, and improved the handling of cases where shuffle key orders do not match. These changes enhance the accuracy and efficiency of join planning in distributed query execution.
zzzxl1993
pushed a commit
to zzzxl1993/doris
that referenced
this pull request
Jan 13, 2026
…le side (apache#59506) Related PR: apache#36784 This pull request refactors the logic in the `visitPhysicalHashJoin` method to improve the handling and enforcement of bucket shuffle join strategies in the Nereids planner. It introduces new checks and flags to more accurately determine when bucket shuffle downgrades are needed, and updates the plan output to reflect the use of `bucketShuffle` joins in various TPC-DS queries. Key changes include: ### Planner logic improvements * Added `shouldCheckLeftBucketDownGrade` and `shouldCheckrightBucketDownGrade` flags to control when to check for bucket shuffle downgrades, leading to more precise enforcement of shuffle strategies. * Refactored conditional logic to use these new flags, replacing previous direct downgrade checks, and improved the handling of cases where shuffle key orders do not match. These changes enhance the accuracy and efficiency of join planning in distributed query execution.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes