[opt](nereids) speedup huge InPredicate for partition pruning #47608
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31594 ms |
TPC-DS: Total hot run time: 190664 ms |
ClickBench: Total hot run time: 30.39 s |
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31839 ms |
TPC-DS: Total hot run time: 190775 ms |
ClickBench: Total hot run time: 30.9 s |
924060929
force-pushed
the
opt-in-filter
branch
from
ac316e8 to
51fa66b
Compare
Contributor
Author
|
run buildall |
924060929
force-pushed
the
opt-in-filter
branch
from
51fa66b to
7d2162d
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31776 ms |
TPC-DS: Total hot run time: 190304 ms |
ClickBench: Total hot run time: 30.44 s |
924060929
force-pushed
the
opt-in-filter
branch
from
7d2162d to
32063f9
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31306 ms |
TPC-DS: Total hot run time: 183992 ms |
ClickBench: Total hot run time: 30.19 s |
924060929
force-pushed
the
opt-in-filter
branch
from
32063f9 to
a4dbc9d
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31698 ms |
TPC-DS: Total hot run time: 183899 ms |
ClickBench: Total hot run time: 30.38 s |
ClickBench: Total hot run time: 31.25 s |
feiniaofeiafei
approved these changes
Mar 6, 2025
Contributor
|
PR approved by anyone and no changes requested. |
morrySnow
approved these changes
Mar 7, 2025
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
This was referenced Mar 24, 2025
Merged
924060929
added a commit
that referenced
this pull request
Mar 26, 2025
) skip run PruneOlapScanTablet when exists lots of InPredicate, follow-up #47608
github-actions bot
pushed a commit
that referenced
this pull request
Mar 26, 2025
) skip run PruneOlapScanTablet when exists lots of InPredicate, follow-up #47608
github-actions bot
pushed a commit
that referenced
this pull request
Mar 26, 2025
) skip run PruneOlapScanTablet when exists lots of InPredicate, follow-up #47608
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
…#47608) This pr can speedup huge InPredicate for partition pruning, for this sql: ```sql select * from tbl where dt in ('2024-01-02 01:00:00', ... , '2024-05-02 02:00:00') -- about 2k literals ``` In my test case, use 2k literals query the table with 20k hour (range / list) partitions, this pr can speed up from 12s to 160ms, if disable binary search filtering, and speed up from 6s to 160ms if enable binary search filtering The changes: 1. add `SessionVariable.enable_binary_search_filtering_partitions` to disable binary search filtering partitions 2. cache the Set/RangeSet of the `InPredicate.options`, instead of foreach options to evaluate it for every partition 3. use big RangeSet to intersect small RangeSet, the order is not trivial, because the TreeRangeSet is red-black tree, the right side need to foreach all items, and the left side can search fast 4. skip prune tablets if the number of InPredicate.options > `Config.max_distribution_pruner_recursion_depth`, because it tend to hit all buckets
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
…che#49386) skip run PruneOlapScanTablet when exists lots of InPredicate, follow-up apache#47608
924060929
added a commit
to 924060929/incubator-doris
that referenced
this pull request
Jul 1, 2025
…#47608) This pr can speedup huge InPredicate for partition pruning, for this sql: ```sql select * from tbl where dt in ('2024-01-02 01:00:00', ... , '2024-05-02 02:00:00') -- about 2k literals ``` In my test case, use 2k literals query the table with 20k hour (range / list) partitions, this pr can speed up from 12s to 160ms, if disable binary search filtering, and speed up from 6s to 160ms if enable binary search filtering The changes: 1. add `SessionVariable.enable_binary_search_filtering_partitions` to disable binary search filtering partitions 2. cache the Set/RangeSet of the `InPredicate.options`, instead of foreach options to evaluate it for every partition 3. use big RangeSet to intersect small RangeSet, the order is not trivial, because the TreeRangeSet is red-black tree, the right side need to foreach all items, and the left side can search fast 4. skip prune tablets if the number of InPredicate.options > `Config.max_distribution_pruner_recursion_depth`, because it tend to hit all buckets (cherry picked from commit 5ec7d34)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
This pr can speedup huge InPredicate for partition pruning, for this sql:
In my test case, use 2k literals query the table with 20k hour (range / list) partitions, this pr can speed up from 12s to 160ms, if disable binary search filtering, and speed up from 6s to 160ms if enable binary search filtering
The changes:
SessionVariable.enable_binary_search_filtering_partitionsto disable binary search filtering partitionsInPredicate.options, instead of foreach options to evaluate it for every partitionConfig.max_distribution_pruner_recursion_depth, because it tend to hit all bucketsCheck List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)