[Feat](nereids) add transform rule MergePercentileToArray #34313
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 187124 ms |
Contributor
|
wait for percentileArray perf opt |
feiniaofeiafei
force-pushed
the
MergePercentileToArray
branch
from
e97e3ff to
64401c7
Compare
Contributor
Author
|
run buildall |
feiniaofeiafei
force-pushed
the
MergePercentileToArray
branch
from
64401c7 to
690d4d1
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41112 ms |
Contributor
Author
|
run xcloud_p1 |
Contributor
|
run cloud_p1 |
Contributor
|
run performance |
TPC-H: Total hot run time: 41695 ms |
TPC-DS: Total hot run time: 169184 ms |
ClickBench: Total hot run time: 30.65 s |
morrySnow
approved these changes
Jun 3, 2024
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
924060929
approved these changes
Jun 3, 2024
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Jun 3, 2024
MergePercentileToArray is to perform a transformation in this case: select ss_item_sk, percentile(ss_quantity,0.9), percentile(ss_quantity,0.6), percentile(ss_quantity,0.3) from store_sales group by ss_item_sk; ===> select ss_item_sk, percentile_array(ss_quantity,[0.3,0.6,0.9]) from store_sales group by ss_item_sk;
dataroaring
pushed a commit
that referenced
this pull request
Jun 4, 2024
MergePercentileToArray is to perform a transformation in this case: select ss_item_sk, percentile(ss_quantity,0.9), percentile(ss_quantity,0.6), percentile(ss_quantity,0.3) from store_sales group by ss_item_sk; ===> select ss_item_sk, percentile_array(ss_quantity,[0.3,0.6,0.9]) from store_sales group by ss_item_sk;
morrySnow
pushed a commit
that referenced
this pull request
Jun 4, 2024
cherry-pick #34313 to branch-2.1 MergePercentileToArray is to perform a transformation in this case: select ss_item_sk, percentile(ss_quantity,0.9), percentile(ss_quantity,0.6), percentile(ss_quantity,0.3) from store_sales group by ss_item_sk; ==> select ss_item_sk, percentile_array(ss_quantity,[0.3,0.6,0.9]) from store_sales group by ss_item_sk;
seawinde
pushed a commit
to seawinde/doris
that referenced
this pull request
Jun 5, 2024
MergePercentileToArray is to perform a transformation in this case: select ss_item_sk, percentile(ss_quantity,0.9), percentile(ss_quantity,0.6), percentile(ss_quantity,0.3) from store_sales group by ss_item_sk; ===> select ss_item_sk, percentile_array(ss_quantity,[0.3,0.6,0.9]) from store_sales group by ss_item_sk;
16 tasks
starocean999
pushed a commit
that referenced
this pull request
Dec 2, 2024
…on (#44783) Related PR: #34313 Problem Summary The original PR did not handle the following scenario: ```sql SELECT SUM(a), PERCENTILE(pk, 0.1) AS c1, PERCENTILE(pk, 0.1) AS c2, PERCENTILE(pk, 0.4) AS c3 FROM test_merge_percentile; ``` In this case, the aggregate outputs include two identical functions (PERCENTILE(pk, 0.1)). When constructing the LogicalProject, a map was used where the key is the child of an Alias and the value is the Alias itself. However, this approach loses information when two Aliases share the same child. This PR modifies the map structure to use the child of an Alias as the key and a list of Alias objects as the value. This ensures that all Alias instances with the same child are preserved, resolving the issue of lost information in such cases.
github-actions bot
pushed a commit
that referenced
this pull request
Dec 2, 2024
…on (#44783) Related PR: #34313 Problem Summary The original PR did not handle the following scenario: ```sql SELECT SUM(a), PERCENTILE(pk, 0.1) AS c1, PERCENTILE(pk, 0.1) AS c2, PERCENTILE(pk, 0.4) AS c3 FROM test_merge_percentile; ``` In this case, the aggregate outputs include two identical functions (PERCENTILE(pk, 0.1)). When constructing the LogicalProject, a map was used where the key is the child of an Alias and the value is the Alias itself. However, this approach loses information when two Aliases share the same child. This PR modifies the map structure to use the child of an Alias as the key and a list of Alias objects as the value. This ensures that all Alias instances with the same child are preserved, resolving the issue of lost information in such cases.
github-actions bot
pushed a commit
that referenced
this pull request
Dec 2, 2024
…on (#44783) Related PR: #34313 Problem Summary The original PR did not handle the following scenario: ```sql SELECT SUM(a), PERCENTILE(pk, 0.1) AS c1, PERCENTILE(pk, 0.1) AS c2, PERCENTILE(pk, 0.4) AS c3 FROM test_merge_percentile; ``` In this case, the aggregate outputs include two identical functions (PERCENTILE(pk, 0.1)). When constructing the LogicalProject, a map was used where the key is the child of an Alias and the value is the Alias itself. However, this approach loses information when two Aliases share the same child. This PR modifies the map structure to use the child of an Alias as the key and a list of Alias objects as the value. This ensures that all Alias instances with the same child are preserved, resolving the issue of lost information in such cases.
16 tasks
morrySnow
pushed a commit
that referenced
this pull request
May 6, 2025
### What problem does this PR solve? Related PR: #34313 Problem Summary: When the second argument of percentile is not literal, will report error ：index out of range. This pr fix this bug, taking the second argument of percentile directly, and changing the arguments of percentilearray from array literal to array expression.
github-actions bot
pushed a commit
that referenced
this pull request
May 6, 2025
### What problem does this PR solve? Related PR: #34313 Problem Summary: When the second argument of percentile is not literal, will report error ：index out of range. This pr fix this bug, taking the second argument of percentile directly, and changing the arguments of percentilearray from array literal to array expression.
github-actions bot
pushed a commit
that referenced
this pull request
May 6, 2025
### What problem does this PR solve? Related PR: #34313 Problem Summary: When the second argument of percentile is not literal, will report error ：index out of range. This pr fix this bug, taking the second argument of percentile directly, and changing the arguments of percentilearray from array literal to array expression.
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
### What problem does this PR solve? Related PR: apache#34313 Problem Summary: When the second argument of percentile is not literal, will report error ：index out of range. This pr fix this bug, taking the second argument of percentile directly, and changing the arguments of percentilearray from array literal to array expression.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MergePercentileToArray is to perform a transformation in this case:
select ss_item_sk, percentile(ss_quantity,0.9), percentile(ss_quantity,0.6), percentile(ss_quantity,0.3) from store_sales group by ss_item_sk;
=》
select ss_item_sk, percentile_array(ss_quantity,[0.3,0.6,0.9]) from store_sales group by ss_item_sk;