Skip to content

[enhancement](Nereids): optimize GroupExpressionMatching #26130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jackwener merged 1 commit into from
Oct 31, 2023
Merged

[enhancement](Nereids): optimize GroupExpressionMatching #26130

jackwener merged 1 commit into from
Oct 31, 2023

Conversation

@jackwener
Copy link
Member

@jackwener jackwener commented Oct 30, 2023 •
edited
Loading

Proposed changes

pattern can't be SubTreePattern in CBO phase.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@jackwener jackwener force-pushed the groupexpression_matching branch from d01fc9e to 4b04585 Compare October 30, 2023 14:51
@jackwener
Copy link
Member Author

jackwener commented Oct 30, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Oct 30, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.29 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162187686 Bytes

@jackwener
Copy link
Member Author

jackwener commented Oct 30, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Oct 30, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.08 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17161953442 Bytes

@jackwener jackwener force-pushed the groupexpression_matching branch from 767ec08 to 2b2d48d Compare October 31, 2023 02:39
@jackwener
Copy link
Member Author

jackwener commented Oct 31, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Oct 31, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.95 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162381413 Bytes

@jackwener jackwener force-pushed the groupexpression_matching branch from 2b2d48d to 0bc72b3 Compare October 31, 2023 06:03
@jackwener
Copy link
Member Author

jackwener commented Oct 31, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Oct 31, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.16 seconds
stream load tsv: 549 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162117566 Bytes

@github-actions
Copy link
Contributor

github-actions bot commented Oct 31, 2023

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 31, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Oct 31, 2023

PR approved by at least one committer and no changes requested.

@jackwener jackwener merged commit 19122b5 into apache:master Oct 31, 2023
@jackwener jackwener deleted the groupexpression_matching branch October 31, 2023 07:47
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Oct 31, 2023
@morrySnow
Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apa…
947ca3f 
…che#26130)"

This reverts commit 19122b5.
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Oct 31, 2023
@morrySnow
Revert "Revert "[enhancement](Nereids): optimize GroupExpressionMatch…
b49ae4f 
…ing (apache#26130)""

This reverts commit 947ca3f.
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Oct 31, 2023
@morrySnow
Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apa…
fa163fc 
…che#26130)"

This reverts commit 19122b5.
@morrySnow morrySnow mentioned this pull request Oct 31, 2023
Merged
morrySnow added a commit that referenced this pull request Nov 1, 2023
@morrySnow
[fix](compile) fe compile failed when generate doc and FE UT failed (#…
18dabe7 
…26164)

1. FE could not compile because below error. Intro by PR #25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - #25951
> - #26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR #25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (#26145)"
> This reverts commit 8d7abf6.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (#26130)"
> This reverts commit 19122b5.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (#26084)"
> This reverts commit 0d956e9.
dutyu pushed a commit to dutyu/doris that referenced this pull request Nov 4, 2023
@jackwener
[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)
f6e0368 
1. pattern can't be SubTreePattern in CBO phase.
2. optimize getInputSlotExprId()
dutyu pushed a commit to dutyu/doris that referenced this pull request Nov 4, 2023
@morrySnow
[fix](compile) fe compile failed when generate doc and FE UT failed (a…
0ee2870 
…pache#26164)

1. FE could not compile because below error. Intro by PR apache#25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - apache#25951
> - apache#26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR apache#25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (apache#26145)"
> This reverts commit 8d7abf6.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)"
> This reverts commit 19122b5.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (apache#26084)"
> This reverts commit 0d956e9.
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 13, 2023
@jackwener @seawinde
[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)
fc531c5 
1. pattern can't be SubTreePattern in CBO phase.
2. optimize getInputSlotExprId()
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 13, 2023
@morrySnow @seawinde
[fix](compile) fe compile failed when generate doc and FE UT failed (a…
4677a82 
…pache#26164)

1. FE could not compile because below error. Intro by PR apache#25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - apache#25951
> - apache#26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR apache#25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (apache#26145)"
> This reverts commit 8d7abf6.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)"
> This reverts commit 19122b5.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (apache#26084)"
> This reverts commit 0d956e9.
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@jackwener
[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)
81d4b4d 
1. pattern can't be SubTreePattern in CBO phase.
2. optimize getInputSlotExprId()
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@morrySnow
[fix](compile) fe compile failed when generate doc and FE UT failed (a…
ae8163e 
…pache#26164)

1. FE could not compile because below error. Intro by PR apache#25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - apache#25951
> - apache#26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR apache#25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (apache#26145)"
> This reverts commit 8d7abf6.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)"
> This reverts commit 19122b5.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (apache#26084)"
> This reverts commit 0d956e9.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

+1 more reviewer

@keanji-x keanji-x keanji-x approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jackwener @doris-robot @keanji-x @starocean999