Skip to content

[refactor](Nereids) mv top-n two phase read rule from post processor to rewriter #22487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jackwener merged 1 commit into from
Aug 3, 2023
Merged

[refactor](Nereids) mv top-n two phase read rule from post processor to rewriter #22487

jackwener merged 1 commit into from
Aug 3, 2023

Conversation

@morrySnow
Copy link
Contributor

@morrySnow morrySnow commented Aug 2, 2023

Proposed changes

Issue Number: close #xxx

use three new plan node to represent defer materialize of TopN.
Example:

-- SQL
select * from t1 order by c1 limit 10;

-- PLAN
+------------------------------------------+
| Explain String                           |
+------------------------------------------+
| PhysicalDeferMaterializeResultSink       |
| --PhysicalDeferMaterializeTopN           |
| ----PhysicalDistribute                   |
| ------PhysicalDeferMaterializeTopN       |
| --------PhysicalDeferMaterializeOlapScan |
+------------------------------------------+

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morrySnow morrySnow changed the title [refactor] mv top-n two phase read rule from post processor to rewriter [refactor](Nereids) mv top-n two phase read rule from post processor to rewriter Aug 2, 2023
@morrySnow
Copy link
Contributor Author

morrySnow commented Aug 2, 2023

run buildall

@morrySnow
Copy link
Contributor Author

morrySnow commented Aug 2, 2023

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 2, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.71 seconds
stream load tsv: 514 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162257663 Bytes

@morrySnow
Copy link
Contributor Author

morrySnow commented Aug 2, 2023

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 2, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.6 seconds
stream load tsv: 517 seconds loaded 74807831229 Bytes, about 137 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.6 seconds inserted 10000000 Rows, about 337K ops/s
storage size: 17162131516 Bytes

@eldenmoon
Copy link
Member

eldenmoon commented Aug 2, 2023

run p0

eldenmoon
eldenmoon approved these changes Aug 2, 2023
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 2, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2023

PR approved by anyone and no changes requested.

@jackwener jackwener merged commit 3961b8d into apache:master Aug 3, 2023
@morrySnow morrySnow deleted the topn_phase1 branch August 11, 2023 10:47
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Aug 11, 2023
@morrySnow
[refactor](Nereids) mv top-n two phase read rule from post processor …
3c4be62 
…to rewriter (apache#22487)

use three new plan node to represent defer materialize of TopN.
Example:

```
-- SQL
select * from t1 order by c1 limit 10;

-- PLAN
+------------------------------------------+
| Explain String                           |
+------------------------------------------+
| PhysicalDeferMaterializeResultSink       |
| --PhysicalDeferMaterializeTopN           |
| ----PhysicalDistribute                   |
| ------PhysicalDeferMaterializeTopN       |
| --------PhysicalDeferMaterializeOlapScan |
+------------------------------------------+
```
@morrySnow morrySnow mentioned this pull request Aug 11, 2023
Merged
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Aug 11, 2023
@morrySnow
[refactor](Nereids) mv top-n two phase read rule from post processor …
1aaa0e3 
…to rewriter (apache#22487)

use three new plan node to represent defer materialize of TopN.
Example:

```
-- SQL
select * from t1 order by c1 limit 10;

-- PLAN
+------------------------------------------+
| Explain String                           |
+------------------------------------------+
| PhysicalDeferMaterializeResultSink       |
| --PhysicalDeferMaterializeTopN           |
| ----PhysicalDistribute                   |
| ------PhysicalDeferMaterializeTopN       |
| --------PhysicalDeferMaterializeOlapScan |
+------------------------------------------+
```
xiaokang pushed a commit that referenced this pull request Aug 11, 2023
@morrySnow
[refactor](Nereids) mv top-n two phase read rule from post processor …
4ef0f53 
…to rewriter (#22487) (#22894)

use three new plan node to represent defer materialize of TopN.
Example:

```
-- SQL
select * from t1 order by c1 limit 10;

-- PLAN
+------------------------------------------+
| Explain String                           |
+------------------------------------------+
| PhysicalDeferMaterializeResultSink       |
| --PhysicalDeferMaterializeTopN           |
| ----PhysicalDistribute                   |
| ------PhysicalDeferMaterializeTopN       |
| --------PhysicalDeferMaterializeOlapScan |
+------------------------------------------+
```
@morrySnow morrySnow mentioned this pull request Mar 6, 2025
Merged
16 tasks
morrySnow added a commit that referenced this pull request Mar 20, 2025
@morrySnow
[opt](Nereids) support more pattern in DeferMaterializeTopNResult (#4…
947463d 
…8747)

### What problem does this PR solve?

Related PR: #47661, #22487

Problem Summary:

after this PR we support:

1. TopN(Scan)
2. Project(TopN(Project(Scan)))
3. TopN(Filter(Scan))
4. TopN(Project(Filter(Scan)))
5. Project(TopN(Project(Filter(Scan))))
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@morrySnow @koarz
[opt](Nereids) support more pattern in DeferMaterializeTopNResult (ap…
bd2a11f 
…ache#48747)

### What problem does this PR solve?

Related PR: apache#47661, apache#22487

Problem Summary:

after this PR we support:

1. TopN(Scan)
2. Project(TopN(Project(Scan)))
3. TopN(Filter(Scan))
4. TopN(Project(Filter(Scan)))
5. Project(TopN(Project(Filter(Scan))))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jackwener jackwener jackwener approved these changes

@eldenmoon eldenmoon eldenmoon approved these changes

@starocean999 starocean999 Awaiting requested review from starocean999

@englefly englefly Awaiting requested review from englefly

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.1-merged merge_conflict reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morrySnow @hello-stephen @eldenmoon @jackwener @xiaokang