Skip to content

[Feature](Nereids)support select into outfile #21197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 28 commits into from
Jul 13, 2023
Merged

[Feature](Nereids)support select into outfile #21197

morrySnow merged 28 commits into from
Jul 13, 2023

Conversation

@sohardforaname
Copy link
Contributor

@sohardforaname sohardforaname commented Jun 26, 2023

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@sohardforaname
Copy link
Contributor Author

sohardforaname commented Jun 26, 2023

run buildall

@sohardforaname
Copy link
Contributor Author

sohardforaname commented Jun 27, 2023

run p0

@sohardforaname
Copy link
Contributor Author

sohardforaname commented Jun 27, 2023

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 27, 2023

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 40.36 seconds
stream load tsv: 459 seconds loaded 74807831229 Bytes, about 155 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 64.9 seconds inserted 10000000 Rows, about 154K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230627154814_clickbench_pr_168935.html

@sohardforaname
Copy link
Contributor Author

sohardforaname commented Jun 28, 2023

run buildall

1 similar comment
@sohardforaname
Copy link
Contributor Author

sohardforaname commented Jun 29, 2023

run buildall

@morrySnow morrySnow added the dev/2.0.0 2.0.0 release label Jun 30, 2023
@sohardforaname
Copy link
Contributor Author

sohardforaname commented Jul 3, 2023

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 3, 2023

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 40.2 seconds
stream load tsv: 456 seconds loaded 74807831229 Bytes, about 156 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 65.9 seconds inserted 10000000 Rows, about 151K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230703043144_clickbench_pr_171145.html

@morrySnow morrySnow added not-merge/2.0 do not merge into 2.0 branch and removed dev/2.0.0 2.0.0 release labels Jul 3, 2023
@morrySnow morrySnow changed the title [Feature](Nereids)supoort select into outfile [Feature](Nereids)support select into outfile Jul 3, 2023
@morrySnow morrySnow added dev/2.0.0 2.0.0 release and removed not-merge/2.0 do not merge into 2.0 branch labels Jul 6, 2023
morrySnow
morrySnow reviewed Jul 6, 2023
View reviewed changes
@sohardforaname
Copy link
Contributor Author

sohardforaname commented Jul 9, 2023

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 10, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51 seconds
stream load tsv: 504 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17162472267 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230710114613_clickbench_pr_175545.html

@sohardforaname
Copy link
Contributor Author

sohardforaname commented Jul 12, 2023

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 12, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 52.91 seconds
stream load tsv: 505 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17162066075 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712171059_clickbench_pr_177087.html

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 12, 2023

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.2 seconds
stream load tsv: 448 seconds loaded 74807831229 Bytes, about 159 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 26.5 seconds inserted 10000000 Rows, about 377K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712103043_clickbench_pr_177086.html

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 13, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jul 13, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 13, 2023

PR approved by anyone and no changes requested.

keanji-x
keanji-x approved these changes Jul 13, 2023
View reviewed changes
Copy link
Contributor

@keanji-x keanji-x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morrySnow morrySnow merged commit d4bdd67 into apache:master Jul 13, 2023
@xiaokang xiaokang added dev/2.0.0-merged and removed dev/2.0.0 2.0.0 release labels Jul 13, 2023
xiaokang pushed a commit that referenced this pull request Jul 13, 2023
@sohardforaname @xiaokang
[Feature](Nereids) support select into outfile (#21197)
02248ea
BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Jul 14, 2023
@sohardforaname @BiteTheDDDDt
[Feature](Nereids) support select into outfile (apache#21197)
46f1a60
@BePPPower BePPPower mentioned this pull request Aug 23, 2023
Merged
morningman pushed a commit that referenced this pull request Aug 25, 2023
@BePPPower
[fix](Outfile/Nereids) fix that csv_with_names and `csv_with_names_…
1809451 
…and_types` file format could not be exported on nereids (#23387)

This problem is casued by #21197

Fixed an issue that `csv_with_names` and `csv_with_names_and_types` file format could not be exported on nereids optimizer when using `select...into outfile`.
xiaokang pushed a commit that referenced this pull request Aug 25, 2023
@BePPPower @xiaokang
[fix](Outfile/Nereids) fix that csv_with_names and `csv_with_names_…
fa3cd04 
…and_types` file format could not be exported on nereids (#23387)

This problem is casued by #21197

Fixed an issue that `csv_with_names` and `csv_with_names_and_types` file format could not be exported on nereids optimizer when using `select...into outfile`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@keanji-x keanji-x keanji-x approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. area/nereids dev/2.0.0-merged kind/test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sohardforaname @hello-stephen @keanji-x @morrySnow @xiaokang