[refactor](join) improve join node output when build table rows is 0 #23713

zhangstar333 · 2023-08-31T11:22:30Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-08-31T11:30:56Z

clang-tidy review says "All clean, LGTM! 👍"

wm1581066

need some test cases

zhangstar333 · 2023-09-01T01:58:30Z

run buildall

hello-stephen · 2023-09-01T02:48:21Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.72 seconds
stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162120915 Bytes

be/src/vec/exec/join/vhash_join_node.cpp

be/src/vec/exec/join/vhash_join_node.h

zhangstar333 · 2023-09-01T03:56:32Z

run buildall

github-actions · 2023-09-01T04:04:48Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-09-01T04:38:55Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.77 seconds
stream load tsv: 547 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162068386 Bytes

zhangstar333 · 2023-09-01T10:48:26Z

run buildall

github-actions · 2023-09-01T10:56:15Z

clang-tidy review says "All clean, LGTM! 👍"

zhangstar333 · 2023-09-01T11:26:16Z

run buildall

hello-stephen · 2023-09-01T11:30:41Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.4 seconds
stream load tsv: 534 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17161946535 Bytes

github-actions · 2023-09-01T11:31:58Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-01T11:33:11Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-09-01T12:08:55Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.66 seconds
stream load tsv: 533 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162128768 Bytes

zhangstar333 · 2023-09-01T14:13:41Z

run buildall

hello-stephen · 2023-09-01T14:58:10Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.99 seconds
stream load tsv: 529 seconds loaded 74807831229 Bytes, about 134 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17161935775 Bytes

zhangstar333 · 2023-09-02T08:10:37Z

run buildall

zhangstar333 · 2023-09-02T08:32:20Z

run buildall

github-actions · 2023-09-02T08:39:47Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-09-02T08:51:14Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.26 seconds
stream load tsv: 530 seconds loaded 74807831229 Bytes, about 134 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162418908 Bytes

hello-stephen · 2023-09-02T09:16:35Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.99 seconds
stream load tsv: 533 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162308770 Bytes

zhangstar333 · 2023-09-02T12:39:27Z

run buildall

github-actions · 2023-09-02T12:46:07Z

clang-tidy review says "All clean, LGTM! 👍"

zhangstar333 · 2023-09-02T13:20:23Z

run buildall

be/src/service/backend_service.cpp

zhangstar333 · 2023-09-03T04:35:42Z

run buildall

github-actions · 2023-09-03T04:43:29Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-09-03T05:07:05Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.79 seconds
stream load tsv: 529 seconds loaded 74807831229 Bytes, about 134 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.7 seconds inserted 10000000 Rows, about 336K ops/s
storage size: 17162118514 Bytes

hello-stephen · 2023-09-03T05:09:20Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.73 seconds
stream load tsv: 542 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17161999731 Bytes

HappenLee

LGTM

github-actions · 2023-09-04T01:42:42Z

PR approved by at least one committer and no changes requested.

wangbo

LGTM

yiguolei

LGTM

…23713)

…ws is 0 (#23713)" This reverts commit 32f5749.

…pache#23713)

…23974 (#24118)

…pache#23713)

…llable (#25626) * [refactor](join) improve join node output when build table rows is 0 (#23713) * [Enhancement](filter) support only min/max runtime filter in BE (#25290) this PR #25193 have achieve about FE. eg: select count() from lineorder join supplier on lo_partkey < s_suppkey; will have a max filter after build hash table , so could use it to filter probe table data. * [bug](cherry-pick) fix bug of converting outer join probe block to nullable

wm1581066 suggested changes Aug 31, 2023

View reviewed changes

yiguolei added dev/2.0.2 usercase Important user case type label labels Sep 1, 2023

HappenLee reviewed Sep 1, 2023

View reviewed changes

be/src/vec/exec/join/vhash_join_node.cpp Show resolved Hide resolved

HappenLee reviewed Sep 1, 2023

View reviewed changes

be/src/vec/exec/join/vhash_join_node.h Show resolved Hide resolved

zhangstar333 force-pushed the left_outer_join branch from 99380fa to 9603d82 Compare September 2, 2023 13:17

zhangstar333 commented Sep 2, 2023

View reviewed changes

be/src/service/backend_service.cpp Outdated Show resolved Hide resolved

fix clear block

52b6cf2

zhangstar333 force-pushed the left_outer_join branch from 9603d82 to 52b6cf2 Compare September 3, 2023 04:35

HappenLee approved these changes Sep 4, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 4, 2023

wangbo approved these changes Sep 4, 2023

View reviewed changes

yiguolei approved these changes Sep 4, 2023

View reviewed changes

yiguolei merged commit 9da9409 into apache:master Sep 4, 2023

xiaokang added dev/2.0.2-merged and removed dev/2.0.2 labels Sep 4, 2023

xiaokang pushed a commit that referenced this pull request Sep 4, 2023

[refactor](join) improve join node output when build table rows is 0 (#…

32f5749

…23713)

xiaokang added a commit that referenced this pull request Sep 4, 2023

Revert "[refactor](join) improve join node output when build table ro…

33ebe47

…ws is 0 (#23713)" This reverts commit 32f5749.

xiaokang added merge_conflict dev/2.0.2 and removed dev/2.0.2-merged labels Sep 5, 2023

zhangstar333 mentioned this pull request Sep 6, 2023

[bug](join) fix outer join not add tuple is null column when build rows is 0 #23974

Merged

BiteTheDDDDt mentioned this pull request Sep 8, 2023

[Bug](join) fix query failed on short_circuit_for_probe_and_additional_data #24109

Closed

zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Sep 8, 2023

[refactor](join) improve join node output when build table rows is 0 (a…

5c96e1a

…pache#23713)

zhangstar333 mentioned this pull request Sep 8, 2023

[pick](branch-2.0) cherry-pick two pr from master to branch2.0 #24118

Merged

xiaokang pushed a commit that referenced this pull request Sep 8, 2023

[pick](branch-2.0) cherry-pick two pr from master to branch2.0 #23713 #…

454e2c0

…23974 (#24118)

xiaokang added dev/2.0.2-merged and removed dev/2.0.2 labels Sep 8, 2023

Gabriel39 mentioned this pull request Oct 12, 2023

[pipelineX](pick) pick PRs from pipeline #25340

Merged

zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Oct 13, 2023

[refactor](join) improve join node output when build table rows is 0 (a…

78ecf7c

…pache#23713)

zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Oct 19, 2023

[refactor](join) improve join node output when build table rows is 0 (a…

a021634

…pache#23713)

[refactor](join) improve join node output when build table rows is 0 #23713

[refactor](join) improve join node output when build table rows is 0 #23713

Uh oh!

Conversation

zhangstar333 commented Aug 31, 2023

Proposed changes

Further comments

Uh oh!

github-actions bot commented Aug 31, 2023

Uh oh!

wm1581066 left a comment

Choose a reason for hiding this comment

Uh oh!

zhangstar333 commented Sep 1, 2023

Uh oh!

hello-stephen commented Sep 1, 2023

Uh oh!

Uh oh!

Uh oh!

zhangstar333 commented Sep 1, 2023

Uh oh!

github-actions bot commented Sep 1, 2023

Uh oh!

hello-stephen commented Sep 1, 2023

Uh oh!

zhangstar333 commented Sep 1, 2023

Uh oh!

github-actions bot commented Sep 1, 2023

Uh oh!

zhangstar333 commented Sep 1, 2023

Uh oh!

hello-stephen commented Sep 1, 2023

Uh oh!

github-actions bot commented Sep 1, 2023

Uh oh!

github-actions bot commented Sep 1, 2023

Uh oh!

hello-stephen commented Sep 1, 2023

Uh oh!

zhangstar333 commented Sep 1, 2023

Uh oh!

hello-stephen commented Sep 1, 2023

Uh oh!

zhangstar333 commented Sep 2, 2023

Uh oh!

zhangstar333 commented Sep 2, 2023

Uh oh!

github-actions bot commented Sep 2, 2023

Uh oh!

hello-stephen commented Sep 2, 2023

Uh oh!

hello-stephen commented Sep 2, 2023

Uh oh!

zhangstar333 commented Sep 2, 2023

Uh oh!

github-actions bot commented Sep 2, 2023

Uh oh!

zhangstar333 commented Sep 2, 2023

Uh oh!

Uh oh!

zhangstar333 commented Sep 3, 2023

Uh oh!

github-actions bot commented Sep 3, 2023

Uh oh!

hello-stephen commented Sep 3, 2023

Uh oh!

hello-stephen commented Sep 3, 2023

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 4, 2023

Uh oh!

wangbo left a comment

Choose a reason for hiding this comment

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers