[feat](pipelinex) support parallel scan on pipeline x engine #29070

mrhhsg · 2023-12-26T09:26:57Z

Proposed changes

Following of #28967

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-12-26T09:34:43Z

clang-tidy review says "All clean, LGTM! 👍"

mrhhsg · 2023-12-26T09:54:31Z

run buildall

doris-robot · 2023-12-26T10:24:08Z

TeamCity be ut coverage result:
Function Coverage: 36.57% (8557/23399)
Line Coverage: 28.63% (69607/243095)
Region Coverage: 27.63% (35979/130221)
Branch Coverage: 24.36% (18392/75510)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d776fcc974beede0a5808a6f9a58016eddc65a7d_d776fcc974beede0a5808a6f9a58016eddc65a7d/report/index.html

doris-robot · 2023-12-26T10:24:58Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit d776fcc974beede0a5808a6f9a58016eddc65a7d, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4755	4457	4480	4457
q2	398	155	159	155
q3	1500	1314	1226	1226
q4	1147	969	930	930
q5	3168	3159	3166	3159
q6	259	135	134	134
q7	1017	506	487	487
q8	2261	2277	2245	2245
q9	6718	6661	6703	6661
q10	3215	3291	3297	3291
q11	332	210	196	196
q12	358	210	209	209
q13	4527	3794	3820	3794
q14	250	213	217	213
q15	571	516	527	516
q16	446	382	387	382
q17	1050	799	640	640
q18	7156	6831	6924	6831
q19	1587	1602	1602	1602
q20	533	313	299	299
q21	3245	2812	2778	2778
q22	370	304	310	304
Total cold run time: 44863 ms
Total hot run time: 40509 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4417	4432	4401	4401
q2	275	165	177	165
q3	3522	3507	3502	3502
q4	2432	2416	2425	2416
q5	5735	5729	5726	5726
q6	247	126	126	126
q7	2387	1886	1850	1850
q8	3604	3617	3606	3606
q9	9048	9015	9055	9015
q10	3930	4000	4015	4000
q11	494	371	388	371
q12	776	594	610	594
q13	4317	3550	3550	3550
q14	286	251	247	247
q15	571	521	521	521
q16	506	448	457	448
q17	1995	1947	1944	1944
q18	8630	8187	8229	8187
q19	1843	1815	1814	1814
q20	2260	1964	1947	1947
q21	6636	6281	6264	6264
q22	560	496	476	476
Total cold run time: 64471 ms
Total hot run time: 61170 ms

doris-robot · 2023-12-26T10:29:37Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.74 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 20.7 seconds inserted 10000000 Rows, about 483K ops/s
storage size: 17183979759 Bytes

mrhhsg · 2023-12-27T10:24:27Z

run buildall

github-actions · 2023-12-27T10:31:14Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-12-27T10:47:25Z

TeamCity be ut coverage result:
Function Coverage: 36.59% (8581/23454)
Line Coverage: 28.64% (69740/243489)
Region Coverage: 27.61% (36049/130546)
Branch Coverage: 24.33% (18412/75680)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8fc69968339a1fa121f1f259ba914dd2bc86ccd9_8fc69968339a1fa121f1f259ba914dd2bc86ccd9/report/index.html

doris-robot · 2023-12-27T11:04:42Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.45 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17183661357 Bytes

doris-robot · 2023-12-27T12:06:28Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 8fc69968339a1fa121f1f259ba914dd2bc86ccd9, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5046	4651	4655	4651
q2	364	151	159	151
q3	1464	1381	1225	1225
q4	1129	945	879	879
q5	3165	3143	3152	3143
q6	249	129	127	127
q7	991	484	504	484
q8	2254	2234	2248	2234
q9	6716	6727	6716	6716
q10	3179	3269	3280	3269
q11	334	209	198	198
q12	345	213	211	211
q13	4179	3413	3416	3413
q14	244	208	209	208
q15	567	514	534	514
q16	449	376	379	376
q17	1031	787	623	623
q18	7067	6838	6966	6838
q19	1629	1630	1652	1630
q20	619	302	306	302
q21	3184	2656	2682	2656
q22	368	302	308	302
Total cold run time: 44573 ms
Total hot run time: 40150 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4580	4580	4566	4566
q2	269	168	168	168
q3	3377	3358	3357	3357
q4	2235	2215	2215	2215
q5	5731	5725	5719	5719
q6	240	118	118	118
q7	2375	1854	1849	1849
q8	3622	3617	3615	3615
q9	9029	8984	8969	8969
q10	3805	3899	3896	3896
q11	495	376	370	370
q12	757	585	592	585
q13	3897	3166	3200	3166
q14	284	245	249	245
q15	589	527	518	518
q16	494	427	446	427
q17	1964	1962	1951	1951
q18	8694	8254	8303	8254
q19	1767	1760	1757	1757
q20	2244	1916	1919	1916
q21	6123	5819	5786	5786
q22	543	466	459	459
Total cold run time: 63114 ms
Total hot run time: 59906 ms

mrhhsg · 2023-12-28T10:06:02Z

run buildall

github-actions · 2023-12-28T10:13:10Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-12-28T10:41:36Z

TeamCity be ut coverage result:
Function Coverage: 36.58% (8587/23477)
Line Coverage: 28.66% (69856/243760)
Region Coverage: 27.65% (36146/130715)
Branch Coverage: 24.36% (18470/75808)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bc6fbeabb70e566f9928a561c206201dc3ace8c0_bc6fbeabb70e566f9928a561c206201dc3ace8c0/report/index.html

doris-robot · 2023-12-28T11:08:17Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit bc6fbeabb70e566f9928a561c206201dc3ace8c0, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5447	5123	5168	5123
q2	408	190	158	158
q3	1452	1242	1155	1155
q4	1088	881	832	832
q5	3110	3074	3145	3074
q6	236	140	137	137
q7	971	563	542	542
q8	2164	2270	2273	2270
q9	6877	6844	6854	6844
q10	3212	3176	3131	3131
q11	351	224	219	219
q12	388	243	242	242
q13	4434	3640	3638	3638
q14	255	233	224	224
q15	611	571	568	568
q16	468	397	408	397
q17	1050	581	601	581
q18	7090	6790	6728	6728
q19	1660	1581	1467	1467
q20	587	363	362	362
q21	2851	2552	2527	2527
q22	407	314	342	314
Total cold run time: 45117 ms
Total hot run time: 40533 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5152	5128	5092	5092
q2	346	244	262	244
q3	3386	3326	3305	3305
q4	2175	2032	1999	1999
q5	5936	5951	5972	5951
q6	235	136	129	129
q7	2375	1930	1976	1930
q8	3548	3674	3678	3674
q9	9091	9022	9011	9011
q10	3878	3943	3928	3928
q11	595	488	494	488
q12	813	674	672	672
q13	3917	3228	3182	3182
q14	296	289	277	277
q15	626	555	571	555
q16	563	513	499	499
q17	2014	1804	1829	1804
q18	8858	8387	8400	8387
q19	1775	1732	1715	1715
q20	2266	2013	1986	1986
q21	5785	5427	5307	5307
q22	545	477	477	477
Total cold run time: 64175 ms
Total hot run time: 60612 ms

doris-robot · 2023-12-28T11:21:17Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.14 seconds
stream load tsv: 572 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17184269163 Bytes

github-actions · 2023-12-28T13:24:44Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-12-28T13:24:47Z

PR approved by anyone and no changes requested.

yiguolei

lgtm

…29070) * [feat](pipelinex) support parallel scan on pipeline x engine * make parallel scan be independent of shared scan

mrhhsg added 2 commits December 28, 2023 18:00

[feat](pipelinex) support parallel scan on pipeline x engine

8ca0119

make parallel scan be independent of shared scan

bc6fbea

mrhhsg force-pushed the parallel_scan_on_x branch from 8fc6996 to bc6fbea Compare December 28, 2023 10:05

Gabriel39 approved these changes Dec 28, 2023

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Dec 28, 2023

yiguolei approved these changes Dec 28, 2023

View reviewed changes

yiguolei merged commit ffd178f into apache:master Dec 28, 2023

[feat](pipelinex) support parallel scan on pipeline x engine #29070

[feat](pipelinex) support parallel scan on pipeline x engine #29070

Uh oh!

Conversation

mrhhsg commented Dec 26, 2023

Proposed changes

Further comments

Uh oh!

github-actions bot commented Dec 26, 2023

Uh oh!

mrhhsg commented Dec 26, 2023

Uh oh!

doris-robot commented Dec 26, 2023

Uh oh!

doris-robot commented Dec 26, 2023

Uh oh!

doris-robot commented Dec 26, 2023

Uh oh!

mrhhsg commented Dec 27, 2023

Uh oh!

github-actions bot commented Dec 27, 2023

Uh oh!

doris-robot commented Dec 27, 2023

Uh oh!

doris-robot commented Dec 27, 2023

Uh oh!

doris-robot commented Dec 27, 2023

Uh oh!

mrhhsg commented Dec 28, 2023

Uh oh!

github-actions bot commented Dec 28, 2023

Uh oh!

doris-robot commented Dec 28, 2023

Uh oh!

doris-robot commented Dec 28, 2023

Uh oh!

doris-robot commented Dec 28, 2023

Uh oh!

github-actions bot commented Dec 28, 2023

Uh oh!

github-actions bot commented Dec 28, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants