Skip to content

Conversation

@xzj7019
Copy link
Contributor

@xzj7019 xzj7019 commented Jul 15, 2024

Local shuffle depends on the operator node's isColocate flag, to decide the shuffle columns.
Current aggr node's isColocate true setting is too strict to support shuffle-colocate scenarios.
This pr releases the unnecessary restriction (only support table scan) and sets the flag true expectly.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xzj7019
Copy link
Contributor Author

xzj7019 commented Jul 15, 2024

run buildall

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 15, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40129 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7f20fe78415de675dd2b2a268256b33a6e141aea, data reload: false

------ Round 1 ----------------------------------
q1	18151	4486	4403	4403
q2	2575	208	194	194
q3	11862	1171	1130	1130
q4	11158	877	784	784
q5	8296	2735	2695	2695
q6	219	139	136	136
q7	955	592	592	592
q8	9227	2055	2088	2055
q9	8700	6568	6539	6539
q10	8661	3831	3751	3751
q11	466	233	234	233
q12	391	222	216	216
q13	18757	2986	3039	2986
q14	285	245	237	237
q15	534	483	473	473
q16	484	397	374	374
q17	959	670	667	667
q18	8063	7613	7462	7462
q19	7327	1522	1427	1427
q20	719	322	316	316
q21	4977	3175	3250	3175
q22	346	284	287	284
Total cold run time: 123112 ms
Total hot run time: 40129 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4327	4327	4306	4306
q2	377	267	267	267
q3	2984	2728	2744	2728
q4	1883	1560	1676	1560
q5	5308	5328	5315	5315
q6	220	132	139	132
q7	2125	1728	1650	1650
q8	3206	3316	3324	3316
q9	8440	8404	8426	8404
q10	3909	3740	3702	3702
q11	572	503	483	483
q12	762	596	609	596
q13	17529	2989	3025	2989
q14	294	265	283	265
q15	523	474	469	469
q16	465	421	420	420
q17	1790	1492	1463	1463
q18	7790	7528	7446	7446
q19	1669	1455	1481	1455
q20	2019	1769	1796	1769
q21	4875	4661	4543	4543
q22	598	505	516	505
Total cold run time: 71665 ms
Total hot run time: 53783 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172908 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7f20fe78415de675dd2b2a268256b33a6e141aea, data reload: false

query1	914	374	367	367
query2	6448	1871	1825	1825
query3	6654	206	220	206
query4	28457	17473	17181	17181
query5	4213	463	494	463
query6	290	167	169	167
query7	4635	292	279	279
query8	238	196	188	188
query9	8485	2371	2342	2342
query10	448	281	263	263
query11	10753	10181	10064	10064
query12	127	82	85	82
query13	1628	364	348	348
query14	10127	7754	7675	7675
query15	222	170	166	166
query16	7585	302	309	302
query17	1365	541	527	527
query18	1867	270	276	270
query19	188	147	151	147
query20	91	83	81	81
query21	207	141	125	125
query22	4235	4027	3977	3977
query23	33853	33350	33415	33350
query24	11361	2922	2916	2916
query25	637	371	369	369
query26	1594	148	145	145
query27	2896	267	275	267
query28	7401	1967	1961	1961
query29	1009	623	599	599
query30	279	150	153	150
query31	969	741	763	741
query32	94	51	57	51
query33	770	295	312	295
query34	938	478	487	478
query35	679	562	577	562
query36	1097	966	919	919
query37	141	82	79	79
query38	2884	2756	2727	2727
query39	848	833	816	816
query40	273	118	120	118
query41	49	45	47	45
query42	117	98	98	98
query43	498	465	470	465
query44	1295	748	726	726
query45	199	165	161	161
query46	1106	734	714	714
query47	1870	1752	1761	1752
query48	362	294	291	291
query49	1172	405	416	405
query50	789	389	396	389
query51	6919	6847	6774	6774
query52	112	88	97	88
query53	359	283	284	283
query54	1074	443	453	443
query55	75	76	73	73
query56	290	265	275	265
query57	1169	1061	1028	1028
query58	255	242	260	242
query59	2828	2537	2585	2537
query60	296	271	283	271
query61	94	94	93	93
query62	859	662	644	644
query63	320	286	296	286
query64	10629	2227	1661	1661
query65	3199	3245	3154	3154
query66	1389	339	347	339
query67	15407	14909	15023	14909
query68	4550	532	527	527
query69	471	327	325	325
query70	1207	1151	1095	1095
query71	400	292	299	292
query72	6948	5903	5903	5903
query73	750	323	323	323
query74	6034	5761	5678	5678
query75	3421	2734	2640	2640
query76	2473	957	1009	957
query77	485	319	301	301
query78	9576	8978	8934	8934
query79	2265	512	525	512
query80	2153	466	462	462
query81	580	224	223	223
query82	969	133	135	133
query83	296	166	162	162
query84	272	94	90	90
query85	1346	315	291	291
query86	472	317	303	303
query87	3288	3087	3125	3087
query88	4385	2446	2456	2446
query89	487	391	394	391
query90	1784	195	190	190
query91	127	114	167	114
query92	60	49	48	48
query93	2403	493	492	492
query94	1284	208	216	208
query95	407	315	322	315
query96	591	274	275	274
query97	3188	3020	3015	3015
query98	218	199	192	192
query99	1643	1261	1277	1261
Total cold run time: 284164 ms
Total hot run time: 172908 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7f20fe78415de675dd2b2a268256b33a6e141aea, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.67	0.07	0.07
query5	0.53	0.47	0.48
query6	1.13	0.74	0.73
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.54
query11	0.15	0.11	0.11
query12	0.15	0.13	0.13
query13	0.59	0.59	0.58
query14	0.77	0.77	0.78
query15	0.84	0.81	0.81
query16	0.37	0.35	0.36
query17	1.00	1.01	1.03
query18	0.22	0.21	0.22
query19	1.88	1.67	1.75
query20	0.02	0.01	0.00
query21	15.40	0.74	0.65
query22	3.95	6.79	2.62
query23	18.36	1.34	1.26
query24	2.10	0.23	0.23
query25	0.16	0.09	0.09
query26	0.29	0.21	0.20
query27	0.45	0.22	0.22
query28	13.28	1.02	0.99
query29	12.65	3.25	3.29
query30	0.25	0.05	0.06
query31	2.86	0.39	0.39
query32	3.27	0.48	0.48
query33	2.86	2.95	2.93
query34	17.13	4.36	4.35
query35	4.51	4.47	4.49
query36	0.66	0.46	0.47
query37	0.19	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.12	0.13
query41	0.09	0.06	0.04
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.77 s
Total hot run time: 31.31 s

dataroaring pushed a commit that referenced this pull request Jul 15, 2024
…#37789)

## Proposed changes

pick from #37788

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
@xzj7019
Copy link
Contributor Author

xzj7019 commented Jul 15, 2024

run p0

@xzj7019
Copy link
Contributor Author

xzj7019 commented Jul 15, 2024

run p0

@xzj7019 xzj7019 force-pushed the fix_colocate_flag_localshuffle_dep branch from 7f20fe7 to e7823ff Compare July 16, 2024 14:06
@xzj7019
Copy link
Contributor Author

xzj7019 commented Jul 16, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39539 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e7823ff965c8fe387860c164f90cdb141119ada5, data reload: false

------ Round 1 ----------------------------------
q1	17623	4362	4267	4267
q2	2017	190	184	184
q3	10472	1171	1046	1046
q4	10199	797	790	790
q5	7538	2666	2637	2637
q6	217	136	136	136
q7	943	596	604	596
q8	9209	2035	2067	2035
q9	8743	6534	6506	6506
q10	8808	3786	3756	3756
q11	479	239	235	235
q12	427	231	221	221
q13	17761	2982	2989	2982
q14	273	236	242	236
q15	530	498	502	498
q16	497	381	377	377
q17	959	608	688	608
q18	7864	7554	7398	7398
q19	4853	1400	1439	1400
q20	655	308	325	308
q21	4863	3034	3266	3034
q22	343	289	295	289
Total cold run time: 115273 ms
Total hot run time: 39539 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4368	4228	4256	4228
q2	383	263	276	263
q3	3012	2794	2925	2794
q4	2021	1733	1680	1680
q5	5617	5586	5528	5528
q6	222	133	134	133
q7	2179	1862	1828	1828
q8	3381	3386	3421	3386
q9	8779	8727	8883	8727
q10	4119	3952	3682	3682
q11	576	514	493	493
q12	830	641	644	641
q13	16935	3140	3163	3140
q14	310	292	285	285
q15	523	483	481	481
q16	481	456	455	455
q17	1814	1551	1526	1526
q18	8021	7907	7873	7873
q19	1733	1544	1586	1544
q20	2102	1849	1862	1849
q21	5130	4915	4877	4877
q22	626	502	533	502
Total cold run time: 73162 ms
Total hot run time: 55915 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173734 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e7823ff965c8fe387860c164f90cdb141119ada5, data reload: false

query1	916	363	357	357
query2	6463	1921	1891	1891
query3	6647	207	219	207
query4	28405	18369	17483	17483
query5	3660	483	472	472
query6	257	169	165	165
query7	4594	288	281	281
query8	258	194	195	194
query9	8556	2393	2369	2369
query10	449	298	281	281
query11	12460	10107	10084	10084
query12	119	87	82	82
query13	1647	373	367	367
query14	10173	7487	7679	7487
query15	211	170	164	164
query16	7301	330	320	320
query17	1601	564	535	535
query18	1582	287	281	281
query19	209	152	155	152
query20	93	82	87	82
query21	214	132	130	130
query22	4642	4318	4002	4002
query23	34276	33815	33549	33549
query24	10908	2970	2911	2911
query25	626	407	411	407
query26	740	151	158	151
query27	2218	284	283	283
query28	6178	2150	2065	2065
query29	917	615	638	615
query30	255	154	153	153
query31	980	790	780	780
query32	96	52	55	52
query33	628	303	301	301
query34	891	509	505	505
query35	681	595	569	569
query36	1132	978	989	978
query37	142	90	78	78
query38	2979	2838	2834	2834
query39	911	848	840	840
query40	206	122	116	116
query41	47	45	42	42
query42	116	100	100	100
query43	483	457	476	457
query44	1095	732	712	712
query45	195	159	158	158
query46	1080	751	759	751
query47	1884	1797	1815	1797
query48	371	291	284	284
query49	825	399	401	399
query50	784	399	388	388
query51	6859	6824	6697	6697
query52	106	91	97	91
query53	355	279	285	279
query54	840	438	436	436
query55	75	73	73	73
query56	279	258	267	258
query57	1134	1066	1052	1052
query58	235	270	262	262
query59	2779	2749	2599	2599
query60	314	277	275	275
query61	92	94	92	92
query62	776	655	644	644
query63	322	282	281	281
query64	8913	2200	1644	1644
query65	3172	3103	3108	3103
query66	787	335	322	322
query67	15766	14993	14926	14926
query68	5623	536	528	528
query69	722	493	384	384
query70	1210	1128	1106	1106
query71	441	274	272	272
query72	8180	5824	5634	5634
query73	765	330	322	322
query74	6201	5744	5660	5660
query75	3957	2716	2671	2671
query76	3338	945	944	944
query77	721	302	299	299
query78	9714	9124	8992	8992
query79	4092	529	517	517
query80	2596	475	480	475
query81	593	219	225	219
query82	1473	131	132	131
query83	301	169	161	161
query84	271	95	84	84
query85	1411	309	307	307
query86	423	325	308	308
query87	3288	3135	3155	3135
query88	4381	2392	2400	2392
query89	488	389	390	389
query90	1783	193	196	193
query91	128	99	104	99
query92	59	48	47	47
query93	5213	497	497	497
query94	1104	265	213	213
query95	411	318	318	318
query96	614	273	280	273
query97	3235	3024	3048	3024
query98	228	194	202	194
query99	1583	1306	1251	1251
Total cold run time: 289452 ms
Total hot run time: 173734 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e7823ff965c8fe387860c164f90cdb141119ada5, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.09	0.08
query5	0.50	0.48	0.50
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.56	0.55	0.56
query11	0.14	0.11	0.12
query12	0.14	0.12	0.13
query13	0.59	0.60	0.58
query14	0.77	0.78	0.77
query15	0.85	0.82	0.81
query16	0.36	0.36	0.34
query17	1.02	0.97	0.99
query18	0.22	0.22	0.22
query19	1.81	1.67	1.66
query20	0.02	0.01	0.01
query21	15.42	0.76	0.67
query22	4.74	6.63	1.85
query23	18.26	1.37	1.30
query24	2.11	0.23	0.23
query25	0.15	0.09	0.09
query26	0.30	0.20	0.21
query27	0.46	0.23	0.23
query28	13.27	1.03	1.00
query29	12.60	3.35	3.37
query30	0.25	0.06	0.05
query31	2.88	0.38	0.39
query32	3.27	0.46	0.47
query33	2.87	2.92	2.91
query34	17.08	4.37	4.36
query35	4.45	4.40	4.36
query36	0.64	0.46	0.47
query37	0.18	0.16	0.16
query38	0.15	0.16	0.14
query39	0.05	0.04	0.03
query40	0.15	0.13	0.11
query41	0.09	0.05	0.05
query42	0.06	0.06	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.19 s
Total hot run time: 30.56 s

@xzj7019
Copy link
Contributor Author

xzj7019 commented Jul 17, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39893 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e7823ff965c8fe387860c164f90cdb141119ada5, data reload: false

------ Round 1 ----------------------------------
q1	17949	4536	4385	4385
q2	2652	223	191	191
q3	11570	1191	1062	1062
q4	10343	881	836	836
q5	7962	2801	2640	2640
q6	227	142	140	140
q7	975	618	612	612
q8	9357	2046	2075	2046
q9	8648	6528	6540	6528
q10	8642	3814	3794	3794
q11	454	238	248	238
q12	391	224	218	218
q13	17751	2989	2975	2975
q14	271	238	237	237
q15	517	478	492	478
q16	501	387	374	374
q17	953	702	690	690
q18	8016	7474	7351	7351
q19	7913	1376	1312	1312
q20	700	332	323	323
q21	4851	3175	3232	3175
q22	351	293	288	288
Total cold run time: 120994 ms
Total hot run time: 39893 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4398	4220	4221	4220
q2	364	262	278	262
q3	3033	2741	2748	2741
q4	1891	1593	1615	1593
q5	5273	5281	5314	5281
q6	221	131	130	130
q7	2130	1758	1726	1726
q8	3218	3319	3286	3286
q9	8404	8367	8377	8367
q10	3873	3757	3738	3738
q11	594	517	484	484
q12	763	595	603	595
q13	16540	3005	2994	2994
q14	299	278	285	278
q15	518	483	478	478
q16	476	420	431	420
q17	1767	1478	1484	1478
q18	7746	7484	7617	7484
q19	1636	1579	1611	1579
q20	2022	1816	1780	1780
q21	5007	4767	4777	4767
q22	548	500	518	500
Total cold run time: 70721 ms
Total hot run time: 54181 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173554 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e7823ff965c8fe387860c164f90cdb141119ada5, data reload: false

query1	926	385	366	366
query2	6456	1944	1858	1858
query3	6671	212	220	212
query4	25559	17689	17319	17319
query5	4208	511	491	491
query6	264	177	186	177
query7	4591	295	293	293
query8	234	211	199	199
query9	8564	2404	2402	2402
query10	452	299	288	288
query11	10561	10114	10010	10010
query12	140	88	85	85
query13	1668	377	370	370
query14	10563	7886	7745	7745
query15	233	175	177	175
query16	7606	328	324	324
query17	1780	586	563	563
query18	1573	289	282	282
query19	207	162	160	160
query20	90	85	85	85
query21	213	133	135	133
query22	4544	4072	3979	3979
query23	34196	33235	33198	33198
query24	12078	2930	2798	2798
query25	667	395	397	395
query26	1778	156	155	155
query27	2865	277	281	277
query28	7592	2016	2002	2002
query29	1070	636	637	636
query30	291	157	152	152
query31	977	733	764	733
query32	94	55	59	55
query33	790	320	309	309
query34	930	501	494	494
query35	703	687	570	570
query36	1148	944	946	944
query37	258	80	86	80
query38	2929	2808	2766	2766
query39	861	801	822	801
query40	275	122	141	122
query41	53	46	43	43
query42	116	97	101	97
query43	499	470	459	459
query44	1258	748	735	735
query45	189	162	167	162
query46	1113	738	720	720
query47	1892	1817	1799	1799
query48	363	293	293	293
query49	1222	444	411	411
query50	786	405	399	399
query51	6893	6763	6800	6763
query52	112	90	103	90
query53	381	299	301	299
query54	978	450	463	450
query55	75	77	72	72
query56	284	265	275	265
query57	1184	1119	1070	1070
query58	275	257	253	253
query59	2876	2761	2601	2601
query60	301	281	299	281
query61	95	96	95	95
query62	839	647	656	647
query63	324	293	296	293
query64	10547	2224	1702	1702
query65	3193	3120	3102	3102
query66	1331	343	345	343
query67	15447	15138	15146	15138
query68	6000	554	572	554
query69	717	449	349	349
query70	1219	1138	1145	1138
query71	483	286	273	273
query72	8618	5437	5448	5437
query73	791	328	329	328
query74	6311	5703	5693	5693
query75	4535	2691	2696	2691
query76	4091	1002	908	908
query77	703	323	330	323
query78	10523	9141	9021	9021
query79	7912	536	531	531
query80	1096	539	499	499
query81	585	224	231	224
query82	793	135	137	135
query83	293	177	168	168
query84	280	90	84	84
query85	1482	321	303	303
query86	469	310	321	310
query87	3363	3118	3091	3091
query88	5102	2378	2382	2378
query89	560	393	378	378
query90	1933	203	197	197
query91	131	105	104	104
query92	70	50	52	50
query93	6268	515	510	510
query94	1314	215	214	214
query95	424	320	312	312
query96	628	280	272	272
query97	3256	3052	3069	3052
query98	217	207	197	197
query99	1643	1264	1245	1245
Total cold run time: 301041 ms
Total hot run time: 173554 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e7823ff965c8fe387860c164f90cdb141119ada5, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.22	0.05	0.04
query4	1.68	0.08	0.08
query5	0.51	0.49	0.48
query6	1.15	0.72	0.71
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.55	0.49	0.49
query10	0.55	0.54	0.55
query11	0.14	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	0.77	0.77	0.79
query15	0.86	0.81	0.82
query16	0.37	0.35	0.36
query17	0.97	0.99	1.05
query18	0.24	0.22	0.22
query19	1.83	1.75	1.75
query20	0.02	0.01	0.00
query21	15.39	0.75	0.66
query22	3.92	6.38	2.41
query23	18.32	1.36	1.43
query24	2.12	0.23	0.24
query25	0.16	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.24
query28	13.31	1.04	1.01
query29	12.65	3.37	3.35
query30	0.26	0.06	0.06
query31	2.89	0.39	0.40
query32	3.26	0.49	0.50
query33	2.90	2.97	2.95
query34	17.03	4.36	4.35
query35	4.39	4.45	4.46
query36	0.66	0.47	0.50
query37	0.19	0.15	0.16
query38	0.17	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.58 s
Total hot run time: 31.47 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 715af6e into apache:master Jul 17, 2024
yiguolei pushed a commit that referenced this pull request Jul 17, 2024
…38016)

## Proposed changes

pick from #37788

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged p0_w reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants