Skip to content

[enhancement](nereids) enhancement distribute planner for data distribution #37219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
924060929 merged 57 commits into from
Jul 19, 2024
Merged

[enhancement](nereids) enhancement distribute planner for data distribution #37219

924060929 merged 57 commits into from
Jul 19, 2024

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented Jul 3, 2024 •
edited
Loading

  1. add benchmark shape test for distribute planner
  2. support read tables of information_schema
  3. fix can not execute query when all tablets pruned
  4. propagate data distribution when the common column of grouping sets is the distribution column

@doris-robot
Copy link

doris-robot commented Jul 3, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@924060929
Copy link
Contributor Author

924060929 commented Jul 3, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 3, 2024

TPC-H: Total hot run time: 40096 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f13a9e0b78bf6f4c687931b863d8afe6f99cb91, data reload: false

------ Round 1 ----------------------------------
q1	17650	4494	4288	4288
q2	2020	203	194	194
q3	10510	1198	1069	1069
q4	10201	897	749	749
q5	7545	2648	2595	2595
q6	223	136	139	136
q7	966	605	626	605
q8	9247	2085	2112	2085
q9	8962	6974	7001	6974
q10	8953	3715	3726	3715
q11	462	233	225	225
q12	462	227	231	227
q13	17754	2989	2983	2983
q14	261	224	234	224
q15	530	488	474	474
q16	506	390	369	369
q17	965	585	627	585
q18	8038	7309	7427	7309
q19	7261	1447	1424	1424
q20	637	306	294	294
q21	5001	3234	3248	3234
q22	390	341	338	338
Total cold run time: 118544 ms
Total hot run time: 40096 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4378	4219	4235	4219
q2	374	278	270	270
q3	3045	2969	2984	2969
q4	1937	1771	1766	1766
q5	5526	5484	5385	5385
q6	232	140	131	131
q7	2242	1842	1870	1842
q8	3274	3461	3413	3413
q9	9255	9340	9286	9286
q10	4160	3822	3880	3822
q11	584	516	497	497
q12	839	637	635	635
q13	15900	3216	3266	3216
q14	303	295	266	266
q15	540	485	479	479
q16	489	417	426	417
q17	1867	1545	1499	1499
q18	8194	7958	7748	7748
q19	3452	1621	1665	1621
q20	2245	2007	1986	1986
q21	5032	4925	4928	4925
q22	653	567	575	567
Total cold run time: 74521 ms
Total hot run time: 56959 ms

@doris-robot
Copy link

doris-robot commented Jul 3, 2024

TPC-DS: Total hot run time: 177967 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f13a9e0b78bf6f4c687931b863d8afe6f99cb91, data reload: false

query1	926	377	365	365
query2	6430	2508	2332	2332
query3	6629	204	210	204
query4	28385	17350	17293	17293
query5	3655	470	466	466
query6	248	182	165	165
query7	4568	302	304	302
query8	310	291	299	291
query9	8610	2365	2362	2362
query10	580	305	274	274
query11	11370	10028	9855	9855
query12	118	92	82	82
query13	1652	372	374	372
query14	10291	8058	7894	7894
query15	260	195	192	192
query16	7964	281	284	281
query17	1845	576	532	532
query18	2070	286	283	283
query19	201	154	154	154
query20	92	84	83	83
query21	207	142	133	133
query22	4290	3987	4027	3987
query23	34484	33863	33686	33686
query24	11179	3155	3073	3073
query25	602	421	402	402
query26	704	173	167	167
query27	2260	324	335	324
query28	6044	2135	2098	2098
query29	902	675	641	641
query30	244	152	153	152
query31	1112	965	989	965
query32	103	58	65	58
query33	758	333	328	328
query34	912	483	499	483
query35	809	665	692	665
query36	1145	973	970	970
query37	141	75	75	75
query38	2857	2770	2736	2736
query39	3958	3883	3870	3870
query40	203	130	124	124
query41	52	51	51	51
query42	123	99	96	96
query43	579	551	573	551
query44	1320	743	735	735
query45	204	165	173	165
query46	1117	751	742	742
query47	1809	1732	1758	1732
query48	387	306	304	304
query49	841	421	425	421
query50	770	388	377	377
query51	6910	6875	6701	6701
query52	105	90	94	90
query53	362	285	283	283
query54	950	465	451	451
query55	74	75	75	75
query56	287	256	271	256
query57	1133	1022	1051	1022
query58	247	249	236	236
query59	3481	3133	3089	3089
query60	312	275	277	275
query61	94	90	94	90
query62	605	445	472	445
query63	323	293	290	290
query64	8533	2266	1790	1790
query65	3183	3117	3124	3117
query66	705	328	323	323
query67	15567	14945	15093	14945
query68	6148	557	557	557
query69	716	421	352	352
query70	1217	1153	1113	1113
query71	511	287	288	287
query72	8862	5104	5926	5104
query73	808	327	329	327
query74	6146	5485	5442	5442
query75	5202	2689	2675	2675
query76	4697	986	903	903
query77	809	305	308	305
query78	10607	9725	9789	9725
query79	7528	530	542	530
query80	1085	503	488	488
query81	578	225	224	224
query82	1266	107	102	102
query83	369	172	173	172
query84	271	87	85	85
query85	1314	338	304	304
query86	445	353	351	351
query87	3344	3115	3071	3071
query88	4724	2451	2487	2451
query89	510	393	399	393
query90	1974	196	192	192
query91	133	104	102	102
query92	66	51	51	51
query93	6061	522	520	520
query94	1219	204	193	193
query95	417	333	323	323
query96	651	276	272	272
query97	3207	3028	2969	2969
query98	226	206	194	194
query99	1110	856	819	819
Total cold run time: 301412 ms
Total hot run time: 177967 ms

@doris-robot
Copy link

doris-robot commented Jul 3, 2024

ClickBench: Total hot run time: 30.7 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4f13a9e0b78bf6f4c687931b863d8afe6f99cb91, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.23	0.05	0.05
query4	1.67	0.10	0.09
query5	0.49	0.49	0.48
query6	1.16	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.56	0.49	0.48
query10	0.55	0.55	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.80	0.77
query15	0.85	0.81	0.83
query16	0.36	0.37	0.34
query17	0.95	0.96	1.00
query18	0.24	0.23	0.26
query19	1.79	1.74	1.68
query20	0.01	0.01	0.01
query21	15.43	0.78	0.67
query22	4.47	7.27	2.02
query23	18.26	1.44	1.28
query24	2.08	0.24	0.25
query25	0.15	0.08	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.09
query28	13.22	1.03	1.03
query29	12.60	3.35	3.35
query30	0.26	0.06	0.06
query31	2.86	0.39	0.39
query32	3.27	0.49	0.47
query33	2.91	2.94	2.88
query34	17.12	4.39	4.41
query35	4.52	4.52	4.42
query36	0.64	0.48	0.49
query37	0.20	0.17	0.16
query38	0.16	0.15	0.16
query39	0.05	0.03	0.03
query40	0.18	0.14	0.14
query41	0.09	0.04	0.05
query42	0.06	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.65 s
Total hot run time: 30.7 s

@924060929
Copy link
Contributor Author

924060929 commented Jul 3, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 3, 2024

TPC-H: Total hot run time: 40347 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf9df876fde9280d97794bf2c5998e1d07d59b67, data reload: false

------ Round 1 ----------------------------------
q1	17618	4690	4298	4298
q2	2024	191	188	188
q3	10464	1189	1027	1027
q4	10186	757	820	757
q5	7553	2673	2658	2658
q6	225	140	142	140
q7	972	601	600	600
q8	9253	2134	2103	2103
q9	9199	6926	6972	6926
q10	8953	3708	3695	3695
q11	524	238	240	238
q12	463	237	232	232
q13	18737	3002	3005	3002
q14	286	227	232	227
q15	526	488	501	488
q16	507	394	371	371
q17	985	656	636	636
q18	8037	7462	7384	7384
q19	3940	1547	1514	1514
q20	657	320	294	294
q21	5038	3221	3435	3221
q22	408	351	348	348
Total cold run time: 116555 ms
Total hot run time: 40347 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4492	4272	4301	4272
q2	387	288	280	280
q3	3008	2988	3018	2988
q4	2008	1655	1732	1655
q5	5639	5539	5464	5464
q6	231	133	136	133
q7	2224	1904	1893	1893
q8	3314	3427	3468	3427
q9	9212	9291	9287	9287
q10	4116	3763	3846	3763
q11	600	500	500	500
q12	808	655	673	655
q13	16434	3223	3253	3223
q14	309	285	279	279
q15	536	479	504	479
q16	496	428	429	428
q17	1841	1526	1501	1501
q18	8244	8114	7862	7862
q19	3806	1713	1591	1591
q20	2379	2007	1977	1977
q21	9667	4959	4907	4907
q22	684	614	585	585
Total cold run time: 80435 ms
Total hot run time: 57149 ms

@doris-robot
Copy link

doris-robot commented Jul 3, 2024

TPC-DS: Total hot run time: 175097 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf9df876fde9280d97794bf2c5998e1d07d59b67, data reload: false

query1	926	376	366	366
query2	6470	2374	2262	2262
query3	6639	210	217	210
query4	28422	17508	17175	17175
query5	3613	496	502	496
query6	272	172	161	161
query7	4571	320	297	297
query8	337	296	293	293
query9	8505	2411	2418	2411
query10	563	316	272	272
query11	10881	10019	9954	9954
query12	115	91	83	83
query13	1652	358	368	358
query14	10421	8265	7913	7913
query15	247	196	207	196
query16	7368	310	325	310
query17	1749	574	536	536
query18	1779	276	269	269
query19	200	146	153	146
query20	94	85	80	80
query21	206	131	136	131
query22	4190	4173	4118	4118
query23	35556	34020	33802	33802
query24	10794	3064	3142	3064
query25	549	395	416	395
query26	717	173	168	168
query27	2195	335	322	322
query28	5954	2172	2169	2169
query29	860	675	636	636
query30	240	151	144	144
query31	1155	919	868	868
query32	103	56	56	56
query33	689	330	318	318
query34	905	512	508	508
query35	753	653	704	653
query36	1141	961	990	961
query37	153	78	80	78
query38	2944	2898	2801	2801
query39	903	851	844	844
query40	225	134	131	131
query41	53	51	52	51
query42	115	105	111	105
query43	614	573	560	560
query44	1099	725	750	725
query45	206	165	169	165
query46	1143	779	775	775
query47	1839	1750	1703	1703
query48	373	304	288	288
query49	838	424	443	424
query50	771	380	398	380
query51	6843	6842	6770	6770
query52	101	95	95	95
query53	363	293	301	293
query54	949	462	456	456
query55	77	76	74	74
query56	299	275	277	275
query57	1144	1055	1054	1054
query58	253	241	268	241
query59	3446	3166	3232	3166
query60	324	283	278	278
query61	96	90	92	90
query62	580	436	432	432
query63	323	305	302	302
query64	8661	2401	1902	1902
query65	3210	3155	3146	3146
query66	711	342	351	342
query67	15411	15098	14899	14899
query68	4569	560	549	549
query69	642	431	350	350
query70	1206	1127	1171	1127
query71	447	291	291	291
query72	8073	5836	5477	5477
query73	765	323	324	323
query74	6050	5539	5452	5452
query75	3539	2666	2710	2666
query76	3231	990	1010	990
query77	685	318	324	318
query78	9651	9045	8920	8920
query79	2902	521	536	521
query80	3217	497	513	497
query81	595	225	221	221
query82	968	119	113	113
query83	446	177	171	171
query84	280	85	86	85
query85	1866	296	298	296
query86	477	324	346	324
query87	3258	3087	3067	3067
query88	4605	2335	2349	2335
query89	493	400	418	400
query90	1779	190	186	186
query91	130	101	100	100
query92	62	50	49	49
query93	4509	505	495	495
query94	1062	204	214	204
query95	403	321	312	312
query96	630	273	271	271
query97	3155	3004	2991	2991
query98	213	209	203	203
query99	1223	865	847	847
Total cold run time: 286029 ms
Total hot run time: 175097 ms

@doris-robot
Copy link

doris-robot commented Jul 3, 2024

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf9df876fde9280d97794bf2c5998e1d07d59b67, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.05	0.04
query4	1.66	0.07	0.06
query5	0.50	0.48	0.48
query6	1.14	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.48	0.48
query10	0.55	0.53	0.55
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.60	0.58	0.59
query14	0.78	0.77	0.78
query15	0.87	0.82	0.81
query16	0.37	0.36	0.37
query17	1.01	1.02	1.04
query18	0.23	0.24	0.26
query19	1.82	1.70	1.74
query20	0.02	0.01	0.01
query21	15.45	0.79	0.65
query22	3.83	7.19	2.08
query23	18.29	1.45	1.18
query24	2.14	0.25	0.22
query25	0.15	0.09	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.17	1.01	0.99
query29	12.63	3.26	3.26
query30	0.26	0.06	0.06
query31	2.88	0.40	0.39
query32	3.25	0.49	0.48
query33	2.86	2.91	2.92
query34	17.12	4.45	4.41
query35	4.51	4.51	4.46
query36	0.66	0.46	0.48
query37	0.18	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.08	0.05	0.06
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.09 s
Total hot run time: 30.63 s

@924060929
Copy link
Contributor Author

924060929 commented Jul 4, 2024

run buildall

1 similar comment
@924060929
Copy link
Contributor Author

924060929 commented Jul 4, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 4, 2024

TPC-H: Total hot run time: 39885 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3fe2c75cf58d17c15221d8685807c0e747a9fdc, data reload: false

------ Round 1 ----------------------------------
q1	17624	4761	4274	4274
q2	2025	198	200	198
q3	10501	1206	1009	1009
q4	10194	820	679	679
q5	7685	2610	2589	2589
q6	221	141	161	141
q7	969	610	616	610
q8	9259	2083	2056	2056
q9	8869	6989	6936	6936
q10	8914	3662	3678	3662
q11	443	236	238	236
q12	481	234	235	234
q13	17744	2936	2927	2927
q14	258	218	218	218
q15	520	478	481	478
q16	497	399	376	376
q17	947	730	608	608
q18	8064	7394	7286	7286
q19	5073	1538	1559	1538
q20	653	307	301	301
q21	4868	3182	3903	3182
q22	398	347	350	347
Total cold run time: 116207 ms
Total hot run time: 39885 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4326	4269	4201	4201
q2	386	270	280	270
q3	3029	2741	2730	2730
q4	1858	1572	1589	1572
q5	5233	5255	5229	5229
q6	216	134	133	133
q7	2084	1758	1787	1758
q8	3221	3351	3298	3298
q9	8811	8738	8861	8738
q10	3859	3673	3638	3638
q11	602	496	475	475
q12	825	613	591	591
q13	17457	2967	2983	2967
q14	300	267	261	261
q15	524	479	483	479
q16	462	403	415	403
q17	1760	1464	1450	1450
q18	7631	7652	7483	7483
q19	1725	1478	1563	1478
q20	2103	1935	1930	1930
q21	4840	4782	4686	4686
q22	662	599	561	561
Total cold run time: 71914 ms
Total hot run time: 54331 ms

@doris-robot
Copy link

doris-robot commented Jul 4, 2024

TPC-DS: Total hot run time: 170550 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3fe2c75cf58d17c15221d8685807c0e747a9fdc, data reload: false

query1	930	367	359	359
query2	6489	2436	2391	2391
query3	6655	201	209	201
query4	28274	17444	17262	17262
query5	4191	492	481	481
query6	281	177	171	171
query7	4587	315	302	302
query8	323	306	304	304
query9	8589	2398	2380	2380
query10	605	299	270	270
query11	10759	10137	10076	10076
query12	135	87	85	85
query13	1645	366	364	364
query14	10274	7749	7743	7743
query15	242	193	192	192
query16	7788	316	339	316
query17	1750	558	528	528
query18	1855	281	274	274
query19	192	150	148	148
query20	87	83	83	83
query21	210	125	126	125
query22	4420	4116	4153	4116
query23	33723	33199	32989	32989
query24	12038	3032	2990	2990
query25	664	356	351	351
query26	1718	160	162	160
query27	2899	315	317	315
query28	7557	2083	2073	2073
query29	1059	622	609	609
query30	276	140	144	140
query31	1137	884	898	884
query32	91	52	56	52
query33	773	343	304	304
query34	883	464	462	462
query35	775	648	611	611
query36	1095	950	941	941
query37	149	72	71	71
query38	2851	2821	2853	2821
query39	855	793	790	790
query40	273	124	122	122
query41	54	44	53	44
query42	113	95	96	95
query43	582	555	538	538
query44	1228	729	727	727
query45	197	173	169	169
query46	1108	720	730	720
query47	1914	1791	1806	1791
query48	373	294	309	294
query49	1169	412	412	412
query50	773	394	408	394
query51	6812	6782	6748	6748
query52	112	89	93	89
query53	351	287	290	287
query54	994	444	439	439
query55	77	74	74	74
query56	307	297	314	297
query57	1140	1077	1063	1063
query58	254	233	243	233
query59	3505	3237	3010	3010
query60	300	277	281	277
query61	95	93	91	91
query62	638	435	427	427
query63	321	295	300	295
query64	9833	2273	1786	1786
query65	3373	3154	3102	3102
query66	1398	331	315	315
query67	15698	14973	14872	14872
query68	4603	542	558	542
query69	475	314	317	314
query70	1202	1093	1069	1069
query71	372	268	281	268
query72	7076	2850	2650	2650
query73	728	318	326	318
query74	5952	5560	5406	5406
query75	3365	2658	2673	2658
query76	2313	962	943	943
query77	466	299	298	298
query78	9411	8812	8878	8812
query79	2706	508	511	508
query80	2067	481	475	475
query81	582	220	225	220
query82	912	110	104	104
query83	284	188	178	178
query84	281	84	96	84
query85	2099	342	342	342
query86	480	340	361	340
query87	3316	3114	3123	3114
query88	3610	2394	2373	2373
query89	465	384	377	377
query90	1823	186	186	186
query91	140	111	114	111
query92	66	52	53	52
query93	3421	509	515	509
query94	1207	222	225	222
query95	432	333	329	329
query96	599	273	268	268
query97	3203	3005	3013	3005
query98	231	205	200	200
query99	1191	870	830	830
Total cold run time: 286894 ms
Total hot run time: 170550 ms

@doris-robot
Copy link

doris-robot commented Jul 4, 2024

ClickBench: Total hot run time: 30.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3fe2c75cf58d17c15221d8685807c0e747a9fdc, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.09	0.08
query5	0.52	0.49	0.48
query6	1.13	0.73	0.72
query7	0.03	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.48
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.15	0.13	0.13
query13	0.59	0.58	0.58
query14	0.77	0.76	0.80
query15	0.84	0.83	0.81
query16	0.37	0.37	0.35
query17	0.94	0.94	0.98
query18	0.21	0.26	0.24
query19	1.79	1.71	1.68
query20	0.01	0.00	0.01
query21	15.43	0.74	0.66
query22	4.21	7.01	1.99
query23	18.31	1.37	1.20
query24	2.17	0.22	0.22
query25	0.16	0.09	0.08
query26	0.29	0.20	0.21
query27	0.46	0.24	0.24
query28	13.20	1.00	1.00
query29	12.60	3.31	3.32
query30	0.25	0.06	0.05
query31	2.86	0.38	0.38
query32	3.26	0.49	0.47
query33	2.87	2.91	2.92
query34	17.12	4.34	4.34
query35	4.41	4.38	4.39
query36	0.66	0.47	0.51
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.61 s
Total hot run time: 30.54 s

@924060929
Copy link
Contributor Author

924060929 commented Jul 5, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 5, 2024

TPC-H: Total hot run time: 40637 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2b6bdc87e867fbb7b9b0575bba803b8acd8e464, data reload: false

------ Round 1 ----------------------------------
q1	17873	4498	4436	4436
q2	2668	204	198	198
q3	12242	1156	1178	1156
q4	10698	777	733	733
q5	8067	2708	2652	2652
q6	229	141	142	141
q7	950	601	621	601
q8	9224	2089	2139	2089
q9	9191	7025	6990	6990
q10	8939	3780	3738	3738
q11	450	246	244	244
q12	414	235	227	227
q13	17766	2951	2993	2951
q14	277	220	227	220
q15	530	492	474	474
q16	516	376	386	376
q17	981	781	765	765
q18	8006	7518	7421	7421
q19	6971	1536	1387	1387
q20	662	335	343	335
q21	5129	3150	3871	3150
q22	415	353	356	353
Total cold run time: 122198 ms
Total hot run time: 40637 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4416	4234	4287	4234
q2	375	283	275	275
q3	3005	2827	2729	2729
q4	1913	1569	1585	1569
q5	5260	5258	5259	5258
q6	223	137	138	137
q7	2152	1721	1717	1717
q8	3332	3388	3340	3340
q9	8873	8852	8871	8852
q10	3906	3715	3665	3665
q11	591	494	505	494
q12	801	604	612	604
q13	16559	2956	3002	2956
q14	284	266	279	266
q15	524	496	478	478
q16	480	411	430	411
q17	1791	1497	1475	1475
q18	7742	7482	7279	7279
q19	1718	1661	1634	1634
q20	2097	1904	1876	1876
q21	4915	4855	4696	4696
q22	668	568	570	568
Total cold run time: 71625 ms
Total hot run time: 54513 ms

@doris-robot
Copy link

doris-robot commented Jul 5, 2024

TPC-DS: Total hot run time: 173733 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2b6bdc87e867fbb7b9b0575bba803b8acd8e464, data reload: false

query1	930	375	374	374
query2	6475	2507	2517	2507
query3	6683	205	218	205
query4	27914	17486	17261	17261
query5	4170	495	505	495
query6	286	184	159	159
query7	4593	305	302	302
query8	316	283	295	283
query9	8655	2471	2433	2433
query10	602	290	276	276
query11	10464	9995	10128	9995
query12	140	92	82	82
query13	1653	373	380	373
query14	10459	8222	6735	6735
query15	238	189	203	189
query16	7850	341	312	312
query17	1737	570	542	542
query18	2025	290	288	288
query19	214	161	160	160
query20	90	87	84	84
query21	216	128	127	127
query22	4545	4141	4126	4126
query23	33745	33194	33105	33105
query24	12067	3011	3047	3011
query25	664	393	393	393
query26	1677	164	162	162
query27	2934	323	339	323
query28	7612	2115	2100	2100
query29	989	645	641	641
query30	278	149	146	146
query31	1152	937	962	937
query32	101	55	56	55
query33	775	320	332	320
query34	880	492	483	483
query35	759	647	623	623
query36	1056	974	922	922
query37	148	79	77	77
query38	2865	2706	2767	2706
query39	895	800	798	798
query40	271	130	128	128
query41	54	50	53	50
query42	123	101	101	101
query43	606	574	578	574
query44	1188	734	726	726
query45	204	165	169	165
query46	1124	753	752	752
query47	1888	1770	1789	1770
query48	367	298	314	298
query49	1193	428	438	428
query50	783	394	405	394
query51	6905	6889	6884	6884
query52	107	94	101	94
query53	370	298	305	298
query54	1000	469	461	461
query55	77	73	78	73
query56	309	293	290	290
query57	1163	1045	1049	1045
query58	261	246	370	246
query59	3453	3170	3298	3170
query60	316	277	275	275
query61	101	88	115	88
query62	638	445	451	445
query63	322	292	299	292
query64	9913	2258	1769	1769
query65	3369	3099	3115	3099
query66	1439	343	334	334
query67	15727	15081	14972	14972
query68	6956	557	555	555
query69	723	450	342	342
query70	1204	1124	1123	1123
query71	551	285	282	282
query72	8863	5659	5901	5659
query73	796	323	324	323
query74	6046	5626	5543	5543
query75	4783	2670	2697	2670
query76	4860	1006	1007	1006
query77	761	312	312	312
query78	9540	8832	9020	8832
query79	8824	516	514	514
query80	1019	470	465	465
query81	589	228	230	228
query82	751	106	105	105
query83	358	171	168	168
query84	265	85	84	84
query85	1407	291	287	287
query86	464	374	344	344
query87	3296	3087	3081	3081
query88	4694	2515	2433	2433
query89	535	383	390	383
query90	2055	189	189	189
query91	129	102	97	97
query92	67	47	49	47
query93	7074	510	502	502
query94	1329	213	207	207
query95	406	320	317	317
query96	605	268	272	268
query97	3204	3017	3006	3006
query98	221	198	188	188
query99	1180	842	875	842
Total cold run time: 305678 ms
Total hot run time: 173733 ms

@doris-robot
Copy link

doris-robot commented Jul 5, 2024

ClickBench: Total hot run time: 31.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2b6bdc87e867fbb7b9b0575bba803b8acd8e464, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.07	0.07
query5	0.50	0.49	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.53	0.51	0.49
query10	0.55	0.54	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.77	0.76	0.78
query15	0.85	0.81	0.81
query16	0.37	0.38	0.36
query17	0.98	1.04	1.02
query18	0.23	0.25	0.25
query19	1.80	1.66	1.69
query20	0.01	0.01	0.01
query21	15.39	0.74	0.64
query22	4.24	7.03	2.40
query23	18.32	1.37	1.30
query24	2.14	0.23	0.21
query25	0.16	0.08	0.08
query26	0.30	0.22	0.20
query27	0.46	0.24	0.23
query28	13.21	1.02	1.00
query29	12.59	3.38	3.34
query30	0.26	0.06	0.05
query31	2.92	0.39	0.38
query32	3.26	0.48	0.47
query33	2.86	2.92	2.93
query34	17.16	4.35	4.44
query35	4.44	4.40	4.49
query36	0.65	0.47	0.47
query37	0.18	0.17	0.16
query38	0.15	0.15	0.16
query39	0.04	0.04	0.03
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.77 s
Total hot run time: 31.21 s

@924060929
Copy link
Contributor Author

924060929 commented Jul 8, 2024

run buildall

@924060929 924060929 force-pushed the enable_distribute_planner branch from 968b00f to 8d2db03 Compare July 8, 2024 08:17
@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TPC-H: Total hot run time: 40508 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 968b00fe8464ffab59ba455a2ab7a2aeffee1005, data reload: false

------ Round 1 ----------------------------------
q1	17589	4573	4322	4322
q2	2025	193	188	188
q3	10445	1267	1137	1137
q4	10199	803	874	803
q5	7524	2658	2614	2614
q6	222	138	137	137
q7	962	601	597	597
q8	9225	2100	2105	2100
q9	9182	7064	7005	7005
q10	9077	3760	3726	3726
q11	457	240	241	240
q12	444	232	241	232
q13	17763	3024	3026	3024
q14	273	222	220	220
q15	516	500	482	482
q16	509	393	390	390
q17	969	656	616	616
q18	8105	7538	7364	7364
q19	6914	1560	1501	1501
q20	674	325	294	294
q21	4868	3178	3944	3178
q22	401	345	338	338
Total cold run time: 118343 ms
Total hot run time: 40508 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4395	4257	4223	4223
q2	401	303	280	280
q3	3171	2973	3008	2973
q4	1955	1750	1718	1718
q5	5517	5465	5433	5433
q6	226	139	135	135
q7	2279	1864	1867	1864
q8	3323	3469	3432	3432
q9	9343	9422	9454	9422
q10	4010	3791	3861	3791
q11	606	506	493	493
q12	805	630	659	630
q13	16875	3251	3221	3221
q14	308	293	272	272
q15	553	482	483	482
q16	485	434	427	427
q17	1820	1549	1540	1540
q18	8090	8022	7744	7744
q19	1806	1596	1625	1596
q20	2191	2038	2022	2022
q21	5273	4761	4946	4761
q22	638	566	612	566
Total cold run time: 74070 ms
Total hot run time: 57025 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TPC-DS: Total hot run time: 174263 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 968b00fe8464ffab59ba455a2ab7a2aeffee1005, data reload: false

query1	918	385	366	366
query2	6362	2425	2315	2315
query3	6634	205	217	205
query4	26666	17447	17288	17288
query5	3752	488	493	488
query6	292	176	156	156
query7	4583	304	295	295
query8	303	282	269	269
query9	8468	2358	2339	2339
query10	552	289	267	267
query11	10677	10120	10200	10120
query12	114	84	83	83
query13	1651	375	371	371
query14	10082	7735	7700	7700
query15	236	194	192	192
query16	7703	313	297	297
query17	1794	558	557	557
query18	1884	303	274	274
query19	194	148	155	148
query20	93	88	84	84
query21	211	131	125	125
query22	4354	4015	3976	3976
query23	33800	33762	33689	33689
query24	11805	3154	3057	3057
query25	675	379	379	379
query26	1762	171	172	171
query27	2867	341	336	336
query28	8167	2142	2138	2138
query29	1058	653	646	646
query30	236	150	145	145
query31	1191	966	913	913
query32	97	76	52	52
query33	743	301	311	301
query34	898	494	490	490
query35	780	670	667	667
query36	1150	963	998	963
query37	180	82	80	80
query38	2957	2759	2741	2741
query39	840	796	783	783
query40	245	134	122	122
query41	52	49	49	49
query42	122	93	96	93
query43	581	530	520	520
query44	1233	741	730	730
query45	211	175	167	167
query46	1113	765	707	707
query47	1877	1806	1777	1777
query48	379	316	310	310
query49	947	438	428	428
query50	769	383	391	383
query51	6898	6748	6770	6748
query52	105	94	91	91
query53	365	291	300	291
query54	966	475	459	459
query55	74	76	75	75
query56	304	295	289	289
query57	1165	1062	1061	1061
query58	270	275	255	255
query59	3322	3166	3071	3071
query60	292	273	274	273
query61	95	96	92	92
query62	597	454	447	447
query63	320	287	287	287
query64	10316	2221	1647	1647
query65	3230	3134	3146	3134
query66	1248	340	361	340
query67	15418	15040	14831	14831
query68	5891	570	572	570
query69	687	410	351	351
query70	1209	1190	1128	1128
query71	454	279	269	269
query72	7161	5566	5493	5493
query73	768	328	321	321
query74	5898	5608	5497	5497
query75	3797	2659	2748	2659
query76	3542	913	972	913
query77	657	303	297	297
query78	9506	8921	8850	8850
query79	3052	513	512	512
query80	2327	494	482	482
query81	614	227	230	227
query82	1348	108	100	100
query83	303	180	172	172
query84	281	87	84	84
query85	1452	293	278	278
query86	471	393	346	346
query87	3275	3116	3088	3088
query88	4650	2461	2442	2442
query89	482	390	391	390
query90	1784	188	188	188
query91	128	98	98	98
query92	58	48	47	47
query93	4150	516	513	513
query94	1115	203	207	203
query95	405	306	313	306
query96	615	267	263	263
query97	3202	3011	3043	3011
query98	296	204	194	194
query99	1198	828	840	828
Total cold run time: 290019 ms
Total hot run time: 174263 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

ClickBench: Total hot run time: 30.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 968b00fe8464ffab59ba455a2ab7a2aeffee1005, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.22	0.05	0.05
query4	1.68	0.09	0.08
query5	0.51	0.49	0.48
query6	1.13	0.73	0.73
query7	0.03	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.49	0.48
query10	0.53	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.60
query14	0.77	0.79	0.77
query15	0.85	0.82	0.82
query16	0.37	0.35	0.36
query17	1.03	1.06	1.05
query18	0.20	0.23	0.25
query19	1.76	1.80	1.78
query20	0.01	0.00	0.00
query21	15.39	0.75	0.66
query22	4.93	7.26	1.72
query23	18.36	1.47	1.35
query24	2.09	0.22	0.23
query25	0.15	0.08	0.08
query26	0.32	0.21	0.20
query27	0.45	0.23	0.23
query28	13.34	1.01	1.01
query29	12.64	3.31	3.29
query30	0.25	0.06	0.05
query31	2.88	0.38	0.38
query32	3.28	0.47	0.48
query33	2.92	2.91	2.95
query34	17.03	4.37	4.35
query35	4.46	4.45	4.44
query36	0.66	0.46	0.47
query37	0.18	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.58 s
Total hot run time: 30.68 s

@924060929 924060929 force-pushed the enable_distribute_planner branch from c7312f7 to 5aa1f68 Compare July 8, 2024 10:17
@924060929
Copy link
Contributor Author

924060929 commented Jul 8, 2024

run buildall

1 similar comment
@924060929
Copy link
Contributor Author

924060929 commented Jul 8, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TPC-H: Total hot run time: 40134 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ad9c454538f28257e5595f971dfacfa815b7ce7a, data reload: false

------ Round 1 ----------------------------------
q1	18469	4470	4373	4373
q2	2019	191	188	188
q3	10521	1149	1131	1131
q4	10206	785	737	737
q5	7550	2632	2557	2557
q6	222	139	136	136
q7	955	592	616	592
q8	9219	2055	2095	2055
q9	8976	6965	7004	6965
q10	8866	3745	3735	3735
q11	459	231	238	231
q12	407	234	237	234
q13	17746	2996	2979	2979
q14	266	228	231	228
q15	513	490	480	480
q16	521	368	374	368
q17	963	704	646	646
q18	8262	7394	7394	7394
q19	5317	1493	1308	1308
q20	640	310	311	310
q21	4878	3157	3212	3157
q22	391	337	330	330
Total cold run time: 117366 ms
Total hot run time: 40134 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4428	4411	4224	4224
q2	392	278	278	278
q3	3024	2798	2744	2744
q4	1870	1573	1595	1573
q5	5239	5272	5267	5267
q6	225	135	130	130
q7	2114	1798	1785	1785
q8	3183	3335	3313	3313
q9	8799	8856	8848	8848
q10	3853	3641	3616	3616
q11	577	493	481	481
q12	784	646	642	642
q13	16358	3021	2971	2971
q14	291	258	274	258
q15	514	466	473	466
q16	452	412	428	412
q17	1786	1500	1467	1467
q18	7715	7425	7393	7393
q19	1710	1578	1522	1522
q20	2117	1933	1906	1906
q21	4927	4741	4838	4741
q22	619	566	566	566
Total cold run time: 70977 ms
Total hot run time: 54603 ms

@924060929
Copy link
Contributor Author

924060929 commented Jul 8, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TPC-H: Total hot run time: 40426 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 177e8dc65f997a988ac9a67c6185a4b9cb6d02a9, data reload: false

------ Round 1 ----------------------------------
q1	17599	4278	4253	4253
q2	2012	189	182	182
q3	10922	1250	1084	1084
q4	10771	804	773	773
q5	7658	2710	2670	2670
q6	227	141	140	140
q7	981	610	612	610
q8	9483	2123	2099	2099
q9	9172	7129	7089	7089
q10	9014	3698	3696	3696
q11	446	238	236	236
q12	416	231	227	227
q13	18799	2970	2960	2960
q14	263	223	228	223
q15	528	490	489	489
q16	529	383	371	371
q17	982	663	631	631
q18	8096	7572	7509	7509
q19	2717	1346	1571	1346
q20	652	308	302	302
q21	4994	3205	3200	3200
q22	404	336	342	336
Total cold run time: 116665 ms
Total hot run time: 40426 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4368	4247	4212	4212
q2	390	278	284	278
q3	3026	2948	2891	2891
q4	1949	1713	1688	1688
q5	5484	5410	5491	5410
q6	228	132	133	132
q7	2250	1854	1866	1854
q8	3319	3439	3380	3380
q9	9006	9321	9169	9169
q10	4084	3853	3804	3804
q11	603	492	516	492
q12	806	635	630	630
q13	16608	3183	3256	3183
q14	314	274	285	274
q15	540	491	487	487
q16	492	443	434	434
q17	1827	1528	1484	1484
q18	8164	8009	7665	7665
q19	1703	1408	1536	1408
q20	2181	1999	2019	1999
q21	5020	4920	4992	4920
q22	629	590	572	572
Total cold run time: 72991 ms
Total hot run time: 56366 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TPC-DS: Total hot run time: 175182 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 177e8dc65f997a988ac9a67c6185a4b9cb6d02a9, data reload: false

query1	918	364	366	364
query2	6189	2280	2372	2280
query3	6649	201	216	201
query4	24743	17544	17211	17211
query5	3731	502	498	498
query6	288	184	181	181
query7	4581	306	296	296
query8	337	292	302	292
query9	8580	2403	2363	2363
query10	602	311	285	285
query11	10696	10100	9907	9907
query12	146	88	79	79
query13	1672	379	391	379
query14	10253	7804	7789	7789
query15	240	194	196	194
query16	7895	314	320	314
query17	1780	566	542	542
query18	2022	289	292	289
query19	204	156	159	156
query20	94	85	86	85
query21	215	130	134	130
query22	4414	4106	3915	3915
query23	33997	33546	33827	33546
query24	12101	3173	3153	3153
query25	678	440	417	417
query26	1781	164	161	161
query27	2893	287	286	286
query28	7502	2175	2162	2162
query29	1060	671	658	658
query30	244	149	147	147
query31	1181	930	922	922
query32	99	60	83	60
query33	773	343	331	331
query34	896	509	500	500
query35	752	664	639	639
query36	1143	988	983	983
query37	301	90	85	85
query38	2906	2807	2804	2804
query39	877	817	806	806
query40	256	132	131	131
query41	57	53	53	53
query42	123	102	100	100
query43	596	518	537	518
query44	1166	753	742	742
query45	211	171	175	171
query46	1124	776	768	768
query47	1871	1902	1762	1762
query48	382	300	303	300
query49	934	438	437	437
query50	768	396	399	396
query51	6886	6830	6792	6792
query52	108	93	89	89
query53	359	303	292	292
query54	922	467	451	451
query55	75	75	73	73
query56	292	268	268	268
query57	1181	1059	1047	1047
query58	245	232	248	232
query59	3545	3160	3091	3091
query60	306	275	274	274
query61	95	96	96	96
query62	801	652	629	629
query63	323	290	297	290
query64	10349	2262	1652	1652
query65	3197	3137	3165	3137
query66	1232	319	329	319
query67	15852	14772	14850	14772
query68	8935	618	576	576
query69	781	441	369	369
query70	1328	1050	1125	1050
query71	548	292	280	280
query72	9295	5550	5969	5550
query73	2214	327	328	327
query74	6078	5485	5531	5485
query75	5967	2700	2654	2654
query76	5548	1057	1036	1036
query77	771	307	314	307
query78	9647	9074	8953	8953
query79	7718	536	523	523
query80	2139	500	499	499
query81	596	218	228	218
query82	279	139	131	131
query83	298	170	172	170
query84	281	89	86	86
query85	996	294	294	294
query86	351	333	342	333
query87	3391	3081	3083	3081
query88	4947	2460	2479	2460
query89	533	381	373	373
query90	2090	187	188	187
query91	130	100	103	100
query92	62	50	50	50
query93	5893	530	522	522
query94	1391	217	212	212
query95	407	315	308	308
query96	619	269	269	269
query97	3188	3005	3025	3005
query98	215	201	186	186
query99	1736	1311	1296	1296
Total cold run time: 307061 ms
Total hot run time: 175182 ms

@doris-robot
Copy link

doris-robot commented Jul 18, 2024

ClickBench: Total hot run time: 30.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 05684013b5e2bce47815518ae03396a426c435f4, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.50	0.47
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.55	0.50	0.49
query10	0.53	0.53	0.54
query11	0.15	0.12	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.75	0.78	0.78
query15	0.86	0.82	0.81
query16	0.36	0.35	0.40
query17	0.95	1.03	0.96
query18	0.22	0.21	0.22
query19	1.78	1.79	1.82
query20	0.01	0.01	0.01
query21	15.40	0.76	0.66
query22	4.81	6.87	1.79
query23	18.29	1.33	1.22
query24	2.08	0.22	0.22
query25	0.15	0.09	0.08
query26	0.31	0.22	0.20
query27	0.45	0.24	0.23
query28	13.27	1.02	1.01
query29	12.64	3.27	3.28
query30	0.26	0.06	0.06
query31	2.86	0.38	0.39
query32	3.29	0.49	0.47
query33	2.89	2.90	2.91
query34	16.95	4.35	4.42
query35	4.40	4.43	4.44
query36	0.65	0.47	0.49
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.05	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.03 s
Total hot run time: 30.55 s

@924060929
Copy link
Contributor Author

924060929 commented Jul 18, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 18, 2024

TPC-H: Total hot run time: 40171 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d7ee26d860c265f3b6b320d591a8108ff33692fa, data reload: false

------ Round 1 ----------------------------------
q1	17699	4573	4318	4318
q2	2034	195	194	194
q3	10519	1214	1156	1156
q4	10214	814	713	713
q5	7540	2755	2722	2722
q6	225	138	140	138
q7	962	597	599	597
q8	9220	2104	2086	2086
q9	8785	6564	6565	6564
q10	8809	3813	3776	3776
q11	457	240	241	240
q12	387	224	222	222
q13	17752	3019	3013	3013
q14	280	234	236	234
q15	519	484	480	480
q16	515	383	382	382
q17	987	684	668	668
q18	8221	7523	7486	7486
q19	6738	1431	1398	1398
q20	714	333	344	333
q21	5064	3158	3282	3158
q22	356	296	293	293
Total cold run time: 117997 ms
Total hot run time: 40171 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4419	4292	4249	4249
q2	366	269	265	265
q3	3038	2913	2981	2913
q4	2021	1785	1724	1724
q5	5553	5578	5475	5475
q6	219	133	137	133
q7	2319	1857	1876	1857
q8	3320	3475	3424	3424
q9	8819	8975	8829	8829
q10	4144	3768	3894	3768
q11	594	516	510	510
q12	836	659	656	656
q13	17127	3131	3166	3131
q14	317	291	299	291
q15	530	502	495	495
q16	473	445	438	438
q17	1818	1531	1495	1495
q18	8283	8276	7783	7783
q19	1731	1563	1634	1563
q20	2127	1890	1858	1858
q21	6331	4701	5059	4701
q22	590	525	527	525
Total cold run time: 74975 ms
Total hot run time: 56083 ms

@doris-robot
Copy link

doris-robot commented Jul 18, 2024

TPC-DS: Total hot run time: 175497 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d7ee26d860c265f3b6b320d591a8108ff33692fa, data reload: false

query1	916	370	368	368
query2	6375	1908	1915	1908
query3	6662	204	213	204
query4	28551	17722	17550	17550
query5	3677	491	484	484
query6	266	174	167	167
query7	4589	293	289	289
query8	238	206	189	189
query9	8641	2521	2482	2482
query10	430	281	266	266
query11	12108	10244	10145	10145
query12	119	84	89	84
query13	1637	364	364	364
query14	9282	7709	7631	7631
query15	210	168	171	168
query16	7558	319	315	315
query17	1362	532	524	524
query18	1904	282	287	282
query19	211	141	148	141
query20	87	82	79	79
query21	203	122	124	122
query22	4528	4206	4180	4180
query23	34056	33745	33769	33745
query24	11424	2921	2926	2921
query25	614	393	402	393
query26	780	150	148	148
query27	2277	277	276	276
query28	6098	2118	2138	2118
query29	873	636	659	636
query30	257	155	167	155
query31	993	775	788	775
query32	100	52	53	52
query33	762	300	292	292
query34	1025	487	509	487
query35	680	603	590	590
query36	1105	982	983	982
query37	152	85	87	85
query38	2951	2888	2828	2828
query39	877	861	856	856
query40	209	124	119	119
query41	47	45	51	45
query42	115	102	94	94
query43	508	488	462	462
query44	1265	732	735	732
query45	204	164	162	162
query46	1103	745	724	724
query47	1897	1778	1812	1778
query48	368	295	312	295
query49	852	427	458	427
query50	779	391	388	388
query51	6893	6826	6677	6677
query52	101	94	97	94
query53	366	292	283	283
query54	880	448	450	448
query55	77	72	71	71
query56	276	271	280	271
query57	1142	1061	1060	1060
query58	243	289	274	274
query59	2919	2727	2801	2727
query60	312	273	286	273
query61	97	91	97	91
query62	798	645	633	633
query63	321	284	284	284
query64	9177	2274	7454	2274
query65	3144	3092	3113	3092
query66	709	330	338	330
query67	15719	15069	15012	15012
query68	5627	554	565	554
query69	672	419	347	347
query70	1214	1140	1147	1140
query71	440	278	280	278
query72	8068	5917	5700	5700
query73	774	331	327	327
query74	6037	5722	5685	5685
query75	3455	2700	2711	2700
query76	3155	992	892	892
query77	631	312	303	303
query78	10008	8985	8923	8923
query79	3552	524	523	523
query80	2183	484	477	477
query81	577	227	217	217
query82	1346	143	149	143
query83	276	160	168	160
query84	276	94	83	83
query85	1256	426	297	297
query86	448	322	273	273
query87	3275	3108	3083	3083
query88	4457	2489	2465	2465
query89	482	385	383	383
query90	1735	192	192	192
query91	125	100	104	100
query92	55	48	52	48
query93	4806	522	518	518
query94	1069	213	220	213
query95	399	322	323	322
query96	606	280	277	277
query97	3190	3029	3038	3029
query98	212	202	192	192
query99	1538	1277	1252	1252
Total cold run time: 287043 ms
Total hot run time: 175497 ms

@doris-robot
Copy link

doris-robot commented Jul 18, 2024

ClickBench: Total hot run time: 30.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d7ee26d860c265f3b6b320d591a8108ff33692fa, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.07	0.08
query5	0.53	0.52	0.48
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.55	0.49	0.49
query10	0.56	0.54	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	0.76	0.78	0.76
query15	0.86	0.82	0.82
query16	0.37	0.38	0.37
query17	1.03	1.03	1.05
query18	0.23	0.22	0.22
query19	1.89	1.84	1.82
query20	0.02	0.01	0.01
query21	15.42	0.75	0.67
query22	4.52	6.55	2.12
query23	18.29	1.30	1.30
query24	2.15	0.23	0.21
query25	0.16	0.08	0.08
query26	0.30	0.20	0.21
query27	0.45	0.23	0.23
query28	13.29	1.01	1.00
query29	12.61	3.34	3.28
query30	0.25	0.06	0.06
query31	2.86	0.38	0.38
query32	3.30	0.45	0.46
query33	2.82	2.93	2.86
query34	16.82	4.34	4.37
query35	4.41	4.41	4.40
query36	0.66	0.46	0.48
query37	0.19	0.16	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.03
query40	0.15	0.11	0.11
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.95 s
Total hot run time: 30.93 s

@924060929
Copy link
Contributor Author

924060929 commented Jul 18, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 18, 2024

TPC-H: Total hot run time: 39743 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a6176d993f8781ee162a1a9c8336154e46724ff6, data reload: false

------ Round 1 ----------------------------------
q1	17615	4380	4251	4251
q2	2018	192	187	187
q3	10469	1155	1097	1097
q4	10188	751	786	751
q5	7518	2693	2642	2642
q6	220	137	139	137
q7	939	610	608	608
q8	9228	2040	2046	2040
q9	8720	6563	6524	6524
q10	8885	3736	3766	3736
q11	455	250	236	236
q12	413	229	229	229
q13	17777	3011	2980	2980
q14	280	226	242	226
q15	538	483	497	483
q16	481	389	385	385
q17	953	638	679	638
q18	8041	7478	7375	7375
q19	3524	1408	1385	1385
q20	630	326	338	326
q21	4999	3236	3221	3221
q22	350	286	293	286
Total cold run time: 114241 ms
Total hot run time: 39743 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4356	4232	4214	4214
q2	359	270	266	266
q3	2961	2749	2792	2749
q4	1981	1682	1726	1682
q5	5602	5630	5479	5479
q6	229	138	139	138
q7	2218	1834	1797	1797
q8	3274	3407	3397	3397
q9	8656	8735	8859	8735
q10	4078	3919	3802	3802
q11	572	493	491	491
q12	808	634	648	634
q13	15867	3180	3195	3180
q14	309	301	284	284
q15	535	504	487	487
q16	501	449	441	441
q17	1781	1542	1492	1492
q18	8077	8020	7762	7762
q19	2506	1570	1560	1560
q20	2148	1884	1924	1884
q21	9758	4969	4954	4954
q22	575	502	535	502
Total cold run time: 77151 ms
Total hot run time: 55930 ms

@doris-robot
Copy link

doris-robot commented Jul 18, 2024

TPC-DS: Total hot run time: 174589 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a6176d993f8781ee162a1a9c8336154e46724ff6, data reload: false

query1	926	374	368	368
query2	6451	1918	1905	1905
query3	6630	210	211	210
query4	24901	17523	17436	17436
query5	3633	471	488	471
query6	272	174	160	160
query7	4583	290	286	286
query8	247	197	196	196
query9	8635	2517	2490	2490
query10	452	302	267	267
query11	11675	10176	10207	10176
query12	115	83	82	82
query13	1637	365	355	355
query14	10132	7711	7574	7574
query15	225	170	166	166
query16	7727	322	307	307
query17	1800	560	547	547
query18	1968	280	279	279
query19	189	143	146	143
query20	90	81	80	80
query21	204	132	130	130
query22	4429	4052	4001	4001
query23	34299	34215	33693	33693
query24	10549	2960	2976	2960
query25	606	421	436	421
query26	716	164	157	157
query27	2276	279	281	279
query28	6103	2133	2119	2119
query29	902	630	645	630
query30	252	150	157	150
query31	984	770	799	770
query32	94	57	57	57
query33	663	287	306	287
query34	900	505	518	505
query35	696	619	585	585
query36	1138	984	955	955
query37	140	83	80	80
query38	3014	2939	2873	2873
query39	870	823	820	820
query40	197	121	122	121
query41	47	44	45	44
query42	117	101	95	95
query43	518	466	475	466
query44	1078	740	731	731
query45	189	164	157	157
query46	1084	729	738	729
query47	1915	1803	1829	1803
query48	368	288	296	288
query49	836	411	410	410
query50	775	388	389	388
query51	6819	6805	6689	6689
query52	104	95	93	93
query53	352	286	285	285
query54	853	452	453	452
query55	74	71	73	71
query56	288	260	276	260
query57	1135	1057	1079	1057
query58	232	248	256	248
query59	2881	2753	2617	2617
query60	302	275	279	275
query61	108	94	92	92
query62	821	657	643	643
query63	314	280	282	280
query64	9121	2202	1661	1661
query65	3138	3091	3087	3087
query66	738	336	327	327
query67	15391	15038	15218	15038
query68	4543	557	550	550
query69	520	414	328	328
query70	1148	1148	1139	1139
query71	368	285	285	285
query72	7008	5689	5550	5550
query73	741	329	327	327
query74	6178	5756	5715	5715
query75	3405	2694	2726	2694
query76	2140	960	914	914
query77	476	301	306	301
query78	9662	9103	9326	9103
query79	2274	528	522	522
query80	2270	475	476	475
query81	597	223	222	222
query82	1024	134	137	134
query83	293	172	165	165
query84	264	90	90	90
query85	1206	320	303	303
query86	465	313	285	285
query87	3365	3120	3174	3120
query88	3135	2461	2444	2444
query89	474	388	373	373
query90	1744	192	193	192
query91	130	100	186	100
query92	58	49	47	47
query93	2177	518	513	513
query94	1185	221	216	216
query95	403	328	321	321
query96	602	277	275	275
query97	3212	3058	3079	3058
query98	220	195	192	192
query99	1500	1259	1263	1259
Total cold run time: 273705 ms
Total hot run time: 174589 ms

@doris-robot
Copy link

doris-robot commented Jul 18, 2024

ClickBench: Total hot run time: 30.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a6176d993f8781ee162a1a9c8336154e46724ff6, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.47	0.50
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.48	0.49
query10	0.55	0.53	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.74	0.80	0.75
query15	0.86	0.82	0.82
query16	0.36	0.36	0.37
query17	0.98	0.95	0.95
query18	0.23	0.23	0.22
query19	1.81	1.70	1.73
query20	0.03	0.01	0.00
query21	15.43	0.74	0.66
query22	3.67	7.73	1.82
query23	18.32	1.31	1.22
query24	2.13	0.26	0.22
query25	0.14	0.08	0.08
query26	0.29	0.22	0.21
query27	0.46	0.23	0.23
query28	13.19	1.02	0.99
query29	12.65	3.30	3.29
query30	0.25	0.06	0.05
query31	2.88	0.39	0.40
query32	3.25	0.47	0.46
query33	2.89	2.90	2.94
query34	17.18	4.40	4.31
query35	4.46	4.39	4.32
query36	0.65	0.48	0.50
query37	0.18	0.15	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.15	0.11	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.3 s
Total hot run time: 30.27 s

@github-actions
Copy link
Contributor

github-actions bot commented Jul 19, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 19, 2024
Jibing-Li
Jibing-Li approved these changes Jul 19, 2024
View reviewed changes
Copy link
Contributor

@Jibing-Li Jibing-Li left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@924060929 924060929 merged commit d619683 into apache:master Jul 19, 2024
@924060929 924060929 deleted the enable_distribute_planner branch July 19, 2024 04:04
@924060929 924060929 mentioned this pull request Jul 19, 2024
Merged
924060929 added a commit that referenced this pull request Jul 19, 2024
@924060929
[fix](test) fix test which conflict between #37219 and #37951 (#38157)
6eb3eac 
fix test which conflict between #37219 and #37951
dataroaring pushed a commit that referenced this pull request Jul 22, 2024
@924060929 @dataroaring
[enhancement](nereids) enhancement distribute planner for data distri…
8d428b6 
…bution (#37219)

1. add benchmark shape test for distribute planner
2. support read tables of information_schema
3. fix can not execute query when all tablets pruned
4. propagate data distribution when the common column of grouping sets
is the distribution column
dataroaring pushed a commit that referenced this pull request Jul 22, 2024
@924060929 @dataroaring
[fix](test) fix test which conflict between #37219 and #37951 (#38157)
a1d3328 
fix test which conflict between #37219 and #37951
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Nov 19, 2024
@morrySnow
[fix](Nereids) repeat's output properties is not right
5fae4ed 
related PR apache#37219

when child's output is hash distribution, repeat's output
follow child only when all child's hash columns in all
repeat set. otherwise, repeat's output should be any.
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Nov 19, 2024
@morrySnow
[fix](Nereids) repeat's output properties is not right
a0d3389 
related PR apache#37219

when child's output is hash distribution, repeat's output
follow child only when all child's hash columns in all
repeat set. otherwise, repeat's output should be any.
@morrySnow morrySnow mentioned this pull request Nov 19, 2024
Merged
16 tasks
morrySnow added a commit that referenced this pull request Nov 20, 2024
@morrySnow
[fix](Nereids) repeat's output properties is not right (#44297)
dda9e8b 
Related PR: #37219

Problem Summary:
when child's output is hash distribution, repeat's output follow child
only when all child's hash columns in all repeat set. otherwise,
repeat's output should be any.
github-actions bot pushed a commit that referenced this pull request Nov 20, 2024
@morrySnow
[fix](Nereids) repeat's output properties is not right (#44297)
7cd2ecf 
Related PR: #37219

Problem Summary:
when child's output is hash distribution, repeat's output follow child
only when all child's hash columns in all repeat set. otherwise,
repeat's output should be any.
@924060929 924060929 mentioned this pull request Feb 13, 2025
Merged
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jibing-Li Jibing-Li Jibing-Li approved these changes

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@feiniaofeiafei feiniaofeiafei feiniaofeiafei approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.1-merged not-merge/2.1 reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@924060929 @doris-robot @feiniaofeiafei @Jibing-Li @morrySnow @dataroaring