Skip to content

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jun 30, 2024

Let SQL Block rule feature support external table:

create sql_block_rule r1 properties("partition_num" = "100", "global" = "true");
create sql_block_rule r2 properties("tablet_num" = "100", "global" = "true");
create sql_block_rule r3 properties("cardinality" = "100", "global" = "true");

For external table, tablet_num equals to splitNum

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman
Copy link
Contributor Author

run buildall

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39628 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b218c934a18598f80066b74d4000c70c12c0ea50, data reload: false

------ Round 1 ----------------------------------
q1	17601	4296	4278	4278
q2	2005	190	199	190
q3	10461	1249	1028	1028
q4	10187	692	818	692
q5	7486	2655	2630	2630
q6	218	139	143	139
q7	943	598	612	598
q8	9225	2087	2068	2068
q9	8964	6477	6507	6477
q10	8976	3668	3714	3668
q11	449	233	229	229
q12	516	244	230	230
q13	17766	3042	2995	2995
q14	273	236	237	236
q15	527	485	489	485
q16	514	381	379	379
q17	965	554	698	554
q18	7933	7510	7414	7414
q19	9679	1537	1543	1537
q20	672	319	332	319
q21	4874	3144	3932	3144
q22	398	339	338	338
Total cold run time: 120632 ms
Total hot run time: 39628 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4210	4231	4210
q2	371	271	272	271
q3	3163	2951	2902	2902
q4	1968	1746	1712	1712
q5	5460	5473	5480	5473
q6	237	146	143	143
q7	2246	1860	1822	1822
q8	3324	3410	3421	3410
q9	8673	8809	8696	8696
q10	4103	3769	3856	3769
q11	593	490	494	490
q12	858	669	619	619
q13	15919	3152	3215	3152
q14	294	284	259	259
q15	547	483	481	481
q16	500	444	450	444
q17	1858	1537	1523	1523
q18	8153	7877	7715	7715
q19	1800	1703	1585	1585
q20	2132	1868	1848	1848
q21	10195	4906	4781	4781
q22	623	543	582	543
Total cold run time: 77422 ms
Total hot run time: 55848 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175078 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b218c934a18598f80066b74d4000c70c12c0ea50, data reload: false

query1	918	384	377	377
query2	6458	2393	2443	2393
query3	6654	208	211	208
query4	19872	17774	17301	17301
query5	3691	483	481	481
query6	255	168	171	168
query7	4596	295	288	288
query8	321	290	286	286
query9	8707	2393	2373	2373
query10	581	324	298	298
query11	10690	10119	10216	10119
query12	114	84	81	81
query13	1647	378	352	352
query14	9598	7365	7771	7365
query15	249	189	188	188
query16	7762	260	282	260
query17	1603	542	535	535
query18	1948	288	268	268
query19	203	149	156	149
query20	89	83	83	83
query21	211	133	124	124
query22	4375	4054	3969	3969
query23	34024	33703	33706	33703
query24	10949	2922	2870	2870
query25	613	382	374	374
query26	715	161	159	159
query27	2289	320	326	320
query28	5987	2159	2123	2123
query29	900	642	653	642
query30	262	153	160	153
query31	987	784	769	769
query32	94	52	55	52
query33	676	298	291	291
query34	918	477	494	477
query35	763	642	666	642
query36	1132	984	967	967
query37	140	80	91	80
query38	2923	2922	2843	2843
query39	914	848	814	814
query40	206	125	121	121
query41	58	51	51	51
query42	119	108	107	107
query43	603	553	510	510
query44	1123	713	726	713
query45	196	163	162	162
query46	1071	724	730	724
query47	1854	1762	1752	1752
query48	365	300	294	294
query49	835	420	415	415
query50	774	392	382	382
query51	6866	6859	6792	6792
query52	106	96	100	96
query53	361	281	293	281
query54	897	457	452	452
query55	75	75	75	75
query56	285	264	266	264
query57	1122	1059	1040	1040
query58	261	248	256	248
query59	3728	3232	3186	3186
query60	307	302	271	271
query61	96	91	93	91
query62	605	450	451	450
query63	331	295	295	295
query64	8514	2249	1747	1747
query65	3191	3107	3095	3095
query66	749	321	331	321
query67	15285	15064	15199	15064
query68	4921	521	546	521
query69	619	489	336	336
query70	1205	1131	1104	1104
query71	456	289	282	282
query72	7658	5571	5593	5571
query73	767	321	320	320
query74	5916	5597	5562	5562
query75	3650	2680	2691	2680
query76	3548	936	989	936
query77	660	327	320	320
query78	10302	9939	9869	9869
query79	2298	510	514	510
query80	1823	473	479	473
query81	563	223	216	216
query82	1425	113	103	103
query83	309	169	177	169
query84	262	87	87	87
query85	1314	288	270	270
query86	479	323	338	323
query87	3319	3124	3097	3097
query88	4104	2388	2362	2362
query89	473	393	373	373
query90	1794	182	192	182
query91	129	98	97	97
query92	60	48	51	48
query93	2203	507	497	497
query94	1080	188	186	186
query95	402	312	315	312
query96	585	273	263	263
query97	3235	3054	3054	3054
query98	228	199	199	199
query99	1147	864	812	812
Total cold run time: 271745 ms
Total hot run time: 175078 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b218c934a18598f80066b74d4000c70c12c0ea50, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.07	0.07
query5	0.52	0.49	0.49
query6	1.15	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.49
query10	0.54	0.55	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.59
query14	0.80	0.78	0.78
query15	0.84	0.80	0.82
query16	0.38	0.38	0.37
query17	0.97	1.02	1.04
query18	0.24	0.21	0.27
query19	1.80	1.74	1.72
query20	0.02	0.01	0.01
query21	15.43	0.77	0.66
query22	4.10	6.45	2.43
query23	18.31	1.33	1.20
query24	2.09	0.23	0.24
query25	0.16	0.10	0.08
query26	0.26	0.18	0.17
query27	0.07	0.07	0.07
query28	13.18	1.01	1.01
query29	12.63	3.27	3.25
query30	0.25	0.06	0.06
query31	2.88	0.40	0.39
query32	3.28	0.48	0.46
query33	2.88	2.94	2.89
query34	17.04	4.44	4.45
query35	4.47	4.44	4.48
query36	0.65	0.46	0.49
query37	0.18	0.15	0.14
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.19	0.14	0.14
query41	0.09	0.04	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.25 s
Total hot run time: 31.02 s

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39811 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 60e6c7d9cb96e9e62099bc2a25f45aaf06624ef7, data reload: false

------ Round 1 ----------------------------------
q1	18104	4507	4375	4375
q2	2612	195	192	192
q3	12216	1200	1070	1070
q4	10360	764	860	764
q5	7720	2647	2655	2647
q6	226	142	140	140
q7	960	604	619	604
q8	9343	2078	2066	2066
q9	9000	6483	6469	6469
q10	8853	3692	3710	3692
q11	432	237	236	236
q12	433	227	228	227
q13	17860	2964	2973	2964
q14	254	224	218	218
q15	527	493	489	489
q16	523	375	373	373
q17	962	611	639	611
q18	7992	7460	7371	7371
q19	4857	1431	1434	1431
q20	651	332	335	332
q21	4915	3209	3352	3209
q22	392	340	331	331
Total cold run time: 119192 ms
Total hot run time: 39811 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4354	4223	4267	4223
q2	364	273	259	259
q3	2980	2739	2677	2677
q4	1851	1564	1590	1564
q5	5278	5254	5290	5254
q6	216	128	130	128
q7	2131	1742	1732	1732
q8	3182	3318	3293	3293
q9	8330	8310	8359	8310
q10	3887	3651	3711	3651
q11	586	493	491	491
q12	793	608	573	573
q13	16310	2998	2959	2959
q14	292	252	273	252
q15	528	466	480	466
q16	464	413	414	413
q17	1765	1470	1482	1470
q18	7642	7612	7524	7524
q19	1693	1594	1482	1482
q20	1981	1815	1790	1790
q21	4918	4762	4693	4693
q22	607	528	572	528
Total cold run time: 70152 ms
Total hot run time: 53732 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170400 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 60e6c7d9cb96e9e62099bc2a25f45aaf06624ef7, data reload: false

query1	926	375	372	372
query2	6469	2355	2356	2355
query3	6651	210	210	210
query4	19975	17627	17236	17236
query5	4188	500	505	500
query6	283	198	173	173
query7	4603	291	292	291
query8	328	290	292	290
query9	8599	2364	2378	2364
query10	636	298	300	298
query11	10508	10025	10009	10009
query12	136	89	86	86
query13	1639	369	384	369
query14	8546	7329	7020	7020
query15	231	192	186	186
query16	7838	266	269	266
query17	1903	549	527	527
query18	1966	269	269	269
query19	198	158	151	151
query20	94	83	82	82
query21	207	132	122	122
query22	4397	4077	4091	4077
query23	33834	33228	33232	33228
query24	11080	2829	2807	2807
query25	608	360	364	360
query26	1227	149	147	147
query27	3002	317	322	317
query28	7321	2035	2041	2035
query29	890	628	617	617
query30	270	151	154	151
query31	953	769	737	737
query32	100	53	54	53
query33	800	294	287	287
query34	949	463	470	463
query35	783	642	596	596
query36	1092	910	955	910
query37	159	73	76	73
query38	2842	2737	2763	2737
query39	873	827	789	789
query40	203	130	126	126
query41	61	53	52	52
query42	121	104	104	104
query43	581	562	545	545
query44	1169	722	731	722
query45	197	172	170	170
query46	1062	733	716	716
query47	1865	1753	1761	1753
query48	387	302	295	295
query49	1091	410	413	410
query50	767	379	388	379
query51	6783	6874	6737	6737
query52	112	89	96	89
query53	359	297	306	297
query54	916	445	444	444
query55	76	73	76	73
query56	289	278	269	269
query57	1114	1068	1024	1024
query58	257	246	269	246
query59	3372	3116	3046	3046
query60	323	290	290	290
query61	94	91	92	91
query62	662	468	458	458
query63	318	310	297	297
query64	8958	2257	1774	1774
query65	3177	3112	3122	3112
query66	804	332	329	329
query67	15320	14841	14914	14841
query68	4578	531	541	531
query69	464	313	318	313
query70	1177	1156	1142	1142
query71	411	296	282	282
query72	7195	5725	2773	2773
query73	734	324	323	323
query74	6016	5541	5491	5491
query75	3503	2703	2768	2703
query76	2837	1009	856	856
query77	470	321	370	321
query78	10257	9729	9723	9723
query79	1783	513	521	513
query80	1775	480	477	477
query81	573	215	221	215
query82	709	112	107	107
query83	254	168	176	168
query84	277	82	81	81
query85	1380	288	287	287
query86	416	325	319	319
query87	3265	3092	3108	3092
query88	3553	2384	2372	2372
query89	489	390	387	387
query90	1898	186	187	186
query91	124	98	102	98
query92	59	50	52	50
query93	2491	509	511	509
query94	1160	185	184	184
query95	410	310	320	310
query96	597	269	269	269
query97	3221	3024	3085	3024
query98	217	211	201	201
query99	1258	853	844	844
Total cold run time: 271093 ms
Total hot run time: 170400 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 60e6c7d9cb96e9e62099bc2a25f45aaf06624ef7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.09	0.08
query5	0.50	0.49	0.50
query6	1.14	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.55	0.49	0.50
query10	0.54	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.59	0.59	0.60
query14	0.77	0.77	0.81
query15	0.86	0.81	0.82
query16	0.38	0.35	0.37
query17	1.05	0.96	1.04
query18	0.25	0.25	0.24
query19	1.94	1.81	1.70
query20	0.02	0.01	0.00
query21	15.47	0.74	0.66
query22	5.01	6.10	2.17
query23	18.25	1.39	1.33
query24	2.17	0.22	0.21
query25	0.15	0.10	0.08
query26	0.26	0.17	0.17
query27	0.08	0.07	0.09
query28	13.19	1.02	1.00
query29	12.58	3.30	3.25
query30	0.27	0.06	0.05
query31	2.86	0.40	0.39
query32	3.25	0.48	0.48
query33	2.94	2.91	2.94
query34	17.14	4.39	4.42
query35	4.48	4.48	4.50
query36	0.65	0.48	0.47
query37	0.18	0.16	0.15
query38	0.14	0.14	0.15
query39	0.04	0.03	0.03
query40	0.18	0.14	0.16
query41	0.10	0.04	0.06
query42	0.07	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.49 s
Total hot run time: 30.84 s

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39911 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2eacdaf84eb8dddbe001198a6551817e9b86a584, data reload: false

------ Round 1 ----------------------------------
q1	17825	4444	4349	4349
q2	2357	185	187	185
q3	10472	1185	1132	1132
q4	10199	757	783	757
q5	7505	2640	2605	2605
q6	213	133	132	132
q7	952	593	602	593
q8	9227	2051	2057	2051
q9	8876	6463	6461	6461
q10	8981	3755	3687	3687
q11	472	227	232	227
q12	398	230	229	229
q13	18678	3028	2976	2976
q14	246	234	212	212
q15	525	500	498	498
q16	527	375	379	375
q17	959	617	646	617
q18	8064	7436	7401	7401
q19	6610	1419	1417	1417
q20	664	327	328	327
q21	5003	3351	3900	3351
q22	385	331	329	329
Total cold run time: 119138 ms
Total hot run time: 39911 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4352	4270	4198	4198
q2	369	265	276	265
q3	2937	2810	2749	2749
q4	1840	1596	1552	1552
q5	5264	5253	5218	5218
q6	217	132	129	129
q7	2103	1742	1712	1712
q8	3181	3302	3259	3259
q9	8309	8328	8318	8318
q10	3841	3667	3643	3643
q11	572	500	475	475
q12	790	589	605	589
q13	17260	2997	2983	2983
q14	291	252	265	252
q15	521	499	487	487
q16	468	420	404	404
q17	1779	1450	1446	1446
q18	7690	7529	7366	7366
q19	1655	1445	1614	1445
q20	1951	1767	1776	1767
q21	4856	4664	4669	4664
q22	622	518	514	514
Total cold run time: 70868 ms
Total hot run time: 53435 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170295 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2eacdaf84eb8dddbe001198a6551817e9b86a584, data reload: false

query1	912	370	362	362
query2	6451	2368	2323	2323
query3	6653	210	218	210
query4	19601	17464	17196	17196
query5	4149	498	483	483
query6	255	190	169	169
query7	4603	294	290	290
query8	306	279	286	279
query9	8638	2375	2348	2348
query10	602	288	281	281
query11	10692	10102	10039	10039
query12	135	88	83	83
query13	1641	367	367	367
query14	10126	7407	7147	7147
query15	228	186	197	186
query16	7769	279	270	270
query17	1888	552	555	552
query18	1827	284	282	282
query19	195	155	160	155
query20	94	83	86	83
query21	220	135	128	128
query22	4499	4268	4218	4218
query23	33756	32986	33062	32986
query24	11881	2828	2831	2828
query25	690	387	444	387
query26	1760	145	163	145
query27	2923	312	317	312
query28	7218	2067	2037	2037
query29	1087	608	602	602
query30	284	150	151	150
query31	967	757	760	757
query32	94	51	54	51
query33	793	288	286	286
query34	957	467	472	467
query35	763	615	630	615
query36	1067	962	962	962
query37	211	76	75	75
query38	2881	2758	2754	2754
query39	896	796	784	784
query40	280	123	122	122
query41	52	50	51	50
query42	120	96	101	96
query43	570	548	514	514
query44	1248	722	713	713
query45	202	161	159	159
query46	1076	753	720	720
query47	1867	1800	1778	1778
query48	354	295	297	295
query49	1192	402	403	402
query50	772	386	392	386
query51	6789	6661	6740	6661
query52	109	90	97	90
query53	360	297	290	290
query54	1065	448	433	433
query55	73	74	72	72
query56	275	265	273	265
query57	1170	1064	1043	1043
query58	247	241	239	239
query59	3300	3237	3079	3079
query60	305	279	273	273
query61	91	92	92	92
query62	651	463	452	452
query63	324	284	287	284
query64	9824	2269	1750	1750
query65	3193	3095	3097	3095
query66	1353	340	319	319
query67	15342	15266	15083	15083
query68	4729	552	548	548
query69	629	393	326	326
query70	1216	1165	1106	1106
query71	427	278	284	278
query72	7601	2749	2604	2604
query73	747	320	323	320
query74	5900	5452	5426	5426
query75	4254	2679	2692	2679
query76	3140	962	917	917
query77	689	312	294	294
query78	10411	9971	9779	9779
query79	2247	517	508	508
query80	1487	467	460	460
query81	563	218	218	218
query82	854	103	101	101
query83	295	171	173	171
query84	264	94	93	93
query85	2052	277	314	277
query86	474	341	328	328
query87	3300	3102	3075	3075
query88	3943	2377	2364	2364
query89	473	390	392	390
query90	1936	187	184	184
query91	129	97	98	97
query92	61	49	48	48
query93	2269	506	505	505
query94	1324	188	185	185
query95	405	314	315	314
query96	589	274	266	266
query97	3225	3025	2999	2999
query98	226	198	200	198
query99	1132	846	833	833
Total cold run time: 278907 ms
Total hot run time: 170295 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2eacdaf84eb8dddbe001198a6551817e9b86a584, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.05
query3	0.22	0.05	0.04
query4	1.68	0.07	0.07
query5	0.48	0.48	0.49
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.56	0.48	0.49
query10	0.54	0.54	0.54
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.80	0.78	0.77
query15	0.84	0.80	0.82
query16	0.36	0.36	0.36
query17	1.04	0.95	0.98
query18	0.22	0.25	0.26
query19	1.91	1.71	1.70
query20	0.01	0.00	0.00
query21	15.41	0.80	0.65
query22	4.17	7.23	1.74
query23	18.36	1.37	1.28
query24	2.13	0.24	0.23
query25	0.16	0.09	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.22	1.01	1.00
query29	12.64	3.29	3.32
query30	0.25	0.05	0.06
query31	2.89	0.38	0.38
query32	3.29	0.47	0.47
query33	2.88	2.90	2.95
query34	17.18	4.44	4.42
query35	4.45	4.52	4.49
query36	0.65	0.47	0.46
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.14	0.15
query41	0.10	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.63 s
Total hot run time: 30.41 s

@morningman morningman force-pushed the external_partition_block_rule branch from f52522e to c72406b Compare July 6, 2024 16:02
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39622 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c72406bbcd91cc3d26651dafebf7dfb1518c12a5, data reload: false

------ Round 1 ----------------------------------
q1	17606	4308	4293	4293
q2	2033	195	193	193
q3	10449	1181	1022	1022
q4	10202	761	809	761
q5	7487	2618	2645	2618
q6	218	137	139	137
q7	944	603	604	603
q8	9228	2038	2065	2038
q9	9022	6426	6445	6426
q10	9013	3716	3746	3716
q11	466	248	243	243
q12	421	225	230	225
q13	17767	3014	3003	3003
q14	272	226	237	226
q15	520	484	493	484
q16	504	375	371	371
q17	947	587	749	587
q18	7999	7499	7365	7365
q19	3685	1499	1529	1499
q20	661	322	354	322
q21	4954	3155	3908	3155
q22	395	345	335	335
Total cold run time: 114793 ms
Total hot run time: 39622 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4429	4312	4271	4271
q2	369	258	277	258
q3	2946	2762	2865	2762
q4	1947	1632	1690	1632
q5	5560	5612	5493	5493
q6	231	145	134	134
q7	2173	1837	1855	1837
q8	3296	3453	3358	3358
q9	8691	8749	8847	8749
q10	4111	3944	3749	3749
q11	606	510	488	488
q12	825	659	649	649
q13	16320	3208	3216	3208
q14	300	275	291	275
q15	524	484	494	484
q16	490	438	444	438
q17	1796	1533	1526	1526
q18	8090	8119	7974	7974
q19	1867	1594	1585	1585
q20	2588	1886	1874	1874
q21	5032	4866	4865	4865
q22	657	571	554	554
Total cold run time: 72848 ms
Total hot run time: 56163 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172306 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c72406bbcd91cc3d26651dafebf7dfb1518c12a5, data reload: false

query1	909	369	360	360
query2	6460	2524	2583	2524
query3	6645	209	226	209
query4	21990	17322	17330	17322
query5	3668	469	484	469
query6	279	180	167	167
query7	4594	298	297	297
query8	313	296	287	287
query9	8511	2373	2327	2327
query10	561	293	278	278
query11	10907	10076	10140	10076
query12	115	82	90	82
query13	1634	379	378	378
query14	10189	6273	6996	6273
query15	249	182	178	178
query16	7035	311	307	307
query17	1381	586	512	512
query18	1706	281	279	279
query19	201	150	148	148
query20	91	82	82	82
query21	207	128	124	124
query22	4412	4028	4062	4028
query23	33907	33694	34193	33694
query24	10972	2792	2791	2791
query25	602	415	389	389
query26	713	155	155	155
query27	2298	316	325	316
query28	6216	2101	2118	2101
query29	907	664	653	653
query30	243	159	153	153
query31	970	766	762	762
query32	99	55	52	52
query33	661	289	296	289
query34	903	492	493	492
query35	738	618	630	618
query36	1153	976	983	976
query37	141	81	80	80
query38	2994	2816	2868	2816
query39	893	794	823	794
query40	211	136	125	125
query41	59	52	54	52
query42	122	97	102	97
query43	599	564	544	544
query44	1116	738	740	738
query45	194	161	164	161
query46	1067	720	703	703
query47	1826	1748	1766	1748
query48	384	306	297	297
query49	849	426	431	426
query50	767	388	400	388
query51	6954	6812	6732	6732
query52	115	92	97	92
query53	368	290	285	285
query54	904	469	455	455
query55	77	74	77	74
query56	305	285	285	285
query57	1154	1049	1023	1023
query58	260	269	252	252
query59	3520	3247	3249	3247
query60	321	295	291	291
query61	118	114	117	114
query62	594	448	431	431
query63	328	300	297	297
query64	9219	2276	1728	1728
query65	3205	3131	3135	3131
query66	771	356	338	338
query67	15649	15177	14987	14987
query68	6281	533	536	533
query69	744	454	351	351
query70	1193	1137	1107	1107
query71	482	284	280	280
query72	9373	5732	4192	4192
query73	811	323	328	323
query74	6006	5557	5454	5454
query75	4309	2693	2693	2693
query76	4301	921	1010	921
query77	765	307	294	294
query78	11748	9500	9632	9500
query79	10053	519	516	516
query80	948	484	480	480
query81	584	219	221	219
query82	443	108	103	103
query83	346	166	163	163
query84	276	84	86	84
query85	712	319	331	319
query86	428	336	322	322
query87	3340	3085	3081	3081
query88	5429	2441	2475	2441
query89	505	394	371	371
query90	2061	190	188	188
query91	130	102	104	102
query92	69	48	53	48
query93	4406	512	500	500
query94	1284	209	209	209
query95	408	328	319	319
query96	599	272	276	272
query97	3212	3004	3014	3004
query98	223	211	200	200
query99	1088	846	838	838
Total cold run time: 290080 ms
Total hot run time: 172306 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c72406bbcd91cc3d26651dafebf7dfb1518c12a5, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.50	0.49	0.48
query6	1.14	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.54	0.49	0.49
query10	0.54	0.54	0.55
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.59	0.59	0.58
query14	0.77	0.78	0.78
query15	0.86	0.82	0.82
query16	0.36	0.37	0.36
query17	1.01	0.96	1.00
query18	0.22	0.24	0.24
query19	1.85	1.74	1.83
query20	0.02	0.01	0.01
query21	15.40	0.77	0.66
query22	4.29	6.86	1.51
query23	18.35	1.40	1.21
query24	2.11	0.23	0.22
query25	0.16	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.22	0.23
query28	13.30	1.02	1.00
query29	12.64	3.31	3.30
query30	0.25	0.06	0.05
query31	2.86	0.38	0.39
query32	3.30	0.48	0.47
query33	2.91	2.92	2.94
query34	17.23	4.40	4.37
query35	4.45	4.44	4.41
query36	0.64	0.46	0.50
query37	0.18	0.17	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.04
query40	0.14	0.13	0.12
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.15 s
Total hot run time: 30.25 s

create sql_block_rule r1 properties("partition_num" = "100", "global" = "true");
@morningman morningman force-pushed the external_partition_block_rule branch from c72406b to eff8312 Compare July 8, 2024 15:17
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40533 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eff8312f254d7d1296f0f73872f5af1e61c438e7, data reload: false

------ Round 1 ----------------------------------
q1	17630	4308	4233	4233
q2	2021	192	188	188
q3	10459	1196	1093	1093
q4	10191	803	809	803
q5	7516	2637	2574	2574
q6	218	136	137	136
q7	954	612	611	611
q8	9216	2076	2047	2047
q9	8996	6612	6548	6548
q10	9069	3704	3733	3704
q11	476	240	240	240
q12	420	244	236	236
q13	18745	2987	3029	2987
q14	265	225	231	225
q15	529	488	485	485
q16	518	394	374	374
q17	955	730	692	692
q18	7981	7490	7324	7324
q19	4466	1453	1427	1427
q20	699	321	324	321
q21	4936	3955	3944	3944
q22	404	343	341	341
Total cold run time: 116664 ms
Total hot run time: 40533 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4331	4213	4277	4213
q2	387	267	260	260
q3	2975	2935	2890	2890
q4	2000	1714	1748	1714
q5	5611	5467	5413	5413
q6	232	135	141	135
q7	2184	1860	1863	1860
q8	3273	3381	3456	3381
q9	8688	8633	8774	8633
q10	4089	3940	3693	3693
q11	613	512	516	512
q12	815	656	628	628
q13	15974	3173	3187	3173
q14	289	281	256	256
q15	515	484	495	484
q16	482	438	413	413
q17	1786	1507	1452	1452
q18	7650	7591	7492	7492
q19	4866	1580	1556	1556
q20	2129	1801	1777	1777
q21	4926	4848	4717	4717
q22	649	621	567	567
Total cold run time: 74464 ms
Total hot run time: 55219 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176391 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eff8312f254d7d1296f0f73872f5af1e61c438e7, data reload: false

query1	913	378	371	371
query2	6447	2453	2311	2311
query3	7449	280	239	239
query4	22224	18211	18181	18181
query5	4105	502	503	502
query6	289	184	158	158
query7	4860	285	301	285
query8	1111	299	312	299
query9	11503	2437	2405	2405
query10	1203	325	301	301
query11	12261	10120	10079	10079
query12	132	86	85	85
query13	1684	365	366	365
query14	10313	8223	8000	8000
query15	341	184	192	184
query16	8001	305	315	305
query17	1817	543	518	518
query18	1997	291	266	266
query19	321	146	153	146
query20	92	85	82	82
query21	222	131	127	127
query22	4340	4103	4128	4103
query23	34247	33645	33755	33645
query24	10746	2902	2901	2901
query25	613	444	424	424
query26	1104	151	155	151
query27	2412	272	275	272
query28	7384	2139	2114	2114
query29	896	628	649	628
query30	252	158	154	154
query31	980	776	749	749
query32	98	55	55	55
query33	756	291	281	281
query34	1002	512	504	504
query35	689	583	623	583
query36	1130	955	975	955
query37	140	83	86	83
query38	2923	2824	2822	2822
query39	940	832	856	832
query40	200	129	128	128
query41	55	53	53	53
query42	124	105	103	103
query43	611	562	560	560
query44	1277	765	764	764
query45	208	174	170	170
query46	1132	734	733	733
query47	1969	1831	1913	1831
query48	389	309	308	308
query49	879	426	442	426
query50	809	412	419	412
query51	7002	6930	6883	6883
query52	103	97	91	91
query53	373	293	293	293
query54	913	441	442	441
query55	75	74	72	72
query56	287	272	268	268
query57	1158	1013	1022	1013
query58	249	255	249	249
query59	3269	3331	3157	3157
query60	299	285	278	278
query61	99	94	93	93
query62	792	665	650	650
query63	328	296	292	292
query64	9279	2196	1615	1615
query65	3199	3104	3115	3104
query66	709	325	334	325
query67	15497	15126	15228	15126
query68	5071	529	529	529
query69	611	497	365	365
query70	1202	1150	1139	1139
query71	446	287	289	287
query72	7362	5444	5709	5444
query73	761	323	325	323
query74	5985	5448	5492	5448
query75	3624	2663	2680	2663
query76	3699	831	937	831
query77	610	322	319	319
query78	9631	9005	10288	9005
query79	3374	508	508	508
query80	2238	510	504	504
query81	604	221	226	221
query82	1179	149	136	136
query83	309	184	180	180
query84	261	97	98	97
query85	1632	340	304	304
query86	484	313	324	313
query87	3314	3055	3128	3055
query88	3732	2351	2342	2342
query89	477	381	398	381
query90	1861	194	194	194
query91	133	103	109	103
query92	65	49	52	49
query93	2717	497	506	497
query94	1173	209	214	209
query95	416	318	318	318
query96	582	269	261	261
query97	3188	3058	3016	3016
query98	211	198	199	198
query99	1496	1249	1333	1249
Total cold run time: 287669 ms
Total hot run time: 176391 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eff8312f254d7d1296f0f73872f5af1e61c438e7, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.66	0.09	0.08
query5	0.53	0.48	0.47
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.55	0.55	0.55
query11	0.16	0.11	0.11
query12	0.15	0.13	0.13
query13	0.58	0.59	0.58
query14	0.80	0.78	0.78
query15	0.84	0.82	0.81
query16	0.37	0.37	0.37
query17	0.99	1.04	0.97
query18	0.23	0.21	0.21
query19	1.81	1.73	1.73
query20	0.01	0.01	0.01
query21	15.39	0.75	0.64
query22	3.56	6.60	2.69
query23	18.32	1.45	1.25
query24	2.16	0.23	0.22
query25	0.16	0.09	0.08
query26	0.30	0.21	0.20
query27	0.45	0.22	0.22
query28	13.24	1.00	0.99
query29	12.65	3.25	3.24
query30	0.25	0.06	0.06
query31	2.87	0.38	0.39
query32	3.30	0.46	0.46
query33	2.94	2.88	2.93
query34	17.08	4.34	4.33
query35	4.47	4.42	4.40
query36	0.65	0.46	0.48
query37	0.19	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.12	0.12
query41	0.10	0.04	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.32 s
Total hot run time: 31.23 s

morrySnow
morrySnow previously approved these changes Jul 11, 2024
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40156 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d4bc0d192b86e1af1f3cd3815349a26995e14248, data reload: false

------ Round 1 ----------------------------------
q1	18106	4606	4371	4371
q2	2518	215	195	195
q3	11583	1179	1078	1078
q4	10340	794	884	794
q5	8134	2740	2727	2727
q6	219	139	140	139
q7	997	596	591	591
q8	9212	2077	2102	2077
q9	8669	6547	6534	6534
q10	8669	3801	3782	3782
q11	481	246	244	244
q12	398	237	235	235
q13	17770	3034	2996	2996
q14	267	229	246	229
q15	536	475	495	475
q16	499	372	375	372
q17	966	695	724	695
q18	8173	7529	7434	7434
q19	7146	1518	1561	1518
q20	697	333	326	326
q21	4989	3005	4064	3005
q22	387	339	340	339
Total cold run time: 120756 ms
Total hot run time: 40156 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4442	4318	4286	4286
q2	378	266	272	266
q3	3010	2786	2760	2760
q4	1887	1652	1625	1625
q5	5295	5366	5336	5336
q6	215	130	129	129
q7	2179	1739	1742	1739
q8	3228	3327	3342	3327
q9	8435	8385	8385	8385
q10	3885	3711	3709	3709
q11	569	502	484	484
q12	837	609	597	597
q13	17498	3059	2960	2960
q14	300	273	272	272
q15	515	488	467	467
q16	471	408	429	408
q17	1771	1463	1480	1463
q18	7686	7594	7446	7446
q19	3558	1593	1488	1488
q20	1965	1786	1761	1761
q21	4845	4759	4774	4759
q22	613	537	523	523
Total cold run time: 73582 ms
Total hot run time: 54190 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174249 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d4bc0d192b86e1af1f3cd3815349a26995e14248, data reload: false

query1	928	370	361	361
query2	6450	2391	2228	2228
query3	6644	214	222	214
query4	28187	17533	17406	17406
query5	4179	509	488	488
query6	272	174	164	164
query7	4584	293	303	293
query8	323	293	292	292
query9	8408	2470	2460	2460
query10	431	279	286	279
query11	10497	10114	10041	10041
query12	136	86	85	85
query13	1633	376	376	376
query14	10072	7706	7093	7093
query15	237	188	194	188
query16	7721	312	318	312
query17	1796	565	522	522
query18	1606	273	266	266
query19	186	152	142	142
query20	92	83	85	83
query21	204	125	127	125
query22	4247	4100	3996	3996
query23	33840	33317	33165	33165
query24	11687	2878	2755	2755
query25	650	367	365	365
query26	1685	150	148	148
query27	2836	270	274	270
query28	7638	2125	2110	2110
query29	988	624	625	624
query30	288	151	152	151
query31	963	763	744	744
query32	100	55	52	52
query33	770	295	292	292
query34	930	486	487	486
query35	672	578	571	571
query36	1097	912	967	912
query37	183	85	83	83
query38	2858	2726	2735	2726
query39	871	800	794	794
query40	279	119	125	119
query41	55	53	55	53
query42	120	100	99	99
query43	589	555	574	555
query44	1288	730	730	730
query45	197	163	173	163
query46	1097	726	743	726
query47	1838	1733	1791	1733
query48	376	298	302	298
query49	1195	421	409	409
query50	780	410	438	410
query51	6932	6791	6736	6736
query52	114	90	98	90
query53	361	293	294	293
query54	1034	456	451	451
query55	76	75	75	75
query56	283	268	267	267
query57	1138	1063	1070	1063
query58	255	239	238	238
query59	3357	3091	3189	3091
query60	302	272	274	272
query61	97	90	98	90
query62	811	648	647	647
query63	322	293	286	286
query64	10507	2197	7406	2197
query65	3193	3102	3130	3102
query66	1322	336	338	336
query67	15477	14951	14913	14913
query68	4904	561	565	561
query69	637	440	359	359
query70	1181	1106	1088	1088
query71	421	302	290	290
query72	7197	5534	5499	5499
query73	764	328	331	328
query74	5980	5537	5518	5518
query75	3430	2661	2706	2661
query76	3430	1015	951	951
query77	698	316	308	308
query78	9721	9300	9831	9300
query79	3306	526	520	520
query80	1958	487	487	487
query81	580	224	221	221
query82	828	137	131	131
query83	325	169	176	169
query84	280	88	91	88
query85	1612	314	302	302
query86	323	330	297	297
query87	3249	3120	3107	3107
query88	4242	2447	2457	2447
query89	476	372	384	372
query90	1626	197	199	197
query91	132	106	104	104
query92	64	49	49	49
query93	3601	519	525	519
query94	736	217	215	215
query95	401	330	328	328
query96	606	276	347	276
query97	3250	3027	3045	3027
query98	220	196	197	196
query99	1611	1302	1230	1230
Total cold run time: 288073 ms
Total hot run time: 174249 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.2 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d4bc0d192b86e1af1f3cd3815349a26995e14248, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.07	0.07
query5	0.51	0.48	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.55	0.50	0.50
query10	0.55	0.55	0.56
query11	0.15	0.11	0.11
query12	0.14	0.13	0.14
query13	0.59	0.59	0.58
query14	0.76	0.79	0.77
query15	0.85	0.81	0.82
query16	0.34	0.37	0.36
query17	0.96	0.98	0.96
query18	0.22	0.21	0.22
query19	1.78	1.79	1.83
query20	0.02	0.01	0.00
query21	15.44	0.74	0.66
query22	4.30	6.57	2.38
query23	18.28	1.47	1.33
query24	2.06	0.24	0.21
query25	0.16	0.09	0.08
query26	0.30	0.20	0.21
query27	0.46	0.22	0.23
query28	13.28	1.01	1.00
query29	12.71	3.29	3.32
query30	0.26	0.05	0.05
query31	2.88	0.39	0.39
query32	3.25	0.48	0.47
query33	2.90	2.92	3.02
query34	16.93	4.38	4.36
query35	4.39	4.39	4.47
query36	0.65	0.45	0.49
query37	0.18	0.16	0.15
query38	0.17	0.15	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.05
Total cold run time: 109.62 s
Total hot run time: 31.2 s

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 12, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit bd05cba into apache:master Jul 12, 2024
morningman added a commit to morningman/doris that referenced this pull request Jul 14, 2024
…he#37041)

Let SQL Block rule feature support external table:
```
create sql_block_rule r1 properties("partition_num" = "100", "global" = "true");
create sql_block_rule r2 properties("tablet_num" = "100", "global" = "true");
create sql_block_rule r3 properties("cardinality" = "100", "global" = "true");
```

For external table, `tablet_num` equals to `splitNum`
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
…he#37041)

Let SQL Block rule feature support external table:
```
create sql_block_rule r1 properties("partition_num" = "100", "global" = "true");
create sql_block_rule r2 properties("tablet_num" = "100", "global" = "true");
create sql_block_rule r3 properties("cardinality" = "100", "global" = "true");
```

For external table, `tablet_num` equals to `splitNum`
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
Let SQL Block rule feature support external table:
```
create sql_block_rule r1 properties("partition_num" = "100", "global" = "true");
create sql_block_rule r2 properties("tablet_num" = "100", "global" = "true");
create sql_block_rule r3 properties("cardinality" = "100", "global" = "true");
```

For external table, `tablet_num` equals to `splitNum`
morningman added a commit to morningman/doris that referenced this pull request Aug 5, 2024
…he#37041)

Let SQL Block rule feature support external table:
```
create sql_block_rule r1 properties("partition_num" = "100", "global" = "true");
create sql_block_rule r2 properties("tablet_num" = "100", "global" = "true");
create sql_block_rule r3 properties("cardinality" = "100", "global" = "true");
```

For external table, `tablet_num` equals to `splitNum`
morningman added a commit to morningman/doris that referenced this pull request Aug 6, 2024
…he#37041)

Let SQL Block rule feature support external table:
```
create sql_block_rule r1 properties("partition_num" = "100", "global" = "true");
create sql_block_rule r2 properties("tablet_num" = "100", "global" = "true");
create sql_block_rule r3 properties("cardinality" = "100", "global" = "true");
```

For external table, `tablet_num` equals to `splitNum`
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants