Skip to content

[fix] (compaction) fix time series compaction policy #38220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
qidaye merged 1 commit into from
Aug 5, 2024
Merged

[fix] (compaction) fix time series compaction policy #38220

qidaye merged 1 commit into from
Aug 5, 2024

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Jul 22, 2024 •
edited
Loading

Proposed changes

  1. Time series policy does not rely on permits
  2. fix compaction in cloud mode
  3. fix compaction level
    Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 22, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 22, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

TPC-H: Total hot run time: 39866 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5cd701157585178a1a5d79464bdfaa7d8fb401ab, data reload: false

------ Round 1 ----------------------------------
q1	18454	4418	4331	4331
q2	2423	195	197	195
q3	11058	1166	1153	1153
q4	10300	805	823	805
q5	7706	2805	2686	2686
q6	222	138	138	138
q7	958	600	609	600
q8	9591	2102	2090	2090
q9	8844	6567	6560	6560
q10	8804	3809	3730	3730
q11	462	239	237	237
q12	389	225	220	220
q13	17945	2976	2969	2969
q14	277	235	246	235
q15	518	473	476	473
q16	487	373	380	373
q17	962	615	731	615
q18	8005	7494	7379	7379
q19	5721	1266	1303	1266
q20	688	330	340	330
q21	4928	3260	3196	3196
q22	335	288	285	285
Total cold run time: 119077 ms
Total hot run time: 39866 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4466	4226	4222	4222
q2	381	261	269	261
q3	2990	2789	2738	2738
q4	1852	1563	1590	1563
q5	5251	5311	5312	5311
q6	222	130	129	129
q7	2110	1755	1777	1755
q8	3262	3341	3293	3293
q9	8426	8353	8430	8353
q10	3868	3712	3701	3701
q11	599	478	503	478
q12	755	634	613	613
q13	16610	2976	2968	2968
q14	301	280	274	274
q15	512	470	466	466
q16	468	432	414	414
q17	1757	1480	1466	1466
q18	7620	7522	7359	7359
q19	1632	1603	1481	1481
q20	1953	1789	1762	1762
q21	4733	4605	4728	4605
q22	558	488	510	488
Total cold run time: 70326 ms
Total hot run time: 53700 ms

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

TPC-DS: Total hot run time: 172397 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5cd701157585178a1a5d79464bdfaa7d8fb401ab, data reload: false

query1	909	368	364	364
query2	6431	1932	1712	1712
query3	6674	202	215	202
query4	28346	17572	17159	17159
query5	4221	484	460	460
query6	267	174	176	174
query7	4594	284	287	284
query8	241	198	203	198
query9	8478	2458	2435	2435
query10	442	306	271	271
query11	12392	10095	10101	10095
query12	136	85	80	80
query13	1638	365	355	355
query14	10169	7624	6778	6778
query15	218	167	169	167
query16	7838	444	472	444
query17	1596	571	546	546
query18	1975	291	288	288
query19	202	154	162	154
query20	93	84	81	81
query21	216	129	129	129
query22	4486	4150	4062	4062
query23	33979	33376	33305	33305
query24	11815	2936	2935	2935
query25	684	385	396	385
query26	1785	155	151	151
query27	2902	265	272	265
query28	7680	1994	1981	1981
query29	1134	647	636	636
query30	286	151	147	147
query31	945	736	756	736
query32	99	53	56	53
query33	781	339	350	339
query34	913	476	483	476
query35	867	743	725	725
query36	1078	943	927	927
query37	163	84	79	79
query38	2859	2814	2760	2760
query39	852	827	820	820
query40	282	124	122	122
query41	51	49	50	49
query42	130	104	105	104
query43	504	456	469	456
query44	1243	711	709	709
query45	189	162	163	162
query46	1086	719	731	719
query47	1872	1770	1778	1770
query48	365	301	300	300
query49	1179	416	431	416
query50	790	406	391	391
query51	6837	6631	6679	6631
query52	106	92	95	92
query53	358	290	290	290
query54	973	464	453	453
query55	75	76	75	75
query56	396	272	268	268
query57	1162	1076	1039	1039
query58	270	257	256	256
query59	2909	2651	2638	2638
query60	306	292	273	273
query61	95	108	95	95
query62	835	641	654	641
query63	324	286	283	283
query64	10400	2233	1655	1655
query65	3193	3099	3097	3097
query66	1375	330	335	330
query67	15668	15013	15255	15013
query68	6326	534	542	534
query69	745	471	358	358
query70	1175	1168	1112	1112
query71	514	280	276	276
query72	8356	5472	5635	5472
query73	811	342	323	323
query74	6247	5710	5700	5700
query75	5042	2649	2693	2649
query76	4693	946	955	946
query77	769	293	293	293
query78	9631	8946	8924	8924
query79	7592	516	512	512
query80	2094	486	476	476
query81	584	222	219	219
query82	1165	131	134	131
query83	286	168	168	168
query84	278	84	92	84
query85	1419	354	295	295
query86	443	319	318	318
query87	3310	3133	3169	3133
query88	4950	2445	2460	2445
query89	500	378	369	369
query90	1992	189	191	189
query91	134	100	106	100
query92	62	48	50	48
query93	6196	500	489	489
query94	1299	287	289	287
query95	402	312	308	308
query96	600	274	277	274
query97	3213	3007	3041	3007
query98	222	204	198	198
query99	1551	1280	1275	1275
Total cold run time: 305958 ms
Total hot run time: 172397 ms

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

ClickBench: Total hot run time: 31.09 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5cd701157585178a1a5d79464bdfaa7d8fb401ab, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.49	0.47	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.55	0.50	0.48
query10	0.55	0.55	0.55
query11	0.14	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.60	0.58
query14	0.77	0.76	0.78
query15	0.84	0.80	0.81
query16	0.34	0.36	0.35
query17	1.04	0.97	1.05
query18	0.23	0.22	0.22
query19	1.83	1.71	1.72
query20	0.01	0.01	0.01
query21	15.40	0.81	0.67
query22	4.53	6.78	2.57
query23	18.27	1.42	1.23
query24	2.09	0.24	0.22
query25	0.15	0.09	0.07
query26	0.30	0.21	0.22
query27	0.46	0.23	0.23
query28	13.27	1.03	0.99
query29	12.65	3.29	3.28
query30	0.25	0.05	0.06
query31	2.87	0.39	0.38
query32	3.26	0.47	0.47
query33	2.91	2.92	2.87
query34	16.88	4.33	4.32
query35	4.43	4.45	4.41
query36	0.66	0.47	0.47
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.11
query41	0.10	0.04	0.04
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.87 s
Total hot run time: 31.09 s

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 23, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 23, 2024

TPC-H: Total hot run time: 40162 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90767d82bfd28a49622e5528ff44d414936d75f2, data reload: false

------ Round 1 ----------------------------------
q1	18033	4510	4426	4426
q2	2901	196	199	196
q3	11613	1205	1095	1095
q4	11431	881	819	819
q5	8178	2697	2686	2686
q6	223	152	139	139
q7	973	610	590	590
q8	9216	2094	2091	2091
q9	8812	6606	6604	6604
q10	8817	3790	3798	3790
q11	451	233	236	233
q12	398	230	227	227
q13	18907	2977	3022	2977
q14	286	229	232	229
q15	529	478	501	478
q16	516	388	390	388
q17	977	730	644	644
q18	8124	7571	7411	7411
q19	8335	1439	1398	1398
q20	702	331	314	314
q21	4962	3145	3235	3145
q22	348	282	289	282
Total cold run time: 124732 ms
Total hot run time: 40162 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4425	4351	4283	4283
q2	383	263	272	263
q3	3009	2765	2759	2759
q4	1876	1626	1624	1624
q5	5266	5311	5297	5297
q6	223	138	137	137
q7	2073	1753	1770	1753
q8	3219	3379	3319	3319
q9	8522	8428	8442	8428
q10	3912	3756	3768	3756
q11	615	490	495	490
q12	785	639	599	599
q13	17495	2987	2998	2987
q14	288	270	274	270
q15	522	476	479	476
q16	490	414	412	412
q17	1796	1484	1472	1472
q18	7727	7522	7371	7371
q19	1675	1526	1574	1526
q20	1991	1778	1808	1778
q21	4845	4841	4619	4619
q22	605	498	510	498
Total cold run time: 71742 ms
Total hot run time: 54117 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2024

TPC-DS: Total hot run time: 173237 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90767d82bfd28a49622e5528ff44d414936d75f2, data reload: false

query1	920	363	368	363
query2	6437	1890	1752	1752
query3	6655	205	224	205
query4	28417	17673	17528	17528
query5	4260	483	481	481
query6	269	161	164	161
query7	4587	298	296	296
query8	255	201	202	201
query9	8645	2480	2435	2435
query10	451	289	280	280
query11	12168	10142	10042	10042
query12	132	84	81	81
query13	1649	378	372	372
query14	10198	7582	7777	7582
query15	224	163	166	163
query16	7836	485	455	455
query17	1551	548	520	520
query18	1985	273	275	273
query19	191	150	151	150
query20	88	82	80	80
query21	207	128	125	125
query22	4347	4043	4089	4043
query23	33787	33269	33228	33228
query24	12042	2935	2824	2824
query25	660	372	371	371
query26	1809	161	165	161
query27	2714	273	280	273
query28	7457	2031	2052	2031
query29	1104	632	622	622
query30	298	147	153	147
query31	984	731	743	731
query32	99	54	54	54
query33	768	339	338	338
query34	895	485	492	485
query35	870	726	737	726
query36	1099	915	925	915
query37	274	81	80	80
query38	2877	2758	2745	2745
query39	872	822	822	822
query40	273	119	117	117
query41	50	46	48	46
query42	123	102	105	102
query43	494	464	450	450
query44	1298	758	739	739
query45	188	161	163	161
query46	1092	712	747	712
query47	1852	1770	1770	1770
query48	370	293	286	286
query49	1201	410	438	410
query50	781	406	401	401
query51	6908	6785	6771	6771
query52	105	93	95	93
query53	354	287	291	287
query54	996	450	449	449
query55	77	77	75	75
query56	288	263	279	263
query57	1160	1042	1060	1042
query58	262	259	262	259
query59	2764	2549	2562	2549
query60	311	276	276	276
query61	97	93	94	93
query62	845	643	722	643
query63	324	300	309	300
query64	10492	2213	1670	1670
query65	3200	3157	3124	3124
query66	1401	334	332	332
query67	15384	15052	14825	14825
query68	5419	550	563	550
query69	631	432	366	366
query70	1151	1105	1071	1071
query71	433	281	293	281
query72	7711	5571	5426	5426
query73	768	334	330	330
query74	6172	5754	5686	5686
query75	3646	2703	2713	2703
query76	3570	946	958	946
query77	677	315	314	314
query78	9591	8976	8921	8921
query79	2574	519	519	519
query80	2101	473	474	473
query81	585	222	239	222
query82	1379	137	134	134
query83	283	188	169	169
query84	265	93	88	88
query85	1613	323	310	310
query86	478	312	322	312
query87	3276	3061	3166	3061
query88	4136	2403	2420	2403
query89	475	386	384	384
query90	1809	201	198	198
query91	129	98	103	98
query92	63	49	49	49
query93	2349	521	508	508
query94	1243	295	289	289
query95	418	335	410	335
query96	598	274	273	273
query97	3207	3027	3004	3004
query98	225	206	198	198
query99	1604	1272	1230	1230
Total cold run time: 291350 ms
Total hot run time: 173237 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2024

ClickBench: Total hot run time: 30.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 90767d82bfd28a49622e5528ff44d414936d75f2, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.09
query5	0.49	0.49	0.52
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.49
query10	0.55	0.55	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.58	0.59
query14	0.78	0.80	0.76
query15	0.84	0.81	0.81
query16	0.36	0.36	0.36
query17	1.00	0.99	0.96
query18	0.23	0.22	0.23
query19	1.77	1.69	1.75
query20	0.01	0.01	0.01
query21	15.44	0.76	0.66
query22	4.38	7.57	1.54
query23	18.26	1.32	1.22
query24	2.09	0.26	0.22
query25	0.16	0.08	0.08
query26	0.30	0.22	0.21
query27	0.45	0.23	0.24
query28	13.22	1.01	1.00
query29	12.63	3.31	3.29
query30	0.25	0.06	0.06
query31	2.86	0.39	0.39
query32	3.26	0.48	0.47
query33	2.90	2.92	2.88
query34	17.12	4.32	4.30
query35	4.43	4.41	4.40
query36	0.66	0.47	0.47
query37	0.19	0.15	0.16
query38	0.15	0.15	0.14
query39	0.05	0.03	0.03
query40	0.16	0.14	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.88 s
Total hot run time: 30.12 s

@csun5285 csun5285 changed the title [fix] (compaction) Time series policy does not rely on permits [fix] (compaction) fix time series compaction policy Jul 25, 2024
@csun5285
Copy link
Contributor Author

csun5285 commented Jul 25, 2024

run buildall

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 25, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jul 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 25, 2024

TPC-H: Total hot run time: 39839 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7820296764d9262b3ad9e1b50cf29c34fa603871, data reload: false

------ Round 1 ----------------------------------
q1	17850	4522	4387	4387
q2	2545	211	198	198
q3	10753	1282	1132	1132
q4	10284	746	806	746
q5	7709	2786	2714	2714
q6	229	145	143	143
q7	1012	617	615	615
q8	9796	1958	1968	1958
q9	8959	6605	6669	6605
q10	8703	3772	3830	3772
q11	461	247	260	247
q12	514	228	223	223
q13	17783	2994	2990	2990
q14	291	236	258	236
q15	528	485	499	485
q16	508	387	384	384
q17	983	658	670	658
q18	8166	7519	7537	7519
q19	1387	1053	970	970
q20	645	346	334	334
q21	5070	3331	3238	3238
q22	352	296	285	285
Total cold run time: 114528 ms
Total hot run time: 39839 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4358	4292	4214	4214
q2	369	265	272	265
q3	3050	2731	2729	2729
q4	1873	1546	1636	1546
q5	5323	5345	5336	5336
q6	220	133	133	133
q7	2161	1677	1769	1677
q8	3226	3349	3308	3308
q9	8484	8456	8396	8396
q10	3871	3649	3722	3649
q11	600	500	492	492
q12	757	601	613	601
q13	16271	2991	2995	2991
q14	311	273	275	273
q15	520	478	474	474
q16	471	431	412	412
q17	1791	1486	1473	1473
q18	7693	7416	7391	7391
q19	1664	1686	1576	1576
q20	1989	1819	1788	1788
q21	4985	4795	4807	4795
q22	611	503	512	503
Total cold run time: 70598 ms
Total hot run time: 54022 ms

@doris-robot
Copy link

doris-robot commented Jul 25, 2024

TPC-DS: Total hot run time: 172624 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7820296764d9262b3ad9e1b50cf29c34fa603871, data reload: false

query1	917	373	365	365
query2	6444	1875	1847	1847
query3	6645	203	213	203
query4	27334	17516	17534	17516
query5	4208	480	485	480
query6	275	181	160	160
query7	4583	306	292	292
query8	240	194	194	194
query9	8543	2449	2435	2435
query10	436	308	281	281
query11	11671	10026	10087	10026
query12	131	93	85	85
query13	1637	368	375	368
query14	10350	8434	7737	7737
query15	230	171	175	171
query16	7303	495	451	451
query17	1611	567	544	544
query18	1276	296	297	296
query19	194	149	148	148
query20	100	86	87	86
query21	211	103	100	100
query22	4279	4106	3934	3934
query23	33877	33095	33189	33095
query24	11072	2895	2954	2895
query25	647	392	394	392
query26	1474	156	152	152
query27	2476	275	276	275
query28	6703	2042	2026	2026
query29	890	436	429	429
query30	293	151	161	151
query31	978	740	729	729
query32	97	57	59	57
query33	810	344	350	344
query34	892	467	468	467
query35	872	817	711	711
query36	1109	952	947	947
query37	129	77	78	77
query38	2874	2775	2734	2734
query39	847	841	817	817
query40	275	116	110	110
query41	46	43	43	43
query42	117	92	100	92
query43	493	463	458	458
query44	1245	741	724	724
query45	208	173	175	173
query46	1104	742	721	721
query47	1863	1764	1798	1764
query48	359	292	289	289
query49	1189	419	405	405
query50	807	406	394	394
query51	6760	6749	6713	6713
query52	100	87	91	87
query53	250	182	182	182
query54	1000	448	448	448
query55	75	73	74	73
query56	300	268	275	268
query57	1198	1047	1047	1047
query58	266	251	257	251
query59	2870	2763	2592	2592
query60	302	304	282	282
query61	99	93	95	93
query62	842	653	659	653
query63	214	186	191	186
query64	10510	2295	1701	1701
query65	3172	3131	3080	3080
query66	1037	336	333	333
query67	15648	14917	14877	14877
query68	5230	563	554	554
query69	702	400	324	324
query70	1200	1091	1087	1087
query71	452	267	258	258
query72	9102	5423	5573	5423
query73	780	333	331	331
query74	6228	5654	5631	5631
query75	3624	2659	2648	2648
query76	3881	974	908	908
query77	656	307	303	303
query78	9713	8988	8921	8921
query79	3273	518	521	518
query80	2574	480	480	480
query81	598	218	220	218
query82	1142	137	140	137
query83	315	172	178	172
query84	273	79	84	79
query85	1630	319	355	319
query86	481	326	303	303
query87	3337	3129	3091	3091
query88	4723	2418	2413	2413
query89	411	291	289	289
query90	1885	191	187	187
query91	126	100	99	99
query92	63	48	50	48
query93	4272	544	539	539
query94	960	298	294	294
query95	360	255	257	255
query96	604	269	265	265
query97	3215	3025	3014	3014
query98	226	214	207	207
query99	1615	1293	1253	1253
Total cold run time: 290658 ms
Total hot run time: 172624 ms

@doris-robot
Copy link

doris-robot commented Jul 25, 2024

ClickBench: Total hot run time: 30.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7820296764d9262b3ad9e1b50cf29c34fa603871, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.07
query5	0.49	0.48	0.46
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.48
query10	0.54	0.53	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.59	0.58	0.57
query14	0.77	0.76	0.77
query15	0.85	0.81	0.81
query16	0.37	0.36	0.36
query17	0.95	0.99	0.98
query18	0.22	0.21	0.22
query19	1.82	1.67	1.71
query20	0.02	0.01	0.01
query21	15.42	0.78	0.64
query22	4.16	7.76	1.84
query23	18.24	1.43	1.19
query24	2.12	0.23	0.23
query25	0.13	0.09	0.08
query26	0.30	0.21	0.20
query27	0.45	0.23	0.24
query28	13.23	1.01	1.00
query29	12.60	3.30	3.24
query30	0.27	0.06	0.05
query31	2.84	0.39	0.38
query32	3.28	0.47	0.47
query33	2.86	2.88	2.89
query34	17.18	4.34	4.40
query35	4.47	4.42	4.41
query36	0.64	0.47	0.47
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.12
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.64 s
Total hot run time: 30.27 s

xiaokang
xiaokang reviewed Jul 28, 2024
View reviewed changes
@csun5285
Copy link
Contributor Author

csun5285 commented Jul 29, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 29, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 29, 2024

TPC-H: Total hot run time: 41525 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85041a029c1b356072f1488347d54e65d8d5fec8, data reload: false

------ Round 1 ----------------------------------
q1	17640	4232	4079	4079
q2	2018	216	199	199
q3	10463	1306	1354	1306
q4	10184	825	894	825
q5	7649	2994	2961	2961
q6	220	139	141	139
q7	1033	617	606	606
q8	9433	1911	1949	1911
q9	8472	6584	6620	6584
q10	8681	3825	3837	3825
q11	425	252	250	250
q12	416	230	231	230
q13	17749	2943	2915	2915
q14	272	249	246	246
q15	524	482	495	482
q16	490	402	383	383
q17	968	903	886	886
q18	8015	7374	7224	7224
q19	1383	1219	1203	1203
q20	572	323	356	323
q21	5207	4663	4772	4663
q22	352	285	285	285
Total cold run time: 112166 ms
Total hot run time: 41525 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4036	4015	4113	4015
q2	326	223	217	217
q3	3003	3001	3000	3000
q4	1866	1875	1854	1854
q5	5245	5208	5206	5206
q6	216	130	133	130
q7	2046	1721	1709	1709
q8	3223	3311	3290	3290
q9	8336	8263	8298	8263
q10	3789	3805	3856	3805
q11	540	437	443	437
q12	735	579	540	540
q13	15795	2950	2943	2943
q14	285	253	247	247
q15	512	476	471	471
q16	441	405	392	392
q17	1730	1692	1680	1680
q18	7795	7260	7162	7162
q19	1658	1699	1666	1666
q20	1971	1755	1746	1746
q21	5529	5118	5259	5118
q22	511	480	449	449
Total cold run time: 69588 ms
Total hot run time: 54340 ms

@doris-robot
Copy link

doris-robot commented Jul 29, 2024

TPC-DS: Total hot run time: 169527 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85041a029c1b356072f1488347d54e65d8d5fec8, data reload: false

query1	924	366	358	358
query2	6469	1707	1651	1651
query3	6651	232	220	220
query4	19207	17509	17202	17202
query5	4321	508	506	506
query6	276	168	159	159
query7	4613	294	309	294
query8	258	201	198	198
query9	8512	2410	2394	2394
query10	462	289	264	264
query11	10452	10198	9978	9978
query12	136	89	91	89
query13	1629	387	367	367
query14	10462	8152	7703	7703
query15	210	168	165	165
query16	7120	442	465	442
query17	960	572	555	555
query18	1913	294	290	290
query19	190	149	150	149
query20	91	87	86	86
query21	204	101	102	101
query22	4213	3942	4081	3942
query23	33772	33075	33002	33002
query24	10415	3055	3106	3055
query25	711	406	422	406
query26	1767	154	157	154
query27	2911	270	277	270
query28	6909	1975	1958	1958
query29	1332	442	440	440
query30	285	151	151	151
query31	939	765	756	756
query32	100	55	58	55
query33	733	332	328	328
query34	904	481	492	481
query35	857	721	720	720
query36	983	868	883	868
query37	305	81	82	81
query38	2881	2767	2810	2767
query39	871	817	818	817
query40	294	120	118	118
query41	50	47	50	47
query42	128	102	107	102
query43	458	408	418	408
query44	1173	723	724	723
query45	208	182	176	176
query46	1107	788	786	786
query47	1783	1695	1713	1695
query48	367	291	294	291
query49	1200	457	426	426
query50	895	436	446	436
query51	6818	6781	6623	6623
query52	100	89	95	89
query53	251	185	183	183
query54	636	466	460	460
query55	79	80	78	78
query56	283	275	267	267
query57	1156	1079	1031	1031
query58	282	265	271	265
query59	2582	2270	2290	2270
query60	320	283	280	280
query61	239	98	95	95
query62	925	671	666	666
query63	207	183	184	183
query64	5884	1943	1903	1903
query65	3251	3070	3090	3070
query66	1449	345	334	334
query67	15360	14811	14801	14801
query68	6435	556	580	556
query69	723	379	332	332
query70	1167	1110	1050	1050
query71	550	272	277	272
query72	8119	2737	2508	2508
query73	916	338	334	334
query74	6089	5641	5595	5595
query75	4575	2752	2735	2735
query76	4336	1395	1409	1395
query77	770	354	303	303
query78	9486	8896	8787	8787
query79	2827	531	533	531
query80	1075	507	510	507
query81	559	223	226	223
query82	766	128	134	128
query83	338	179	181	179
query84	274	81	83	81
query85	1490	312	336	312
query86	443	315	302	302
query87	3293	3170	3123	3123
query88	3761	2503	2501	2501
query89	410	284	284	284
query90	2085	193	199	193
query91	129	102	104	102
query92	67	48	51	48
query93	3960	597	608	597
query94	1020	295	273	273
query95	397	274	270	270
query96	617	286	284	284
query97	3218	2996	3027	2996
query98	223	202	196	196
query99	1639	1307	1302	1302
Total cold run time: 275190 ms
Total hot run time: 169527 ms

@csun5285
Copy link
Contributor Author

csun5285 commented Aug 2, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Aug 2, 2024

TPC-H: Total hot run time: 41755 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd712d105aa274b9b220dd8f6b6c2c3974ae8706, data reload: false

------ Round 1 ----------------------------------
q1	17604	4153	4113	4113
q2	2030	200	197	197
q3	10551	1336	1353	1336
q4	10267	832	933	832
q5	7680	2957	2989	2957
q6	222	137	136	136
q7	1062	609	604	604
q8	9430	1921	1950	1921
q9	8415	6621	6624	6621
q10	8706	3822	3836	3822
q11	430	248	246	246
q12	408	222	221	221
q13	17764	2953	2949	2949
q14	279	236	243	236
q15	533	493	492	492
q16	534	410	390	390
q17	969	901	906	901
q18	8167	7335	7245	7245
q19	1672	1230	1207	1207
q20	569	318	329	318
q21	5298	4723	4744	4723
q22	344	288	302	288
Total cold run time: 112934 ms
Total hot run time: 41755 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4125	4021	4071	4021
q2	329	228	229	228
q3	3058	3070	3189	3070
q4	2097	2079	2013	2013
q5	5731	5532	5457	5457
q6	220	127	127	127
q7	2141	1762	1823	1762
q8	3318	3375	3363	3363
q9	8620	8643	8802	8643
q10	3919	4034	3908	3908
q11	561	467	474	467
q12	786	584	568	568
q13	16292	3103	3100	3100
q14	310	263	274	263
q15	548	479	495	479
q16	483	399	417	399
q17	1771	1763	1728	1728
q18	8123	7855	7821	7821
q19	1701	1718	1713	1713
q20	2100	1836	1825	1825
q21	5695	5588	5267	5267
q22	537	487	467	467
Total cold run time: 72465 ms
Total hot run time: 56689 ms

@doris-robot
Copy link

doris-robot commented Aug 2, 2024

TPC-DS: Total hot run time: 169790 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd712d105aa274b9b220dd8f6b6c2c3974ae8706, data reload: false

query1	923	371	365	365
query2	6469	1701	1617	1617
query3	6682	214	228	214
query4	19590	17482	17319	17319
query5	3672	502	507	502
query6	273	170	185	170
query7	4608	299	303	299
query8	251	192	185	185
query9	8729	2367	2333	2333
query10	425	279	273	273
query11	10508	10156	10157	10156
query12	120	88	89	88
query13	1614	392	388	388
query14	9761	6752	7160	6752
query15	210	161	164	161
query16	6731	438	501	438
query17	965	584	563	563
query18	1827	294	291	291
query19	203	150	145	145
query20	96	88	90	88
query21	208	102	103	102
query22	4495	4139	4001	4001
query23	33848	33779	33759	33759
query24	10543	3115	3115	3115
query25	684	409	414	409
query26	1721	166	151	151
query27	2904	285	297	285
query28	7538	1996	1980	1980
query29	1170	444	429	429
query30	248	157	157	157
query31	966	777	795	777
query32	101	55	57	55
query33	714	330	330	330
query34	946	500	498	498
query35	878	741	742	741
query36	1063	875	877	875
query37	274	77	76	76
query38	2860	2786	2778	2778
query39	864	795	811	795
query40	256	111	109	109
query41	44	46	43	43
query42	125	100	106	100
query43	451	426	415	415
query44	1167	757	744	744
query45	213	181	178	178
query46	1092	812	778	778
query47	1830	1719	1717	1717
query48	376	297	294	294
query49	938	424	434	424
query50	897	430	424	424
query51	6815	6696	6701	6696
query52	112	92	90	90
query53	259	193	184	184
query54	635	462	455	455
query55	76	77	72	72
query56	282	251	263	251
query57	1211	1039	1014	1014
query58	268	280	284	280
query59	2500	2436	2478	2436
query60	294	272	271	271
query61	98	93	96	93
query62	881	677	668	668
query63	204	175	177	175
query64	5602	1929	1891	1891
query65	3144	3071	3108	3071
query66	1303	328	322	322
query67	15342	14949	14748	14748
query68	4488	573	590	573
query69	652	359	305	305
query70	1095	1040	1078	1040
query71	435	288	277	277
query72	7076	2680	2526	2526
query73	774	330	335	330
query74	5972	5847	5590	5590
query75	3383	2726	2725	2725
query76	2721	1215	1283	1215
query77	472	305	308	305
query78	9553	8980	8948	8948
query79	2705	530	522	522
query80	1162	506	507	506
query81	561	223	226	223
query82	875	137	130	130
query83	234	166	171	166
query84	269	80	80	80
query85	1384	325	302	302
query86	498	300	311	300
query87	3264	3062	3096	3062
query88	3881	2520	2548	2520
query89	386	287	287	287
query90	1820	189	186	186
query91	123	99	103	99
query92	59	50	49	49
query93	2173	609	628	609
query94	802	300	285	285
query95	379	266	274	266
query96	616	283	285	283
query97	3247	3088	3009	3009
query98	218	199	197	197
query99	1593	1272	1276	1272
Total cold run time: 264967 ms
Total hot run time: 169790 ms

@doris-robot
Copy link

doris-robot commented Aug 2, 2024

ClickBench: Total hot run time: 30.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bd712d105aa274b9b220dd8f6b6c2c3974ae8706, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.48	0.48
query6	1.14	0.71	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.52	0.51
query10	0.57	0.58	0.57
query11	0.16	0.12	0.11
query12	0.15	0.13	0.12
query13	0.63	0.59	0.60
query14	0.76	0.79	0.79
query15	0.89	0.86	0.86
query16	0.35	0.36	0.36
query17	1.00	1.00	0.96
query18	0.23	0.21	0.21
query19	1.81	1.79	1.78
query20	0.01	0.00	0.01
query21	15.39	0.75	0.64
query22	4.05	7.20	1.55
query23	17.91	1.35	1.33
query24	2.27	0.21	0.22
query25	0.18	0.08	0.08
query26	0.30	0.21	0.22
query27	0.46	0.23	0.22
query28	13.17	1.02	0.98
query29	12.58	3.31	3.28
query30	0.25	0.06	0.05
query31	2.87	0.41	0.40
query32	3.26	0.49	0.48
query33	2.96	2.90	2.93
query34	15.42	4.26	4.29
query35	4.31	4.28	4.31
query36	0.68	0.48	0.49
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.10	0.04	0.05
query42	0.05	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 107.67 s
Total hot run time: 30.27 s

xiaokang
xiaokang approved these changes Aug 2, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 2, 2024
@qidaye qidaye merged commit 13cb380 into apache:master Aug 5, 2024
@xiaokang xiaokang added the p0_c label Aug 6, 2024
This was referenced Aug 6, 2024
Merged
Merged
airborne12 pushed a commit that referenced this pull request Aug 6, 2024
@csun5285
[fix] (compaction) fix time series compaction policy (#38220) (#38917)
8ce3096 
## Proposed changes

pick from #38220
airborne12 pushed a commit that referenced this pull request Aug 6, 2024
@csun5285
[fix] (compaction) fix time series compaction policy (#38915)
3b9a2b3 
## Proposed changes

pick from master #38220
@csun5285 csun5285 mentioned this pull request Aug 9, 2024
Merged
airborne12 pushed a commit that referenced this pull request Aug 12, 2024
@csun5285
[fix] (compaction) fix time series compaction policy (#39170)
191371d 
## Proposed changes

Fix the issue introduced by #38220 

### BUG
The permits requested for compaction cannot be 0, 0 indicates failure.
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@csun5285 @wyxxxcat
[fix] (compaction) fix time series compaction policy (apache#39170)
25c2fb7 
## Proposed changes

Fix the issue introduced by apache#38220 

### BUG
The permits requested for compaction cannot be 0, 0 indicates failure.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@csun5285
[fix] (compaction) fix time series compaction policy (apache#38915)
665be8c 
## Proposed changes

pick from master apache#38220
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@qidaye qidaye qidaye approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

@airborne12 airborne12 airborne12 left review comments

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.1-merged p0_c reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@csun5285 @doris-robot @xiaokang @airborne12 @qidaye @zzzxl1993 @wm1581066 @dataroaring