[enhance](metrics)add some compaction metrics #50910

koarz · 2025-05-14T10:34:33Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-05-14T10:34:39Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

koarz · 2025-05-14T10:34:45Z

run buildall

doris-robot · 2025-05-14T11:52:40Z

TPC-H: Total hot run time: 34094 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5f3f2480e4172f4a2df3026e295a4a768d81c672, data reload: false

------ Round 1 ----------------------------------
q1	26327	5062	4992	4992
q2	2129	276	184	184
q3	10415	1269	706	706
q4	10216	1030	545	545
q5	7479	2431	2465	2431
q6	184	164	131	131
q7	934	783	611	611
q8	9329	1350	1156	1156
q9	6817	5184	5107	5107
q10	6860	2344	1903	1903
q11	476	291	284	284
q12	362	356	213	213
q13	17776	3737	3080	3080
q14	226	227	217	217
q15	531	485	501	485
q16	433	444	380	380
q17	611	879	368	368
q18	7745	7194	7003	7003
q19	1366	1102	582	582
q20	334	339	229	229
q21	4130	3407	2516	2516
q22	1013	1009	971	971
Total cold run time: 115693 ms
Total hot run time: 34094 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5162	5077	5193	5077
q2	242	316	231	231
q3	2173	2671	2327	2327
q4	1387	1812	1472	1472
q5	4546	4455	4412	4412
q6	211	165	128	128
q7	1972	1880	1739	1739
q8	2670	2594	2521	2521
q9	7151	7061	7119	7061
q10	3040	3187	2777	2777
q11	579	501	494	494
q12	658	788	666	666
q13	3525	3896	3358	3358
q14	274	308	275	275
q15	518	486	469	469
q16	445	483	442	442
q17	1191	1575	1388	1388
q18	7633	7511	7508	7508
q19	847	881	1021	881
q20	2054	1998	1869	1869
q21	5057	4601	4635	4601
q22	1071	1024	978	978
Total cold run time: 52406 ms
Total hot run time: 50674 ms

doris-robot · 2025-05-14T12:04:15Z

TPC-DS: Total hot run time: 186699 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5f3f2480e4172f4a2df3026e295a4a768d81c672, data reload: false

query1	1010	482	492	482
query2	6577	1880	1869	1869
query3	6754	232	232	232
query4	26166	23311	23092	23092
query5	4361	603	452	452
query6	326	216	183	183
query7	4615	496	292	292
query8	284	239	237	237
query9	8645	2595	2618	2595
query10	487	336	294	294
query11	15472	15076	14801	14801
query12	159	113	111	111
query13	1702	524	409	409
query14	8681	5935	6067	5935
query15	201	189	165	165
query16	7189	644	502	502
query17	1209	733	595	595
query18	1985	398	304	304
query19	196	197	165	165
query20	124	116	119	116
query21	219	128	107	107
query22	4172	4282	4070	4070
query23	34008	33278	33239	33239
query24	8421	2398	2388	2388
query25	558	484	418	418
query26	1248	276	159	159
query27	2731	505	344	344
query28	4302	2154	2139	2139
query29	796	573	457	457
query30	284	217	181	181
query31	908	918	755	755
query32	72	66	61	61
query33	545	388	316	316
query34	810	847	519	519
query35	774	807	725	725
query36	938	967	891	891
query37	120	95	77	77
query38	4251	4207	4119	4119
query39	1475	1436	1425	1425
query40	213	118	104	104
query41	58	58	55	55
query42	118	112	110	110
query43	507	533	487	487
query44	1305	821	825	821
query45	172	166	161	161
query46	847	1024	647	647
query47	1792	1799	1722	1722
query48	377	416	302	302
query49	772	487	431	431
query50	666	671	413	413
query51	4139	4171	4098	4098
query52	111	107	100	100
query53	229	250	188	188
query54	580	577	510	510
query55	88	85	80	80
query56	288	298	285	285
query57	1140	1144	1102	1102
query58	273	245	261	245
query59	2524	2775	2640	2640
query60	338	326	299	299
query61	131	161	129	129
query62	810	715	664	664
query63	217	191	182	182
query64	4375	1017	675	675
query65	4301	4319	4272	4272
query66	1187	419	359	359
query67	15831	15706	15669	15669
query68	8908	892	521	521
query69	474	309	258	258
query70	1150	1121	1086	1086
query71	454	316	311	311
query72	5761	4656	4608	4608
query73	693	569	352	352
query74	8819	9221	8859	8859
query75	4185	3216	2750	2750
query76	3636	1186	750	750
query77	814	383	293	293
query78	10144	10149	9330	9330
query79	2193	815	568	568
query80	610	531	448	448
query81	484	254	221	221
query82	437	128	97	97
query83	281	253	231	231
query84	312	106	84	84
query85	791	360	308	308
query86	334	298	309	298
query87	4486	4389	4402	4389
query88	3362	2256	2307	2256
query89	394	324	286	286
query90	1945	210	208	208
query91	140	144	109	109
query92	81	62	56	56
query93	1275	959	589	589
query94	680	419	299	299
query95	386	296	280	280
query96	503	560	275	275
query97	3160	3240	3113	3113
query98	240	223	199	199
query99	1465	1376	1296	1296
Total cold run time: 274761 ms
Total hot run time: 186699 ms

doris-robot · 2025-05-14T12:09:26Z

ClickBench: Total hot run time: 29.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5f3f2480e4172f4a2df3026e295a4a768d81c672, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.11	0.11
query3	0.24	0.19	0.19
query4	1.60	0.19	0.20
query5	0.60	0.59	0.58
query6	1.18	0.71	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.54	0.50
query10	0.56	0.58	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.59	0.60
query14	0.77	0.81	0.80
query15	0.88	0.85	0.87
query16	0.38	0.39	0.38
query17	1.03	1.08	1.10
query18	0.22	0.20	0.22
query19	1.91	1.90	1.88
query20	0.01	0.01	0.02
query21	15.45	0.90	0.53
query22	0.77	1.11	0.72
query23	14.92	1.38	0.64
query24	7.19	1.80	0.70
query25	0.45	0.09	0.14
query26	0.66	0.17	0.15
query27	0.05	0.05	0.05
query28	9.44	0.86	0.45
query29	12.59	3.99	3.31
query30	0.25	0.10	0.07
query31	2.83	0.63	0.38
query32	3.23	0.56	0.47
query33	3.02	3.09	3.12
query34	15.78	5.10	4.51
query35	4.47	4.49	4.46
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.34 s
Total hot run time: 29.37 s

hello-stephen · 2025-05-14T13:30:47Z

BE UT Coverage Report

Increment line coverage 60.00% (33/55) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.85% (14911/26698)
Line Coverage	44.63% (131998/295745)
Region Coverage	43.68% (66352/151895)
Branch Coverage	38.29% (33990/88780)

hello-stephen · 2025-05-14T15:44:31Z

BE Regression && UT Coverage Report

Increment line coverage 70.91% (39/55) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.40% (20854/26264)
Line Coverage	72.54% (214475/295681)
Region Coverage	70.75% (126220/178396)
Branch Coverage	64.46% (65351/101390)

be/src/cloud/cloud_storage_engine.cpp

luwei16 · 2025-05-16T11:59:38Z

be/src/io/cache/cached_remote_file_reader.cpp

    if (read_stats.hit_cache) {
        statis->num_local_io_total++;
        statis->bytes_read_from_local += read_stats.bytes_read;
+        DorisMetrics::instance()->local_compaction_read_bytes_total->increment(


The query path will also go here. This statistic includes the amount of query data read

Yes, there's double counting here. I'll fix it

koarz · 2025-05-19T07:14:36Z

run buildall

doris-robot · 2025-05-19T08:46:01Z

TPC-H: Total hot run time: 34143 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85920d56fac9d54ab626bd9a0730e508b3bb5d5b, data reload: false

------ Round 1 ----------------------------------
q1	26968	5051	5027	5027
q2	2087	291	199	199
q3	10363	1254	710	710
q4	10229	1004	514	514
q5	7549	2395	2368	2368
q6	182	165	134	134
q7	936	729	626	626
q8	9337	1306	1156	1156
q9	6860	5116	5109	5109
q10	6882	2343	1915	1915
q11	507	301	277	277
q12	348	351	209	209
q13	17776	3664	3080	3080
q14	229	232	230	230
q15	535	501	480	480
q16	419	435	371	371
q17	622	867	382	382
q18	7735	7218	7188	7188
q19	1613	955	560	560
q20	338	351	227	227
q21	3910	2532	2406	2406
q22	1065	1000	975	975
Total cold run time: 116490 ms
Total hot run time: 34143 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5174	5078	5015	5015
q2	246	329	225	225
q3	2168	2630	2309	2309
q4	1372	1821	1366	1366
q5	4472	4429	4429	4429
q6	217	169	131	131
q7	1977	1941	1752	1752
q8	2611	2651	2574	2574
q9	7200	7294	7043	7043
q10	3029	3198	2807	2807
q11	602	517	490	490
q12	685	785	600	600
q13	3526	3897	3299	3299
q14	270	306	302	302
q15	533	491	490	490
q16	456	487	467	467
q17	1155	1548	1396	1396
q18	7766	7586	7446	7446
q19	804	800	873	800
q20	2014	2060	1862	1862
q21	4809	4473	4366	4366
q22	1085	1092	1046	1046
Total cold run time: 52171 ms
Total hot run time: 50215 ms

doris-robot · 2025-05-19T08:58:19Z

TPC-DS: Total hot run time: 193524 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85920d56fac9d54ab626bd9a0730e508b3bb5d5b, data reload: false

query1	1420	1090	1075	1075
query2	6295	1803	1827	1803
query3	11032	4583	4569	4569
query4	54749	25771	23123	23123
query5	5036	579	454	454
query6	332	210	213	210
query7	4897	513	309	309
query8	314	259	254	254
query9	5309	2699	2692	2692
query10	417	313	277	277
query11	15109	15068	14852	14852
query12	169	109	103	103
query13	1017	534	405	405
query14	10199	6362	6407	6362
query15	211	199	189	189
query16	7149	673	523	523
query17	1094	768	624	624
query18	1549	432	331	331
query19	209	198	179	179
query20	139	141	126	126
query21	213	132	111	111
query22	4436	4442	4271	4271
query23	34416	33474	33822	33474
query24	6670	2428	2527	2428
query25	490	474	423	423
query26	706	276	153	153
query27	2320	522	347	347
query28	3059	2195	2193	2193
query29	590	574	442	442
query30	267	225	195	195
query31	938	898	815	815
query32	73	66	62	62
query33	470	375	331	331
query34	805	918	555	555
query35	810	850	764	764
query36	961	1029	896	896
query37	112	112	83	83
query38	4314	4381	4343	4343
query39	1549	1511	1481	1481
query40	215	127	114	114
query41	60	53	59	53
query42	132	117	114	114
query43	503	521	474	474
query44	1376	869	856	856
query45	189	183	169	169
query46	878	1045	662	662
query47	1831	1888	1816	1816
query48	425	458	338	338
query49	690	534	463	463
query50	665	708	423	423
query51	4276	4275	4242	4242
query52	122	111	102	102
query53	235	259	216	216
query54	593	580	517	517
query55	94	95	82	82
query56	326	331	295	295
query57	1179	1193	1151	1151
query58	277	274	261	261
query59	2654	2796	2648	2648
query60	351	337	323	323
query61	137	124	126	124
query62	717	719	640	640
query63	234	194	192	192
query64	1765	1036	684	684
query65	4395	4263	4215	4215
query66	729	398	340	340
query67	15947	15887	15449	15449
query68	7114	891	525	525
query69	544	320	277	277
query70	1261	1160	1097	1097
query71	493	330	302	302
query72	5716	4876	4889	4876
query73	1334	658	367	367
query74	8962	9303	8807	8807
query75	4131	3212	2699	2699
query76	4316	1201	749	749
query77	758	376	295	295
query78	10151	10223	9373	9373
query79	2512	798	593	593
query80	659	523	452	452
query81	502	254	222	222
query82	434	130	104	104
query83	250	251	234	234
query84	296	112	81	81
query85	835	356	311	311
query86	363	320	287	287
query87	4492	4500	4398	4398
query88	3741	2365	2384	2365
query89	413	331	287	287
query90	1813	215	209	209
query91	209	147	111	111
query92	70	61	57	57
query93	2029	953	585	585
query94	659	402	304	304
query95	365	295	291	291
query96	506	581	293	293
query97	2775	2786	2710	2710
query98	238	206	206	206
query99	1598	1396	1267	1267
Total cold run time: 300379 ms
Total hot run time: 193524 ms

doris-robot · 2025-05-19T09:03:37Z

ClickBench: Total hot run time: 29.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85920d56fac9d54ab626bd9a0730e508b3bb5d5b, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.12
query3	0.25	0.20	0.19
query4	1.59	0.20	0.20
query5	0.47	0.44	0.44
query6	1.17	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.60	0.52	0.52
query10	0.57	0.58	0.56
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.62
query14	0.79	0.81	0.82
query15	0.89	0.85	0.88
query16	0.39	0.39	0.39
query17	1.05	1.05	1.08
query18	0.22	0.21	0.21
query19	1.96	1.91	1.88
query20	0.01	0.01	0.01
query21	15.40	0.88	0.56
query22	0.76	1.09	0.67
query23	15.04	1.38	0.66
query24	7.06	1.30	0.47
query25	0.53	0.16	0.17
query26	0.64	0.17	0.14
query27	0.05	0.05	0.05
query28	9.13	0.96	0.47
query29	12.66	3.96	3.28
query30	0.25	0.09	0.06
query31	2.81	0.63	0.39
query32	3.23	0.55	0.46
query33	3.11	3.12	3.04
query34	15.79	5.06	4.50
query35	4.54	4.51	4.53
query36	0.69	0.51	0.48
query37	0.09	0.06	0.07
query38	0.06	0.05	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.32 s
Total hot run time: 29.03 s

hello-stephen · 2025-05-19T11:21:00Z

BE UT Coverage Report

Increment line coverage 62.50% (35/56) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.94% (14920/26673)
Line Coverage	44.78% (132463/295801)
Region Coverage	43.86% (66614/151880)
Branch Coverage	38.45% (34127/88768)

hello-stephen · 2025-05-19T12:37:17Z

BE Regression && UT Coverage Report

Increment line coverage 83.93% (47/56) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.38% (20843/26257)
Line Coverage	72.66% (214917/295798)
Region Coverage	70.85% (126419/178426)
Branch Coverage	64.63% (65538/101400)

koarz · 2025-05-23T08:59:18Z

run buildall

koarz · 2025-05-23T09:15:38Z

run buildall

doris-robot · 2025-05-23T10:16:09Z

TPC-H: Total hot run time: 33988 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef77c692a7bce3bcf05a2ef0932046ba9fa7d8e3, data reload: false

------ Round 1 ----------------------------------
q1	26419	5018	4985	4985
q2	2079	282	190	190
q3	10379	1272	732	732
q4	10247	1026	530	530
q5	7572	2408	2384	2384
q6	183	161	130	130
q7	918	743	606	606
q8	9307	1342	1022	1022
q9	6861	5119	5132	5119
q10	6808	2343	1893	1893
q11	480	296	282	282
q12	356	368	219	219
q13	17762	3729	3123	3123
q14	244	239	215	215
q15	531	485	500	485
q16	430	435	374	374
q17	595	866	373	373
q18	7714	7119	7115	7115
q19	2078	964	562	562
q20	342	354	234	234
q21	4120	3315	2408	2408
q22	1031	1029	1007	1007
Total cold run time: 116456 ms
Total hot run time: 33988 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5212	5091	5084	5084
q2	243	322	233	233
q3	2187	2652	2311	2311
q4	1347	1790	1511	1511
q5	4550	4457	4387	4387
q6	209	168	127	127
q7	1984	1903	1731	1731
q8	2593	2646	2601	2601
q9	7099	7155	7099	7099
q10	3039	3180	2750	2750
q11	583	528	509	509
q12	722	814	606	606
q13	3566	3933	3266	3266
q14	291	295	267	267
q15	525	482	484	482
q16	457	496	456	456
q17	1170	1534	1403	1403
q18	7844	7502	7540	7502
q19	853	807	889	807
q20	1948	2028	1928	1928
q21	5150	4324	4239	4239
q22	1089	1031	974	974
Total cold run time: 52661 ms
Total hot run time: 50273 ms

doris-robot · 2025-05-23T10:27:38Z

TPC-DS: Total hot run time: 186361 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef77c692a7bce3bcf05a2ef0932046ba9fa7d8e3, data reload: false

query1	1006	493	495	493
query2	6548	1906	1883	1883
query3	6755	218	219	218
query4	26280	24078	23173	23173
query5	4349	634	480	480
query6	301	238	193	193
query7	4621	502	285	285
query8	297	245	245	245
query9	8599	2646	2675	2646
query10	491	338	277	277
query11	15560	15013	14796	14796
query12	157	125	111	111
query13	1664	537	413	413
query14	8854	6233	6066	6066
query15	205	184	171	171
query16	7147	624	497	497
query17	1182	730	586	586
query18	1957	393	285	285
query19	188	185	151	151
query20	118	116	119	116
query21	214	137	109	109
query22	4176	4223	4063	4063
query23	33906	33184	33506	33184
query24	8383	2390	2445	2390
query25	546	462	405	405
query26	1231	269	152	152
query27	2753	497	350	350
query28	4343	2125	2100	2100
query29	731	551	435	435
query30	286	215	183	183
query31	914	866	799	799
query32	70	66	60	60
query33	551	368	313	313
query34	821	849	530	530
query35	768	800	755	755
query36	959	991	880	880
query37	108	100	73	73
query38	4177	4108	4085	4085
query39	1506	1461	1407	1407
query40	213	118	109	109
query41	55	57	51	51
query42	121	110	113	110
query43	530	514	517	514
query44	1350	820	836	820
query45	173	172	168	168
query46	861	1030	661	661
query47	1748	1782	1710	1710
query48	378	417	314	314
query49	775	503	428	428
query50	657	706	406	406
query51	4191	4114	4107	4107
query52	108	111	100	100
query53	230	255	190	190
query54	588	571	504	504
query55	85	83	86	83
query56	321	310	308	308
query57	1123	1141	1083	1083
query58	265	254	261	254
query59	2689	2718	2606	2606
query60	342	321	299	299
query61	149	130	124	124
query62	801	711	658	658
query63	244	195	197	195
query64	4283	1003	744	744
query65	4327	4269	4262	4262
query66	1158	414	304	304
query67	15924	15769	15432	15432
query68	6207	883	521	521
query69	488	303	273	273
query70	1180	1108	1142	1108
query71	409	334	317	317
query72	5571	4786	4751	4751
query73	661	614	355	355
query74	9262	9146	8933	8933
query75	3183	3228	2737	2737
query76	3225	1238	767	767
query77	482	393	291	291
query78	10174	10195	9356	9356
query79	1281	790	570	570
query80	645	521	441	441
query81	477	261	219	219
query82	202	127	102	102
query83	264	250	244	244
query84	248	112	85	85
query85	754	353	322	322
query86	319	292	277	277
query87	4401	4457	4281	4281
query88	2963	2311	2319	2311
query89	387	315	286	286
query90	1809	211	215	211
query91	143	147	110	110
query92	69	62	57	57
query93	1069	934	588	588
query94	619	413	301	301
query95	374	295	286	286
query96	495	575	284	284
query97	2670	2755	2612	2612
query98	222	210	205	205
query99	1358	1393	1258	1258
Total cold run time: 268022 ms
Total hot run time: 186361 ms

doris-robot · 2025-05-23T10:32:50Z

ClickBench: Total hot run time: 28.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef77c692a7bce3bcf05a2ef0932046ba9fa7d8e3, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.26	0.19	0.19
query4	1.60	0.19	0.11
query5	0.43	0.42	0.43
query6	1.15	0.67	0.66
query7	0.02	0.02	0.01
query8	0.05	0.03	0.04
query9	0.58	0.52	0.50
query10	0.57	0.58	0.59
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.78	0.82	0.81
query15	0.88	0.86	0.86
query16	0.38	0.40	0.38
query17	1.07	1.03	1.00
query18	0.22	0.21	0.20
query19	1.96	1.83	1.80
query20	0.02	0.01	0.01
query21	15.39	0.92	0.55
query22	0.75	1.17	0.63
query23	14.97	1.36	0.60
query24	7.66	1.12	0.33
query25	0.67	0.27	0.20
query26	0.87	0.16	0.14
query27	0.06	0.05	0.05
query28	9.42	0.89	0.44
query29	12.59	3.98	3.35
query30	0.24	0.09	0.07
query31	2.83	0.61	0.39
query32	3.22	0.54	0.46
query33	3.01	3.06	3.13
query34	15.85	5.06	4.45
query35	4.51	4.53	4.47
query36	0.65	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.05	0.04
query39	0.03	0.03	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.25 s
Total hot run time: 28.49 s

hello-stephen · 2025-05-23T12:20:42Z

BE UT Coverage Report

Increment line coverage 37.68% (26/69) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.05% (14934/26643)
Line Coverage	44.85% (132891/296327)
Region Coverage	43.91% (66781/152081)
Branch Coverage	38.52% (34253/88912)

hello-stephen · 2025-05-23T13:17:04Z

BE Regression && UT Coverage Report

Increment line coverage 28.79% (19/66) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.57% (14270/26150)
Line Coverage	43.77% (128660/293939)
Region Coverage	41.12% (72898/177264)
Branch Coverage	35.63% (35875/100684)

koarz · 2025-05-24T03:22:13Z

run buildall

koarz · 2025-05-27T02:02:41Z

run beut

koarz · 2025-05-27T02:02:50Z

run external

koarz · 2025-05-27T07:10:05Z

run buildall

doris-robot · 2025-05-27T09:00:48Z

TPC-H: Total hot run time: 34052 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b1cea20cfcf22a1ba4775625464b7c7fadd0a61, data reload: false

------ Round 1 ----------------------------------
q1	26310	5076	5112	5076
q2	2086	280	197	197
q3	10496	1236	701	701
q4	10241	982	543	543
q5	7639	2371	2378	2371
q6	190	165	132	132
q7	913	733	619	619
q8	9313	1241	1068	1068
q9	6816	5061	5102	5061
q10	6825	2295	1905	1905
q11	498	292	279	279
q12	340	347	207	207
q13	17785	3644	3108	3108
q14	231	240	218	218
q15	526	483	500	483
q16	434	440	372	372
q17	612	863	389	389
q18	7741	7263	7207	7207
q19	1620	955	559	559
q20	331	331	225	225
q21	3795	3165	2363	2363
q22	1012	1030	969	969
Total cold run time: 115754 ms
Total hot run time: 34052 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5190	5107	5129	5107
q2	244	331	233	233
q3	2191	2689	2357	2357
q4	1335	1799	1331	1331
q5	4615	4451	4417	4417
q6	205	169	123	123
q7	2046	1921	1722	1722
q8	2569	2502	2533	2502
q9	7338	7165	7109	7109
q10	3029	3186	2783	2783
q11	567	541	490	490
q12	668	766	612	612
q13	3546	4003	3196	3196
q14	281	303	269	269
q15	506	493	490	490
q16	442	484	452	452
q17	1200	1506	1416	1416
q18	7839	7419	7546	7419
q19	827	858	1065	858
q20	1955	2058	1836	1836
q21	4814	4409	4293	4293
q22	1081	1066	1000	1000
Total cold run time: 52488 ms
Total hot run time: 50015 ms

doris-robot · 2025-05-27T09:12:22Z

TPC-DS: Total hot run time: 184388 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0b1cea20cfcf22a1ba4775625464b7c7fadd0a61, data reload: false

query1	1007	487	491	487
query2	6585	1836	1824	1824
query3	6751	226	221	221
query4	26361	23659	23396	23396
query5	4322	594	444	444
query6	293	202	216	202
query7	4615	497	285	285
query8	286	248	228	228
query9	8617	2641	2640	2640
query10	477	320	263	263
query11	15365	15161	14811	14811
query12	161	109	117	109
query13	1661	522	425	425
query14	8801	6195	6202	6195
query15	212	193	169	169
query16	7150	643	484	484
query17	1142	737	586	586
query18	1975	413	307	307
query19	195	190	164	164
query20	125	117	122	117
query21	211	126	110	110
query22	4143	4024	4007	4007
query23	34049	33383	33362	33362
query24	8488	2346	2374	2346
query25	554	459	410	410
query26	1222	270	160	160
query27	2745	515	336	336
query28	4325	2138	2123	2123
query29	775	532	431	431
query30	330	213	185	185
query31	932	869	790	790
query32	74	65	65	65
query33	565	360	318	318
query34	789	840	524	524
query35	782	798	728	728
query36	969	989	883	883
query37	110	100	82	82
query38	4166	4155	4043	4043
query39	1516	1425	1422	1422
query40	217	121	109	109
query41	59	58	53	53
query42	121	102	111	102
query43	504	507	475	475
query44	1285	809	802	802
query45	182	173	168	168
query46	834	1026	634	634
query47	1775	1805	1718	1718
query48	388	419	313	313
query49	804	511	454	454
query50	654	675	401	401
query51	4138	4073	4068	4068
query52	114	108	99	99
query53	235	253	188	188
query54	583	575	507	507
query55	85	85	80	80
query56	295	296	298	296
query57	1146	1140	1070	1070
query58	267	265	253	253
query59	2548	2591	2570	2570
query60	331	319	304	304
query61	127	149	127	127
query62	793	720	651	651
query63	219	188	187	187
query64	4383	1002	670	670
query65	4377	4272	4227	4227
query66	1184	412	316	316
query67	15708	15677	15571	15571
query68	8473	883	522	522
query69	534	302	269	269
query70	1234	1147	1093	1093
query71	451	342	319	319
query72	5662	2461	5048	2461
query73	796	750	353	353
query74	9176	8960	9016	8960
query75	4039	3201	2705	2705
query76	3746	1198	770	770
query77	776	363	284	284
query78	10196	10152	9367	9367
query79	2472	827	596	596
query80	625	521	468	468
query81	496	255	229	229
query82	456	128	101	101
query83	288	257	243	243
query84	309	103	91	91
query85	786	364	322	322
query86	392	321	260	260
query87	4347	4467	4315	4315
query88	3453	2277	2304	2277
query89	446	367	284	284
query90	1833	217	213	213
query91	144	159	113	113
query92	83	65	54	54
query93	1849	977	588	588
query94	681	427	316	316
query95	384	294	300	294
query96	512	569	285	285
query97	2729	2788	2676	2676
query98	224	212	204	204
query99	1458	1380	1261	1261
Total cold run time: 275252 ms
Total hot run time: 184388 ms

doris-robot · 2025-05-27T09:17:34Z

ClickBench: Total hot run time: 29.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0b1cea20cfcf22a1ba4775625464b7c7fadd0a61, data reload: false

query1	0.04	0.03	0.04
query2	0.12	0.11	0.11
query3	0.27	0.19	0.19
query4	1.59	0.19	0.11
query5	0.44	0.42	0.41
query6	1.15	0.66	0.67
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.58	0.51	0.51
query10	0.58	0.59	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.62	0.59
query14	0.78	0.80	0.82
query15	0.86	0.85	0.86
query16	0.37	0.40	0.40
query17	1.03	1.04	1.05
query18	0.23	0.21	0.21
query19	1.95	1.82	1.87
query20	0.01	0.02	0.01
query21	15.41	0.89	0.56
query22	0.75	1.19	0.74
query23	14.84	1.36	0.68
query24	6.87	1.53	0.85
query25	0.47	0.08	0.13
query26	0.71	0.17	0.14
query27	0.05	0.05	0.06
query28	9.28	0.87	0.45
query29	12.54	4.05	3.36
query30	0.25	0.09	0.06
query31	2.82	0.59	0.39
query32	3.24	0.55	0.47
query33	3.04	3.00	3.10
query34	15.78	5.13	4.53
query35	4.55	4.59	4.56
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 102.75 s
Total hot run time: 29.29 s

koarz · 2025-05-28T07:47:50Z

run buildall

doris-robot · 2025-05-28T09:18:10Z

TPC-H: Total hot run time: 33801 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee5b3bd4ae549ade118587b339c94a3cdc278942, data reload: false

------ Round 1 ----------------------------------
q1	25960	5038	5062	5038
q2	2077	298	188	188
q3	10480	1280	691	691
q4	10227	996	520	520
q5	7804	2431	2358	2358
q6	181	163	132	132
q7	913	737	598	598
q8	9303	1314	1057	1057
q9	6853	5134	5155	5134
q10	6879	2305	1907	1907
q11	494	290	268	268
q12	345	346	216	216
q13	17800	3660	3039	3039
q14	230	234	216	216
q15	528	484	493	484
q16	426	414	393	393
q17	623	874	376	376
q18	7842	7129	7246	7129
q19	1569	964	561	561
q20	347	325	226	226
q21	3709	3151	2332	2332
q22	1051	1025	938	938
Total cold run time: 115641 ms
Total hot run time: 33801 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5217	5039	5085	5039
q2	239	322	231	231
q3	2185	2652	2307	2307
q4	1337	1787	1356	1356
q5	4478	4381	4383	4381
q6	218	165	133	133
q7	1988	1911	1761	1761
q8	2606	2563	2550	2550
q9	7392	7263	6939	6939
q10	3033	3207	2752	2752
q11	581	522	503	503
q12	670	756	634	634
q13	3502	3863	3229	3229
q14	284	288	283	283
q15	510	504	480	480
q16	449	481	446	446
q17	1137	1542	1386	1386
q18	7903	7582	7417	7417
q19	790	865	1045	865
q20	1978	2056	1896	1896
q21	4759	4500	4605	4500
q22	1125	1050	1008	1008
Total cold run time: 52381 ms
Total hot run time: 50096 ms

doris-robot · 2025-05-28T09:30:18Z

TPC-DS: Total hot run time: 192388 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee5b3bd4ae549ade118587b339c94a3cdc278942, data reload: false

query1	1419	1112	1082	1082
query2	6196	1817	1778	1778
query3	11174	4597	4621	4597
query4	54541	25006	23042	23042
query5	5114	536	457	457
query6	337	207	196	196
query7	4910	509	290	290
query8	316	242	224	224
query9	5751	2618	2602	2602
query10	394	306	269	269
query11	15152	15146	14829	14829
query12	164	105	105	105
query13	1044	495	378	378
query14	10185	6353	6197	6197
query15	186	209	182	182
query16	7127	697	534	534
query17	1070	740	606	606
query18	1529	388	302	302
query19	205	196	165	165
query20	126	120	118	118
query21	216	126	112	112
query22	4242	4528	4299	4299
query23	34415	33597	33620	33597
query24	6766	2471	2466	2466
query25	460	486	411	411
query26	666	282	163	163
query27	2196	510	349	349
query28	3211	2156	2158	2156
query29	570	568	426	426
query30	280	215	181	181
query31	851	837	783	783
query32	79	70	71	70
query33	494	367	325	325
query34	776	879	538	538
query35	807	830	729	729
query36	939	1006	904	904
query37	104	101	74	74
query38	4198	4248	4133	4133
query39	1523	1444	1459	1444
query40	212	120	111	111
query41	60	57	57	57
query42	131	110	120	110
query43	517	484	472	472
query44	1310	829	830	829
query45	185	179	169	169
query46	869	1081	651	651
query47	1836	1882	1783	1783
query48	396	426	345	345
query49	681	539	440	440
query50	679	682	418	418
query51	4268	4236	4203	4203
query52	109	105	97	97
query53	232	271	187	187
query54	604	588	529	529
query55	85	84	83	83
query56	303	309	297	297
query57	1158	1174	1123	1123
query58	270	256	259	256
query59	2653	2693	2621	2621
query60	377	330	315	315
query61	138	130	129	129
query62	742	722	663	663
query63	222	187	184	184
query64	1470	1068	720	720
query65	4404	4259	4254	4254
query66	721	402	296	296
query67	15722	15660	15477	15477
query68	6000	888	523	523
query69	538	297	266	266
query70	1183	1147	1178	1147
query71	439	322	301	301
query72	5353	4762	4943	4762
query73	1243	638	351	351
query74	9427	8941	9015	8941
query75	3206	3207	2722	2722
query76	3876	1186	738	738
query77	551	368	284	284
query78	10016	10171	9370	9370
query79	2029	783	581	581
query80	668	554	438	438
query81	479	265	219	219
query82	259	125	97	97
query83	256	255	227	227
query84	295	114	79	79
query85	753	363	323	323
query86	353	301	279	279
query87	4393	4387	4299	4299
query88	2994	2271	2306	2271
query89	407	312	286	286
query90	1732	202	208	202
query91	155	143	113	113
query92	74	60	58	58
query93	2474	927	580	580
query94	716	423	317	317
query95	366	293	287	287
query96	507	563	292	292
query97	2745	2744	2674	2674
query98	232	210	191	191
query99	1315	1412	1238	1238
Total cold run time: 295121 ms
Total hot run time: 192388 ms

doris-robot · 2025-05-28T09:35:32Z

ClickBench: Total hot run time: 28.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ee5b3bd4ae549ade118587b339c94a3cdc278942, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.26	0.19	0.20
query4	1.59	0.20	0.10
query5	0.43	0.40	0.42
query6	1.15	0.66	0.66
query7	0.03	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.50	0.51
query10	0.56	0.58	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.59	0.60
query14	0.79	0.80	0.81
query15	0.88	0.85	0.88
query16	0.38	0.39	0.38
query17	1.05	1.03	1.02
query18	0.22	0.21	0.21
query19	1.91	1.86	1.81
query20	0.01	0.01	0.01
query21	15.40	0.93	0.54
query22	0.76	1.13	0.64
query23	14.99	1.37	0.66
query24	6.76	2.36	0.44
query25	0.46	0.13	0.27
query26	0.52	0.17	0.14
query27	0.05	0.05	0.04
query28	10.17	0.86	0.45
query29	12.57	3.97	3.28
query30	0.26	0.10	0.06
query31	2.81	0.58	0.38
query32	3.23	0.54	0.48
query33	3.08	3.04	3.01
query34	15.72	5.15	4.50
query35	4.60	4.56	4.56
query36	0.68	0.50	0.49
query37	0.09	0.06	0.07
query38	0.04	0.03	0.04
query39	0.04	0.02	0.03
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.51 s
Total hot run time: 28.61 s

koarz · 2025-05-28T10:42:42Z

run beut

github-actions · 2025-05-29T02:02:34Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

github-actions · 2025-05-29T12:34:35Z

PR approved by at least one committer and no changes requested.

) Cherry-picked from #50910 Co-authored-by: koarz <lihao@selectdb.com>

[enhance](metrics)add some compaction metrics

5f3f248

koarz requested review from dataroaring and gavinchou as code owners May 14, 2025 10:34

luwei16 reviewed May 16, 2025

View reviewed changes

be/src/cloud/cloud_storage_engine.cpp Outdated Show resolved Hide resolved

luwei16 reviewed May 16, 2025

View reviewed changes

koarz added 2 commits May 19, 2025 15:13

fix

f829093

Merge branch 'master' into compaction_metrics

85920d5

fix

ce62262

1

ef77c69

fix

ee5b3bd

koarz force-pushed the compaction_metrics branch from 0b1cea2 to ee5b3bd Compare May 28, 2025 07:47

Yukang-Lian approved these changes May 29, 2025

View reviewed changes

github-actions bot added the reviewed label May 29, 2025

luwei16 approved these changes May 29, 2025

View reviewed changes

dataroaring added the dev/3.0.x label May 29, 2025

dataroaring approved these changes May 29, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 29, 2025

dataroaring merged commit 8ff490f into apache:master Jun 4, 2025
28 of 29 checks passed

github-actions bot pushed a commit that referenced this pull request Jun 4, 2025

[enhance](metrics)add some compaction metrics (#50910)

4a21260

github-actions bot mentioned this pull request Jun 4, 2025

branch-3.0: [enhance](metrics)add some compaction metrics #50910 #51487

Merged

koarz deleted the compaction_metrics branch June 4, 2025 10:30

dataroaring pushed a commit that referenced this pull request Jun 11, 2025

branch-3.0: [enhance](metrics)add some compaction metrics #50910 (#51487

06d7514

) Cherry-picked from #50910 Co-authored-by: koarz <lihao@selectdb.com>

dataroaring added dev/3.0.7-merged and removed dev/3.0.x labels Jun 11, 2025

morrySnow added the dev/3.1.0-merged label Jun 17, 2025

gavinchou mentioned this pull request Aug 15, 2025

Release Note 3.0.7 #54868

Open

[enhance](metrics)add some compaction metrics #50910

[enhance](metrics)add some compaction metrics #50910

Uh oh!

Conversation

koarz commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented May 14, 2025

Uh oh!

koarz commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

hello-stephen commented May 14, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 14, 2025

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

luwei16 May 16, 2025

Choose a reason for hiding this comment

Uh oh!

koarz May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

koarz commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

hello-stephen commented May 19, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 19, 2025

BE Regression && UT Coverage Report

Uh oh!

koarz commented May 23, 2025

Uh oh!

koarz commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

hello-stephen commented May 23, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 23, 2025

BE Regression && UT Coverage Report

Uh oh!

koarz commented May 24, 2025

Uh oh!

koarz commented May 27, 2025

Uh oh!

koarz commented May 27, 2025

Uh oh!

koarz commented May 27, 2025

Uh oh!

doris-robot commented May 27, 2025

Uh oh!

doris-robot commented May 27, 2025

Uh oh!

doris-robot commented May 27, 2025

Uh oh!

koarz commented May 28, 2025

koarz commented May 14, 2025 •

edited

Loading

koarz May 19, 2025 •

edited

Loading