Skip to content

[fix](path gc) Fix path gc race with publish task #50343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 5 commits into from
Apr 28, 2025
Merged

[fix](path gc) Fix path gc race with publish task #50343

dataroaring merged 5 commits into from
Apr 28, 2025

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented Apr 23, 2025 •
edited
Loading

What problem does this PR solve?

Fix path gc race with publish task and checkpoint task,
if not fix it,

  1. very low probability of data loss
  2. run test_path_gc_with_publish_version case will get err
java.sql.SQLException: errCode = 2, detailMessage = (175.40.51.1)[NOT_FOUND]failed to get file size /opt/apache-doris/be/storage/1.HDD/data/22/1745498446653/1868719407/0200000000000019214e72da3f1fe5cfc12
c5efad2bf05bf_0.dat: No such file or directory

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Apr 23, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng
Copy link
Contributor Author

deardeng commented Apr 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 24, 2025

TPC-H: Total hot run time: 33972 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd31bdd74983096ebb653eed28ab2a9d2c0ad209, data reload: false

------ Round 1 ----------------------------------
q1	26094	5284	4962	4962
q2	2064	277	182	182
q3	10400	1248	700	700
q4	10218	1046	522	522
q5	7529	2464	2330	2330
q6	188	163	134	134
q7	926	742	613	613
q8	9332	1327	1115	1115
q9	6847	5157	5151	5151
q10	6828	2317	1914	1914
q11	478	282	261	261
q12	342	345	220	220
q13	17767	3816	3166	3166
q14	231	222	206	206
q15	525	485	504	485
q16	455	444	395	395
q17	584	856	368	368
q18	7559	7351	7034	7034
q19	1218	937	543	543
q20	349	324	227	227
q21	4273	3313	2456	2456
q22	1065	1029	988	988
Total cold run time: 115272 ms
Total hot run time: 33972 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5129	5096	5105	5096
q2	237	341	225	225
q3	2163	2668	2238	2238
q4	1463	1853	1439	1439
q5	4573	4436	4443	4436
q6	217	172	127	127
q7	2008	1898	1720	1720
q8	2589	2561	2548	2548
q9	7304	7169	7190	7169
q10	2978	3198	2702	2702
q11	582	493	512	493
q12	676	759	598	598
q13	3590	3948	3236	3236
q14	278	292	265	265
q15	521	485	476	476
q16	452	506	458	458
q17	1126	1482	1417	1417
q18	7844	7520	7354	7354
q19	803	789	932	789
q20	1999	2009	1864	1864
q21	5247	4680	4626	4626
q22	1095	1058	1015	1015
Total cold run time: 52874 ms
Total hot run time: 50291 ms

@doris-robot
Copy link

doris-robot commented Apr 24, 2025

TPC-DS: Total hot run time: 185700 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd31bdd74983096ebb653eed28ab2a9d2c0ad209, data reload: false

query1	1015	476	481	476
query2	6555	1818	1820	1818
query3	6751	222	223	222
query4	26702	23337	23360	23337
query5	4317	616	462	462
query6	291	196	194	194
query7	4625	487	278	278
query8	288	235	250	235
query9	8592	2554	2577	2554
query10	451	339	263	263
query11	15780	15202	14808	14808
query12	158	110	108	108
query13	1661	514	396	396
query14	8825	6314	6308	6308
query15	207	194	165	165
query16	7162	648	484	484
query17	1212	749	571	571
query18	1975	412	294	294
query19	193	191	166	166
query20	128	117	116	116
query21	224	128	111	111
query22	4162	4208	3973	3973
query23	33920	33071	33100	33071
query24	8540	2394	2382	2382
query25	581	451	391	391
query26	1225	264	146	146
query27	2765	500	329	329
query28	4322	2073	2075	2073
query29	773	556	423	423
query30	283	213	183	183
query31	897	817	753	753
query32	73	63	63	63
query33	576	363	311	311
query34	784	858	528	528
query35	790	786	705	705
query36	959	965	905	905
query37	110	99	77	77
query38	4183	4160	4119	4119
query39	1441	1382	1372	1372
query40	213	120	109	109
query41	66	56	56	56
query42	124	108	112	108
query43	488	493	464	464
query44	1283	810	817	810
query45	177	169	170	169
query46	841	1020	622	622
query47	1723	1811	1730	1730
query48	358	429	293	293
query49	778	517	394	394
query50	644	674	418	418
query51	4139	4151	4060	4060
query52	103	102	97	97
query53	222	256	183	183
query54	560	573	491	491
query55	86	83	82	82
query56	331	294	290	290
query57	1148	1168	1089	1089
query58	261	247	246	246
query59	2652	2693	2479	2479
query60	315	344	301	301
query61	126	125	122	122
query62	786	728	666	666
query63	231	185	194	185
query64	4367	994	693	693
query65	4369	4275	4192	4192
query66	1140	415	328	328
query67	15904	15583	15399	15399
query68	7938	883	513	513
query69	468	312	260	260
query70	1178	1113	1069	1069
query71	484	319	307	307
query72	5626	4796	4912	4796
query73	740	676	353	353
query74	8928	8820	8625	8625
query75	3922	3204	2707	2707
query76	3714	1190	775	775
query77	788	373	290	290
query78	10009	10101	9277	9277
query79	2272	799	572	572
query80	582	529	455	455
query81	486	245	219	219
query82	458	132	97	97
query83	246	248	236	236
query84	254	97	88	88
query85	796	354	380	354
query86	385	322	273	273
query87	4387	4443	4265	4265
query88	3712	2193	2210	2193
query89	385	313	276	276
query90	1836	212	211	211
query91	142	154	115	115
query92	72	67	58	58
query93	1871	928	579	579
query94	663	419	319	319
query95	377	289	290	289
query96	485	563	271	271
query97	3147	3207	3147	3147
query98	232	203	217	203
query99	1436	1396	1262	1262
Total cold run time: 274950 ms
Total hot run time: 185700 ms

@doris-robot
Copy link

doris-robot commented Apr 24, 2025

ClickBench: Total hot run time: 29.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dd31bdd74983096ebb653eed28ab2a9d2c0ad209, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.27	0.19	0.19
query4	1.59	0.19	0.11
query5	0.57	0.55	0.54
query6	1.20	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.53	0.51
query10	0.57	0.56	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	1.19	1.16	1.16
query15	0.88	0.86	0.85
query16	0.38	0.40	0.39
query17	1.03	1.02	1.02
query18	0.22	0.20	0.20
query19	1.91	1.80	1.81
query20	0.02	0.01	0.02
query21	15.39	0.91	0.55
query22	0.75	1.13	0.70
query23	14.90	1.36	0.68
query24	7.42	1.16	0.48
query25	0.47	0.18	0.13
query26	0.58	0.18	0.14
query27	0.05	0.06	0.04
query28	9.38	0.90	0.45
query29	12.54	3.95	3.34
query30	0.26	0.10	0.07
query31	2.81	0.59	0.38
query32	3.23	0.54	0.47
query33	3.07	3.07	3.03
query34	15.76	5.15	4.51
query35	4.56	4.55	4.51
query36	0.66	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.03	0.02	0.03
Total cold run time: 103.83 s
Total hot run time: 29.33 s

@deardeng
Copy link
Contributor Author

deardeng commented Apr 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-H: Total hot run time: 34250 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 38f8e8d579b138481e31618156fda6c846537bba, data reload: false

------ Round 1 ----------------------------------
q1	25852	5106	5029	5029
q2	2072	274	186	186
q3	10389	1250	688	688
q4	10225	998	540	540
q5	7491	2380	2397	2380
q6	191	163	131	131
q7	915	752	618	618
q8	9332	1404	1203	1203
q9	6861	5173	5102	5102
q10	6868	2318	1898	1898
q11	494	295	270	270
q12	347	357	222	222
q13	17797	3784	3121	3121
q14	232	232	204	204
q15	534	487	482	482
q16	446	447	422	422
q17	588	833	375	375
q18	7472	7187	7161	7161
q19	1601	936	554	554
q20	328	327	216	216
q21	4008	3498	2468	2468
q22	1093	1038	980	980
Total cold run time: 115136 ms
Total hot run time: 34250 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5178	5050	5075	5050
q2	232	323	230	230
q3	2154	2618	2275	2275
q4	1457	1847	1421	1421
q5	4450	4398	4381	4381
q6	218	169	127	127
q7	2014	1917	1731	1731
q8	2575	2531	2732	2531
q9	7251	7339	7061	7061
q10	2972	3181	2777	2777
q11	582	499	491	491
q12	688	781	640	640
q13	3549	3846	3231	3231
q14	273	286	294	286
q15	520	487	482	482
q16	464	501	459	459
q17	1157	1562	1394	1394
q18	7673	7458	7405	7405
q19	805	838	939	838
q20	1937	2009	1835	1835
q21	5249	4980	4948	4948
q22	1124	1071	1026	1026
Total cold run time: 52522 ms
Total hot run time: 50619 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-DS: Total hot run time: 192056 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 38f8e8d579b138481e31618156fda6c846537bba, data reload: false

query1	1408	1080	1057	1057
query2	6098	1845	1834	1834
query3	11168	4539	4661	4539
query4	54261	24701	22991	22991
query5	5133	532	445	445
query6	335	207	204	204
query7	4911	492	293	293
query8	335	263	239	239
query9	5765	2575	2582	2575
query10	431	336	279	279
query11	15053	15004	14763	14763
query12	159	114	109	109
query13	1044	531	390	390
query14	10093	6352	6329	6329
query15	202	215	181	181
query16	7094	686	539	539
query17	1098	733	608	608
query18	1548	425	327	327
query19	202	202	172	172
query20	136	127	122	122
query21	209	133	119	119
query22	4370	4499	4291	4291
query23	34093	33468	33582	33468
query24	6756	2421	2420	2420
query25	477	460	411	411
query26	719	272	151	151
query27	2289	523	336	336
query28	3387	2129	2116	2116
query29	574	594	456	456
query30	276	225	199	199
query31	874	906	817	817
query32	71	64	62	62
query33	463	370	313	313
query34	760	857	540	540
query35	778	834	765	765
query36	972	984	903	903
query37	119	99	101	99
query38	4188	4205	4204	4204
query39	1510	1427	1424	1424
query40	219	123	107	107
query41	55	55	52	52
query42	124	108	111	108
query43	498	523	488	488
query44	1329	821	812	812
query45	180	179	179	179
query46	855	1026	634	634
query47	1877	1889	1799	1799
query48	379	408	316	316
query49	683	527	440	440
query50	664	704	417	417
query51	4171	4160	4233	4160
query52	114	103	97	97
query53	235	262	187	187
query54	588	595	522	522
query55	90	89	90	89
query56	331	316	301	301
query57	1178	1171	1121	1121
query58	269	254	265	254
query59	2729	2871	2662	2662
query60	336	333	316	316
query61	151	129	127	127
query62	750	765	694	694
query63	252	195	188	188
query64	1913	1057	698	698
query65	4478	4226	4251	4226
query66	709	403	304	304
query67	15863	15664	15350	15350
query68	7435	890	511	511
query69	533	294	287	287
query70	1174	1106	1059	1059
query71	492	323	284	284
query72	5995	4851	4758	4758
query73	1383	619	341	341
query74	8977	8884	8641	8641
query75	3867	3197	2714	2714
query76	4305	1187	752	752
query77	646	372	271	271
query78	10116	9986	9299	9299
query79	4174	814	556	556
query80	683	519	445	445
query81	486	252	216	216
query82	497	126	98	98
query83	335	250	231	231
query84	299	110	84	84
query85	803	360	313	313
query86	388	296	280	280
query87	4323	4438	4278	4278
query88	3555	2232	2242	2232
query89	432	322	277	277
query90	1906	209	214	209
query91	144	144	112	112
query92	78	60	55	55
query93	3023	914	572	572
query94	671	420	301	301
query95	376	291	287	287
query96	487	561	274	274
query97	3182	3200	3110	3110
query98	226	204	207	204
query99	1407	1388	1267	1267
Total cold run time: 302777 ms
Total hot run time: 192056 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

ClickBench: Total hot run time: 29.98 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 38f8e8d579b138481e31618156fda6c846537bba, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.11	0.10
query3	0.25	0.20	0.20
query4	1.60	0.19	0.20
query5	0.60	0.59	0.60
query6	1.18	0.71	0.71
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.53	0.51
query10	0.58	0.58	0.56
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	1.15	1.18	1.17
query15	0.86	0.86	0.88
query16	0.38	0.39	0.39
query17	1.06	1.03	1.03
query18	0.21	0.20	0.19
query19	1.85	1.80	1.87
query20	0.01	0.01	0.01
query21	15.39	0.92	0.55
query22	0.78	1.18	0.74
query23	14.81	1.41	0.62
query24	6.73	1.38	0.99
query25	0.51	0.21	0.12
query26	0.64	0.16	0.14
query27	0.06	0.05	0.05
query28	9.83	0.88	0.45
query29	12.56	4.00	3.33
query30	0.25	0.09	0.07
query31	2.83	0.60	0.39
query32	3.22	0.54	0.46
query33	2.96	3.12	3.10
query34	15.90	5.13	4.53
query35	4.56	4.51	4.46
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.62 s
Total hot run time: 29.98 s

@deardeng
Copy link
Contributor Author

deardeng commented Apr 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-H: Total hot run time: 33858 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f210c2b6d095df463f870da9cb687415173963d, data reload: false

------ Round 1 ----------------------------------
q1	26428	5018	4986	4986
q2	2063	285	183	183
q3	10387	1233	686	686
q4	10236	1017	537	537
q5	7548	2368	2333	2333
q6	178	161	129	129
q7	908	724	603	603
q8	9302	1227	1098	1098
q9	6831	5098	5069	5069
q10	6827	2308	1897	1897
q11	479	284	266	266
q12	357	356	212	212
q13	17771	3701	3093	3093
q14	228	235	213	213
q15	531	483	494	483
q16	460	462	402	402
q17	607	851	369	369
q18	7443	7121	7088	7088
q19	1252	943	539	539
q20	335	315	211	211
q21	4124	2665	2498	2498
q22	1054	1016	963	963
Total cold run time: 115349 ms
Total hot run time: 33858 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5128	5076	5093	5076
q2	232	332	230	230
q3	2141	2629	2286	2286
q4	1471	1807	1427	1427
q5	4408	4412	4371	4371
q6	218	173	128	128
q7	1996	1910	1782	1782
q8	2594	2531	2550	2531
q9	7287	7272	7006	7006
q10	3008	3229	2728	2728
q11	576	511	488	488
q12	688	750	632	632
q13	3554	3894	3270	3270
q14	287	287	294	287
q15	541	491	487	487
q16	473	499	455	455
q17	1125	1559	1413	1413
q18	7669	7712	7537	7537
q19	826	830	964	830
q20	1980	1991	1826	1826
q21	5308	4892	4757	4757
q22	1120	1094	1068	1068
Total cold run time: 52630 ms
Total hot run time: 50615 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-DS: Total hot run time: 192897 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f210c2b6d095df463f870da9cb687415173963d, data reload: false

query1	1405	1103	1056	1056
query2	6218	1824	1825	1824
query3	11152	4595	4731	4595
query4	25335	23214	23410	23214
query5	5142	628	449	449
query6	310	197	185	185
query7	3986	498	278	278
query8	293	235	240	235
query9	8515	2587	2600	2587
query10	510	346	263	263
query11	15157	15229	14816	14816
query12	166	106	105	105
query13	1549	520	385	385
query14	8951	6172	6249	6172
query15	204	189	172	172
query16	7281	628	479	479
query17	1158	733	616	616
query18	2041	444	332	332
query19	203	196	179	179
query20	134	135	122	122
query21	209	131	115	115
query22	4426	4550	4422	4422
query23	34495	33707	33682	33682
query24	8559	2467	2408	2408
query25	528	478	398	398
query26	1233	279	150	150
query27	2724	516	345	345
query28	4868	2146	2107	2107
query29	727	571	439	439
query30	277	219	187	187
query31	911	884	780	780
query32	71	63	70	63
query33	596	366	316	316
query34	812	900	544	544
query35	820	844	749	749
query36	951	1010	900	900
query37	113	101	80	80
query38	4314	4315	4143	4143
query39	1510	1442	1406	1406
query40	243	116	104	104
query41	55	55	52	52
query42	124	106	106	106
query43	482	510	489	489
query44	1321	839	821	821
query45	195	186	165	165
query46	844	1017	641	641
query47	1822	1905	1780	1780
query48	391	418	318	318
query49	750	510	404	404
query50	644	700	407	407
query51	4208	4252	4253	4252
query52	115	113	93	93
query53	227	256	189	189
query54	597	585	541	541
query55	83	81	81	81
query56	338	307	281	281
query57	1194	1185	1082	1082
query58	267	256	261	256
query59	2717	2768	2645	2645
query60	340	317	327	317
query61	135	132	128	128
query62	770	746	659	659
query63	228	193	192	192
query64	3998	1048	754	754
query65	4478	4402	4351	4351
query66	1147	423	333	333
query67	16132	15439	15558	15439
query68	8937	879	503	503
query69	494	312	265	265
query70	1219	1099	1063	1063
query71	477	319	287	287
query72	5597	4695	4655	4655
query73	696	575	351	351
query74	9244	9141	8954	8954
query75	3996	3206	2692	2692
query76	3766	1177	740	740
query77	803	372	273	273
query78	9882	9959	9340	9340
query79	4658	784	552	552
query80	688	518	435	435
query81	483	263	216	216
query82	683	127	94	94
query83	271	257	238	238
query84	294	105	86	86
query85	774	425	324	324
query86	342	303	288	288
query87	4324	4506	4381	4381
query88	2836	2235	2224	2224
query89	455	330	290	290
query90	2048	213	215	213
query91	143	144	130	130
query92	80	62	57	57
query93	2541	959	586	586
query94	681	397	313	313
query95	380	294	287	287
query96	508	554	269	269
query97	3144	3257	3133	3133
query98	226	209	201	201
query99	1426	1412	1279	1279
Total cold run time: 283795 ms
Total hot run time: 192897 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

ClickBench: Total hot run time: 29.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1f210c2b6d095df463f870da9cb687415173963d, data reload: false

query1	0.04	0.04	0.02
query2	0.12	0.11	0.11
query3	0.24	0.20	0.20
query4	1.59	0.20	0.20
query5	0.59	0.59	0.59
query6	1.18	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.51	0.51
query10	0.57	0.56	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.59	0.60
query14	1.24	1.21	1.21
query15	0.87	0.85	0.85
query16	0.38	0.38	0.37
query17	1.07	1.01	1.02
query18	0.22	0.20	0.19
query19	1.84	1.78	1.81
query20	0.01	0.01	0.01
query21	15.39	0.94	0.52
query22	0.75	1.18	0.68
query23	14.95	1.40	0.58
query24	7.20	1.09	0.58
query25	0.53	0.09	0.19
query26	0.63	0.17	0.13
query27	0.04	0.05	0.06
query28	9.42	0.89	0.44
query29	12.51	3.96	3.27
query30	0.24	0.09	0.06
query31	2.83	0.60	0.38
query32	3.23	0.56	0.47
query33	3.09	3.06	2.99
query34	15.86	5.16	4.49
query35	4.54	4.56	4.57
query36	0.68	0.50	0.49
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 103.92 s
Total hot run time: 29.3 s

dataroaring
dataroaring reviewed Apr 25, 2025
View reviewed changes
dataroaring
dataroaring previously approved these changes Apr 25, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 25, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 25, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 25, 2025

PR approved by anyone and no changes requested.

@deardeng
Copy link
Contributor Author

deardeng commented Apr 25, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 25, 2025
@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-H: Total hot run time: 34805 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf0387c1fff0bfc040a58b7bd2fc6e1bd3560d07, data reload: false

------ Round 1 ----------------------------------
q1	25801	5073	5049	5049
q2	2072	269	176	176
q3	10409	1248	686	686
q4	10230	1004	538	538
q5	7501	2341	2701	2341
q6	185	165	135	135
q7	918	727	600	600
q8	9326	1281	1119	1119
q9	6811	5209	5134	5134
q10	6814	2282	1905	1905
q11	464	282	273	273
q12	351	355	226	226
q13	17787	3616	3151	3151
q14	223	221	210	210
q15	531	486	484	484
q16	445	455	405	405
q17	606	866	366	366
q18	7483	7074	7161	7074
q19	1232	975	558	558
q20	334	327	230	230
q21	4320	3501	3181	3181
q22	1080	980	964	964
Total cold run time: 114923 ms
Total hot run time: 34805 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5225	5141	5091	5091
q2	249	345	230	230
q3	2164	2647	2265	2265
q4	1449	1841	1471	1471
q5	4466	4408	4436	4408
q6	227	167	137	137
q7	2022	1953	1784	1784
q8	2614	2617	2577	2577
q9	7445	7372	7301	7301
q10	3062	3257	2820	2820
q11	595	520	511	511
q12	707	807	647	647
q13	3731	4104	3622	3622
q14	297	312	260	260
q15	520	500	501	500
q16	530	525	495	495
q17	1257	1635	1453	1453
q18	8516	8123	7961	7961
q19	867	828	915	828
q20	2017	1975	1872	1872
q21	5372	4953	4731	4731
q22	1113	1045	1036	1036
Total cold run time: 54445 ms
Total hot run time: 52000 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-DS: Total hot run time: 192310 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf0387c1fff0bfc040a58b7bd2fc6e1bd3560d07, data reload: false

query1	1413	1089	1055	1055
query2	6339	1789	1805	1789
query3	11006	4495	4446	4446
query4	53598	24415	23173	23173
query5	5237	503	462	462
query6	382	215	205	205
query7	5017	495	284	284
query8	341	273	246	246
query9	6479	2579	2569	2569
query10	421	322	258	258
query11	15261	15047	14917	14917
query12	160	119	113	113
query13	1117	534	399	399
query14	10233	6482	6354	6354
query15	195	188	175	175
query16	7013	659	470	470
query17	1084	746	570	570
query18	1510	406	306	306
query19	195	193	175	175
query20	132	130	125	125
query21	207	126	123	123
query22	4254	4521	4354	4354
query23	34156	33451	33573	33451
query24	6732	2420	2433	2420
query25	464	469	402	402
query26	669	279	157	157
query27	2225	504	339	339
query28	2999	2138	2130	2130
query29	585	556	427	427
query30	274	235	191	191
query31	859	881	789	789
query32	77	67	67	67
query33	469	404	319	319
query34	806	877	528	528
query35	817	828	740	740
query36	959	990	890	890
query37	122	104	82	82
query38	4205	4238	4209	4209
query39	1491	1438	1460	1438
query40	216	122	114	114
query41	62	51	54	51
query42	119	111	110	110
query43	505	485	477	477
query44	1404	836	850	836
query45	191	177	165	165
query46	865	1061	658	658
query47	1812	1830	1787	1787
query48	400	431	323	323
query49	698	516	432	432
query50	715	691	411	411
query51	4207	4256	4229	4229
query52	117	115	102	102
query53	243	261	203	203
query54	606	583	525	525
query55	81	83	79	79
query56	326	323	294	294
query57	1197	1218	1115	1115
query58	282	290	253	253
query59	2644	2755	2581	2581
query60	336	321	316	316
query61	141	126	128	126
query62	774	775	685	685
query63	237	196	190	190
query64	1412	1031	692	692
query65	4467	4251	4211	4211
query66	728	404	298	298
query67	15690	15386	15245	15245
query68	5446	897	516	516
query69	527	312	283	283
query70	1199	1128	1145	1128
query71	458	322	303	303
query72	5880	4796	4945	4796
query73	1132	692	358	358
query74	8834	8899	8680	8680
query75	3535	3246	2757	2757
query76	3644	1203	763	763
query77	554	477	297	297
query78	9924	9934	9212	9212
query79	2603	811	579	579
query80	640	510	434	434
query81	492	255	219	219
query82	626	121	94	94
query83	248	248	246	246
query84	294	101	84	84
query85	815	364	362	362
query86	430	291	296	291
query87	4368	4475	4322	4322
query88	3552	2228	2219	2219
query89	411	314	278	278
query90	1886	216	221	216
query91	146	139	113	113
query92	73	58	59	58
query93	2509	929	567	567
query94	684	412	296	296
query95	369	297	283	283
query96	487	573	278	278
query97	3152	3217	3118	3118
query98	235	206	203	203
query99	1336	1397	1302	1302
Total cold run time: 296467 ms
Total hot run time: 192310 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

ClickBench: Total hot run time: 29.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf0387c1fff0bfc040a58b7bd2fc6e1bd3560d07, data reload: false

query1	0.04	0.05	0.03
query2	0.12	0.11	0.11
query3	0.26	0.19	0.19
query4	1.59	0.19	0.19
query5	0.58	0.58	0.59
query6	1.19	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.03	0.04
query9	0.57	0.53	0.51
query10	0.56	0.55	0.57
query11	0.16	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.59	0.60
query14	1.18	1.18	1.21
query15	0.87	0.84	0.86
query16	0.40	0.38	0.37
query17	1.01	1.01	1.04
query18	0.21	0.20	0.19
query19	1.91	1.74	1.87
query20	0.01	0.02	0.01
query21	15.42	0.90	0.55
query22	0.76	1.20	0.65
query23	14.91	1.35	0.62
query24	7.29	1.30	1.11
query25	0.46	0.15	0.12
query26	0.66	0.16	0.13
query27	0.06	0.04	0.05
query28	9.13	0.89	0.43
query29	12.58	3.95	3.29
query30	0.25	0.10	0.05
query31	2.82	0.59	0.40
query32	3.23	0.54	0.47
query33	3.06	3.05	3.01
query34	15.84	5.14	4.45
query35	4.52	4.53	4.53
query36	0.67	0.52	0.49
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.62 s
Total hot run time: 29.71 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 25, 2025

BE UT Coverage Report

Increment line coverage 20.34% (12/59) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.13% (14684/27125)
Line Coverage 43.00% (127867/297363)
Region Coverage 41.85% (65470/156454)
Branch Coverage 36.44% (33002/90568)

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 25, 2025

BE Regression P0 && UT Coverage Report

Increment line coverage 38.98% (23/59) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.08% (14668/26631)
Line Coverage 44.47% (132014/296871)
Region Coverage 41.53% (75955/182879)
Branch Coverage 35.61% (36721/103126)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 27, 2025

BE Regression P0 && UT Coverage Report

Increment line coverage 38.98% (23/59) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.08% (14668/26631)
Line Coverage 44.47% (132014/296871)
Region Coverage 41.53% (75955/182879)
Branch Coverage 35.61% (36721/103126)

dataroaring
dataroaring approved these changes Apr 27, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Apr 27, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 27, 2025
liaoxin01
liaoxin01 approved these changes Apr 27, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 4e9f140 into apache:master Apr 28, 2025
25 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request Apr 28, 2025
@deardeng
[fix](path gc) Fix path gc race with publish task (#50343)
67db391 
### What problem does this PR solve?

Fix path gc race with publish task and checkpoint task,
if not fix it, 
1. very low probability of data loss
2. run `test_path_gc_with_publish_version` case will get err
```
java.sql.SQLException: errCode = 2, detailMessage = (175.40.51.1)[NOT_FOUND]failed to get file size /opt/apache-doris/be/storage/1.HDD/data/22/1745498446653/1868719407/0200000000000019214e72da3f1fe5cfc12
c5efad2bf05bf_0.dat: No such file or directory
```
@github-actions github-actions bot mentioned this pull request Apr 28, 2025
Merged
dataroaring pushed a commit that referenced this pull request Apr 28, 2025
@github-actions @deardeng
branch-3.0: [fix](path gc) Fix path gc race with publish task #50343 (#…
7b70b72 
…50487)

Cherry-picked from #50343

Co-authored-by: deardeng <dengxin@selectdb.com>
@deardeng deardeng mentioned this pull request Apr 28, 2025
Merged
deardeng added a commit to deardeng/incubator-doris that referenced this pull request Apr 28, 2025
@deardeng
[fix](path gc) Fix path gc race with publish task (apache#50343)
edae705 
Fix path gc race with publish task and checkpoint task,
if not fix it,
1. very low probability of data loss
2. run `test_path_gc_with_publish_version` case will get err
```
java.sql.SQLException: errCode = 2, detailMessage = (175.40.51.1)[NOT_FOUND]failed to get file size /opt/apache-doris/be/storage/1.HDD/data/22/1745498446653/1868719407/0200000000000019214e72da3f1fe5cfc12
c5efad2bf05bf_0.dat: No such file or directory
```
yiguolei pushed a commit that referenced this pull request Apr 29, 2025
@deardeng
branch-2.1: [fix](path gc) Fix path gc race with publish task #50343 (#…
3660139 
…50520)

cherry pick from #50343
deardeng added a commit to deardeng/incubator-doris that referenced this pull request Apr 30, 2025
@deardeng
branch-2.1: [fix](path gc) Fix path gc race with publish task apache#…
035cb22 
…50343 (apache#50520)

cherry pick from apache#50343
@yiguolei yiguolei mentioned this pull request May 13, 2025
Closed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@deardeng @koarz
[fix](path gc) Fix path gc race with publish task (apache#50343)
69be295 
### What problem does this PR solve?

Fix path gc race with publish task and checkpoint task,
if not fix it, 
1. very low probability of data loss
2. run `test_path_gc_with_publish_version` case will get err
```
java.sql.SQLException: errCode = 2, detailMessage = (175.40.51.1)[NOT_FOUND]failed to get file size /opt/apache-doris/be/storage/1.HDD/data/22/1745498446653/1868719407/0200000000000019214e72da3f1fe5cfc12
c5efad2bf05bf_0.dat: No such file or directory
```
@gavinchou gavinchou mentioned this pull request Jun 11, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.6-merged p0_w reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@deardeng @Thearas @doris-robot @hello-stephen @liaoxin01 @dataroaring @yiguolei