Skip to content

branch-3.0: [fix](path gc) Fix path gc race with publish task #50343 #50487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Apr 28, 2025
Merged

branch-3.0: [fix](path gc) Fix path gc race with publish task #50343 #50487

dataroaring merged 1 commit into from
Apr 28, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Apr 28, 2025

Cherry-picked from #50343

@deardeng
[fix](path gc) Fix path gc race with publish task (#50343)
67db391 
### What problem does this PR solve?

Fix path gc race with publish task and checkpoint task,
if not fix it, 
1. very low probability of data loss
2. run `test_path_gc_with_publish_version` case will get err
```
java.sql.SQLException: errCode = 2, detailMessage = (175.40.51.1)[NOT_FOUND]failed to get file size /opt/apache-doris/be/storage/1.HDD/data/22/1745498446653/1868719407/0200000000000019214e72da3f1fe5cfc12
c5efad2bf05bf_0.dat: No such file or directory
```
@github-actions github-actions bot requested a review from dataroaring as a code owner April 28, 2025 03:05
@Thearas
Copy link
Contributor

Thearas commented Apr 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Apr 28, 2025
@Thearas
Copy link
Contributor

Thearas commented Apr 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 28, 2025

TPC-H: Total hot run time: 40316 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 67db3911013aef737d8cdbef4408f45484768ce3, data reload: false

------ Round 1 ----------------------------------
q1	17613	6778	6628	6628
q2	2066	188	195	188
q3	10603	1087	1148	1087
q4	10803	717	744	717
q5	7782	2920	2889	2889
q6	233	139	134	134
q7	1018	633	618	618
q8	9388	1948	2037	1948
q9	6626	6427	6412	6412
q10	7021	2301	2272	2272
q11	460	263	255	255
q12	398	214	215	214
q13	17807	2977	3016	2977
q14	249	217	207	207
q15	513	458	462	458
q16	674	606	585	585
q17	969	595	600	595
q18	7335	6729	6572	6572
q19	1396	1134	1111	1111
q20	475	202	209	202
q21	4200	3335	3270	3270
q22	1128	1013	977	977
Total cold run time: 108757 ms
Total hot run time: 40316 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6633	6579	6571	6571
q2	329	232	230	230
q3	2934	2782	2952	2782
q4	2065	1824	1796	1796
q5	5781	5731	5708	5708
q6	206	125	130	125
q7	2237	1791	1790	1790
q8	3401	3577	3509	3509
q9	8995	8878	8794	8794
q10	3587	3500	3506	3500
q11	594	491	490	490
q12	814	664	597	597
q13	9135	3202	3157	3157
q14	304	285	271	271
q15	513	471	456	456
q16	674	653	648	648
q17	1830	1639	1610	1610
q18	8148	7748	7881	7748
q19	1710	1566	1417	1417
q20	2053	1855	1861	1855
q21	5492	5337	5382	5337
q22	1122	1060	1023	1023
Total cold run time: 68557 ms
Total hot run time: 59414 ms

@doris-robot
Copy link

doris-robot commented Apr 28, 2025

BE UT Coverage Report

Increment line coverage 20.34% (12/59) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 40.21% (10558/26256)
Line Coverage 31.00% (89403/288413)
Region Coverage 30.12% (46097/153064)
Branch Coverage 26.63% (23573/88516)

@doris-robot
Copy link

doris-robot commented Apr 28, 2025

TPC-DS: Total hot run time: 197393 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 67db3911013aef737d8cdbef4408f45484768ce3, data reload: false

query1	1310	911	906	906
query2	6314	1999	1973	1973
query3	10810	4400	4207	4207
query4	60367	29231	23436	23436
query5	5188	459	443	443
query6	403	179	171	171
query7	5459	318	308	308
query8	308	250	226	226
query9	8443	2626	2601	2601
query10	461	273	278	273
query11	17604	15443	15612	15443
query12	175	107	104	104
query13	1483	439	443	439
query14	10551	7036	6882	6882
query15	208	182	181	181
query16	7124	460	464	460
query17	1246	586	602	586
query18	1860	321	320	320
query19	218	167	176	167
query20	120	111	114	111
query21	212	107	105	105
query22	4634	4324	4782	4324
query23	34610	34363	34374	34363
query24	6170	2946	2966	2946
query25	542	416	416	416
query26	660	177	174	174
query27	1827	367	367	367
query28	4635	2453	2454	2453
query29	716	454	443	443
query30	237	172	161	161
query31	983	834	802	802
query32	72	55	59	55
query33	468	283	294	283
query34	886	512	540	512
query35	870	747	729	729
query36	1086	963	970	963
query37	117	69	73	69
query38	4119	3970	4080	3970
query39	1566	1460	1454	1454
query40	201	103	99	99
query41	51	51	51	51
query42	119	105	103	103
query43	551	494	504	494
query44	1188	805	802	802
query45	184	168	173	168
query46	1144	732	759	732
query47	2019	1930	1917	1917
query48	494	406	386	386
query49	735	414	408	408
query50	855	432	433	432
query51	7455	7215	7172	7172
query52	110	92	90	90
query53	269	208	188	188
query54	579	466	481	466
query55	80	82	81	81
query56	266	260	266	260
query57	1299	1149	1161	1149
query58	215	214	223	214
query59	3355	3128	3005	3005
query60	306	264	259	259
query61	127	111	106	106
query62	807	691	697	691
query63	226	201	196	196
query64	1367	671	636	636
query65	3271	3189	3214	3189
query66	717	303	311	303
query67	15899	15662	15642	15642
query68	4263	568	562	562
query69	439	269	266	266
query70	1171	1133	1076	1076
query71	365	267	256	256
query72	6666	3975	3994	3975
query73	766	343	356	343
query74	10268	8880	9367	8880
query75	3393	2662	2645	2645
query76	2015	1070	1177	1070
query77	471	278	267	267
query78	10682	9563	9611	9563
query79	1383	593	584	584
query80	861	457	437	437
query81	507	244	236	236
query82	1292	89	89	89
query83	177	142	144	142
query84	294	85	75	75
query85	862	305	285	285
query86	345	288	301	288
query87	4495	4293	4248	4248
query88	3699	2440	2365	2365
query89	415	296	294	294
query90	2061	190	188	188
query91	178	152	152	152
query92	64	53	52	52
query93	1810	554	549	549
query94	754	302	299	299
query95	355	262	260	260
query96	613	275	280	275
query97	3283	3122	3109	3109
query98	216	203	201	201
query99	1581	1297	1305	1297
Total cold run time: 313695 ms
Total hot run time: 197393 ms

@doris-robot
Copy link

doris-robot commented Apr 28, 2025

ClickBench: Total hot run time: 32.07 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 67db3911013aef737d8cdbef4408f45484768ce3, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.11
query5	0.52	0.49	0.50
query6	1.14	0.72	0.73
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.48
query10	0.55	0.54	0.54
query11	0.14	0.10	0.10
query12	0.14	0.12	0.12
query13	0.61	0.61	0.59
query14	2.73	2.72	2.83
query15	0.90	0.83	0.84
query16	0.40	0.38	0.40
query17	1.02	1.12	1.05
query18	0.24	0.21	0.22
query19	1.88	1.73	2.00
query20	0.02	0.01	0.01
query21	15.36	0.62	0.59
query22	2.45	3.12	2.18
query23	16.95	0.93	0.81
query24	3.24	0.62	1.67
query25	0.25	0.08	0.12
query26	0.46	0.15	0.14
query27	0.05	0.05	0.04
query28	10.12	0.56	0.47
query29	12.55	3.24	3.22
query30	0.25	0.07	0.06
query31	2.87	0.40	0.40
query32	3.23	0.47	0.46
query33	2.96	3.03	3.03
query34	17.23	4.51	4.47
query35	4.52	4.48	4.51
query36	0.65	0.49	0.48
query37	0.09	0.06	0.07
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.48 s
Total hot run time: 32.07 s

dataroaring
dataroaring approved these changes Apr 28, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 7b70b72 into branch-3.0 Apr 28, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-50343-branch-3.0 branch April 28, 2025 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Thearas @doris-robot @dataroaring @deardeng