Skip to content

branch-3.0: [fix](cloud) potential data race when retrying prepare/commit rowset for load #51129 #51483

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 11, 2025
Merged

branch-3.0: [fix](cloud) potential data race when retrying prepare/commit rowset for load #51129 #51483

dataroaring merged 1 commit into from
Jun 11, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jun 4, 2025

Cherry-picked from #51129

@liaoxin01
[fix](cloud) potential data race when retrying prepare/commit rowset …
f8ad47a 
…for load (#51129)

If the transaction has been finished, it means the prepare/commit rowset
is a timeout retry request. In this case, do not write the recycle key
again, otherwise it may cause data loss.
**Notice**:
There is a compatibility issue with multi-statement transaction write
operations during upgrades that could cause write operations to fail
either before or during the meta service upgrade process.
@github-actions github-actions bot requested a review from dataroaring as a code owner June 4, 2025 08:34
@Thearas
Copy link
Contributor

Thearas commented Jun 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jun 4, 2025
@dataroaring dataroaring reopened this Jun 4, 2025
@Thearas
Copy link
Contributor

Thearas commented Jun 4, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 4, 2025

Cloud UT Coverage Report

Increment line coverage 66.15% (43/65) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.14% (1095/1317)
Line Coverage 66.39% (18293/27554)
Region Coverage 66.12% (9054/13693)
Branch Coverage 56.01% (4905/8758)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 4, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.08% (10864/26445)
Line Coverage 31.91% (92869/290996)
Region Coverage 31.00% (47912/154566)
Branch Coverage 27.48% (24545/89306)

@doris-robot
Copy link

doris-robot commented Jun 4, 2025

TPC-H: Total hot run time: 40013 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f8ad47af301db548edf60f605bbce60795a4ead6, data reload: false

------ Round 1 ----------------------------------
q1	17650	6776	6657	6657
q2	2088	169	185	169
q3	10629	1086	1224	1086
q4	10564	714	772	714
q5	7748	2875	2825	2825
q6	220	134	132	132
q7	967	624	589	589
q8	9351	1961	2060	1961
q9	6618	6442	6435	6435
q10	6997	2274	2280	2274
q11	463	264	241	241
q12	396	208	209	208
q13	17816	2955	2958	2955
q14	231	203	211	203
q15	500	465	454	454
q16	470	365	379	365
q17	976	536	562	536
q18	7348	6773	6786	6773
q19	1390	1128	1152	1128
q20	471	196	189	189
q21	4138	3316	3126	3126
q22	1132	1005	993	993
Total cold run time: 108163 ms
Total hot run time: 40013 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6618	6524	6559	6524
q2	338	236	230	230
q3	2935	2769	2842	2769
q4	2086	1798	1773	1773
q5	5755	5745	5783	5745
q6	203	129	124	124
q7	2222	1782	1779	1779
q8	3383	3599	3564	3564
q9	8941	8733	8910	8733
q10	3538	3519	3527	3519
q11	583	490	487	487
q12	781	583	592	583
q13	9936	3117	3136	3117
q14	316	277	278	277
q15	509	459	457	457
q16	493	452	436	436
q17	1839	1611	1623	1611
q18	8260	7731	7714	7714
q19	1685	1694	1511	1511
q20	2134	1868	1848	1848
q21	5336	5358	5266	5266
q22	1156	1040	1058	1040
Total cold run time: 69047 ms
Total hot run time: 59107 ms

@doris-robot
Copy link

doris-robot commented Jun 4, 2025

TPC-DS: Total hot run time: 195506 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f8ad47af301db548edf60f605bbce60795a4ead6, data reload: false

query1	1267	896	885	885
query2	6409	1929	1916	1916
query3	10990	4326	4350	4326
query4	61887	29921	23246	23246
query5	5251	441	444	441
query6	403	166	175	166
query7	5475	319	312	312
query8	305	226	218	218
query9	8490	2575	2566	2566
query10	483	263	269	263
query11	17503	15131	16080	15131
query12	175	103	113	103
query13	1430	441	430	430
query14	10839	7501	6787	6787
query15	199	181	181	181
query16	7239	502	484	484
query17	1182	599	611	599
query18	1841	317	312	312
query19	218	161	160	160
query20	120	108	106	106
query21	211	105	110	105
query22	4838	4754	4523	4523
query23	34431	33638	34106	33638
query24	6171	2907	2934	2907
query25	540	406	442	406
query26	650	163	161	161
query27	1833	340	360	340
query28	4370	2167	2154	2154
query29	680	446	430	430
query30	241	160	154	154
query31	973	810	827	810
query32	62	55	53	53
query33	394	274	269	269
query34	912	494	500	494
query35	848	711	744	711
query36	1055	958	943	943
query37	109	66	70	66
query38	3972	4013	3910	3910
query39	1520	1453	1454	1453
query40	200	95	99	95
query41	48	47	46	46
query42	106	99	105	99
query43	523	492	501	492
query44	1197	821	842	821
query45	183	169	171	169
query46	1132	723	716	716
query47	2043	1920	1991	1920
query48	478	389	387	387
query49	746	383	386	383
query50	845	419	414	414
query51	7367	7266	7188	7188
query52	102	85	87	85
query53	262	181	185	181
query54	570	463	452	452
query55	79	78	76	76
query56	253	264	252	252
query57	1285	1175	1160	1160
query58	221	214	212	212
query59	3292	3135	3014	3014
query60	285	253	251	251
query61	110	107	104	104
query62	789	684	684	684
query63	215	188	187	187
query64	1403	681	628	628
query65	3220	3204	3150	3150
query66	706	288	290	288
query67	15908	15743	15613	15613
query68	4118	580	569	569
query69	408	271	260	260
query70	1131	1037	1109	1037
query71	337	255	245	245
query72	6428	4073	4099	4073
query73	756	340	356	340
query74	10505	9011	9126	9011
query75	3304	2609	2664	2609
query76	1944	1061	1009	1009
query77	492	269	272	269
query78	10521	9681	9649	9649
query79	1538	605	589	589
query80	863	441	416	416
query81	487	216	214	214
query82	1212	87	86	86
query83	165	139	136	136
query84	282	80	71	71
query85	880	299	281	281
query86	341	271	294	271
query87	4407	4174	4194	4174
query88	3530	2374	2337	2337
query89	420	292	292	292
query90	1991	183	181	181
query91	179	146	143	143
query92	65	48	48	48
query93	1936	562	558	558
query94	726	290	288	288
query95	349	250	245	245
query96	604	278	288	278
query97	3315	3124	3154	3124
query98	210	204	199	199
query99	1544	1322	1278	1278
Total cold run time: 314072 ms
Total hot run time: 195506 ms

@doris-robot
Copy link

doris-robot commented Jun 4, 2025

ClickBench: Total hot run time: 31.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f8ad47af301db548edf60f605bbce60795a4ead6, data reload: false

query1	0.03	0.03	0.03
query2	0.10	0.05	0.05
query3	0.23	0.05	0.06
query4	1.64	0.08	0.08
query5	0.51	0.52	0.51
query6	1.13	0.75	0.75
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.55	0.52	0.50
query10	0.56	0.55	0.57
query11	0.16	0.11	0.12
query12	0.16	0.13	0.12
query13	0.61	0.60	0.59
query14	0.77	0.79	0.81
query15	0.85	0.84	0.83
query16	0.38	0.37	0.38
query17	1.05	1.06	1.07
query18	0.19	0.18	0.20
query19	1.99	1.88	1.87
query20	0.02	0.01	0.02
query21	15.42	0.67	0.66
query22	4.31	6.79	2.22
query23	18.27	1.34	1.21
query24	2.20	0.23	0.23
query25	0.16	0.09	0.08
query26	0.27	0.17	0.19
query27	0.08	0.09	0.07
query28	13.29	0.60	0.59
query29	12.70	3.39	3.37
query30	0.25	0.05	0.05
query31	2.85	0.41	0.39
query32	3.25	0.50	0.48
query33	2.97	3.03	3.03
query34	16.94	4.49	4.59
query35	4.61	4.61	4.64
query36	0.66	0.51	0.47
query37	0.19	0.16	0.17
query38	0.17	0.16	0.15
query39	0.06	0.04	0.04
query40	0.16	0.13	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 110.02 s
Total hot run time: 31.25 s

dataroaring
dataroaring approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 7247a23 into branch-3.0 Jun 11, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-51129-branch-3.0 branch June 11, 2025 03:00
dataroaring pushed a commit that referenced this pull request Jun 18, 2025
@github-actions @liaoxin01
branch-3.0: [fix](cloud) potential data race when retrying prepare/co…
7bc546c 
…mmit rowset for load #51129 (#51483)

Cherry-picked from #51129

Co-authored-by: Xin Liao <liaoxin@selectdb.com>
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @hello-stephen @dataroaring @liaoxin01