Skip to content

branch-3.1: [Opt](cloud-mow) Retry to commit txn when encounter stale calc delete bitmap response regardless of status code #52547 #52847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 8, 2025
Merged

branch-3.1: [Opt](cloud-mow) Retry to commit txn when encounter stale calc delete bitmap response regardless of status code #52547 #52847

morrySnow merged 1 commit into from
Jul 8, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Jul 7, 2025 •
edited by morrySnow
Loading

pick #52547

@bobhan1 bobhan1 requested a review from morrySnow as a code owner July 7, 2025 02:43
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 7, 2025

run buildall

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 7, 2025

run feut

@bobhan1
[Opt](cloud-mow) Retry to commit txn when encounter stale calc delete…
2ac2cf0 
… bitmap response regardless of status code (apache#52547)

apache#49710 add a check in MS to forbid
stale calc delete bitmap task to wrongly update delete bitmaps in MS.
But this may lead to load fail due to the check on FE.
This PR let FE retry to commit the txn when encounter stale calc delete
bitmap response regardless of task's status code to avoid the problem.
@bobhan1 bobhan1 force-pushed the branch-3.1-pick-52547 branch from b644a31 to 2ac2cf0 Compare July 8, 2025 02:09
@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 8, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-H: Total hot run time: 41977 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ac2cf0ed7773c25ad93d3e93e5cbcd7486a3790, data reload: false

------ Round 1 ----------------------------------
q1	17574	7658	7558	7558
q2	2097	175	187	175
q3	10513	1208	1220	1208
q4	10207	838	755	755
q5	7778	3068	2977	2977
q6	250	137	135	135
q7	1089	635	613	613
q8	9356	2087	2109	2087
q9	7061	6858	6833	6833
q10	7062	2327	2336	2327
q11	480	266	270	266
q12	550	221	218	218
q13	17809	3010	3001	3001
q14	264	215	214	214
q15	524	482	470	470
q16	510	374	386	374
q17	1026	682	574	574
q18	7357	6723	6653	6653
q19	1351	1222	1037	1037
q20	488	203	197	197
q21	4116	3537	3358	3358
q22	1118	947	974	947
Total cold run time: 108580 ms
Total hot run time: 41977 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7471	7379	7473	7379
q2	368	247	234	234
q3	3275	3085	3132	3085
q4	2135	1883	1829	1829
q5	5824	5950	5973	5950
q6	226	131	127	127
q7	2353	1752	1830	1752
q8	3624	3718	3790	3718
q9	9008	9048	8987	8987
q10	3750	3659	3715	3659
q11	626	510	496	496
q12	848	677	616	616
q13	8982	3168	3176	3168
q14	307	274	292	274
q15	529	469	485	469
q16	520	442	454	442
q17	2026	1746	1725	1725
q18	8321	7794	7714	7714
q19	1850	1585	1592	1585
q20	2179	1869	1905	1869
q21	5446	5177	5296	5177
q22	1131	997	1004	997
Total cold run time: 70799 ms
Total hot run time: 61252 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-DS: Total hot run time: 197034 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ac2cf0ed7773c25ad93d3e93e5cbcd7486a3790, data reload: false

query1	1288	939	907	907
query2	6166	1894	1856	1856
query3	10828	4315	4235	4235
query4	33067	23561	23593	23561
query5	3351	465	484	465
query6	284	186	191	186
query7	3992	317	325	317
query8	303	229	232	229
query9	9245	2605	2595	2595
query10	467	264	270	264
query11	17861	15685	15181	15181
query12	165	100	105	100
query13	1558	428	425	425
query14	10028	7374	7442	7374
query15	262	184	205	184
query16	7999	483	489	483
query17	1656	579	581	579
query18	2120	309	301	301
query19	225	158	161	158
query20	126	117	124	117
query21	210	111	111	111
query22	4651	4616	4445	4445
query23	35919	34218	33821	33821
query24	11919	2966	2921	2921
query25	681	425	410	410
query26	1727	172	175	172
query27	2806	351	358	351
query28	7994	2183	2193	2183
query29	1046	490	470	470
query30	274	161	161	161
query31	1044	839	825	825
query32	98	58	63	58
query33	796	327	314	314
query34	1219	522	528	522
query35	859	757	734	734
query36	1092	961	987	961
query37	265	71	87	71
query38	3967	4008	3951	3951
query39	1505	1501	1453	1453
query40	259	103	99	99
query41	49	47	49	47
query42	113	103	103	103
query43	504	479	504	479
query44	1306	851	820	820
query45	192	177	177	177
query46	1181	728	748	728
query47	2031	1875	1918	1875
query48	449	346	346	346
query49	1169	403	388	388
query50	857	431	433	431
query51	7493	7218	7109	7109
query52	105	95	88	88
query53	257	179	182	179
query54	1334	477	497	477
query55	81	79	81	79
query56	299	256	258	256
query57	1336	1192	1195	1192
query58	260	215	236	215
query59	3252	3183	2837	2837
query60	297	270	264	264
query61	112	107	119	107
query62	829	668	667	667
query63	215	185	190	185
query64	5149	741	719	719
query65	3319	3217	3200	3200
query66	1168	313	323	313
query67	15974	15513	15821	15513
query68	4957	591	579	579
query69	418	265	274	265
query70	1163	1110	1092	1092
query71	326	283	271	271
query72	6391	4180	4100	4100
query73	757	351	367	351
query74	10255	9089	9243	9089
query75	3379	2622	2640	2622
query76	2591	1049	1072	1049
query77	404	285	269	269
query78	10649	9659	9543	9543
query79	2299	611	635	611
query80	992	438	439	438
query81	548	225	228	225
query82	619	94	84	84
query83	230	139	143	139
query84	235	84	77	77
query85	1872	307	298	298
query86	498	296	289	289
query87	4386	4276	4280	4276
query88	4106	2400	2375	2375
query89	423	295	286	286
query90	1986	198	190	190
query91	142	117	110	110
query92	66	51	51	51
query93	2452	553	562	553
query94	757	288	307	288
query95	364	254	254	254
query96	622	290	281	281
query97	3312	3144	3153	3144
query98	230	215	204	204
query99	1477	1299	1287	1287
Total cold run time: 307559 ms
Total hot run time: 197034 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

ClickBench: Total hot run time: 30.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ac2cf0ed7773c25ad93d3e93e5cbcd7486a3790, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.02
query3	0.23	0.06	0.06
query4	1.63	0.10	0.10
query5	0.52	0.50	0.52
query6	1.14	0.75	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.50
query10	0.56	0.57	0.56
query11	0.14	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.61	0.60
query14	0.80	0.80	0.79
query15	0.85	0.84	0.82
query16	0.40	0.37	0.37
query17	1.03	1.02	0.97
query18	0.23	0.22	0.23
query19	1.89	1.73	1.79
query20	0.02	0.01	0.01
query21	15.40	0.61	0.59
query22	2.27	2.39	1.76
query23	16.88	0.88	0.87
query24	2.84	0.93	2.29
query25	0.22	0.11	0.12
query26	0.62	0.15	0.13
query27	0.04	0.04	0.04
query28	9.40	0.48	0.46
query29	12.66	3.27	3.22
query30	0.24	0.06	0.06
query31	2.88	0.38	0.39
query32	3.25	0.46	0.46
query33	2.94	3.03	3.06
query34	16.81	4.52	4.52
query35	4.51	4.49	4.54
query36	0.69	0.48	0.48
query37	0.08	0.07	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.04 s
Total hot run time: 30.1 s

@morrySnow morrySnow changed the title branch-3.1: [Opt](cloud-mow) Retry to commit txn when encounter stale calc delete bitmap response regardless of status code (#52547) branch-3.1: [Opt](cloud-mow) Retry to commit txn when encounter stale calc delete bitmap response regardless of status code #52547 Jul 8, 2025
@morrySnow morrySnow merged commit 35e31d7 into apache:branch-3.1 Jul 8, 2025
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bobhan1 @hello-stephen @doris-robot @morrySnow