Skip to content

branch-3.0: [Fix](cloud-mow) Fix FE's wrong handling when low version MS don't set tablet states for GetDeleteBitmapUpdateLockResponse #49165 #49187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Mar 18, 2025
Merged

branch-3.0: [Fix](cloud-mow) Fix FE's wrong handling when low version MS don't set tablet states for GetDeleteBitmapUpdateLockResponse #49165 #49187

dataroaring merged 1 commit into from
Mar 18, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Mar 18, 2025

Cherry-picked from #49165

@bobhan1
[Fix](cloud-mow) Fix FE's wrong handling when low version MS don't se…
4228f2d 
…t tablet states for `GetDeleteBitmapUpdateLockResponse` (#49165)

### What problem does this PR solve?

fix for #48400, when fe send
`GetDeleteBitmapUpdateLock` rpc to low version MS which will not set
tablet states field and get response from it, FE will encounter
`IndexOutOfBoundsException`.
```
2025-03-17 18:05:35,224 WARN (thrift-server-pool-77|200) [FrontendServiceImpl.loadTxnCommit():1676] catch unknown result.
java.lang.IndexOutOfBoundsException: Index:0, Size:0
        at com.google.protobuf.LongArrayList.ensureIndexInRange(LongArrayList.java:288) ~[protobuf-java-3.24.3.jar:?]
        at com.google.protobuf.LongArrayList.getLong(LongArrayList.java:136) ~[protobuf-java-3.24.3.jar:?]
        at com.google.protobuf.LongArrayList.get(LongArrayList.java:131) ~[protobuf-java-3.24.3.jar:?]
        at com.google.protobuf.LongArrayList.get(LongArrayList.java:45) ~[protobuf-java-3.24.3.jar:?]
        at org.apache.doris.cloud.transaction.CloudGlobalTransactionMgr.getDeleteBitmapUpdateLock(CloudGlobalTransactionMgr.java:949) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.cloud.transaction.CloudGlobalTransactionMgr.commitTransaction(CloudGlobalTransactionMgr.java:361) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.cloud.transaction.CloudGlobalTransactionMgr.commitAndPublishTransaction(CloudGlobalTransactionMgr.java:1203) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.service.FrontendServiceImpl.loadTxnCommitImpl(FrontendServiceImpl.java:1730) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.service.FrontendServiceImpl.loadTxnCommit(FrontendServiceImpl.java:1660) ~[doris-fe.jar:1.2-SNAPSHOT]
        at jdk.internal.reflect.GeneratedMethodAccessor121.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT]
        at jdk.proxy2.$Proxy45.loadTxnCommit(Unknown Source) ~[?:?]
        at org.apache.doris.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:4282) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.doris.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:4262) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?] 
```
@github-actions github-actions bot requested a review from dataroaring as a code owner March 18, 2025 05:38
@Thearas
Copy link
Contributor

Thearas commented Mar 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Mar 18, 2025
@Thearas
Copy link
Contributor

Thearas commented Mar 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

TPC-H: Total hot run time: 40069 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4228f2df3a74f3fca28cf75d8632a9ab8e6e7ce8, data reload: false

------ Round 1 ----------------------------------
q1	17577	6760	6583	6583
q2	2059	178	168	168
q3	10615	1051	1200	1051
q4	10573	706	831	706
q5	7728	2868	2785	2785
q6	220	134	131	131
q7	960	614	595	595
q8	9579	1938	2033	1938
q9	7501	6387	6398	6387
q10	6997	2271	2228	2228
q11	455	256	269	256
q12	398	219	212	212
q13	17766	3023	3009	3009
q14	230	201	221	201
q15	519	460	461	460
q16	646	581	586	581
q17	969	653	578	578
q18	7236	6765	7061	6765
q19	1401	1134	1057	1057
q20	486	215	204	204
q21	4043	3183	3304	3183
q22	1098	1000	991	991
Total cold run time: 109056 ms
Total hot run time: 40069 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6583	6546	6571	6546
q2	328	237	230	230
q3	2898	2833	2941	2833
q4	2031	1837	1841	1837
q5	5712	5739	5730	5730
q6	208	125	129	125
q7	2243	1823	1850	1823
q8	3344	3562	3509	3509
q9	8830	8854	8826	8826
q10	3599	3537	3517	3517
q11	590	493	480	480
q12	844	608	615	608
q13	8527	3144	3211	3144
q14	291	277	294	277
q15	521	469	462	462
q16	669	658	645	645
q17	1852	1611	1593	1593
q18	8179	7772	7583	7583
q19	1664	1610	1485	1485
q20	2114	1888	1838	1838
q21	5543	5453	5345	5345
q22	1130	1073	1036	1036
Total cold run time: 67700 ms
Total hot run time: 59472 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

TPC-DS: Total hot run time: 196730 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4228f2df3a74f3fca28cf75d8632a9ab8e6e7ce8, data reload: false

query1	1284	931	934	931
query2	6225	2021	2001	2001
query3	10830	4280	4218	4218
query4	65115	28713	23242	23242
query5	5034	443	445	443
query6	408	167	161	161
query7	5573	301	313	301
query8	295	209	214	209
query9	8725	2606	2600	2600
query10	475	273	257	257
query11	17511	15447	15890	15447
query12	158	110	103	103
query13	1498	447	444	444
query14	11010	7681	6837	6837
query15	208	188	188	188
query16	7175	497	512	497
query17	1097	599	599	599
query18	2032	296	308	296
query19	229	153	161	153
query20	121	110	111	110
query21	211	102	111	102
query22	4685	4720	4551	4551
query23	34385	34088	34024	34024
query24	6063	2951	2869	2869
query25	515	400	412	400
query26	673	171	164	164
query27	2043	351	356	351
query28	4346	2464	2434	2434
query29	702	464	461	461
query30	242	161	172	161
query31	965	787	839	787
query32	77	59	59	59
query33	427	314	313	313
query34	938	528	517	517
query35	845	760	739	739
query36	1108	952	965	952
query37	123	73	77	73
query38	4113	4173	4039	4039
query39	1521	1483	1466	1466
query40	219	102	100	100
query41	49	49	55	49
query42	118	101	101	101
query43	527	495	483	483
query44	1167	821	844	821
query45	183	174	173	173
query46	1185	763	734	734
query47	2032	1931	1936	1931
query48	469	401	413	401
query49	759	409	421	409
query50	850	427	425	425
query51	7410	7090	7176	7090
query52	103	89	92	89
query53	262	184	181	181
query54	550	462	462	462
query55	76	80	78	78
query56	274	253	258	253
query57	1239	1145	1090	1090
query58	220	217	214	214
query59	3190	3055	2928	2928
query60	279	263	263	263
query61	133	147	105	105
query62	750	661	670	661
query63	218	192	202	192
query64	1351	675	676	675
query65	3241	3188	3235	3188
query66	701	304	300	300
query67	15944	15657	15449	15449
query68	4083	572	566	566
query69	426	272	259	259
query70	1158	1020	1118	1020
query71	371	251	253	251
query72	6390	3966	4029	3966
query73	754	349	348	348
query74	10402	9246	9011	9011
query75	3373	2622	2653	2622
query76	1983	1031	1152	1031
query77	544	267	273	267
query78	10511	9606	9580	9580
query79	1614	599	606	599
query80	865	434	414	414
query81	523	255	243	243
query82	1301	90	84	84
query83	162	147	140	140
query84	291	78	77	77
query85	885	301	297	297
query86	335	303	283	283
query87	4495	4310	4315	4310
query88	3819	2382	2330	2330
query89	411	286	302	286
query90	1994	189	185	185
query91	181	166	147	147
query92	70	48	49	48
query93	1928	566	558	558
query94	769	285	271	271
query95	356	260	260	260
query96	612	283	288	283
query97	3339	3183	3166	3166
query98	220	205	201	201
query99	1580	1283	1288	1283
Total cold run time: 318229 ms
Total hot run time: 196730 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

ClickBench: Total hot run time: 32.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4228f2df3a74f3fca28cf75d8632a9ab8e6e7ce8, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.02	0.03
query3	0.24	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.50	0.52
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.03
query9	0.55	0.49	0.49
query10	0.54	0.54	0.56
query11	0.14	0.09	0.10
query12	0.15	0.11	0.10
query13	0.60	0.61	0.60
query14	2.73	2.85	2.80
query15	0.88	0.83	0.84
query16	0.40	0.38	0.38
query17	1.06	1.08	1.03
query18	0.23	0.23	0.21
query19	1.97	1.81	1.99
query20	0.01	0.01	0.02
query21	15.36	0.61	0.59
query22	2.25	2.69	2.02
query23	16.97	1.02	0.82
query24	3.60	0.91	1.80
query25	0.17	0.09	0.16
query26	0.50	0.14	0.13
query27	0.05	0.05	0.04
query28	9.40	0.51	0.51
query29	12.62	3.26	3.25
query30	0.24	0.05	0.06
query31	2.88	0.40	0.40
query32	3.23	0.46	0.45
query33	2.97	3.02	2.99
query34	17.06	4.46	4.49
query35	4.46	4.49	4.53
query36	0.65	0.49	0.48
query37	0.09	0.06	0.05
query38	0.06	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.88 s
Total hot run time: 32.33 s

dataroaring
dataroaring approved these changes Mar 18, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit cf1406e into branch-3.0 Mar 18, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-49165-branch-3.0 branch March 18, 2025 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @bobhan1