[fix](replica) skip missing version should care catchup #49999

dataroaring · 2025-04-12T08:59:01Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-04-12T08:59:06Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

dataroaring · 2025-04-12T08:59:09Z

run buildall

Copilot

Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (1)

fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java:308

The removal of the dedicated check for replicas with a failed version (lastFailedVersion > 0) might alter the intended behavior. Please verify that the new condition combined with allowMissingVersion accurately handles both missing and failed versions.

boolean allowMissingVersion) {

doris-robot · 2025-04-12T09:37:14Z

TPC-H: Total hot run time: 34806 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1d980d4a82bba40b41041e27d4b65526b3e50f51, data reload: false

------ Round 1 ----------------------------------
q1	26160	5114	5085	5085
q2	2070	270	185	185
q3	10390	1225	678	678
q4	10219	1011	517	517
q5	7520	2353	2341	2341
q6	187	161	133	133
q7	1002	735	608	608
q8	9319	1290	1127	1127
q9	6703	5067	5181	5067
q10	6871	2296	1881	1881
q11	481	276	269	269
q12	360	357	225	225
q13	17764	3641	3119	3119
q14	222	219	221	219
q15	536	475	479	475
q16	631	635	597	597
q17	594	849	346	346
q18	7425	7086	7061	7061
q19	1211	961	586	586
q20	346	334	221	221
q21	4031	3391	3104	3104
q22	1043	999	962	962
Total cold run time: 115085 ms
Total hot run time: 34806 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5150	5147	5472	5147
q2	239	326	225	225
q3	2171	2621	2279	2279
q4	1415	1792	1476	1476
q5	4512	4400	4558	4400
q6	225	179	139	139
q7	2066	1984	1789	1789
q8	2643	2632	2534	2534
q9	7466	7295	7252	7252
q10	3017	3144	2763	2763
q11	575	514	493	493
q12	702	770	636	636
q13	3494	3887	3343	3343
q14	280	290	284	284
q15	521	489	475	475
q16	665	669	661	661
q17	1164	1499	1439	1439
q18	7674	7655	7332	7332
q19	827	828	947	828
q20	1969	1965	1833	1833
q21	5332	4902	4869	4869
q22	1125	1033	1047	1033
Total cold run time: 53232 ms
Total hot run time: 51230 ms

doris-robot · 2025-04-12T09:49:29Z

TPC-DS: Total hot run time: 192761 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1d980d4a82bba40b41041e27d4b65526b3e50f51, data reload: false

query1	1424	1128	1067	1067
query2	6288	1871	1920	1871
query3	11033	4344	4270	4270
query4	53819	25756	23166	23166
query5	5220	507	472	472
query6	357	203	182	182
query7	4973	506	285	285
query8	321	261	254	254
query9	6151	2570	2578	2570
query10	437	333	287	287
query11	15572	15095	14863	14863
query12	154	112	108	108
query13	1081	492	373	373
query14	10140	6554	6442	6442
query15	209	183	184	183
query16	7089	686	555	555
query17	1090	714	573	573
query18	1551	396	316	316
query19	204	203	169	169
query20	126	123	130	123
query21	208	124	102	102
query22	4308	4483	4458	4458
query23	34294	33576	33317	33317
query24	6607	2435	2404	2404
query25	491	479	423	423
query26	684	286	166	166
query27	2198	508	355	355
query28	2970	2453	2423	2423
query29	620	582	451	451
query30	280	229	191	191
query31	883	908	810	810
query32	77	68	63	63
query33	451	388	301	301
query34	786	866	535	535
query35	810	825	781	781
query36	959	1014	914	914
query37	118	103	75	75
query38	4188	4283	4168	4168
query39	1553	1438	1427	1427
query40	215	123	111	111
query41	53	51	51	51
query42	123	108	119	108
query43	492	504	478	478
query44	1331	836	812	812
query45	187	172	171	171
query46	843	1047	643	643
query47	1823	1878	1808	1808
query48	396	427	303	303
query49	678	527	439	439
query50	665	711	404	404
query51	4293	4304	4311	4304
query52	107	108	98	98
query53	227	267	201	201
query54	595	634	539	539
query55	87	88	92	88
query56	320	307	313	307
query57	1216	1237	1128	1128
query58	263	264	272	264
query59	2680	2754	2653	2653
query60	356	319	316	316
query61	130	125	134	125
query62	718	717	724	717
query63	240	196	190	190
query64	1485	1067	699	699
query65	4425	4223	4232	4223
query66	778	401	295	295
query67	15922	15613	15226	15226
query68	8112	882	508	508
query69	536	308	265	265
query70	1203	1050	1040	1040
query71	483	307	288	288
query72	5692	4795	5005	4795
query73	1357	693	342	342
query74	9027	9327	8785	8785
query75	3871	3202	2668	2668
query76	4267	1323	730	730
query77	610	371	284	284
query78	10148	10200	9282	9282
query79	2137	827	581	581
query80	646	522	449	449
query81	492	257	227	227
query82	373	131	100	100
query83	356	257	235	235
query84	288	112	79	79
query85	754	364	319	319
query86	344	304	311	304
query87	4603	4453	4362	4362
query88	2831	2211	2201	2201
query89	397	313	280	280
query90	2048	211	207	207
query91	138	143	113	113
query92	78	60	58	58
query93	1438	951	577	577
query94	710	406	314	314
query95	369	293	287	287
query96	487	562	276	276
query97	3174	3276	3106	3106
query98	231	212	200	200
query99	1446	1397	1305	1305
Total cold run time: 299018 ms
Total hot run time: 192761 ms

doris-robot · 2025-04-12T09:54:48Z

ClickBench: Total hot run time: 31.28 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1d980d4a82bba40b41041e27d4b65526b3e50f51, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.11
query3	0.25	0.20	0.18
query4	1.59	0.20	0.18
query5	0.62	0.61	0.60
query6	1.18	0.71	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.53	0.51
query10	0.58	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.60	0.60	0.59
query14	2.83	2.73	2.70
query15	0.93	0.87	0.85
query16	0.37	0.39	0.39
query17	1.02	1.01	1.02
query18	0.21	0.20	0.19
query19	1.93	1.94	1.80
query20	0.02	0.01	0.01
query21	15.35	0.93	0.53
query22	0.75	1.27	0.65
query23	14.86	1.42	0.62
query24	7.19	1.43	0.96
query25	0.50	0.24	0.16
query26	0.63	0.15	0.13
query27	0.06	0.06	0.05
query28	9.50	0.85	0.44
query29	12.58	4.07	3.30
query30	0.25	0.08	0.06
query31	2.84	0.60	0.38
query32	3.23	0.56	0.48
query33	3.05	3.06	3.11
query34	15.81	5.09	4.47
query35	4.53	4.50	4.48
query36	0.66	0.50	0.50
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.02	0.02	0.02
query40	0.17	0.15	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.51 s
Total hot run time: 31.28 s

github-actions · 2025-04-12T11:06:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-12T11:06:49Z

PR approved by anyone and no changes requested.

yujun777

LGTM

yujun777 · 2025-04-14T01:44:01Z

fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java

                continue;
            }

-            // Skip the missing version replica


if not allow missing version, must check last fail version < 0, line 318 ~ 321 cannot deleted, keep it

yujun777

need update

yujun777 · 2025-04-14T01:49:35Z

fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java

-                continue;
-            }
-
-            if (!replica.checkVersionCatchUp(visibleVersion, false)) {


if be not found rowset [1, visible version], be maybe also throw a error

yujun777

LGTM

dataroaring · 2025-04-16T02:18:26Z

run buildall

github-actions · 2025-04-16T02:40:32Z

PR approved by at least one committer and no changes requested.

doris-robot · 2025-04-16T03:26:17Z

TPC-H: Total hot run time: 34599 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46efe5cdb68f90d79c103674d81961c951038b81, data reload: false

------ Round 1 ----------------------------------
q1	26004	5112	5089	5089
q2	2068	277	177	177
q3	10400	1250	694	694
q4	10209	1014	547	547
q5	7986	2602	2403	2403
q6	182	163	137	137
q7	936	768	606	606
q8	9303	1294	1006	1006
q9	6784	5094	5170	5094
q10	6866	2322	1885	1885
q11	483	297	275	275
q12	364	357	228	228
q13	17793	3673	3071	3071
q14	224	226	216	216
q15	536	497	498	497
q16	440	449	406	406
q17	599	866	365	365
q18	7751	7152	7057	7057
q19	1593	957	565	565
q20	332	339	230	230
q21	4020	3347	3111	3111
q22	1072	1026	940	940
Total cold run time: 115945 ms
Total hot run time: 34599 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5126	5062	5075	5062
q2	239	329	230	230
q3	2107	2658	2315	2315
q4	1448	1801	1519	1519
q5	4474	4422	4392	4392
q6	213	170	134	134
q7	1930	1944	1747	1747
q8	2589	2692	2562	2562
q9	7296	7261	7026	7026
q10	2987	3196	2740	2740
q11	584	513	496	496
q12	668	779	601	601
q13	3470	3884	3355	3355
q14	278	316	263	263
q15	520	474	476	474
q16	464	510	464	464
q17	1170	1524	1385	1385
q18	7821	7552	7470	7470
q19	835	869	959	869
q20	1925	1948	1801	1801
q21	5149	4880	4779	4779
q22	1123	1057	1033	1033
Total cold run time: 52416 ms
Total hot run time: 50717 ms

doris-robot · 2025-04-16T03:38:35Z

TPC-DS: Total hot run time: 193276 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46efe5cdb68f90d79c103674d81961c951038b81, data reload: false

query1	1439	1082	1069	1069
query2	6080	1848	1841	1841
query3	11012	4605	4388	4388
query4	57378	26125	23628	23628
query5	4978	449	447	447
query6	346	189	184	184
query7	4880	493	287	287
query8	321	268	243	243
query9	5735	2586	2581	2581
query10	437	314	271	271
query11	15126	15016	14919	14919
query12	158	107	108	107
query13	1053	521	389	389
query14	10213	6465	6209	6209
query15	205	190	175	175
query16	7112	653	519	519
query17	1065	741	565	565
query18	1563	411	312	312
query19	194	191	165	165
query20	132	124	123	123
query21	206	125	111	111
query22	4342	4658	4359	4359
query23	34109	33640	33663	33640
query24	6529	2450	2407	2407
query25	458	464	402	402
query26	695	273	150	150
query27	2330	524	342	342
query28	3015	2161	2140	2140
query29	573	562	424	424
query30	275	220	199	199
query31	851	854	797	797
query32	78	68	69	68
query33	514	374	317	317
query34	758	862	532	532
query35	813	848	749	749
query36	957	1027	916	916
query37	115	98	72	72
query38	4226	4309	4190	4190
query39	1550	1426	1434	1426
query40	226	126	112	112
query41	54	56	51	51
query42	130	115	104	104
query43	498	521	496	496
query44	1336	814	837	814
query45	181	176	165	165
query46	845	1036	635	635
query47	1865	1885	1772	1772
query48	383	426	309	309
query49	674	500	447	447
query50	663	685	412	412
query51	4263	4314	4211	4211
query52	114	114	108	108
query53	247	258	188	188
query54	591	566	530	530
query55	87	93	88	88
query56	333	311	284	284
query57	1166	1210	1101	1101
query58	270	251	261	251
query59	2709	2694	2664	2664
query60	353	339	333	333
query61	156	151	161	151
query62	742	728	672	672
query63	233	198	211	198
query64	1609	1057	690	690
query65	4403	4242	4217	4217
query66	713	401	312	312
query67	15829	15484	15340	15340
query68	8014	823	510	510
query69	526	311	263	263
query70	1190	1117	1080	1080
query71	490	308	297	297
query72	5756	4799	4808	4799
query73	1480	626	340	340
query74	9282	9040	9077	9040
query75	3846	3176	2712	2712
query76	4169	1211	766	766
query77	608	340	284	284
query78	9928	10315	9334	9334
query79	1986	806	568	568
query80	637	524	453	453
query81	466	249	215	215
query82	448	125	98	98
query83	256	257	232	232
query84	286	108	83	83
query85	805	352	308	308
query86	393	309	308	308
query87	4453	4505	4317	4317
query88	3644	2203	2185	2185
query89	403	308	288	288
query90	1902	200	212	200
query91	195	158	111	111
query92	74	63	55	55
query93	1289	942	598	598
query94	655	404	309	309
query95	363	292	285	285
query96	484	575	274	274
query97	3168	3231	3162	3162
query98	230	206	204	204
query99	1656	1395	1280	1280
Total cold run time: 301576 ms
Total hot run time: 193276 ms

doris-robot · 2025-04-16T03:43:45Z

ClickBench: Total hot run time: 29.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46efe5cdb68f90d79c103674d81961c951038b81, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.11	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.20
query5	0.60	0.59	0.57
query6	1.16	0.71	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.58	0.53	0.52
query10	0.58	0.57	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	1.18	1.19	1.17
query15	0.87	0.87	0.84
query16	0.39	0.38	0.39
query17	1.04	1.04	1.02
query18	0.21	0.19	0.20
query19	1.85	1.78	1.78
query20	0.02	0.01	0.01
query21	15.40	0.95	0.56
query22	0.77	1.12	0.63
query23	15.04	1.38	0.62
query24	7.51	0.76	0.39
query25	0.38	0.17	0.12
query26	0.63	0.16	0.13
query27	0.05	0.05	0.05
query28	9.25	0.89	0.43
query29	12.59	3.93	3.31
query30	0.25	0.09	0.06
query31	2.82	0.56	0.38
query32	3.23	0.54	0.47
query33	3.05	3.10	3.08
query34	15.64	5.11	4.46
query35	4.53	4.51	4.47
query36	0.65	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.14
query41	0.07	0.03	0.03
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.72 s
Total hot run time: 29.1 s

…49999 (#50087) Cherry-picked from #49999 Co-authored-by: Yongqiang YANG <yangyongqiang@selectdb.com>

) (#52386) pick (#49999) (#50094)

[fix](replica) skip missing version should care catchup

1d980d4

dataroaring added dev/3.0.x dev/2.1.x labels Apr 12, 2025

dataroaring requested a review from Copilot April 12, 2025 09:04

Copilot AI reviewed Apr 12, 2025

View reviewed changes

gavinchou previously approved these changes Apr 12, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 12, 2025

github-actions bot added the reviewed label Apr 12, 2025

yujun777 approved these changes Apr 14, 2025

View reviewed changes

yujun777 reviewed Apr 14, 2025

View reviewed changes

yujun777 suggested changes Apr 14, 2025

View reviewed changes

yujun777 reviewed Apr 14, 2025

View reviewed changes

resolve comment

46efe5c

dataroaring dismissed gavinchou’s stale review via 46efe5c April 14, 2025 05:55

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 14, 2025

yujun777 approved these changes Apr 14, 2025

View reviewed changes

gavinchou approved these changes Apr 16, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 16, 2025

gavinchou merged commit bb4bacd into apache:master Apr 16, 2025
26 of 27 checks passed

github-actions bot pushed a commit that referenced this pull request Apr 16, 2025

[fix](replica) skip missing version should care catchup (#49999)

525a702

github-actions bot mentioned this pull request Apr 16, 2025

branch-3.0: [fix](replica) skip missing version should care catchup #49999 #50087

Merged

github-actions bot added the dev/2.1.x-conflict label Apr 16, 2025

seawinde pushed a commit to seawinde/doris that referenced this pull request Apr 17, 2025

[fix](replica) skip missing version should care catchup (apache#49999)

695bbf4

dataroaring added a commit that referenced this pull request Apr 28, 2025

branch-3.0: [fix](replica) skip missing version should care catchup #…

23623e7

…49999 (#50087) Cherry-picked from #49999 Co-authored-by: Yongqiang YANG <yangyongqiang@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels Apr 28, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[fix](replica) skip missing version should care catchup (apache#49999)

c862e4d

dataroaring mentioned this pull request Jun 26, 2025

[fix](replica) skip missing version should care catchup (#49999) (#50094) #52386

Merged

dataroaring added a commit to dataroaring/incubator-doris that referenced this pull request Jun 26, 2025

[fix](replica) skip missing version should care catchup (apache#49999)

15762e1

yiguolei pushed a commit to dataroaring/incubator-doris that referenced this pull request Jun 28, 2025

[fix](replica) skip missing version should care catchup (apache#49999)

1c518c5

yiguolei pushed a commit that referenced this pull request Jul 3, 2025

[fix](replica) skip missing version should care catchup (#49999) (#50094

d374b6e

) (#52386) pick (#49999) (#50094)

yiguolei added dev/2.1.11-merged and removed dev/2.1.x dev/2.1.x-conflict labels Jul 3, 2025

dataroaring mentioned this pull request Sep 9, 2025

[Bug] Doris BE 宕机后数据无法导入和查询 #55754

Closed

3 tasks

[fix](replica) skip missing version should care catchup #49999

[fix](replica) skip missing version should care catchup #49999

Uh oh!

Conversation

dataroaring commented Apr 12, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 12, 2025

Uh oh!

dataroaring commented Apr 12, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Apr 12, 2025

Uh oh!

doris-robot commented Apr 12, 2025

Uh oh!

doris-robot commented Apr 12, 2025

Uh oh!

github-actions bot commented Apr 12, 2025

Uh oh!

github-actions bot commented Apr 12, 2025

Uh oh!

yujun777 left a comment

Choose a reason for hiding this comment

Uh oh!

yujun777 Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yujun777 left a comment

Choose a reason for hiding this comment

Uh oh!

yujun777 Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

yujun777 left a comment

Choose a reason for hiding this comment

Uh oh!

dataroaring commented Apr 16, 2025

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

doris-robot commented Apr 16, 2025

Uh oh!

doris-robot commented Apr 16, 2025

Uh oh!

doris-robot commented Apr 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yujun777 Apr 14, 2025 •

edited

Loading