[fix](load) fix routine load job progress fallback after FE master node restart #50221

sollhui · 2025-04-21T06:07:25Z

What problem does this PR solve?

If user creates routine load job by specifying partitions:

FROM KAFKA
(
       "kafka_partitions" = "0",
       "kafka_offsets" = "XXX"
 );

job progress fallback after FE master node restart in share-storage mode.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-04-21T06:07:30Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

sollhui · 2025-04-21T06:08:53Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java

sollhui · 2025-04-21T06:37:18Z

run buildall

doris-robot · 2025-04-21T07:38:44Z

TPC-H: Total hot run time: 33895 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3149780eee2c25f9ec3717151b1a5523e12d902b, data reload: false

------ Round 1 ----------------------------------
q1	26181	5106	5013	5013
q2	2085	287	181	181
q3	10489	1292	687	687
q4	10232	1017	527	527
q5	8266	2466	2300	2300
q6	182	160	130	130
q7	913	740	602	602
q8	9316	1363	1109	1109
q9	6824	5112	5083	5083
q10	6831	2316	1906	1906
q11	461	283	278	278
q12	342	355	216	216
q13	17769	3701	3056	3056
q14	223	235	209	209
q15	540	499	493	493
q16	461	445	405	405
q17	598	934	368	368
q18	7752	7507	7169	7169
q19	1592	954	562	562
q20	334	324	217	217
q21	4226	2648	2443	2443
q22	1044	1010	941	941
Total cold run time: 116661 ms
Total hot run time: 33895 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5199	5078	5095	5078
q2	237	349	241	241
q3	2156	2657	2265	2265
q4	1452	1812	1525	1525
q5	4553	4421	4355	4355
q6	210	164	123	123
q7	1979	1879	1756	1756
q8	2588	2511	2522	2511
q9	7204	7201	7223	7201
q10	2975	3184	2730	2730
q11	563	491	500	491
q12	689	732	633	633
q13	3457	3963	3308	3308
q14	278	295	273	273
q15	528	479	470	470
q16	467	505	452	452
q17	1131	1573	1433	1433
q18	7596	7582	7565	7565
q19	815	819	859	819
q20	1983	1995	1861	1861
q21	5197	4706	4540	4540
q22	1052	1022	1014	1014
Total cold run time: 52309 ms
Total hot run time: 50644 ms

doris-robot · 2025-04-21T07:50:20Z

TPC-DS: Total hot run time: 186228 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3149780eee2c25f9ec3717151b1a5523e12d902b, data reload: false

query1	1002	489	503	489
query2	6558	1844	1812	1812
query3	6742	226	213	213
query4	26224	23368	23004	23004
query5	4334	642	454	454
query6	301	208	203	203
query7	4609	484	278	278
query8	278	238	227	227
query9	8610	2540	2566	2540
query10	475	324	262	262
query11	15667	15143	14895	14895
query12	164	109	108	108
query13	1666	550	406	406
query14	9450	6347	6359	6347
query15	218	193	169	169
query16	7226	647	505	505
query17	1192	720	599	599
query18	1995	409	313	313
query19	204	188	161	161
query20	128	118	118	118
query21	211	126	109	109
query22	4257	4228	4026	4026
query23	34120	33071	32967	32967
query24	8463	2415	2450	2415
query25	584	460	382	382
query26	1233	269	150	150
query27	2753	493	329	329
query28	4348	2116	2091	2091
query29	786	565	433	433
query30	282	216	187	187
query31	933	839	782	782
query32	73	69	62	62
query33	544	376	305	305
query34	803	873	500	500
query35	817	808	745	745
query36	936	970	891	891
query37	115	102	80	80
query38	4184	4147	4124	4124
query39	1455	1403	1408	1403
query40	211	122	104	104
query41	56	52	52	52
query42	123	106	105	105
query43	497	497	473	473
query44	1338	809	817	809
query45	185	171	162	162
query46	873	1033	633	633
query47	1739	1762	1701	1701
query48	380	414	291	291
query49	787	520	427	427
query50	668	679	402	402
query51	4134	4096	4031	4031
query52	110	113	97	97
query53	238	260	194	194
query54	576	586	512	512
query55	89	79	82	79
query56	346	283	285	283
query57	1160	1144	1074	1074
query58	271	249	257	249
query59	2641	2655	2579	2579
query60	312	330	299	299
query61	126	125	124	124
query62	806	715	651	651
query63	220	187	190	187
query64	4378	1017	674	674
query65	4329	4221	4267	4221
query66	1146	414	306	306
query67	15634	15637	15546	15546
query68	8320	891	512	512
query69	461	300	268	268
query70	1222	1139	1139	1139
query71	477	323	292	292
query72	5543	4728	4876	4728
query73	748	646	355	355
query74	8877	9180	8917	8917
query75	3872	3199	2721	2721
query76	3674	1193	837	837
query77	808	372	294	294
query78	9984	9991	9307	9307
query79	2915	819	573	573
query80	635	504	435	435
query81	492	253	225	225
query82	477	128	99	99
query83	286	257	248	248
query84	295	104	89	89
query85	815	365	313	313
query86	390	284	323	284
query87	4441	4488	4348	4348
query88	3381	2213	2186	2186
query89	402	312	282	282
query90	1864	217	208	208
query91	141	144	109	109
query92	83	62	57	57
query93	2180	945	589	589
query94	668	410	297	297
query95	379	290	288	288
query96	478	564	276	276
query97	3146	3268	3120	3120
query98	253	206	202	202
query99	1442	1382	1281	1281
Total cold run time: 276462 ms
Total hot run time: 186228 ms

doris-robot · 2025-04-21T07:55:35Z

ClickBench: Total hot run time: 29.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3149780eee2c25f9ec3717151b1a5523e12d902b, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.19	0.19
query4	1.59	0.20	0.10
query5	0.59	0.57	0.54
query6	1.19	0.71	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.51
query10	0.58	0.56	0.57
query11	0.16	0.10	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.60
query14	1.18	1.20	1.20
query15	0.87	0.83	0.85
query16	0.38	0.37	0.38
query17	1.03	1.03	1.08
query18	0.22	0.20	0.20
query19	1.88	1.80	1.85
query20	0.01	0.02	0.02
query21	15.40	0.92	0.56
query22	0.76	1.21	0.64
query23	14.91	1.38	0.65
query24	6.75	0.94	1.09
query25	0.51	0.14	0.14
query26	0.59	0.16	0.13
query27	0.05	0.05	0.06
query28	10.30	0.91	0.44
query29	12.58	3.92	3.27
query30	0.25	0.08	0.06
query31	2.83	0.59	0.37
query32	3.21	0.54	0.47
query33	3.08	3.10	3.07
query34	15.80	5.12	4.46
query35	4.53	4.57	4.46
query36	0.66	0.50	0.48
query37	0.08	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.16 s
Total hot run time: 29.56 s

liaoxin01

LGTM

github-actions · 2025-04-21T07:57:20Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-21T07:57:23Z

PR approved by anyone and no changes requested.

sollhui · 2025-04-21T09:31:54Z

run buildall

doris-robot · 2025-04-21T10:04:06Z

TPC-H: Total hot run time: 33697 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3149780eee2c25f9ec3717151b1a5523e12d902b, data reload: false

------ Round 1 ----------------------------------
q1	26287	5015	4977	4977
q2	2061	280	198	198
q3	10378	1224	699	699
q4	10216	969	523	523
q5	7513	2298	2366	2298
q6	180	164	133	133
q7	891	742	599	599
q8	9304	1293	1013	1013
q9	6700	5037	5049	5037
q10	6796	2299	1890	1890
q11	471	288	274	274
q12	335	352	219	219
q13	17762	3709	3101	3101
q14	220	232	208	208
q15	542	488	488	488
q16	439	443	404	404
q17	582	845	371	371
q18	7518	7226	7084	7084
q19	1565	937	553	553
q20	337	331	219	219
q21	4019	3444	2450	2450
q22	1041	997	959	959
Total cold run time: 115157 ms
Total hot run time: 33697 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5157	5105	5139	5105
q2	231	330	235	235
q3	2186	2664	2287	2287
q4	1398	1809	1405	1405
q5	4507	4476	4396	4396
q6	218	166	136	136
q7	2037	1902	1785	1785
q8	2623	2593	2521	2521
q9	7429	7573	7580	7573
q10	3047	3297	2843	2843
q11	636	523	531	523
q12	767	794	661	661
q13	3942	4263	3424	3424
q14	289	311	283	283
q15	521	485	483	483
q16	479	488	453	453
q17	1160	1592	1405	1405
q18	7780	7463	7581	7463
q19	778	806	849	806
q20	2041	2040	1888	1888
q21	5185	4811	4765	4765
q22	1057	1067	1024	1024
Total cold run time: 53468 ms
Total hot run time: 51464 ms

doris-robot · 2025-04-21T10:16:15Z

TPC-DS: Total hot run time: 191848 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3149780eee2c25f9ec3717151b1a5523e12d902b, data reload: false

query1	1442	1088	1063	1063
query2	6426	1778	1777	1777
query3	11029	4458	4380	4380
query4	52594	24234	23123	23123
query5	5110	571	438	438
query6	338	196	192	192
query7	4942	505	294	294
query8	308	252	240	240
query9	5559	2580	2601	2580
query10	437	323	253	253
query11	15496	14947	14779	14779
query12	150	108	102	102
query13	1067	505	374	374
query14	10014	6342	6389	6342
query15	234	193	169	169
query16	7097	659	502	502
query17	1096	768	605	605
query18	1568	428	354	354
query19	204	205	169	169
query20	138	118	120	118
query21	213	129	108	108
query22	4499	4493	4309	4309
query23	34177	33269	33428	33269
query24	6546	2470	2452	2452
query25	466	466	389	389
query26	705	280	153	153
query27	2288	503	342	342
query28	2958	2147	2141	2141
query29	550	559	441	441
query30	278	224	187	187
query31	842	861	821	821
query32	72	65	61	61
query33	479	358	308	308
query34	764	853	552	552
query35	824	850	748	748
query36	948	1004	914	914
query37	110	102	77	77
query38	4243	4330	4268	4268
query39	1486	1424	1600	1424
query40	207	128	109	109
query41	57	54	53	53
query42	127	102	107	102
query43	500	520	466	466
query44	1325	808	823	808
query45	190	175	172	172
query46	845	1035	642	642
query47	1871	1829	1775	1775
query48	391	420	328	328
query49	665	521	419	419
query50	674	720	425	425
query51	4166	4184	4215	4184
query52	109	113	103	103
query53	241	272	199	199
query54	573	587	523	523
query55	82	89	89	89
query56	316	293	302	293
query57	1140	1198	1106	1106
query58	270	255	263	255
query59	2713	2812	2649	2649
query60	321	318	324	318
query61	131	130	125	125
query62	764	823	681	681
query63	231	193	200	193
query64	1773	1010	691	691
query65	4342	4230	4237	4230
query66	721	396	303	303
query67	16041	15344	15412	15344
query68	7123	898	512	512
query69	534	299	267	267
query70	1165	1095	1063	1063
query71	538	316	279	279
query72	5836	4795	4883	4795
query73	1498	650	351	351
query74	8868	9196	8677	8677
query75	4008	3200	2692	2692
query76	4170	1183	750	750
query77	772	371	299	299
query78	10031	10193	9265	9265
query79	2442	821	577	577
query80	682	489	462	462
query81	491	266	225	225
query82	445	129	94	94
query83	254	255	227	227
query84	293	109	84	84
query85	769	422	303	303
query86	370	326	306	306
query87	4479	4399	4307	4307
query88	3302	2188	2188	2188
query89	401	315	284	284
query90	1801	212	205	205
query91	142	143	111	111
query92	77	60	61	60
query93	1893	964	591	591
query94	663	406	314	314
query95	368	291	282	282
query96	489	570	271	271
query97	3142	3195	3150	3150
query98	235	214	199	199
query99	1410	1390	1255	1255
Total cold run time: 297174 ms
Total hot run time: 191848 ms

doris-robot · 2025-04-21T10:21:28Z

ClickBench: Total hot run time: 29.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3149780eee2c25f9ec3717151b1a5523e12d902b, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.19	0.20
query4	1.59	0.19	0.19
query5	0.60	0.62	0.61
query6	1.19	0.73	0.71
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.53	0.52
query10	0.56	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.61	0.60	0.60
query14	1.15	1.16	1.17
query15	0.87	0.86	0.85
query16	0.38	0.40	0.39
query17	1.05	1.07	1.05
query18	0.20	0.20	0.20
query19	1.86	1.87	1.85
query20	0.01	0.01	0.01
query21	15.43	0.91	0.57
query22	0.75	1.20	0.62
query23	14.96	1.42	0.62
query24	6.90	1.39	0.48
query25	0.50	0.13	0.14
query26	0.70	0.16	0.13
query27	0.05	0.06	0.05
query28	10.04	0.91	0.44
query29	12.56	3.94	3.33
query30	0.24	0.08	0.06
query31	2.82	0.58	0.39
query32	3.23	0.54	0.47
query33	2.97	3.05	3.05
query34	15.74	5.09	4.48
query35	4.53	4.50	4.48
query36	0.68	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.04	0.02
Total cold run time: 103.98 s
Total hot run time: 29.39 s

dataroaring

LGTM

…de restart (#50221) ### What problem does this PR solve? If user creates routine load job by specifying partitions: ``` FROM KAFKA ( "kafka_partitions" = "0", "kafka_offsets" = "XXX" ); ``` job progress fallback after FE master node restart in share-storage mode. ### Release note None ### Check List (For Author) - Test  - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…FE master node restart #50221 (#50282) Cherry-picked from #50221 Co-authored-by: hui lai <laihui@selectdb.com>

…de restart (apache#50221) ### What problem does this PR solve? If user creates routine load job by specifying partitions: ``` FROM KAFKA ( "kafka_partitions" = "0", "kafka_offsets" = "XXX" ); ``` job progress fallback after FE master node restart in share-storage mode. ### Release note None ### Check List (For Author) - Test  - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…pache#3924) pick apache#50221

dataroaring added dev/3.0.x dev/2.1.x labels Apr 21, 2025

liaoxin01 reviewed Apr 21, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java Show resolved Hide resolved

fix routine load job progress fallback after FE master node restart

3149780

sollhui force-pushed the fix_rl_progress_fallback branch from 8368f4d to 3149780 Compare April 21, 2025 06:36

liaoxin01 approved these changes Apr 21, 2025

View reviewed changes

liaoxin01 removed the dev/2.1.x label Apr 21, 2025

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 21, 2025

dataroaring approved these changes Apr 22, 2025

View reviewed changes

dataroaring merged commit 917da4d into apache:master Apr 22, 2025
25 of 26 checks passed

github-actions bot mentioned this pull request Apr 22, 2025

branch-3.0: [fix](load) fix routine load job progress fallback after FE master node restart #50221 #50282

Merged

dataroaring pushed a commit that referenced this pull request Apr 23, 2025

branch-3.0: [fix](load) fix routine load job progress fallback after …

681ac90

…FE master node restart #50221 (#50282) Cherry-picked from #50221 Co-authored-by: hui lai <laihui@selectdb.com>

dataroaring added p0_l dev/3.0.6-merged and removed dev/3.0.x labels Apr 23, 2025

gavinchou mentioned this pull request Jun 11, 2025

Release Note 3.0.6 #51585

Closed

suxiaogang223 pushed a commit to suxiaogang223/doris that referenced this pull request Jul 15, 2025

fix routine load job progress fallback after FE master node restart (a…

3344d8b

…pache#3924) pick apache#50221

Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Jul 21, 2025

fix routine load job progress fallback after FE master node restart (a…

641762c

…pache#3924) pick apache#50221

[fix](load) fix routine load job progress fallback after FE master node restart #50221

[fix](load) fix routine load job progress fallback after FE master node restart #50221

Uh oh!

Conversation

sollhui commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 21, 2025

Uh oh!

sollhui commented Apr 21, 2025

Uh oh!

Uh oh!

sollhui commented Apr 21, 2025

Uh oh!

doris-robot commented Apr 21, 2025

Uh oh!

doris-robot commented Apr 21, 2025

Uh oh!

doris-robot commented Apr 21, 2025

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 21, 2025

Uh oh!

github-actions bot commented Apr 21, 2025

Uh oh!

sollhui commented Apr 21, 2025

Uh oh!

doris-robot commented Apr 21, 2025

Uh oh!

doris-robot commented Apr 21, 2025

Uh oh!

doris-robot commented Apr 21, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sollhui commented Apr 21, 2025 •

edited

Loading