[feat](refactor-param) refactor routineLoad's code about fileformat #50552

BePPPower · 2025-04-29T13:11:40Z

Issue Number: #50238

Problem Summary:

Previously, we refactored the code of the fileFormat attribute (#50225). However, we only added the relevant code without modifying the business code. This pull request modifies the code of the RoutineLoad feature that is related to the fileformat.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-04-29T13:11:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

BePPPower · 2025-04-29T13:11:54Z

run buildall

starocean999 · 2025-04-29T13:29:39Z

stream load, routine load and broker load has been migrate to nereids planner( #49116 ), please update the code for the new planner too.

BePPPower · 2025-04-30T07:24:40Z

run buildall

doris-robot · 2025-04-30T08:56:05Z

TPC-H: Total hot run time: 33989 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99de336c00b3d9f21c940b2ad7f866ffd20f5531, data reload: false

------ Round 1 ----------------------------------
q1	26352	5022	5019	5019
q2	2111	290	208	208
q3	10554	1264	711	711
q4	10267	1030	538	538
q5	7944	2439	2325	2325
q6	181	166	134	134
q7	910	732	607	607
q8	9316	1298	1040	1040
q9	6809	5052	5070	5052
q10	6811	2296	1901	1901
q11	475	284	280	280
q12	352	346	216	216
q13	17808	3744	3145	3145
q14	245	240	229	229
q15	554	483	515	483
q16	434	442	365	365
q17	614	866	388	388
q18	7947	7198	7153	7153
q19	1574	964	548	548
q20	338	340	234	234
q21	3951	2642	2436	2436
q22	1029	1015	977	977
Total cold run time: 116576 ms
Total hot run time: 33989 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5129	5084	5090	5084
q2	235	328	246	246
q3	2141	2626	2278	2278
q4	1324	1800	1353	1353
q5	4601	4516	4361	4361
q6	216	175	130	130
q7	1920	1887	1742	1742
q8	2587	2509	2511	2509
q9	7243	7069	7193	7069
q10	2962	3216	2720	2720
q11	571	493	483	483
q12	663	751	598	598
q13	3496	3871	3327	3327
q14	274	298	276	276
q15	509	479	476	476
q16	427	473	437	437
q17	1170	1593	1340	1340
q18	7733	7666	7341	7341
q19	808	777	869	777
q20	1952	2062	1880	1880
q21	5102	4575	4560	4560
q22	1032	1034	964	964
Total cold run time: 52095 ms
Total hot run time: 49951 ms

doris-robot · 2025-04-30T09:07:34Z

TPC-DS: Total hot run time: 185352 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99de336c00b3d9f21c940b2ad7f866ffd20f5531, data reload: false

query1	1025	473	466	466
query2	6552	1812	1781	1781
query3	6764	218	221	218
query4	26783	23970	23118	23118
query5	4368	628	457	457
query6	307	215	197	197
query7	4621	494	277	277
query8	308	253	244	244
query9	8673	2555	2554	2554
query10	486	324	276	276
query11	15293	15029	14797	14797
query12	156	110	101	101
query13	1646	513	381	381
query14	8830	6054	6093	6054
query15	199	182	175	175
query16	7154	614	482	482
query17	1182	711	556	556
query18	1971	385	297	297
query19	185	184	150	150
query20	119	113	118	113
query21	209	134	107	107
query22	4069	4116	4008	4008
query23	33862	32847	33019	32847
query24	8445	2402	2380	2380
query25	571	480	424	424
query26	1251	275	166	166
query27	2721	506	333	333
query28	4349	2114	2082	2082
query29	749	588	428	428
query30	281	210	189	189
query31	916	848	766	766
query32	70	68	63	63
query33	547	354	294	294
query34	792	844	529	529
query35	770	789	757	757
query36	942	989	868	868
query37	109	99	76	76
query38	4136	4115	4138	4115
query39	1438	1391	1389	1389
query40	210	119	105	105
query41	57	56	53	53
query42	120	105	109	105
query43	490	472	454	454
query44	1280	805	805	805
query45	174	191	166	166
query46	822	996	623	623
query47	1747	1799	1757	1757
query48	381	399	309	309
query49	766	494	405	405
query50	627	672	416	416
query51	4165	4087	4069	4069
query52	106	102	101	101
query53	230	251	182	182
query54	581	571	494	494
query55	82	82	84	82
query56	311	294	314	294
query57	1130	1146	1075	1075
query58	259	252	249	249
query59	2516	2667	2528	2528
query60	328	327	298	298
query61	133	129	131	129
query62	823	704	674	674
query63	217	179	181	179
query64	4399	1002	716	716
query65	4301	4247	4226	4226
query66	1159	407	343	343
query67	15635	15577	15373	15373
query68	7841	861	515	515
query69	500	305	269	269
query70	1207	1137	1054	1054
query71	409	319	293	293
query72	5597	4765	4848	4765
query73	675	632	342	342
query74	8940	8863	8873	8863
query75	3174	3182	2697	2697
query76	3153	1175	743	743
query77	461	357	292	292
query78	9860	9986	9277	9277
query79	2027	807	556	556
query80	608	542	475	475
query81	485	262	224	224
query82	314	131	101	101
query83	266	258	261	258
query84	258	122	94	94
query85	873	452	316	316
query86	361	297	286	286
query87	4389	4402	4314	4314
query88	3557	2193	2232	2193
query89	384	307	291	291
query90	1889	210	213	210
query91	146	143	112	112
query92	80	64	55	55
query93	1694	964	596	596
query94	686	372	314	314
query95	376	289	289	289
query96	473	561	274	274
query97	3132	3251	3084	3084
query98	228	273	205	205
query99	1314	1388	1270	1270
Total cold run time: 271255 ms
Total hot run time: 185352 ms

doris-robot · 2025-04-30T09:12:46Z

ClickBench: Total hot run time: 28.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 99de336c00b3d9f21c940b2ad7f866ffd20f5531, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.11	0.11
query3	0.25	0.18	0.19
query4	1.59	0.19	0.12
query5	0.59	0.55	0.56
query6	1.18	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.54	0.52
query10	0.57	0.57	0.56
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.60	0.59
query14	0.77	0.79	0.80
query15	0.88	0.86	0.86
query16	0.37	0.37	0.38
query17	1.07	0.99	1.02
query18	0.22	0.20	0.20
query19	1.92	1.82	1.78
query20	0.01	0.01	0.01
query21	15.40	0.89	0.56
query22	0.78	1.14	0.72
query23	14.91	1.40	0.67
query24	6.87	1.72	0.46
query25	0.49	0.21	0.13
query26	0.62	0.16	0.14
query27	0.05	0.05	0.06
query28	9.80	0.94	0.45
query29	12.61	3.91	3.24
query30	0.25	0.09	0.07
query31	2.81	0.60	0.38
query32	3.22	0.55	0.46
query33	3.12	3.04	3.04
query34	15.80	5.13	4.53
query35	4.53	4.51	4.47
query36	0.66	0.49	0.49
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.54 s
Total hot run time: 28.83 s

BePPPower · 2025-05-08T03:41:33Z

run buildall

doris-robot · 2025-05-08T04:25:58Z

TPC-H: Total hot run time: 33750 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c496d856aac32d383ccbb38dc2da849cac942bd5, data reload: false

------ Round 1 ----------------------------------
q1	25986	5146	5022	5022
q2	2067	274	178	178
q3	10393	1230	683	683
q4	10232	1006	509	509
q5	7527	2233	2370	2233
q6	180	163	134	134
q7	900	776	628	628
q8	9327	1247	1073	1073
q9	6733	5132	5086	5086
q10	6862	2281	1901	1901
q11	499	284	270	270
q12	346	347	209	209
q13	17773	3657	3059	3059
q14	235	219	209	209
q15	524	498	490	490
q16	430	432	372	372
q17	590	848	365	365
q18	7657	7183	7260	7183
q19	1494	959	556	556
q20	323	339	221	221
q21	4009	3437	2449	2449
q22	985	1012	920	920
Total cold run time: 115072 ms
Total hot run time: 33750 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5197	5061	5074	5061
q2	270	338	234	234
q3	2303	2770	2425	2425
q4	1399	1829	1340	1340
q5	4384	4433	4436	4433
q6	224	170	127	127
q7	2008	1969	1777	1777
q8	2558	2555	2547	2547
q9	7133	7121	7012	7012
q10	2991	3174	2733	2733
q11	578	494	505	494
q12	647	782	615	615
q13	3513	3838	3285	3285
q14	279	297	302	297
q15	534	489	462	462
q16	432	494	452	452
q17	1154	1577	1370	1370
q18	7791	7547	7410	7410
q19	835	838	913	838
q20	1988	1946	1832	1832
q21	5057	4725	4622	4622
q22	1092	1030	985	985
Total cold run time: 52367 ms
Total hot run time: 50351 ms

doris-robot · 2025-05-08T04:38:09Z

TPC-DS: Total hot run time: 192231 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c496d856aac32d383ccbb38dc2da849cac942bd5, data reload: false

query1	1432	1073	1071	1071
query2	6333	1796	1752	1752
query3	10971	4556	4501	4501
query4	53185	24746	23400	23400
query5	5335	512	470	470
query6	399	209	198	198
query7	5211	518	283	283
query8	333	266	241	241
query9	6800	2535	2540	2535
query10	454	317	270	270
query11	15064	15150	14916	14916
query12	156	118	106	106
query13	1369	532	390	390
query14	10138	6288	6173	6173
query15	206	195	193	193
query16	6981	660	510	510
query17	1096	736	605	605
query18	1549	414	336	336
query19	240	185	169	169
query20	132	129	120	120
query21	205	131	110	110
query22	4323	4429	4321	4321
query23	34144	33161	33276	33161
query24	6552	2458	2407	2407
query25	463	456	397	397
query26	724	285	155	155
query27	2290	505	338	338
query28	3253	2097	2092	2092
query29	585	565	419	419
query30	268	228	189	189
query31	871	885	761	761
query32	67	67	65	65
query33	438	365	311	311
query34	776	898	533	533
query35	788	846	763	763
query36	953	1002	878	878
query37	109	98	79	79
query38	4160	4313	4213	4213
query39	1505	1448	1437	1437
query40	216	121	121	121
query41	57	58	52	52
query42	131	109	105	105
query43	531	498	479	479
query44	1338	818	815	815
query45	185	172	178	172
query46	861	1017	642	642
query47	1838	1844	1788	1788
query48	392	409	313	313
query49	699	530	428	428
query50	663	718	424	424
query51	4182	4262	4198	4198
query52	116	110	103	103
query53	221	254	179	179
query54	577	569	514	514
query55	88	85	79	79
query56	298	302	308	302
query57	1201	1213	1131	1131
query58	261	253	256	253
query59	2510	2733	2642	2642
query60	341	335	313	313
query61	133	137	125	125
query62	708	751	710	710
query63	226	202	186	186
query64	1785	1072	693	693
query65	4429	4249	4243	4243
query66	731	395	297	297
query67	15869	15393	15341	15341
query68	7011	903	509	509
query69	531	301	269	269
query70	1163	1056	1094	1056
query71	491	330	317	317
query72	5891	4902	4960	4902
query73	1221	664	348	348
query74	8896	9236	8785	8785
query75	3764	3190	2704	2704
query76	4235	1204	780	780
query77	638	380	282	282
query78	10078	10078	9251	9251
query79	1918	839	557	557
query80	593	518	453	453
query81	483	258	223	223
query82	446	127	97	97
query83	256	252	229	229
query84	292	112	85	85
query85	787	348	317	317
query86	400	297	275	275
query87	4382	4419	4359	4359
query88	3682	2245	2247	2245
query89	404	329	279	279
query90	1902	215	216	215
query91	142	138	113	113
query92	73	65	57	57
query93	1487	958	577	577
query94	666	401	308	308
query95	363	294	295	294
query96	486	569	272	272
query97	3096	3224	3072	3072
query98	230	213	202	202
query99	1432	1408	1283	1283
Total cold run time: 297884 ms
Total hot run time: 192231 ms

doris-robot · 2025-05-08T04:43:22Z

ClickBench: Total hot run time: 28.88 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c496d856aac32d383ccbb38dc2da849cac942bd5, data reload: false

query1	0.04	0.03	0.03
query2	0.14	0.11	0.11
query3	0.27	0.20	0.20
query4	1.60	0.19	0.10
query5	0.56	0.55	0.56
query6	1.18	0.72	0.73
query7	0.03	0.02	0.01
query8	0.05	0.04	0.03
query9	0.58	0.52	0.53
query10	0.58	0.57	0.55
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.60
query14	0.78	0.81	0.79
query15	0.88	0.84	0.84
query16	0.40	0.40	0.38
query17	1.06	1.01	1.04
query18	0.21	0.19	0.19
query19	1.94	1.82	1.80
query20	0.02	0.01	0.01
query21	15.42	0.90	0.53
query22	0.76	1.21	0.73
query23	14.79	1.41	0.65
query24	7.36	1.87	0.42
query25	0.51	0.22	0.19
query26	0.62	0.16	0.13
query27	0.06	0.06	0.05
query28	9.17	0.90	0.42
query29	12.54	3.91	3.27
query30	0.25	0.10	0.07
query31	2.81	0.61	0.38
query32	3.23	0.55	0.48
query33	3.07	3.08	3.10
query34	15.57	5.13	4.53
query35	4.50	4.54	4.47
query36	0.65	0.52	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.03 s
Total hot run time: 28.88 s

BePPPower · 2025-05-08T08:21:00Z

run p0

BePPPower · 2025-05-08T10:14:01Z

run buildall

BePPPower · 2025-05-08T14:27:43Z

run feut, performance

BePPPower · 2025-05-08T15:01:41Z

run buildall

doris-robot · 2025-05-08T15:25:08Z

TPC-H: Total hot run time: 33856 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0568b20a1c498369d7f76a961d28fe9304ca8e2f, data reload: false

------ Round 1 ----------------------------------
q1	26158	5136	5024	5024
q2	2079	278	195	195
q3	10401	1256	709	709
q4	10244	1003	555	555
q5	8197	2347	2377	2347
q6	273	162	133	133
q7	923	769	614	614
q8	9324	1239	1059	1059
q9	6846	5085	5106	5085
q10	6802	2299	1876	1876
q11	468	284	277	277
q12	349	342	210	210
q13	17765	3645	3162	3162
q14	245	242	212	212
q15	544	490	483	483
q16	437	439	383	383
q17	595	875	362	362
q18	7721	7131	6976	6976
q19	1568	955	567	567
q20	344	338	234	234
q21	3827	3451	2442	2442
q22	1004	996	951	951
Total cold run time: 116114 ms
Total hot run time: 33856 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5176	5121	5141	5121
q2	243	324	232	232
q3	2203	2682	2313	2313
q4	1355	1788	1421	1421
q5	4543	4476	4350	4350
q6	212	163	123	123
q7	1964	1959	1748	1748
q8	2590	2813	2537	2537
q9	7086	7017	7140	7017
q10	2987	3219	2740	2740
q11	565	501	482	482
q12	675	756	585	585
q13	3545	3895	3307	3307
q14	291	284	273	273
q15	527	472	474	472
q16	423	483	462	462
q17	1166	1558	1385	1385
q18	7759	7592	7383	7383
q19	813	892	912	892
q20	1977	1983	1821	1821
q21	4985	4604	4562	4562
q22	1048	1015	972	972
Total cold run time: 52133 ms
Total hot run time: 50198 ms

doris-robot · 2025-05-08T15:36:45Z

TPC-DS: Total hot run time: 185485 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0568b20a1c498369d7f76a961d28fe9304ca8e2f, data reload: false

query1	1010	451	512	451
query2	6561	1813	1850	1813
query3	6748	231	217	217
query4	26321	23916	22931	22931
query5	4433	662	466	466
query6	308	196	186	186
query7	4641	490	281	281
query8	285	249	234	234
query9	8623	2546	2549	2546
query10	485	306	260	260
query11	15712	15469	14767	14767
query12	160	118	111	111
query13	1659	521	390	390
query14	9375	6083	6038	6038
query15	195	182	167	167
query16	7236	621	486	486
query17	1203	745	632	632
query18	1966	397	302	302
query19	188	180	165	165
query20	118	120	118	118
query21	216	120	101	101
query22	3997	4121	4078	4078
query23	33863	33044	33146	33044
query24	8400	2397	2419	2397
query25	544	434	378	378
query26	1232	267	156	156
query27	2755	492	332	332
query28	4344	2091	2067	2067
query29	773	583	458	458
query30	282	215	185	185
query31	944	872	766	766
query32	71	66	65	65
query33	552	359	340	340
query34	811	839	519	519
query35	783	773	720	720
query36	936	956	900	900
query37	113	101	82	82
query38	4250	4120	4160	4120
query39	1483	1373	1381	1373
query40	214	121	106	106
query41	60	57	55	55
query42	130	103	103	103
query43	518	522	483	483
query44	1313	796	804	796
query45	176	170	162	162
query46	839	1039	633	633
query47	1760	1777	1731	1731
query48	371	415	301	301
query49	773	526	426	426
query50	653	688	396	396
query51	4124	4181	4017	4017
query52	109	104	100	100
query53	240	250	183	183
query54	588	563	509	509
query55	91	83	80	80
query56	306	295	311	295
query57	1133	1160	1083	1083
query58	272	258	264	258
query59	2679	2680	2626	2626
query60	331	339	316	316
query61	161	156	152	152
query62	782	761	667	667
query63	225	190	186	186
query64	4475	1112	819	819
query65	4280	4253	4241	4241
query66	1144	423	313	313
query67	15749	15478	15176	15176
query68	8628	883	507	507
query69	465	305	264	264
query70	1193	1127	1075	1075
query71	472	328	292	292
query72	5780	4782	4831	4782
query73	723	635	345	345
query74	8886	9066	8881	8881
query75	4058	3244	2715	2715
query76	3683	1176	765	765
query77	803	367	279	279
query78	10051	9945	9235	9235
query79	3329	802	545	545
query80	657	570	443	443
query81	464	263	225	225
query82	495	126	97	97
query83	293	245	233	233
query84	298	114	92	92
query85	795	361	312	312
query86	338	304	291	291
query87	4461	4408	4337	4337
query88	2996	2211	2191	2191
query89	438	315	282	282
query90	1955	211	204	204
query91	141	141	112	112
query92	75	58	57	57
query93	2177	942	574	574
query94	671	391	311	311
query95	376	288	292	288
query96	483	553	271	271
query97	3140	3215	3160	3160
query98	229	212	208	208
query99	1472	1362	1257	1257
Total cold run time: 277097 ms
Total hot run time: 185485 ms

doris-robot · 2025-05-08T15:41:57Z

ClickBench: Total hot run time: 28.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0568b20a1c498369d7f76a961d28fe9304ca8e2f, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.10
query3	0.26	0.18	0.19
query4	1.59	0.19	0.20
query5	0.57	0.56	0.55
query6	1.18	0.71	0.74
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.57	0.53	0.51
query10	0.57	0.57	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.61	0.60	0.59
query14	0.79	0.80	0.80
query15	0.87	0.87	0.85
query16	0.38	0.38	0.39
query17	1.05	1.07	1.03
query18	0.21	0.20	0.20
query19	1.88	1.78	1.80
query20	0.01	0.01	0.01
query21	15.40	0.93	0.56
query22	0.75	1.16	0.64
query23	15.01	1.38	0.62
query24	7.61	0.91	0.66
query25	0.48	0.19	0.08
query26	0.54	0.16	0.14
query27	0.05	0.05	0.06
query28	9.48	0.90	0.43
query29	12.55	3.87	3.23
query30	0.25	0.09	0.06
query31	2.82	0.59	0.39
query32	3.23	0.54	0.46
query33	3.02	3.01	3.09
query34	15.76	5.07	4.49
query35	4.49	4.52	4.47
query36	0.69	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.72 s
Total hot run time: 28.83 s

BePPPower · 2025-05-08T16:09:51Z

run buildall

morningman · 2025-05-12T01:57:43Z

run buildall

doris-robot · 2025-05-12T02:40:27Z

TPC-H: Total hot run time: 33961 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1ec8e5a113c1a937dbb4ac9059bd9344efd051b, data reload: false

------ Round 1 ----------------------------------
q1	26079	5061	5003	5003
q2	2074	275	196	196
q3	10384	1220	683	683
q4	10241	1013	521	521
q5	7535	2331	2344	2331
q6	188	164	129	129
q7	883	725	608	608
q8	9318	1236	1117	1117
q9	6883	5077	5112	5077
q10	6866	2292	1881	1881
q11	472	288	273	273
q12	331	351	209	209
q13	17781	3664	3125	3125
q14	224	227	218	218
q15	541	480	479	479
q16	423	436	370	370
q17	605	862	372	372
q18	7461	7408	7205	7205
q19	1583	962	535	535
q20	348	338	231	231
q21	4004	3277	2443	2443
q22	1058	1003	955	955
Total cold run time: 115282 ms
Total hot run time: 33961 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5121	5007	5063	5007
q2	243	329	230	230
q3	2182	2668	2253	2253
q4	1334	1744	1342	1342
q5	4405	4406	4393	4393
q6	217	162	129	129
q7	1995	1930	1765	1765
q8	2596	2483	2463	2463
q9	7243	7142	7048	7048
q10	3067	3245	2751	2751
q11	574	501	486	486
q12	667	780	633	633
q13	3569	3943	3277	3277
q14	314	309	285	285
q15	518	498	468	468
q16	445	480	456	456
q17	1160	1490	1395	1395
q18	7730	7488	7326	7326
q19	780	760	823	760
q20	1945	1964	1828	1828
q21	5143	4829	4720	4720
q22	1039	1055	1003	1003
Total cold run time: 52287 ms
Total hot run time: 50018 ms

doris-robot · 2025-05-12T02:52:44Z

TPC-DS: Total hot run time: 193438 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1ec8e5a113c1a937dbb4ac9059bd9344efd051b, data reload: false

query1	1391	1092	1072	1072
query2	6294	1813	1826	1813
query3	11137	4627	4535	4535
query4	54391	25533	23406	23406
query5	4921	587	485	485
query6	349	204	197	197
query7	4894	520	298	298
query8	302	257	244	244
query9	5359	2658	2672	2658
query10	435	338	276	276
query11	15255	14948	14929	14929
query12	161	116	108	108
query13	1025	517	403	403
query14	10138	6206	6351	6206
query15	221	195	179	179
query16	7112	674	497	497
query17	1070	736	569	569
query18	1580	427	316	316
query19	224	189	163	163
query20	123	126	120	120
query21	204	131	104	104
query22	4330	4338	4462	4338
query23	34460	33520	33530	33520
query24	6620	2418	2390	2390
query25	453	527	430	430
query26	716	278	167	167
query27	2364	528	362	362
query28	2860	2178	2184	2178
query29	582	581	473	473
query30	291	227	192	192
query31	885	860	800	800
query32	74	66	70	66
query33	450	389	336	336
query34	783	868	561	561
query35	816	834	814	814
query36	958	1014	919	919
query37	113	103	80	80
query38	4282	4270	4279	4270
query39	1488	1480	1480	1480
query40	202	119	107	107
query41	55	52	54	52
query42	122	111	107	107
query43	505	505	494	494
query44	1386	835	850	835
query45	180	175	168	168
query46	887	1020	675	675
query47	1842	1891	1747	1747
query48	389	434	315	315
query49	686	535	398	398
query50	662	688	422	422
query51	4289	4179	4157	4157
query52	117	111	109	109
query53	235	274	193	193
query54	608	585	529	529
query55	91	86	84	84
query56	321	312	271	271
query57	1164	1178	1152	1152
query58	289	283	261	261
query59	2661	2744	2550	2550
query60	347	345	314	314
query61	127	121	128	121
query62	760	750	684	684
query63	233	194	200	194
query64	1758	1022	729	729
query65	4437	4236	4250	4236
query66	702	399	305	305
query67	15919	15691	15234	15234
query68	7314	902	527	527
query69	552	305	271	271
query70	1244	1131	1110	1110
query71	526	324	307	307
query72	5789	4870	4974	4870
query73	1466	706	360	360
query74	8969	8901	8802	8802
query75	4084	3267	2789	2789
query76	4252	1189	746	746
query77	691	387	289	289
query78	9998	10120	9254	9254
query79	3868	824	559	559
query80	638	542	466	466
query81	483	263	229	229
query82	469	134	102	102
query83	313	252	237	237
query84	292	102	92	92
query85	862	344	316	316
query86	350	305	310	305
query87	4469	4599	4389	4389
query88	3278	2301	2309	2301
query89	444	323	289	289
query90	1982	214	214	214
query91	141	146	115	115
query92	76	62	57	57
query93	2550	947	576	576
query94	671	427	314	314
query95	380	300	293	293
query96	485	578	283	283
query97	3163	3208	3131	3131
query98	229	291	206	206
query99	1452	1414	1242	1242
Total cold run time: 301565 ms
Total hot run time: 193438 ms

doris-robot · 2025-05-12T02:57:58Z

ClickBench: Total hot run time: 29.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e1ec8e5a113c1a937dbb4ac9059bd9344efd051b, data reload: false

query1	0.03	0.04	0.03
query2	0.13	0.10	0.11
query3	0.26	0.19	0.20
query4	1.59	0.19	0.11
query5	0.57	0.55	0.55
query6	1.21	0.72	0.74
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.52	0.51
query10	0.57	0.58	0.57
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	0.81	0.81	0.83
query15	0.88	0.86	0.86
query16	0.37	0.37	0.41
query17	1.05	1.04	1.05
query18	0.23	0.21	0.21
query19	1.92	1.77	1.87
query20	0.02	0.01	0.01
query21	15.40	0.89	0.54
query22	0.75	1.22	0.76
query23	14.78	1.36	0.63
query24	6.80	1.96	1.27
query25	0.51	0.17	0.06
query26	0.52	0.17	0.16
query27	0.05	0.05	0.05
query28	10.12	0.83	0.48
query29	12.53	4.06	3.29
query30	0.26	0.09	0.07
query31	2.83	0.60	0.38
query32	3.23	0.56	0.47
query33	3.12	3.04	3.07
query34	15.89	5.17	4.49
query35	4.51	4.56	4.52
query36	0.69	0.50	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.13	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.65 s
Total hot run time: 29.74 s

morningman

LGTM

github-actions · 2025-05-12T03:13:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-12T03:13:38Z

PR approved by anyone and no changes requested.

…pache#50552) Issue Number: apache#50238 Problem Summary: Previously, we refactored the code of the fileFormat attribute (apache#50225). However, we only added the relevant code without modifying the business code. This pull request modifies the code of the `RoutineLoad` feature that is related to the fileformat.

…ileformat (apache#50552) Issue Number: apache#50238 Problem Summary: Previously, we refactored the code of the fileFormat attribute (apache#50225). However, we only added the relevant code without modifying the business code. This pull request modifies the code of the `RoutineLoad` feature that is related to the fileformat. (cherry picked from commit b3abfab)

…fileformat #50552 (#52173) bp #50552 --------- Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>

BePPPower force-pushed the refactorRoutineLoad branch from 0e75500 to c496d85 Compare May 8, 2025 03:40

BePPPower added 5 commits May 12, 2025 09:57

refactor routineLoad

96840dd

refactor 1

28b6e98

refactor 2

882c2ab

refactor 3

567aca9

refactor 4

e1ec8e5

morningman force-pushed the refactorRoutineLoad branch from c3805dd to e1ec8e5 Compare May 12, 2025 01:57

morningman approved these changes May 12, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 12, 2025

github-actions bot added the reviewed label May 12, 2025

suxiaogang223 approved these changes May 13, 2025

View reviewed changes

wuwenchi approved these changes May 13, 2025

View reviewed changes

morningman merged commit b3abfab into apache:master May 14, 2025
26 of 28 checks passed

CalvinKirs added the dev/3.1.x label Jun 23, 2025

github-actions bot added the dev/3.1.x-conflict label Jun 23, 2025

CalvinKirs mentioned this pull request Jun 24, 2025

branch-3.1: [feat](refactor-param) refactor routineLoad's code about fileformat #50552 #52173

Merged

morrySnow pushed a commit that referenced this pull request Jun 25, 2025

branch-3.1: [feat](refactor-param) refactor routineLoad's code about …

c219df6

…fileformat #50552 (#52173) bp #50552 --------- Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jun 25, 2025

[feat](refactor-param) refactor routineLoad's code about fileformat #50552

[feat](refactor-param) refactor routineLoad's code about fileformat #50552

Uh oh!

Conversation

BePPPower commented Apr 29, 2025

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Apr 29, 2025

Uh oh!

BePPPower commented Apr 29, 2025

Uh oh!

starocean999 commented Apr 29, 2025

Uh oh!

BePPPower commented Apr 30, 2025

Uh oh!

doris-robot commented Apr 30, 2025

Uh oh!

doris-robot commented Apr 30, 2025

Uh oh!

doris-robot commented Apr 30, 2025

Uh oh!

BePPPower commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

BePPPower commented May 8, 2025

Uh oh!

BePPPower commented May 8, 2025

Uh oh!

BePPPower commented May 8, 2025

Uh oh!

BePPPower commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

BePPPower commented May 8, 2025

Uh oh!

morningman commented May 12, 2025

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants