Skip to content

[feat](refactor-param) refactor brokerLoad's code about fileformat #50882

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 8 commits into from
May 29, 2025
Merged

[feat](refactor-param) refactor brokerLoad's code about fileformat #50882

morningman merged 8 commits into from
May 29, 2025

Conversation

@BePPPower
Copy link
Contributor

@BePPPower BePPPower commented May 14, 2025

Issue Number:#50238

Problem Summary:

Previously, we refactored the code of the fileFormat attribute (#50225). However, we only added the relevant code without modifying the business code. This pull request modifies the code of the BrokerLoad feature that is related to the fileformat.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented May 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@BePPPower
Copy link
Contributor Author

BePPPower commented May 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-H: Total hot run time: 34291 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ec024d2b422a74630ec5817ec0504b0baccff72, data reload: false

------ Round 1 ----------------------------------
q1	26076	5121	5077	5077
q2	2073	274	180	180
q3	10408	1259	723	723
q4	10250	1062	525	525
q5	7579	2446	2379	2379
q6	187	160	135	135
q7	937	734	626	626
q8	9336	1294	1117	1117
q9	6703	5099	5064	5064
q10	6900	2343	1929	1929
q11	483	293	296	293
q12	355	361	218	218
q13	17780	3708	3132	3132
q14	227	243	218	218
q15	529	485	486	485
q16	423	441	386	386
q17	591	859	371	371
q18	7723	7158	7138	7138
q19	1451	956	569	569
q20	337	334	215	215
q21	4342	3476	2521	2521
q22	1104	1053	990	990
Total cold run time: 115794 ms
Total hot run time: 34291 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5232	5158	5121	5121
q2	245	325	230	230
q3	2231	2685	2319	2319
q4	1393	1864	1488	1488
q5	4441	4430	4449	4430
q6	218	175	135	135
q7	2028	1929	1809	1809
q8	2616	2613	2556	2556
q9	7262	7118	7104	7104
q10	3043	3205	2802	2802
q11	636	527	491	491
q12	697	793	626	626
q13	3561	3961	3382	3382
q14	282	292	260	260
q15	534	477	479	477
q16	442	503	439	439
q17	1174	1558	1413	1413
q18	7930	7601	7536	7536
q19	838	813	845	813
q20	2042	1976	1841	1841
q21	5167	4912	4796	4796
q22	1157	1103	1057	1057
Total cold run time: 53169 ms
Total hot run time: 51125 ms

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-DS: Total hot run time: 193146 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ec024d2b422a74630ec5817ec0504b0baccff72, data reload: false

query1	1425	1077	1030	1030
query2	6228	1904	1909	1904
query3	11201	4500	4742	4500
query4	25849	24014	23032	23032
query5	4577	615	453	453
query6	316	229	204	204
query7	3985	504	294	294
query8	310	276	243	243
query9	8510	2670	2670	2670
query10	486	322	293	293
query11	15620	15334	14863	14863
query12	177	117	112	112
query13	1562	538	417	417
query14	9703	6304	6155	6155
query15	208	197	169	169
query16	7609	671	484	484
query17	1138	790	603	603
query18	2029	421	328	328
query19	207	203	171	171
query20	130	120	126	120
query21	208	132	115	115
query22	4441	4735	4385	4385
query23	34555	33565	33583	33565
query24	8690	2431	2414	2414
query25	522	498	419	419
query26	1194	272	155	155
query27	2944	518	350	350
query28	4452	2167	2169	2167
query29	729	562	439	439
query30	272	214	185	185
query31	915	866	774	774
query32	85	66	67	66
query33	529	375	321	321
query34	785	895	557	557
query35	778	829	753	753
query36	997	988	896	896
query37	110	96	77	77
query38	4195	4319	4160	4160
query39	1504	1452	1584	1452
query40	210	125	110	110
query41	57	60	53	53
query42	124	110	114	110
query43	514	509	501	501
query44	1358	829	822	822
query45	195	177	174	174
query46	846	1038	631	631
query47	1807	1876	1763	1763
query48	395	426	330	330
query49	736	517	467	467
query50	678	693	420	420
query51	4246	4303	4221	4221
query52	112	115	103	103
query53	227	253	184	184
query54	595	597	545	545
query55	91	90	83	83
query56	331	335	308	308
query57	1205	1261	1124	1124
query58	282	273	256	256
query59	2814	2810	2647	2647
query60	328	332	312	312
query61	133	124	126	124
query62	811	748	683	683
query63	229	197	192	192
query64	4274	1054	681	681
query65	4472	4370	4371	4370
query66	1035	419	322	322
query67	16120	15534	15531	15531
query68	8788	891	570	570
query69	478	305	258	258
query70	1180	1074	1118	1074
query71	488	331	298	298
query72	5693	4714	4716	4714
query73	677	575	359	359
query74	9055	9167	8792	8792
query75	3964	3236	2701	2701
query76	3755	1208	773	773
query77	788	379	282	282
query78	10005	10362	9376	9376
query79	2287	823	586	586
query80	623	500	453	453
query81	485	262	218	218
query82	407	126	95	95
query83	283	251	235	235
query84	292	116	89	89
query85	775	347	311	311
query86	349	288	280	280
query87	4426	4444	4310	4310
query88	2983	2256	2242	2242
query89	402	311	294	294
query90	1918	215	214	214
query91	141	144	110	110
query92	77	60	55	55
query93	1183	924	580	580
query94	665	411	308	308
query95	378	294	290	290
query96	489	559	278	278
query97	3129	3216	3151	3151
query98	240	210	198	198
query99	1432	1420	1290	1290
Total cold run time: 281235 ms
Total hot run time: 193146 ms

@doris-robot
Copy link

doris-robot commented May 14, 2025

ClickBench: Total hot run time: 29.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ec024d2b422a74630ec5817ec0504b0baccff72, data reload: false

query1	0.04	0.04	0.03
query2	0.15	0.10	0.11
query3	0.35	0.21	0.19
query4	1.59	0.20	0.08
query5	0.56	0.55	0.55
query6	1.19	0.74	0.73
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.62	0.51	0.54
query10	0.57	0.60	0.57
query11	0.26	0.12	0.13
query12	0.26	0.13	0.13
query13	0.65	0.62	0.62
query14	0.80	0.82	0.82
query15	0.95	0.89	0.88
query16	0.37	0.38	0.37
query17	1.05	1.05	1.08
query18	0.18	0.18	0.18
query19	1.98	1.87	1.91
query20	0.02	0.01	0.01
query21	15.40	0.98	0.68
query22	0.92	1.00	0.83
query23	14.69	1.61	0.78
query24	5.42	0.59	0.29
query25	0.17	0.10	0.08
query26	0.56	0.22	0.18
query27	0.08	0.08	0.09
query28	11.04	1.23	0.58
query29	12.54	4.11	3.48
query30	0.28	0.08	0.07
query31	2.81	0.63	0.42
query32	3.23	0.60	0.50
query33	3.03	3.06	3.16
query34	16.56	5.14	4.48
query35	4.54	4.46	4.53
query36	0.65	0.51	0.50
query37	0.20	0.17	0.17
query38	0.16	0.15	0.16
query39	0.04	0.04	0.04
query40	0.20	0.16	0.16
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.05	0.05
Total cold run time: 104.4 s
Total hot run time: 29.99 s

@BePPPower
Copy link
Contributor Author

BePPPower commented May 15, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 15, 2025

TPC-H: Total hot run time: 33898 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 718db6e49ac78b41aa888244d92a82eed0fff72b, data reload: false

------ Round 1 ----------------------------------
q1	25969	5072	5053	5053
q2	2063	270	197	197
q3	10394	1226	689	689
q4	10218	1011	527	527
q5	7560	2372	2570	2372
q6	185	168	136	136
q7	945	769	613	613
q8	9337	1324	1110	1110
q9	6720	5073	5145	5073
q10	6826	2289	1886	1886
q11	505	292	274	274
q12	350	350	209	209
q13	17764	3666	3032	3032
q14	228	227	213	213
q15	528	497	498	497
q16	433	428	385	385
q17	630	863	378	378
q18	7571	7234	7207	7207
q19	1216	959	561	561
q20	355	344	230	230
q21	3995	3134	2311	2311
q22	1001	1027	945	945
Total cold run time: 114793 ms
Total hot run time: 33898 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5232	5085	5079	5079
q2	235	321	228	228
q3	2200	2661	2238	2238
q4	1376	1783	1332	1332
q5	4497	4414	4378	4378
q6	213	169	126	126
q7	2002	1928	1769	1769
q8	2590	2514	2463	2463
q9	7045	7123	7081	7081
q10	3012	3182	2767	2767
q11	575	500	498	498
q12	665	747	608	608
q13	3554	3807	3248	3248
q14	292	302	272	272
q15	519	478	485	478
q16	444	485	438	438
q17	1153	1514	1396	1396
q18	7792	7368	7456	7368
q19	854	788	869	788
q20	2030	2040	1887	1887
q21	4863	4337	4271	4271
q22	1044	1016	1002	1002
Total cold run time: 52187 ms
Total hot run time: 49715 ms

@doris-robot
Copy link

doris-robot commented May 15, 2025

TPC-DS: Total hot run time: 185342 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 718db6e49ac78b41aa888244d92a82eed0fff72b, data reload: false

query1	1010	473	476	473
query2	6553	1776	1775	1775
query3	6742	223	222	222
query4	26714	23268	23492	23268
query5	4342	629	452	452
query6	291	200	212	200
query7	4621	493	292	292
query8	280	235	230	230
query9	8588	2614	2600	2600
query10	473	314	277	277
query11	15409	15064	14780	14780
query12	154	108	105	105
query13	1643	555	400	400
query14	8636	6148	6052	6052
query15	191	188	172	172
query16	7113	658	497	497
query17	952	722	596	596
query18	1976	394	305	305
query19	195	188	162	162
query20	121	120	118	118
query21	215	129	110	110
query22	4132	4253	4180	4180
query23	33894	33104	33069	33069
query24	8470	2403	2377	2377
query25	567	498	406	406
query26	1247	273	157	157
query27	2746	507	337	337
query28	4387	2109	2087	2087
query29	823	562	422	422
query30	279	216	186	186
query31	952	846	792	792
query32	72	65	62	62
query33	544	357	326	326
query34	797	832	507	507
query35	782	805	739	739
query36	954	1046	870	870
query37	109	101	77	77
query38	4097	4129	4040	4040
query39	1468	1411	1389	1389
query40	210	123	104	104
query41	57	54	53	53
query42	133	109	105	105
query43	491	507	480	480
query44	1306	802	796	796
query45	173	175	177	175
query46	875	1032	639	639
query47	1738	1799	1684	1684
query48	400	426	299	299
query49	770	522	417	417
query50	681	711	398	398
query51	4141	4163	4050	4050
query52	114	109	106	106
query53	229	249	182	182
query54	589	588	500	500
query55	87	85	78	78
query56	285	310	273	273
query57	1154	1133	1084	1084
query58	258	265	257	257
query59	2541	2509	2483	2483
query60	339	321	304	304
query61	133	123	156	123
query62	798	720	683	683
query63	226	186	184	184
query64	4380	998	660	660
query65	4284	4216	4282	4216
query66	1143	417	301	301
query67	15802	15694	15334	15334
query68	7905	898	530	530
query69	491	305	259	259
query70	1203	1111	1089	1089
query71	447	309	294	294
query72	5543	4722	4716	4716
query73	700	599	350	350
query74	9070	9124	8692	8692
query75	3761	3188	2673	2673
query76	3642	1188	780	780
query77	802	378	293	293
query78	10039	10159	9262	9262
query79	2037	794	580	580
query80	596	521	445	445
query81	466	259	223	223
query82	426	129	100	100
query83	275	261	235	235
query84	295	110	82	82
query85	765	352	366	352
query86	337	296	282	282
query87	4347	4575	4430	4430
query88	3159	2284	2283	2283
query89	406	328	283	283
query90	1917	228	225	225
query91	144	143	115	115
query92	75	63	63	63
query93	1124	953	585	585
query94	680	402	315	315
query95	377	294	283	283
query96	502	574	288	288
query97	2692	2739	2678	2678
query98	230	218	207	207
query99	1438	1399	1273	1273
Total cold run time: 271767 ms
Total hot run time: 185342 ms

@doris-robot
Copy link

doris-robot commented May 15, 2025

ClickBench: Total hot run time: 29.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 718db6e49ac78b41aa888244d92a82eed0fff72b, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.20	0.19
query4	1.60	0.18	0.20
query5	0.45	0.43	0.46
query6	1.19	0.66	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.52	0.54
query10	0.58	0.58	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	0.79	0.82	0.81
query15	0.88	0.88	0.85
query16	0.40	0.36	0.39
query17	1.04	1.04	1.07
query18	0.23	0.22	0.21
query19	1.96	1.79	1.81
query20	0.02	0.01	0.01
query21	15.41	0.89	0.54
query22	0.74	1.10	0.67
query23	14.98	1.38	0.63
query24	6.87	0.93	0.89
query25	0.48	0.15	0.09
query26	0.57	0.17	0.13
query27	0.06	0.06	0.04
query28	9.79	0.95	0.46
query29	12.53	4.03	3.39
query30	0.26	0.09	0.07
query31	2.83	0.61	0.38
query32	3.22	0.55	0.48
query33	2.99	3.10	3.08
query34	15.67	5.13	4.45
query35	4.52	4.53	4.49
query36	0.66	0.49	0.49
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.19 s
Total hot run time: 29.22 s

morningman
morningman reviewed May 15, 2025
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/analysis/CopyStmt.java
getOrigStmt() != null ? getOrigStmt().originStmt : "", copyFromParam.getFileColumns(),
copyFromParam.getColumnMappingList(), copyFromParam.getFileFilterExpr());
}
dataDescProperties.put(FileFormatProperties.PROP_COMPRESS_TYPE, copyIntoProperties.getCompression());
Copy link
Contributor

@morningman morningman May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some format properties in copyIntoProperties. I think we should remove them too. But may be in next PR.

@BePPPower
Copy link
Contributor Author

BePPPower commented May 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 16, 2025

TPC-H: Total hot run time: 34298 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c2bc920efa32b2a55fdb5f725bdc11ca8cbb078, data reload: false

------ Round 1 ----------------------------------
q1	26000	4981	4938	4938
q2	2075	275	182	182
q3	10394	1251	696	696
q4	10236	964	513	513
q5	7529	2317	2318	2317
q6	176	166	131	131
q7	904	736	606	606
q8	9310	1226	1033	1033
q9	6830	5082	5139	5082
q10	6850	2333	1920	1920
q11	488	291	271	271
q12	347	338	208	208
q13	17794	3659	3059	3059
q14	243	239	211	211
q15	544	485	491	485
q16	418	421	379	379
q17	608	856	372	372
q18	7456	7172	7210	7172
q19	1873	962	562	562
q20	340	334	235	235
q21	3998	3297	2950	2950
q22	1042	1016	976	976
Total cold run time: 115455 ms
Total hot run time: 34298 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5133	5054	5041	5041
q2	238	321	233	233
q3	2152	2616	2305	2305
q4	1310	1763	1357	1357
q5	4421	4364	4389	4364
q6	212	179	128	128
q7	2046	1908	1771	1771
q8	2580	2468	2439	2439
q9	7248	7245	6993	6993
q10	3014	3154	2720	2720
q11	579	516	487	487
q12	678	745	614	614
q13	3505	3791	3276	3276
q14	307	296	285	285
q15	513	471	482	471
q16	432	472	430	430
q17	1118	1496	1418	1418
q18	7742	7533	7445	7445
q19	781	849	1005	849
q20	1953	2059	1827	1827
q21	4729	4435	4335	4335
q22	1082	1080	1032	1032
Total cold run time: 51773 ms
Total hot run time: 49820 ms

@doris-robot
Copy link

doris-robot commented May 16, 2025

TPC-DS: Total hot run time: 193435 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c2bc920efa32b2a55fdb5f725bdc11ca8cbb078, data reload: false

query1	1384	1109	1068	1068
query2	6251	1903	1880	1880
query3	11002	4420	4379	4379
query4	55345	24369	23465	23465
query5	4999	518	471	471
query6	359	230	203	203
query7	4922	518	297	297
query8	321	245	236	236
query9	5890	2632	2662	2632
query10	438	331	261	261
query11	15142	15109	14868	14868
query12	157	115	105	105
query13	1034	543	412	412
query14	10166	6310	6305	6305
query15	215	200	190	190
query16	7169	663	485	485
query17	1106	737	600	600
query18	1632	418	345	345
query19	201	189	169	169
query20	129	134	118	118
query21	213	130	109	109
query22	4506	4333	4255	4255
query23	34470	33597	33554	33554
query24	6687	2468	2467	2467
query25	490	491	416	416
query26	681	277	157	157
query27	2304	530	352	352
query28	2930	2168	2169	2168
query29	588	585	441	441
query30	276	235	228	228
query31	859	885	805	805
query32	73	61	60	60
query33	466	373	320	320
query34	809	905	537	537
query35	815	865	761	761
query36	962	1018	908	908
query37	113	102	80	80
query38	4195	4372	4353	4353
query39	1549	1443	1465	1443
query40	222	133	107	107
query41	58	56	55	55
query42	132	112	118	112
query43	517	516	506	506
query44	1386	837	843	837
query45	185	177	167	167
query46	879	1049	672	672
query47	1859	1884	1828	1828
query48	411	431	339	339
query49	692	527	451	451
query50	686	729	431	431
query51	4243	4232	4189	4189
query52	118	113	101	101
query53	231	266	192	192
query54	591	577	516	516
query55	90	86	86	86
query56	314	316	314	314
query57	1208	1212	1134	1134
query58	286	275	275	275
query59	2824	2930	2746	2746
query60	345	369	340	340
query61	167	128	120	120
query62	736	751	687	687
query63	228	191	200	191
query64	1516	1043	712	712
query65	4307	4252	4279	4252
query66	793	402	306	306
query67	15825	15779	15351	15351
query68	7497	906	522	522
query69	550	328	272	272
query70	1222	1117	1100	1100
query71	526	332	296	296
query72	5877	4756	4846	4756
query73	1446	653	363	363
query74	8955	9200	8844	8844
query75	3779	3201	2729	2729
query76	4186	1196	790	790
query77	611	375	303	303
query78	10192	10144	9264	9264
query79	2536	839	593	593
query80	804	515	462	462
query81	483	258	220	220
query82	489	127	93	93
query83	330	249	235	235
query84	284	107	86	86
query85	780	354	314	314
query86	400	301	278	278
query87	4418	4475	4403	4403
query88	3588	2339	2331	2331
query89	401	322	284	284
query90	1754	215	217	215
query91	145	149	112	112
query92	77	61	56	56
query93	2261	941	568	568
query94	657	413	301	301
query95	366	294	289	289
query96	521	595	290	290
query97	2757	2751	2660	2660
query98	231	205	216	205
query99	1437	1415	1281	1281
Total cold run time: 301257 ms
Total hot run time: 193435 ms

@doris-robot
Copy link

doris-robot commented May 16, 2025

ClickBench: Total hot run time: 28.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c2bc920efa32b2a55fdb5f725bdc11ca8cbb078, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.11
query3	0.25	0.20	0.20
query4	1.60	0.19	0.21
query5	0.46	0.44	0.44
query6	1.16	0.66	0.66
query7	0.03	0.01	0.01
query8	0.04	0.04	0.03
query9	0.59	0.52	0.53
query10	0.57	0.56	0.57
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.60
query14	0.78	0.81	0.80
query15	0.88	0.84	0.87
query16	0.39	0.37	0.37
query17	1.08	1.00	1.05
query18	0.22	0.21	0.21
query19	1.96	1.83	1.91
query20	0.01	0.01	0.02
query21	15.41	0.91	0.55
query22	0.77	1.22	0.73
query23	14.80	1.38	0.61
query24	7.00	0.94	0.58
query25	0.48	0.22	0.16
query26	0.63	0.15	0.13
query27	0.05	0.06	0.05
query28	9.69	0.98	0.46
query29	12.95	3.98	3.33
query30	0.26	0.09	0.07
query31	2.82	0.58	0.39
query32	3.23	0.54	0.49
query33	3.08	3.02	3.04
query34	15.78	5.09	4.44
query35	4.52	4.54	4.47
query36	0.67	0.50	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.75 s
Total hot run time: 28.92 s

@BePPPower
Copy link
Contributor Author

BePPPower commented May 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 19, 2025

TPC-H: Total hot run time: 33601 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 797b36d0200bec6084de75e98f01a5a895a6d586, data reload: false

------ Round 1 ----------------------------------
q1	25660	5169	5037	5037
q2	2062	284	204	204
q3	10376	1246	672	672
q4	10217	1007	526	526
q5	7530	2404	2278	2278
q6	176	161	133	133
q7	909	742	617	617
q8	9317	1251	1054	1054
q9	6774	5130	5152	5130
q10	6886	2301	1913	1913
q11	492	288	264	264
q12	351	348	209	209
q13	17786	3671	3073	3073
q14	242	225	212	212
q15	529	498	497	497
q16	415	441	379	379
q17	579	849	367	367
q18	7461	7186	6946	6946
q19	1414	944	546	546
q20	340	323	212	212
q21	3862	3159	2344	2344
q22	1055	1027	988	988
Total cold run time: 114433 ms
Total hot run time: 33601 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5152	5004	5058	5004
q2	230	334	227	227
q3	2164	2632	2254	2254
q4	1355	1750	1371	1371
q5	4368	4348	4475	4348
q6	219	171	133	133
q7	1977	1952	1859	1859
q8	2644	2720	2613	2613
q9	7402	7292	7072	7072
q10	3103	3200	2765	2765
q11	575	549	486	486
q12	674	751	643	643
q13	3524	3907	3342	3342
q14	297	328	269	269
q15	525	457	478	457
q16	445	504	447	447
q17	1157	1570	1360	1360
q18	7639	7567	7300	7300
q19	821	812	808	808
q20	1979	1967	1851	1851
q21	4847	4498	4467	4467
q22	1109	1060	1013	1013
Total cold run time: 52206 ms
Total hot run time: 50089 ms

@doris-robot
Copy link

doris-robot commented May 19, 2025

TPC-DS: Total hot run time: 192319 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 797b36d0200bec6084de75e98f01a5a895a6d586, data reload: false

query1	1411	1063	1030	1030
query2	6091	1923	1875	1875
query3	11116	4565	4593	4565
query4	54689	25433	22853	22853
query5	5029	496	462	462
query6	347	217	206	206
query7	4913	516	304	304
query8	323	262	232	232
query9	5808	2627	2623	2623
query10	436	335	279	279
query11	15112	15000	14765	14765
query12	158	109	102	102
query13	1047	540	411	411
query14	10155	6368	6374	6368
query15	215	203	176	176
query16	7066	641	510	510
query17	1112	730	607	607
query18	1553	419	322	322
query19	211	208	175	175
query20	130	128	123	123
query21	251	125	109	109
query22	4393	4466	4204	4204
query23	34169	33438	33541	33438
query24	6622	2390	2432	2390
query25	453	467	428	428
query26	666	276	159	159
query27	2191	510	352	352
query28	3058	2152	2139	2139
query29	576	552	447	447
query30	291	229	186	186
query31	871	845	770	770
query32	69	70	68	68
query33	453	381	332	332
query34	776	864	551	551
query35	793	833	752	752
query36	962	987	904	904
query37	113	100	80	80
query38	4245	4269	4164	4164
query39	1540	1492	1501	1492
query40	210	118	102	102
query41	55	53	52	52
query42	130	123	118	118
query43	531	509	508	508
query44	1378	857	873	857
query45	187	178	165	165
query46	866	1028	664	664
query47	1821	1857	1772	1772
query48	402	453	362	362
query49	687	500	444	444
query50	678	682	407	407
query51	4235	4280	4252	4252
query52	114	118	98	98
query53	229	263	198	198
query54	608	597	545	545
query55	86	95	83	83
query56	319	295	294	294
query57	1195	1199	1129	1129
query58	266	251	253	251
query59	2728	2792	2745	2745
query60	346	334	311	311
query61	127	122	143	122
query62	722	730	679	679
query63	240	194	203	194
query64	1435	1015	670	670
query65	4311	4203	4239	4203
query66	714	411	295	295
query67	15731	15718	15481	15481
query68	7447	887	524	524
query69	548	304	273	273
query70	1230	1175	1141	1141
query71	502	318	298	298
query72	5700	4904	4963	4904
query73	1400	724	367	367
query74	9001	8930	8730	8730
query75	3770	3196	2644	2644
query76	4240	1192	757	757
query77	629	364	281	281
query78	9975	10116	9370	9370
query79	1709	816	591	591
query80	654	501	442	442
query81	487	252	217	217
query82	401	129	93	93
query83	379	256	234	234
query84	343	107	88	88
query85	761	350	302	302
query86	372	305	297	297
query87	4438	4408	4282	4282
query88	2962	2357	2334	2334
query89	397	315	282	282
query90	2038	212	202	202
query91	135	138	111	111
query92	70	63	56	56
query93	1187	916	564	564
query94	699	466	303	303
query95	359	293	279	279
query96	504	572	286	286
query97	2707	2732	2655	2655
query98	226	209	196	196
query99	1451	1399	1262	1262
Total cold run time: 296486 ms
Total hot run time: 192319 ms

@doris-robot
Copy link

doris-robot commented May 19, 2025

ClickBench: Total hot run time: 28.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 797b36d0200bec6084de75e98f01a5a895a6d586, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.19	0.20
query4	1.59	0.19	0.19
query5	0.44	0.45	0.44
query6	1.52	0.66	0.67
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.51	0.50
query10	0.60	0.58	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.61	0.60	0.61
query14	0.79	0.79	0.81
query15	0.86	0.83	0.86
query16	0.38	0.37	0.37
query17	1.04	1.00	1.06
query18	0.23	0.21	0.21
query19	1.93	1.81	1.80
query20	0.02	0.01	0.01
query21	15.39	0.90	0.55
query22	0.74	1.02	0.63
query23	15.17	1.36	0.56
query24	7.00	1.29	0.56
query25	0.51	0.30	0.07
query26	0.52	0.17	0.13
query27	0.06	0.05	0.04
query28	9.22	0.89	0.43
query29	12.52	4.05	3.40
query30	0.24	0.09	0.06
query31	2.84	0.60	0.39
query32	3.23	0.53	0.46
query33	3.08	3.01	3.08
query34	15.85	5.10	4.50
query35	4.56	4.57	4.59
query36	0.67	0.49	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.02	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 103.45 s
Total hot run time: 28.72 s

@BePPPower
Copy link
Contributor Author

BePPPower commented May 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 20, 2025

TPC-H: Total hot run time: 34431 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 329c614a6638ed30e68c9da7822e59ee74709da7, data reload: false

------ Round 1 ----------------------------------
q1	26194	5217	5074	5074
q2	2074	284	186	186
q3	10384	1212	713	713
q4	10233	1030	568	568
q5	7542	2299	2346	2299
q6	198	165	132	132
q7	909	764	617	617
q8	9323	1269	1121	1121
q9	6900	5116	5055	5055
q10	6882	2321	1928	1928
q11	546	302	295	295
q12	371	366	220	220
q13	17864	3821	3223	3223
q14	241	233	219	219
q15	541	504	497	497
q16	433	449	385	385
q17	607	866	389	389
q18	7590	7384	7349	7349
q19	1554	970	565	565
q20	338	329	215	215
q21	4100	3180	2394	2394
q22	1069	1003	987	987
Total cold run time: 115893 ms
Total hot run time: 34431 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5201	5289	5090	5090
q2	237	333	229	229
q3	2195	2677	2316	2316
q4	1412	1836	1401	1401
q5	4418	4371	4386	4371
q6	226	166	124	124
q7	2035	1957	1796	1796
q8	2609	2594	2625	2594
q9	7263	7096	7086	7086
q10	3010	3189	2744	2744
q11	576	519	489	489
q12	665	776	650	650
q13	3522	3896	3378	3378
q14	297	290	281	281
q15	524	473	466	466
q16	454	479	450	450
q17	1162	1560	1390	1390
q18	7835	7587	7417	7417
q19	828	912	1012	912
q20	1925	1973	1862	1862
q21	4710	4587	4458	4458
q22	1123	1056	997	997
Total cold run time: 52227 ms
Total hot run time: 50501 ms

@doris-robot
Copy link

doris-robot commented May 20, 2025

TPC-DS: Total hot run time: 192903 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 329c614a6638ed30e68c9da7822e59ee74709da7, data reload: false

query1	1429	1106	1065	1065
query2	6143	1828	1817	1817
query3	11021	4607	4467	4467
query4	52467	24670	23301	23301
query5	5243	532	466	466
query6	389	211	205	205
query7	5032	530	294	294
query8	338	263	237	237
query9	6083	2643	2604	2604
query10	445	352	263	263
query11	15202	15091	14784	14784
query12	167	114	104	104
query13	1113	535	420	420
query14	10109	6395	6292	6292
query15	204	204	180	180
query16	6967	653	480	480
query17	1095	772	608	608
query18	1555	411	334	334
query19	213	199	182	182
query20	135	126	129	126
query21	207	136	120	120
query22	4297	4411	4235	4235
query23	34460	33739	33859	33739
query24	6541	2439	2421	2421
query25	471	500	430	430
query26	744	272	156	156
query27	2380	510	344	344
query28	3417	2120	2112	2112
query29	597	555	475	475
query30	272	220	191	191
query31	867	879	786	786
query32	74	62	67	62
query33	459	371	330	330
query34	796	876	548	548
query35	806	830	754	754
query36	939	1006	915	915
query37	120	106	81	81
query38	4380	4346	4205	4205
query39	1508	1448	1479	1448
query40	214	124	111	111
query41	58	58	53	53
query42	121	111	108	108
query43	511	513	489	489
query44	1351	846	852	846
query45	186	173	170	170
query46	863	1030	653	653
query47	1831	1869	1798	1798
query48	402	449	345	345
query49	700	504	442	442
query50	704	724	426	426
query51	4269	4258	4251	4251
query52	113	117	110	110
query53	240	269	187	187
query54	612	590	537	537
query55	89	87	83	83
query56	316	334	297	297
query57	1163	1229	1168	1168
query58	268	279	259	259
query59	2664	2715	2563	2563
query60	332	343	322	322
query61	135	127	129	127
query62	761	758	647	647
query63	220	191	179	179
query64	1705	1054	683	683
query65	4298	4232	4255	4232
query66	753	399	308	308
query67	15769	15656	15523	15523
query68	7735	894	520	520
query69	552	309	275	275
query70	1213	1138	1124	1124
query71	492	341	294	294
query72	5932	4869	4951	4869
query73	1287	746	354	354
query74	9307	9130	8699	8699
query75	3913	3212	2713	2713
query76	4404	1207	781	781
query77	631	395	279	279
query78	10134	10094	9244	9244
query79	1961	840	600	600
query80	720	514	459	459
query81	473	251	223	223
query82	408	135	99	99
query83	363	254	245	245
query84	289	98	88	88
query85	795	356	320	320
query86	341	303	276	276
query87	4430	4438	4425	4425
query88	3016	2316	2317	2316
query89	413	400	278	278
query90	1971	218	213	213
query91	136	143	111	111
query92	72	61	57	57
query93	1174	965	578	578
query94	686	414	310	310
query95	374	296	299	296
query96	501	573	279	279
query97	2744	2719	2635	2635
query98	231	202	199	199
query99	1463	1402	1246	1246
Total cold run time: 297494 ms
Total hot run time: 192903 ms

@doris-robot
Copy link

doris-robot commented May 20, 2025

ClickBench: Total hot run time: 29.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 329c614a6638ed30e68c9da7822e59ee74709da7, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.11
query3	0.26	0.19	0.19
query4	1.58	0.20	0.10
query5	0.42	0.40	0.45
query6	1.19	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.59	0.53	0.52
query10	0.59	0.59	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.59
query14	0.78	0.82	0.81
query15	0.88	0.84	0.84
query16	0.39	0.37	0.39
query17	1.06	1.05	1.08
query18	0.22	0.22	0.21
query19	1.89	1.80	1.76
query20	0.02	0.01	0.02
query21	15.40	0.92	0.55
query22	0.75	1.15	0.76
query23	14.88	1.39	0.60
query24	7.46	1.00	0.96
query25	0.46	0.14	0.08
query26	0.67	0.17	0.13
query27	0.05	0.06	0.06
query28	8.95	0.84	0.46
query29	12.63	4.03	3.30
query30	0.25	0.10	0.07
query31	2.81	0.59	0.38
query32	3.25	0.55	0.48
query33	3.12	3.08	3.04
query34	15.85	5.07	4.51
query35	4.55	4.51	4.51
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.26 s
Total hot run time: 29.17 s

@BePPPower
Copy link
Contributor Author

BePPPower commented May 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 20, 2025

TPC-H: Total hot run time: 34195 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3610cb09587ec762e1c529327533df67d92ff5e, data reload: false

------ Round 1 ----------------------------------
q1	26227	5064	5015	5015
q2	2069	281	191	191
q3	10382	1238	725	725
q4	10241	1005	536	536
q5	7499	2913	2391	2391
q6	182	160	134	134
q7	895	737	616	616
q8	9314	1345	1123	1123
q9	6768	5142	5060	5060
q10	6826	2317	1932	1932
q11	468	286	276	276
q12	348	342	218	218
q13	17795	3720	3166	3166
q14	226	229	213	213
q15	535	503	496	496
q16	411	430	367	367
q17	594	855	374	374
q18	7695	7284	7256	7256
q19	1226	965	567	567
q20	353	341	223	223
q21	3794	2551	2361	2361
q22	1041	1010	955	955
Total cold run time: 114889 ms
Total hot run time: 34195 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5065	5068	5060	5060
q2	238	330	226	226
q3	2174	2651	2281	2281
q4	1368	1786	1334	1334
q5	4514	4444	4397	4397
q6	221	173	131	131
q7	1954	1920	1746	1746
q8	2555	2503	2525	2503
q9	7207	7110	7076	7076
q10	2954	3177	2745	2745
q11	567	496	532	496
q12	679	773	611	611
q13	3525	3887	3283	3283
q14	279	301	278	278
q15	531	490	497	490
q16	426	481	456	456
q17	1173	1626	1359	1359
q18	7935	7579	7378	7378
q19	856	839	823	823
q20	1966	2044	1885	1885
q21	4931	4449	4342	4342
q22	1115	994	993	993
Total cold run time: 52233 ms
Total hot run time: 49893 ms

@doris-robot
Copy link

doris-robot commented May 20, 2025

TPC-DS: Total hot run time: 186551 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3610cb09587ec762e1c529327533df67d92ff5e, data reload: false

query1	1005	485	500	485
query2	6581	1927	1938	1927
query3	6747	233	218	218
query4	26130	23658	23145	23145
query5	4308	603	461	461
query6	293	202	183	183
query7	4622	493	281	281
query8	279	234	244	234
query9	8629	2617	2619	2617
query10	486	341	276	276
query11	15346	15643	14836	14836
query12	168	114	106	106
query13	1663	522	396	396
query14	8890	6098	6343	6098
query15	220	196	169	169
query16	7139	634	482	482
query17	1187	712	560	560
query18	1962	389	296	296
query19	189	182	158	158
query20	120	119	121	119
query21	214	126	101	101
query22	4069	4079	4037	4037
query23	34019	33091	33306	33091
query24	8396	2421	2379	2379
query25	553	453	401	401
query26	1243	268	156	156
query27	2766	509	345	345
query28	4371	2117	2103	2103
query29	762	545	453	453
query30	287	218	190	190
query31	912	859	772	772
query32	74	63	65	63
query33	548	356	311	311
query34	800	846	529	529
query35	778	801	733	733
query36	974	961	878	878
query37	114	102	80	80
query38	4178	4152	4054	4054
query39	1459	1393	1405	1393
query40	215	130	118	118
query41	62	58	60	58
query42	127	114	112	112
query43	514	497	505	497
query44	1308	840	836	836
query45	183	174	170	170
query46	843	1006	638	638
query47	1791	1797	1728	1728
query48	407	419	320	320
query49	812	568	453	453
query50	655	679	457	457
query51	4097	4147	4071	4071
query52	111	103	97	97
query53	222	247	190	190
query54	594	570	519	519
query55	88	84	86	84
query56	314	308	301	301
query57	1124	1143	1083	1083
query58	273	278	255	255
query59	2700	2778	2735	2735
query60	332	332	309	309
query61	162	134	123	123
query62	793	760	655	655
query63	224	188	195	188
query64	4361	1008	680	680
query65	4285	4276	4258	4258
query66	1149	418	313	313
query67	15996	15418	15239	15239
query68	7778	874	524	524
query69	464	307	268	268
query70	1167	1121	1084	1084
query71	444	331	304	304
query72	5637	4890	4717	4717
query73	657	601	363	363
query74	8801	9184	9108	9108
query75	3677	3193	2724	2724
query76	3403	1193	763	763
query77	771	391	295	295
query78	9942	10082	9505	9505
query79	2242	785	574	574
query80	661	545	445	445
query81	483	253	230	230
query82	468	126	97	97
query83	253	255	232	232
query84	252	106	84	84
query85	791	356	311	311
query86	391	287	294	287
query87	4389	4442	4398	4398
query88	3766	2353	2343	2343
query89	395	326	283	283
query90	1892	207	209	207
query91	139	138	173	138
query92	77	63	58	58
query93	1926	947	577	577
query94	653	417	307	307
query95	375	285	281	281
query96	507	580	283	283
query97	2703	2702	2669	2669
query98	240	218	212	212
query99	1353	1370	1277	1277
Total cold run time: 272920 ms
Total hot run time: 186551 ms

@morningman morningman force-pushed the refactorBrokerLoad4 branch from ad81c8d to 3a0ecca Compare May 24, 2025 02:44
@morningman
Copy link
Contributor

morningman commented May 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 24, 2025

TPC-H: Total hot run time: 34076 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a0eccaf768b29fe21618fd8f6b0475d667ee690, data reload: false

------ Round 1 ----------------------------------
q1	27137	5029	5071	5029
q2	2093	281	191	191
q3	10390	1251	688	688
q4	10222	991	540	540
q5	7527	2398	2287	2287
q6	180	162	130	130
q7	883	739	615	615
q8	9329	1272	1114	1114
q9	6964	5129	5286	5129
q10	6851	2304	1891	1891
q11	487	286	300	286
q12	345	344	215	215
q13	17767	3703	3134	3134
q14	223	231	206	206
q15	525	503	490	490
q16	443	437	382	382
q17	652	865	403	403
q18	7539	7197	7267	7197
q19	1232	949	543	543
q20	337	347	229	229
q21	4023	3154	2384	2384
q22	1076	1019	993	993
Total cold run time: 116225 ms
Total hot run time: 34076 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5075	5125	5062	5062
q2	242	323	230	230
q3	2162	2612	2300	2300
q4	1362	1808	1459	1459
q5	4430	4419	4411	4411
q6	225	169	129	129
q7	2022	1951	1785	1785
q8	2551	2524	2573	2524
q9	7379	7260	7195	7195
q10	3043	3205	2759	2759
q11	582	521	508	508
q12	668	765	606	606
q13	3538	3963	3463	3463
q14	287	284	269	269
q15	513	494	475	475
q16	447	493	479	479
q17	1168	1578	1389	1389
q18	7830	7606	7381	7381
q19	866	851	898	851
q20	1917	1966	1868	1868
q21	4902	4478	4497	4478
q22	1147	1080	1060	1060
Total cold run time: 52356 ms
Total hot run time: 50681 ms

@doris-robot
Copy link

doris-robot commented May 24, 2025

TPC-DS: Total hot run time: 195464 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a0eccaf768b29fe21618fd8f6b0475d667ee690, data reload: false

query1	1439	1139	1099	1099
query2	6230	1840	1863	1840
query3	11089	4771	4531	4531
query4	25537	23738	23309	23309
query5	4953	650	466	466
query6	318	220	212	212
query7	3996	531	310	310
query8	308	256	250	250
query9	8554	2811	2802	2802
query10	514	334	290	290
query11	15277	15041	14827	14827
query12	167	108	111	108
query13	1550	554	420	420
query14	9298	6344	6349	6344
query15	212	189	173	173
query16	7607	618	483	483
query17	1150	739	611	611
query18	2018	401	308	308
query19	206	190	165	165
query20	128	120	121	120
query21	206	124	114	114
query22	4593	4475	4320	4320
query23	34927	33857	33839	33839
query24	8758	2476	2473	2473
query25	529	487	404	404
query26	1227	281	155	155
query27	2892	534	355	355
query28	4751	2312	2251	2251
query29	730	591	497	497
query30	278	224	199	199
query31	919	928	801	801
query32	71	69	67	67
query33	565	409	357	357
query34	865	906	569	569
query35	821	831	775	775
query36	1041	1046	962	962
query37	126	108	83	83
query38	4335	4311	4238	4238
query39	1567	1492	1478	1478
query40	227	131	125	125
query41	62	59	61	59
query42	134	113	116	113
query43	532	526	504	504
query44	1384	870	893	870
query45	189	173	170	170
query46	887	1053	681	681
query47	1864	1887	1805	1805
query48	435	466	343	343
query49	742	539	433	433
query50	687	725	433	433
query51	4269	4237	4323	4237
query52	112	113	108	108
query53	245	264	192	192
query54	638	607	553	553
query55	95	91	99	91
query56	311	342	344	342
query57	1209	1190	1152	1152
query58	287	271	300	271
query59	2826	2830	2760	2760
query60	350	329	330	329
query61	127	129	125	125
query62	793	771	679	679
query63	236	205	202	202
query64	4070	1059	695	695
query65	4434	4404	4357	4357
query66	1061	418	320	320
query67	16208	15874	15728	15728
query68	8729	939	566	566
query69	492	320	275	275
query70	1274	1148	1101	1101
query71	499	325	311	311
query72	5402	4774	4851	4774
query73	739	665	384	384
query74	9138	9198	8981	8981
query75	3880	3191	2696	2696
query76	3734	1207	764	764
query77	790	382	298	298
query78	10187	10321	9492	9492
query79	2454	810	594	594
query80	697	519	482	482
query81	478	257	226	226
query82	472	127	99	99
query83	286	267	242	242
query84	297	110	93	93
query85	798	365	319	319
query86	330	306	299	299
query87	4510	4491	4373	4373
query88	3409	2438	2472	2438
query89	421	319	297	297
query90	1913	215	214	214
query91	152	146	112	112
query92	78	59	57	57
query93	1243	987	627	627
query94	668	422	314	314
query95	381	299	296	296
query96	557	583	299	299
query97	2735	2794	2663	2663
query98	234	217	218	217
query99	1431	1388	1279	1279
Total cold run time: 282574 ms
Total hot run time: 195464 ms

@doris-robot
Copy link

doris-robot commented May 24, 2025

ClickBench: Total hot run time: 29.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a0eccaf768b29fe21618fd8f6b0475d667ee690, data reload: false

query1	0.05	0.04	0.03
query2	0.12	0.10	0.11
query3	0.26	0.19	0.19
query4	1.60	0.19	0.11
query5	0.43	0.41	0.42
query6	1.14	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.52	0.52
query10	0.58	0.58	0.57
query11	0.16	0.10	0.11
query12	0.15	0.12	0.11
query13	0.62	0.60	0.60
query14	0.80	0.82	0.82
query15	0.89	0.88	0.89
query16	0.38	0.38	0.38
query17	1.10	1.07	1.05
query18	0.23	0.21	0.21
query19	1.93	1.84	1.79
query20	0.02	0.01	0.01
query21	15.40	0.90	0.54
query22	0.78	1.31	1.00
query23	14.71	1.38	0.65
query24	7.51	0.88	0.52
query25	0.46	0.11	0.09
query26	0.64	0.18	0.14
query27	0.06	0.06	0.05
query28	9.02	0.96	0.46
query29	12.53	4.18	3.47
query30	0.26	0.10	0.07
query31	2.82	0.61	0.40
query32	3.24	0.54	0.48
query33	3.05	3.10	3.08
query34	15.81	5.14	4.53
query35	4.53	4.58	4.54
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.14
query41	0.09	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.1 s
Total hot run time: 29.41 s

@BePPPower
Copy link
Contributor Author

BePPPower commented May 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 26, 2025

TPC-H: Total hot run time: 34440 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb91ee4d35bfe0a9505a475c8ce8189f73b2e92f, data reload: false

------ Round 1 ----------------------------------
q1	26282	5077	5264	5077
q2	2071	282	184	184
q3	10401	1263	703	703
q4	10223	1021	517	517
q5	7530	2453	2317	2317
q6	181	165	135	135
q7	934	741	602	602
q8	9323	1252	1055	1055
q9	6847	5077	5132	5077
q10	6811	2289	1921	1921
q11	501	289	286	286
q12	332	342	207	207
q13	17783	3709	3137	3137
q14	231	224	213	213
q15	532	480	476	476
q16	410	434	372	372
q17	582	885	360	360
q18	7774	7160	7083	7083
q19	1222	975	563	563
q20	338	342	229	229
q21	3719	3134	2967	2967
q22	1054	1009	959	959
Total cold run time: 115081 ms
Total hot run time: 34440 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5131	5165	5153	5153
q2	242	323	227	227
q3	2236	2679	2323	2323
q4	1359	1761	1374	1374
q5	4539	4458	4361	4361
q6	215	171	129	129
q7	1975	1920	1748	1748
q8	2616	2417	2584	2417
q9	7087	7152	7055	7055
q10	3026	3210	2738	2738
q11	566	509	499	499
q12	686	769	602	602
q13	3467	3916	3299	3299
q14	277	309	288	288
q15	528	470	473	470
q16	429	501	438	438
q17	1159	1513	1400	1400
q18	7719	7445	7372	7372
q19	820	874	990	874
q20	1993	2012	1943	1943
q21	4997	4301	4254	4254
q22	1074	993	972	972
Total cold run time: 52141 ms
Total hot run time: 49936 ms

@doris-robot
Copy link

doris-robot commented May 26, 2025

TPC-DS: Total hot run time: 185738 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb91ee4d35bfe0a9505a475c8ce8189f73b2e92f, data reload: false

query1	1019	500	513	500
query2	6571	1807	1794	1794
query3	6767	230	221	221
query4	26576	23615	22932	22932
query5	4302	629	446	446
query6	296	200	207	200
query7	4621	492	296	296
query8	301	240	227	227
query9	8654	2609	2586	2586
query10	475	317	271	271
query11	15743	15392	14823	14823
query12	162	110	103	103
query13	1648	547	389	389
query14	8811	6268	6234	6234
query15	205	200	168	168
query16	7120	629	448	448
query17	1187	719	562	562
query18	1962	388	303	303
query19	191	190	177	177
query20	123	117	120	117
query21	208	123	111	111
query22	4050	4170	4033	4033
query23	34024	33224	33078	33078
query24	8408	2409	2357	2357
query25	526	451	399	399
query26	1228	276	165	165
query27	2728	506	346	346
query28	4354	2100	2089	2089
query29	781	595	458	458
query30	285	218	196	196
query31	935	857	775	775
query32	82	67	67	67
query33	601	400	327	327
query34	799	852	537	537
query35	788	810	739	739
query36	956	973	897	897
query37	113	103	83	83
query38	4114	4113	4133	4113
query39	1489	1423	1404	1404
query40	213	125	111	111
query41	63	60	60	60
query42	123	116	113	113
query43	515	504	494	494
query44	1299	843	830	830
query45	193	177	173	173
query46	869	1024	645	645
query47	1723	1801	1703	1703
query48	377	409	318	318
query49	787	524	442	442
query50	626	694	415	415
query51	4105	4181	4131	4131
query52	108	108	102	102
query53	225	248	184	184
query54	584	576	513	513
query55	87	93	86	86
query56	319	305	310	305
query57	1193	1142	1111	1111
query58	267	276	256	256
query59	2590	2610	2510	2510
query60	321	329	319	319
query61	128	141	134	134
query62	776	750	651	651
query63	227	183	181	181
query64	4325	1007	678	678
query65	4325	4249	4257	4249
query66	1146	406	311	311
query67	15923	15598	15426	15426
query68	7771	880	521	521
query69	478	313	271	271
query70	1185	1152	1085	1085
query71	454	335	301	301
query72	5559	4762	4666	4666
query73	666	578	354	354
query74	8790	9012	8695	8695
query75	3768	3179	2804	2804
query76	3514	1189	748	748
query77	776	390	289	289
query78	9923	10072	9311	9311
query79	2011	814	590	590
query80	582	559	447	447
query81	475	268	222	222
query82	418	127	99	99
query83	261	246	237	237
query84	243	113	96	96
query85	781	351	317	317
query86	332	302	299	299
query87	4392	4484	4356	4356
query88	3487	2330	2312	2312
query89	388	314	291	291
query90	1928	217	216	216
query91	144	145	113	113
query92	83	61	59	59
query93	1083	920	587	587
query94	674	413	293	293
query95	385	296	282	282
query96	490	563	286	286
query97	2744	2818	2689	2689
query98	242	206	207	206
query99	1650	1406	1274	1274
Total cold run time: 272317 ms
Total hot run time: 185738 ms

@doris-robot
Copy link

doris-robot commented May 26, 2025

ClickBench: Total hot run time: 29.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb91ee4d35bfe0a9505a475c8ce8189f73b2e92f, data reload: false

query1	0.04	0.04	0.03
query2	0.14	0.10	0.12
query3	0.26	0.20	0.20
query4	1.59	0.20	0.10
query5	0.43	0.40	0.43
query6	1.16	0.66	0.66
query7	0.02	0.02	0.01
query8	0.05	0.03	0.04
query9	0.58	0.52	0.51
query10	0.57	0.59	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	0.78	0.80	0.82
query15	0.88	0.85	0.86
query16	0.38	0.39	0.38
query17	1.04	1.07	1.02
query18	0.23	0.21	0.21
query19	1.89	1.84	1.85
query20	0.01	0.02	0.01
query21	15.41	0.88	0.56
query22	0.75	1.24	0.67
query23	14.91	1.41	0.63
query24	6.75	1.04	0.84
query25	0.49	0.27	0.10
query26	0.53	0.16	0.14
query27	0.06	0.06	0.05
query28	9.99	0.94	0.45
query29	12.53	3.97	3.38
query30	0.26	0.09	0.06
query31	2.83	0.60	0.38
query32	3.23	0.55	0.47
query33	3.08	3.10	3.12
query34	15.80	5.08	4.48
query35	4.50	4.53	4.50
query36	0.67	0.50	0.48
query37	0.09	0.07	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.25 s
Total hot run time: 29.15 s

@BePPPower
Copy link
Contributor Author

BePPPower commented May 27, 2025

run external

morningman
morningman approved these changes May 27, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented May 27, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 27, 2025

PR approved by anyone and no changes requested.

liaoxin01
liaoxin01 approved these changes May 28, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring approved these changes May 29, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 0b95a4d into apache:master May 29, 2025
27 of 28 checks passed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@BePPPower @koarz
[feat](refactor-param) refactor brokerLoad's code about fileformat (a…
4465506 
…pache#50882)

Issue Number:apache#50238

Problem Summary:

Previously, we refactored the code of the fileFormat attribute (apache#50225).
However, we only added the relevant code without modifying the business
code. This pull request modifies the code of the BrokerLoad feature that
is related to the fileformat.
CalvinKirs pushed a commit to CalvinKirs/incubator-doris that referenced this pull request Jun 27, 2025
@BePPPower @CalvinKirs
branch-3.1:[feat](refactor-param) refactor brokerLoad's code about fi…
1852669 
…leformat (apache#50882)

apache#50882

(cherry picked from commit 0b95a4d)
@CalvinKirs CalvinKirs mentioned this pull request Jun 27, 2025
Closed
morningman pushed a commit to morningman/doris that referenced this pull request Jul 12, 2025
@BePPPower @morningman
[feat](refactor-param) refactor brokerLoad's code about fileformat (a…
e4ab5e0 
…pache#50882)

Issue Number:apache#50238

Problem Summary:

Previously, we refactored the code of the fileFormat attribute (apache#50225).
However, we only added the relevant code without modifying the business
code. This pull request modifies the code of the BrokerLoad feature that
is related to the fileformat.
This was referenced Jul 12, 2025
Merged
Merged
morrySnow pushed a commit that referenced this pull request Jul 14, 2025
@morningman @BePPPower
branch-3.1: [feat](refactor-param) refactor brokerLoad's code about f…
b0f0573 
…ileformat #50882 #53159 (#53145)

bp #50882 #53159

---------

Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
morningman added a commit that referenced this pull request Jul 15, 2025
@morningman
[fix](csv) fix failed to get prop from connection context (#53159)
8bce836 
### What problem does this PR solve?

Related PR: #50882
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

@yiguolei yiguolei Awaiting requested review from yiguolei

@xiaokang xiaokang Awaiting requested review from xiaokang

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@BePPPower @hello-stephen @doris-robot @morningman @liaoxin01 @dataroaring @morrySnow