Skip to content

[Fix](outfile) Fix the timing of setting the _is_closed flag in Parquet/ORC writer #34669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
May 16, 2024
Merged

[Fix](outfile) Fix the timing of setting the _is_closed flag in Parquet/ORC writer #34669

morningman merged 2 commits into from
May 16, 2024

Conversation

@BePPPower
Copy link
Contributor

@BePPPower BePPPower commented May 10, 2024

Proposed changes

Issue Number: close #xxx

backport: #34668

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented May 10, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BePPPower
Copy link
Contributor Author

BePPPower commented May 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 10, 2024

TPC-H: Total hot run time: 50213 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae, data reload: false

------ Round 1 ----------------------------------
q1	17527	4397	4362	4362
q2	2042	152	143	143
q3	10294	1867	1919	1867
q4	10103	1277	1322	1277
q5	8559	3926	3918	3918
q6	242	126	128	126
q7	2069	1611	1608	1608
q8	9268	2741	2721	2721
q9	10608	11141	10537	10537
q10	8650	3554	3512	3512
q11	423	250	244	244
q12	473	309	301	301
q13	18359	3969	4048	3969
q14	351	328	316	316
q15	501	475	472	472
q16	678	570	601	570
q17	1154	978	986	978
q18	7233	6972	6971	6971
q19	1720	1565	1567	1565
q20	519	298	307	298
q21	4449	4162	4068	4068
q22	506	390	390	390
Total cold run time: 115728 ms
Total hot run time: 50213 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4334	4304	4301	4301
q2	320	219	224	219
q3	4193	4140	4174	4140
q4	2760	2747	2752	2747
q5	7214	7052	7128	7052
q6	237	121	118	118
q7	3285	2847	2871	2847
q8	4360	4467	4484	4467
q9	16910	16783	16682	16682
q10	4218	4310	4276	4276
q11	753	680	658	658
q12	1017	861	872	861
q13	4257	3747	3757	3747
q14	448	424	414	414
q15	508	468	461	461
q16	734	688	672	672
q17	3839	3855	3876	3855
q18	8764	8818	8788	8788
q19	1735	1716	1684	1684
q20	2343	2131	2088	2088
q21	8518	8521	8496	8496
q22	1059	948	926	926
Total cold run time: 81806 ms
Total hot run time: 79499 ms

@doris-robot
Copy link

doris-robot commented May 10, 2024

TeamCity be ut coverage result:
Function Coverage: 37.79% (8075/21367)
Line Coverage: 29.45% (65913/223838)
Region Coverage: 28.91% (33924/117338)
Branch Coverage: 24.77% (17407/70282)
Coverage Report: http://coverage.selectdb-in.cc/coverage/eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae_eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae/report/index.html

@doris-robot
Copy link

doris-robot commented May 10, 2024

TPC-DS: Total hot run time: 202089 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae, data reload: false

query1	922	388	373	373
query2	6520	2926	2786	2786
query3	6927	216	217	216
query4	20219	17972	17982	17972
query5	19737	6490	6462	6462
query6	288	215	225	215
query7	4146	312	312	312
query8	245	233	233	233
query9	3092	2698	2632	2632
query10	411	299	296	296
query11	11481	10637	10680	10637
query12	126	81	70	70
query13	5591	706	675	675
query14	17530	13117	13727	13117
query15	359	219	221	219
query16	6474	284	257	257
query17	1711	1470	869	869
query18	2314	426	412	412
query19	204	151	149	149
query20	81	77	80	77
query21	183	96	91	91
query22	5240	5072	5022	5022
query23	32464	31815	32009	31815
query24	7050	6556	6495	6495
query25	528	439	408	408
query26	528	164	162	162
query27	1884	298	297	297
query28	6210	2416	2353	2353
query29	2953	2707	2595	2595
query30	238	163	165	163
query31	920	749	733	733
query32	72	63	54	54
query33	392	265	246	246
query34	860	482	505	482
query35	1096	923	918	918
query36	1361	1135	1330	1135
query37	89	58	64	58
query38	3036	2896	2931	2896
query39	1378	1332	1340	1332
query40	197	95	93	93
query41	40	37	36	36
query42	83	80	88	80
query43	774	742	693	693
query44	1111	700	720	700
query45	244	228	233	228
query46	1229	980	992	980
query47	2028	1824	1647	1647
query48	1012	726	712	712
query49	627	384	376	376
query50	878	615	597	597
query51	4771	4719	4616	4616
query52	91	73	83	73
query53	434	329	325	325
query54	2666	2449	2461	2449
query55	99	80	92	80
query56	208	205	209	205
query57	1267	1057	1076	1057
query58	215	201	208	201
query59	4400	4263	3839	3839
query60	210	202	197	197
query61	85	81	89	81
query62	830	486	465	465
query63	476	341	333	333
query64	2567	1518	1306	1306
query65	3632	3535	3582	3535
query66	818	374	374	374
query67	16103	15254	15172	15172
query68	8208	657	635	635
query69	565	373	347	347
query70	1572	1342	1255	1255
query71	398	306	300	300
query72	6438	3419	3422	3419
query73	736	328	317	317
query74	6303	5859	5854	5854
query75	4543	3668	3781	3668
query76	4618	1170	1202	1170
query77	603	247	248	247
query78	12490	11672	11562	11562
query79	7909	641	636	636
query80	1938	384	387	384
query81	517	234	240	234
query82	1493	97	97	97
query83	184	130	128	128
query84	264	68	68	68
query85	1334	292	293	292
query86	356	296	297	296
query87	3267	3039	3012	3012
query88	5099	2369	2348	2348
query89	376	307	279	279
query90	1741	204	206	204
query91	166	151	135	135
query92	62	53	53	53
query93	4741	584	577	577
query94	869	196	192	192
query95	1107	1048	1026	1026
query96	658	339	335	335
query97	6460	6267	6388	6267
query98	199	178	175	175
query99	2951	893	903	893
Total cold run time: 310120 ms
Total hot run time: 202089 ms

@doris-robot
Copy link

doris-robot commented May 10, 2024

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.04
query4	1.80	0.06	0.07
query5	0.54	0.53	0.52
query6	1.25	0.61	0.67
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.47	0.48
query10	0.55	0.54	0.55
query11	0.12	0.08	0.09
query12	0.11	0.10	0.09
query13	0.62	0.62	0.62
query14	0.77	0.80	0.78
query15	0.78	0.76	0.76
query16	0.36	0.36	0.35
query17	1.01	0.99	1.01
query18	0.26	0.23	0.26
query19	1.85	1.85	1.79
query20	0.01	0.01	0.01
query21	15.48	0.61	0.55
query22	2.12	2.05	1.58
query23	16.98	1.09	0.99
query24	7.92	1.21	0.71
query25	0.37	0.09	0.07
query26	0.82	0.14	0.14
query27	0.04	0.04	0.03
query28	5.45	0.74	0.74
query29	12.74	2.36	2.37
query30	0.56	0.54	0.54
query31	2.82	0.38	0.38
query32	3.38	0.49	0.50
query33	3.04	3.08	3.04
query34	15.27	4.81	4.80
query35	4.84	4.82	4.84
query36	1.07	1.01	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.15
query41	0.06	0.02	0.02
query42	0.02	0.01	0.01
query43	0.03	0.02	0.01
Total cold run time: 104.23 s
Total hot run time: 30.51 s

@doris-robot
Copy link

doris-robot commented May 10, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@BePPPower
Copy link
Contributor Author

BePPPower commented May 13, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 13, 2024

TPC-H: Total hot run time: 49591 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae, data reload: false

------ Round 1 ----------------------------------
q1	17871	4418	4323	4323
q2	2035	159	147	147
q3	10473	1904	1954	1904
q4	10373	1263	1345	1263
q5	8394	3891	3902	3891
q6	242	123	125	123
q7	2031	1605	1595	1595
q8	9501	2717	2698	2698
q9	13571	10222	10103	10103
q10	8649	3515	3537	3515
q11	413	234	247	234
q12	471	311	310	310
q13	18347	3970	4057	3970
q14	363	324	336	324
q15	521	469	465	465
q16	684	564	563	563
q17	1127	938	938	938
q18	7281	6878	6882	6878
q19	1702	1621	1572	1572
q20	522	303	298	298
q21	4382	4190	4088	4088
q22	502	389	404	389
Total cold run time: 119455 ms
Total hot run time: 49591 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4332	4282	4318	4282
q2	318	227	216	216
q3	4131	4172	4129	4129
q4	2765	2728	2745	2728
q5	7091	6947	7152	6947
q6	239	123	124	123
q7	3197	2848	2802	2802
q8	4316	4395	4434	4395
q9	16800	16690	16683	16683
q10	4233	4253	4289	4253
q11	753	683	666	666
q12	1027	849	853	849
q13	6776	3778	3763	3763
q14	452	428	417	417
q15	509	469	456	456
q16	746	697	684	684
q17	3758	3823	3802	3802
q18	8820	8732	8726	8726
q19	1748	1676	1683	1676
q20	2354	2103	2150	2103
q21	8416	8467	8523	8467
q22	1050	954	921	921
Total cold run time: 83831 ms
Total hot run time: 79088 ms

@doris-robot
Copy link

doris-robot commented May 13, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8077/21368)
Line Coverage: 29.46% (65941/223864)
Region Coverage: 28.93% (33949/117353)
Branch Coverage: 24.78% (17421/70292)
Coverage Report: http://coverage.selectdb-in.cc/coverage/eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae_eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae/report/index.html

@doris-robot
Copy link

doris-robot commented May 13, 2024

TPC-DS: Total hot run time: 201921 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae, data reload: false

query1	939	383	373	373
query2	6534	2803	2670	2670
query3	6918	212	205	205
query4	20127	17888	17818	17818
query5	19740	6523	6532	6523
query6	291	217	220	217
query7	4153	299	303	299
query8	256	258	267	258
query9	3101	2668	2576	2576
query10	423	281	292	281
query11	11368	10758	10757	10757
query12	117	78	69	69
query13	5579	681	665	665
query14	17654	13128	13487	13128
query15	381	216	224	216
query16	6461	280	252	252
query17	1757	1463	875	875
query18	2301	399	401	399
query19	210	151	142	142
query20	81	82	80	80
query21	190	87	95	87
query22	5275	5131	5028	5028
query23	32468	31599	31883	31599
query24	6892	6466	6563	6466
query25	537	422	404	404
query26	526	158	157	157
query27	1884	295	288	288
query28	6203	2366	2304	2304
query29	2977	2793	2694	2694
query30	236	160	170	160
query31	906	746	714	714
query32	66	64	62	62
query33	394	262	252	252
query34	840	470	477	470
query35	1133	936	943	936
query36	1325	1190	1128	1128
query37	91	59	62	59
query38	3073	2956	2947	2947
query39	1351	1321	1317	1317
query40	195	90	95	90
query41	43	37	37	37
query42	85	80	85	80
query43	700	715	708	708
query44	1144	704	713	704
query45	236	230	224	224
query46	1234	986	949	949
query47	1848	1747	1859	1747
query48	1004	714	686	686
query49	623	378	361	361
query50	876	636	609	609
query51	4726	4682	4706	4682
query52	95	81	81	81
query53	449	314	313	313
query54	2647	2471	2459	2459
query55	81	82	81	81
query56	232	225	214	214
query57	1334	1130	1117	1117
query58	200	202	193	193
query59	3981	4164	3853	3853
query60	203	207	194	194
query61	89	83	84	83
query62	726	469	462	462
query63	480	333	332	332
query64	2454	1490	1366	1366
query65	3622	3554	3539	3539
query66	764	356	375	356
query67	15277	17129	14894	14894
query68	9405	660	655	655
query69	572	343	339	339
query70	1569	1630	1545	1545
query71	413	305	312	305
query72	6439	3392	3407	3392
query73	752	333	322	322
query74	6355	5998	5880	5880
query75	5345	3650	3603	3603
query76	5910	1169	1173	1169
query77	998	255	245	245
query78	12571	11626	11781	11626
query79	7194	648	621	621
query80	1087	383	386	383
query81	496	233	229	229
query82	1414	101	94	94
query83	173	129	129	129
query84	253	70	69	69
query85	843	292	292	292
query86	318	298	284	284
query87	3221	3006	2990	2990
query88	4479	2352	2340	2340
query89	394	286	288	286
query90	1961	202	201	201
query91	167	149	137	137
query92	59	51	51	51
query93	6314	600	556	556
query94	697	201	199	199
query95	1083	1056	1040	1040
query96	641	336	332	332
query97	6465	6291	6358	6291
query98	182	177	169	169
query99	2941	933	858	858
Total cold run time: 310817 ms
Total hot run time: 201921 ms

@doris-robot
Copy link

doris-robot commented May 13, 2024

ClickBench: Total hot run time: 31.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.04
query4	1.80	0.06	0.07
query5	0.53	0.53	0.53
query6	1.32	0.62	0.61
query7	0.01	0.00	0.01
query8	0.03	0.02	0.02
query9	0.53	0.48	0.46
query10	0.54	0.55	0.53
query11	0.12	0.09	0.08
query12	0.12	0.09	0.09
query13	0.61	0.60	0.61
query14	0.79	0.78	0.79
query15	0.78	0.75	0.76
query16	0.37	0.38	0.38
query17	1.01	1.02	0.99
query18	0.21	0.28	0.22
query19	1.91	1.78	1.83
query20	0.02	0.01	0.01
query21	15.51	0.55	0.54
query22	2.29	2.62	2.24
query23	17.21	0.98	1.10
query24	7.15	1.00	1.12
query25	0.38	0.12	0.05
query26	0.81	0.14	0.15
query27	0.03	0.03	0.04
query28	5.89	0.74	0.71
query29	12.60	2.14	2.32
query30	0.62	0.53	0.53
query31	2.81	0.38	0.37
query32	3.39	0.49	0.49
query33	3.03	3.11	3.07
query34	15.28	4.80	4.81
query35	4.86	4.83	4.85
query36	1.07	1.01	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.15	0.15
query41	0.07	0.01	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.01
Total cold run time: 104.37 s
Total hot run time: 31.16 s

@doris-robot
Copy link

doris-robot commented May 13, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit eb982630d5b8c76f71b6b09a3ee57f7e8914e8ae with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 14, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 14, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 14, 2024

PR approved by anyone and no changes requested.

@BePPPower
Copy link
Contributor Author

BePPPower commented May 15, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 15, 2024

TeamCity be ut coverage result:
Function Coverage: 37.79% (8075/21368)
Line Coverage: 29.45% (65934/223864)
Region Coverage: 28.92% (33941/117353)
Branch Coverage: 24.78% (17421/70292)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f8d629039e940d5ebe9737f26ef0d3f7ca3e305c_f8d629039e940d5ebe9737f26ef0d3f7ca3e305c/report/index.html

@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-H: Total hot run time: 49665 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f8d629039e940d5ebe9737f26ef0d3f7ca3e305c, data reload: false

------ Round 1 ----------------------------------
q1	17727	4390	4385	4385
q2	2046	158	150	150
q3	10330	1862	1920	1862
q4	10123	1268	1369	1268
q5	8485	3931	3979	3931
q6	235	127	127	127
q7	2019	1607	1611	1607
q8	9351	2736	2724	2724
q9	10515	10248	10411	10248
q10	8643	3512	3471	3471
q11	419	249	252	249
q12	469	305	300	300
q13	18343	3965	4036	3965
q14	350	335	328	328
q15	521	466	467	466
q16	664	576	569	569
q17	1149	987	941	941
q18	7246	6792	6896	6792
q19	1720	1594	1498	1498
q20	515	311	293	293
q21	4471	4152	4095	4095
q22	496	398	396	396
Total cold run time: 115837 ms
Total hot run time: 49665 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4329	4292	4296	4292
q2	324	252	224	224
q3	4185	4191	4124	4124
q4	2772	2772	2758	2758
q5	7117	7135	7117	7117
q6	236	122	120	120
q7	3250	2922	2895	2895
q8	4367	4463	4517	4463
q9	16929	16843	16684	16684
q10	4283	4271	4262	4262
q11	781	690	692	690
q12	1044	858	842	842
q13	7124	3760	3746	3746
q14	449	412	420	412
q15	522	456	466	456
q16	736	685	689	685
q17	3835	3864	3888	3864
q18	8881	8823	8867	8823
q19	1715	1691	1681	1681
q20	2419	2167	2108	2108
q21	8477	8476	8455	8455
q22	988	922	944	922
Total cold run time: 84763 ms
Total hot run time: 79623 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-DS: Total hot run time: 202881 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f8d629039e940d5ebe9737f26ef0d3f7ca3e305c, data reload: false

query1	929	386	374	374
query2	6559	2748	2615	2615
query3	6918	204	202	202
query4	21185	17994	17907	17907
query5	19726	6569	6492	6492
query6	290	220	217	217
query7	4154	301	303	301
query8	277	281	250	250
query9	3093	2662	2606	2606
query10	411	309	293	293
query11	11390	10648	10681	10648
query12	120	80	68	68
query13	5584	693	680	680
query14	17630	13589	14142	13589
query15	362	223	230	223
query16	6475	272	262	262
query17	1714	1444	876	876
query18	2329	428	402	402
query19	210	148	144	144
query20	79	76	79	76
query21	192	88	90	88
query22	5209	5027	5036	5027
query23	32517	32067	31903	31903
query24	7065	6591	6519	6519
query25	516	424	414	414
query26	627	163	161	161
query27	2039	292	294	292
query28	6195	2372	2322	2322
query29	2921	2758	2763	2758
query30	239	169	165	165
query31	911	744	742	742
query32	72	61	55	55
query33	387	254	237	237
query34	863	470	474	470
query35	1122	869	912	869
query36	1300	1131	1207	1131
query37	91	66	63	63
query38	3130	2936	2943	2936
query39	1392	1324	1326	1324
query40	245	99	91	91
query41	38	37	36	36
query42	86	85	83	83
query43	757	834	668	668
query44	1134	712	719	712
query45	248	229	232	229
query46	1213	948	973	948
query47	1852	1712	1827	1712
query48	1033	694	724	694
query49	648	365	385	365
query50	874	594	600	594
query51	4766	4656	4624	4624
query52	94	85	91	85
query53	448	319	311	311
query54	2675	2441	2482	2441
query55	87	82	87	82
query56	235	221	224	221
query57	1154	1135	1126	1126
query58	212	199	204	199
query59	4694	3975	3784	3784
query60	204	199	220	199
query61	90	82	82	82
query62	852	559	429	429
query63	474	332	336	332
query64	2412	1507	1482	1482
query65	3635	3568	3572	3568
query66	813	378	375	375
query67	15237	15039	15748	15039
query68	12340	646	663	646
query69	577	331	334	331
query70	1961	1549	1429	1429
query71	416	308	306	306
query72	6624	3458	3405	3405
query73	2354	342	325	325
query74	6370	5847	5836	5836
query75	5479	3694	3604	3604
query76	6955	1181	1220	1181
query77	1180	252	254	252
query78	12682	12367	11698	11698
query79	9799	672	666	666
query80	677	392	388	388
query81	463	236	237	236
query82	604	98	100	98
query83	174	131	134	131
query84	260	69	69	69
query85	795	295	316	295
query86	327	291	285	285
query87	3291	3024	3060	3024
query88	4136	2364	2350	2350
query89	448	299	291	291
query90	1930	211	207	207
query91	183	149	135	135
query92	57	53	53	53
query93	5808	605	525	525
query94	726	213	205	205
query95	1138	1047	1043	1043
query96	654	331	330	330
query97	6496	6332	6357	6332
query98	186	179	156	156
query99	2942	837	950	837
Total cold run time: 320465 ms
Total hot run time: 202881 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

ClickBench: Total hot run time: 30.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f8d629039e940d5ebe9737f26ef0d3f7ca3e305c, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.26	0.06	0.07
query4	1.75	0.08	0.08
query5	0.53	0.52	0.51
query6	1.32	0.63	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.50	0.48
query10	0.54	0.54	0.54
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.61	0.61	0.62
query14	0.79	0.78	0.78
query15	0.77	0.77	0.78
query16	0.37	0.36	0.38
query17	1.02	0.97	1.03
query18	0.22	0.26	0.24
query19	1.92	1.85	1.82
query20	0.02	0.01	0.01
query21	15.48	0.54	0.53
query22	2.36	2.34	1.28
query23	16.72	1.10	0.99
query24	5.68	0.99	1.19
query25	0.40	0.06	0.04
query26	0.67	0.15	0.15
query27	0.05	0.03	0.03
query28	7.29	0.77	0.71
query29	12.64	2.38	2.33
query30	0.56	0.53	0.53
query31	2.80	0.38	0.37
query32	3.37	0.49	0.49
query33	3.06	3.08	3.06
query34	15.23	4.79	4.77
query35	4.85	4.82	4.84
query36	1.05	1.01	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 103.61 s
Total hot run time: 30.43 s

@doris-robot
Copy link

doris-robot commented May 15, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f8d629039e940d5ebe9737f26ef0d3f7ca3e305c with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@morningman morningman merged commit ccf8be8 into apache:branch-2.0 May 16, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@BePPPower
[Fix](outfile) Fix the timing of setting the _is_closed flag in Parqu…
01534c2 
…et/ORC writer (apache#34669)

backport: apache#34668
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@BePPPower @doris-robot @morningman