Skip to content

Conversation

@zxealous
Copy link
Contributor

@zxealous zxealous commented Feb 19, 2024

Proposed changes

Issue Number: close #xxx

cherry-pick #30703
be/src/io/fs/s3_file_writer.cpp has some diff, but the logic is the same.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zxealous
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (8035/21247)
Line Coverage: 29.48% (65497/222155)
Region Coverage: 28.96% (33703/116394)
Branch Coverage: 24.81% (17299/69720)
Coverage Report: http://coverage.selectdb-in.cc/coverage/317e135e43f9687c32b2fc436fe236dfb19d824e_317e135e43f9687c32b2fc436fe236dfb19d824e/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 50378 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 317e135e43f9687c32b2fc436fe236dfb19d824e, data reload: false

------ Round 1 ----------------------------------
q1	18054	4410	4304	4304
q2	2049	147	149	147
q3	10450	1900	1965	1900
q4	11095	1266	1283	1266
q5	8501	3941	4109	3941
q6	230	121	123	121
q7	2066	1600	1599	1599
q8	9518	2721	2715	2715
q9	13770	10551	10492	10492
q10	8876	3693	3664	3664
q11	432	268	255	255
q12	473	306	303	303
q13	19101	4210	4175	4175
q14	362	334	333	333
q15	521	466	462	462
q16	712	601	585	585
q17	1139	949	908	908
q18	7381	6894	6935	6894
q19	2119	1542	1498	1498
q20	521	313	323	313
q21	4398	4119	4140	4119
q22	508	384	400	384
Total cold run time: 122276 ms
Total hot run time: 50378 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4293	4281	4252	4252
q2	317	220	228	220
q3	4158	4137	4105	4105
q4	2751	2725	2754	2725
q5	7176	7158	7156	7156
q6	232	117	117	117
q7	3223	2831	2790	2790
q8	4351	4440	4496	4440
q9	17002	16882	16762	16762
q10	4217	4206	4228	4206
q11	757	682	669	669
q12	1018	829	842	829
q13	6596	3758	3729	3729
q14	450	434	415	415
q15	506	441	460	441
q16	748	705	695	695
q17	3829	3873	3789	3789
q18	8787	8677	8817	8677
q19	1705	1623	1696	1623
q20	2403	2098	2096	2096
q21	8441	8423	8436	8423
q22	1009	990	946	946
Total cold run time: 83969 ms
Total hot run time: 79105 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 236376 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 317e135e43f9687c32b2fc436fe236dfb19d824e, data reload: false

query1	918	382	373	373
query2	6522	2229	2225	2225
query3	6918	198	195	195
query4	20750	18313	17793	17793
query5	19712	6532	6498	6498
query6	293	211	236	211
query7	4336	308	298	298
query8	258	223	221	221
query9	3125	2658	2584	2584
query10	425	273	288	273
query11	11296	10574	10629	10574
query12	118	79	75	75
query13	5577	646	614	614
query14	17955	13360	13425	13360
query15	375	235	227	227
query16	6399	259	257	257
query17	1530	1473	855	855
query18	2287	402	402	402
query19	196	139	147	139
query20	81	81	76	76
query21	184	95	93	93
query22	5086	5202	4952	4952
query23	32343	31869	31857	31857
query24	6906	6645	6660	6645
query25	517	427	423	423
query26	466	167	157	157
query27	1647	300	296	296
query28	6032	2185	2165	2165
query29	2993	2714	2788	2714
query30	249	160	160	160
query31	873	718	719	718
query32	65	57	57	57
query33	394	253	239	239
query34	853	463	478	463
query35	1090	905	953	905
query36	1681	1548	1404	1404
query37	87	58	58	58
query38	3096	2900	2907	2900
query39	1337	1320	1317	1317
query40	205	91	93	91
query41	35	32	35	32
query42	92	81	90	81
query43	678	575	693	575
query44	1128	706	711	706
query45	237	220	231	220
query46	1226	965	957	957
query47	1815	1844	1937	1844
query48	974	692	662	662
query49	597	368	368	368
query50	863	630	621	621
query51	5586	5450	5477	5450
query52	99	81	91	81
query53	439	316	320	316
query54	2655	2454	2451	2451
query55	96	80	83	80
query56	213	206	174	174
query57	1183	1112	1083	1083
query58	204	200	203	200
query59	3461	3482	3175	3175
query60	190	184	208	184
query61	81	83	79	79
query62	879	501	485	485
query63	468	321	334	321
query64	2188	1436	1344	1344
query65	3616	3551	3861	3551
query66	743	365	365	365
query67	16979	15276	17187	15276
query68	8841	635	659	635
query69	559	345	344	344
query70	2109	1553	1802	1553
query71	422	314	310	310
query72	6472	3386	3433	3386
query73	719	318	311	311
query74	6304	5812	5798	5798
query75	5410	3775	3611	3611
query76	5545	1164	1202	1164
query77	922	241	248	241
query78	32712	51843	45832	45832
query79	15031	617	652	617
query80	4953	374	377	374
query81	564	229	230	229
query82	775	96	95	95
query83	326	136	129	129
query84	258	68	68	68
query85	2459	283	269	269
query86	480	373	374	373
query87	3306	3001	2990	2990
query88	6361	2314	2322	2314
query89	388	275	295	275
query90	2486	206	216	206
query91	155	113	127	113
query92	68	53	55	53
query93	4607	556	552	552
query94	1806	203	203	203
query95	1099	1056	1067	1056
query96	652	334	330	330
query97	6463	6279	6360	6279
query98	199	189	172	172
query99	3907	995	949	949
Total cold run time: 348258 ms
Total hot run time: 236376 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 317e135e43f9687c32b2fc436fe236dfb19d824e, data reload: false

query1	0.03	0.02	0.01
query2	0.06	0.02	0.02
query3	0.25	0.05	0.04
query4	1.83	0.07	0.06
query5	0.53	0.53	0.52
query6	1.22	0.61	0.63
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.54	0.47	0.47
query10	0.54	0.54	0.53
query11	0.11	0.09	0.08
query12	0.12	0.09	0.09
query13	0.61	0.61	0.61
query14	0.78	0.78	0.78
query15	0.78	0.75	0.77
query16	0.36	0.35	0.36
query17	0.99	1.02	0.99
query18	0.24	0.25	0.24
query19	1.94	1.82	1.83
query20	0.01	0.01	0.02
query21	15.48	0.55	0.55
query22	1.87	1.83	1.95
query23	17.16	1.03	1.04
query24	7.36	1.05	3.06
query25	1.57	0.12	0.11
query26	0.47	0.15	0.13
query27	0.11	0.11	0.11
query28	4.57	0.73	0.72
query29	12.67	2.36	2.34
query30	0.62	0.54	0.53
query31	2.80	0.39	0.40
query32	3.43	0.49	0.50
query33	3.08	3.07	3.08
query34	15.25	4.79	4.79
query35	4.87	4.82	4.82
query36	1.04	1.01	1.01
query37	0.05	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.01
Total cold run time: 103.72 s
Total hot run time: 31.22 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 317e135e43f9687c32b2fc436fe236dfb19d824e with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.0 seconds inserted 10000000 Rows, about 476K ops/s

@morningman morningman force-pushed the fix-outfile-empty-data-2.0 branch from 317e135 to ca29622 Compare February 19, 2024 09:56
@morningman
Copy link
Contributor

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.83% (8037/21247)
Line Coverage: 29.49% (65520/222155)
Region Coverage: 28.97% (33715/116394)
Branch Coverage: 24.82% (17307/69720)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ca29622f5b0f2ff13ac8e24d3819ba0384e1b5a5_ca29622f5b0f2ff13ac8e24d3819ba0384e1b5a5/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 49907 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca29622f5b0f2ff13ac8e24d3819ba0384e1b5a5, data reload: false

------ Round 1 ----------------------------------
q1	17853	4380	4359	4359
q2	2051	152	144	144
q3	10338	1856	1928	1856
q4	10100	1292	1337	1292
q5	8427	3978	3992	3978
q6	225	122	123	122
q7	2046	1618	1624	1618
q8	9301	2716	2715	2715
q9	10959	10388	10479	10388
q10	8607	3509	3505	3505
q11	414	251	240	240
q12	464	301	301	301
q13	18344	3967	4090	3967
q14	353	331	320	320
q15	501	453	457	453
q16	702	594	595	594
q17	1119	973	964	964
q18	7236	6975	6822	6822
q19	1643	1564	1491	1491
q20	502	315	319	315
q21	4453	4093	4075	4075
q22	515	400	388	388
Total cold run time: 116153 ms
Total hot run time: 49907 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4333	4313	4310	4310
q2	315	222	226	222
q3	4128	4182	4146	4146
q4	2750	2730	2764	2730
q5	7358	7181	7223	7181
q6	234	118	120	118
q7	3229	2860	2872	2860
q8	4349	4463	4474	4463
q9	17109	17020	17013	17013
q10	4226	4263	4226	4226
q11	739	693	686	686
q12	1046	861	872	861
q13	6923	3758	3784	3758
q14	448	416	412	412
q15	506	452	464	452
q16	756	703	717	703
q17	3869	3849	3825	3825
q18	8810	8777	8862	8777
q19	1709	1684	1639	1639
q20	2395	2150	2109	2109
q21	8501	8549	8463	8463
q22	1087	960	997	960
Total cold run time: 84820 ms
Total hot run time: 79914 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 236920 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca29622f5b0f2ff13ac8e24d3819ba0384e1b5a5, data reload: false

query1	914	396	386	386
query2	6534	2282	2067	2067
query3	6915	198	199	198
query4	19924	18018	18051	18018
query5	19804	6569	6544	6544
query6	289	211	228	211
query7	4195	286	292	286
query8	268	235	266	235
query9	3083	2664	2587	2587
query10	414	280	309	280
query11	11306	10669	10687	10669
query12	125	77	72	72
query13	5584	637	626	626
query14	18187	13524	13497	13497
query15	379	240	231	231
query16	6416	263	263	263
query17	1573	1470	857	857
query18	2282	405	391	391
query19	202	149	146	146
query20	80	77	79	77
query21	188	94	90	90
query22	5261	5148	5067	5067
query23	32706	31973	31893	31893
query24	6866	6527	6501	6501
query25	532	419	408	408
query26	539	162	165	162
query27	1880	285	285	285
query28	6087	2179	2181	2179
query29	2904	2780	2775	2775
query30	239	164	160	160
query31	886	712	751	712
query32	68	52	56	52
query33	392	252	231	231
query34	854	452	504	452
query35	1126	850	931	850
query36	1556	1332	1454	1332
query37	87	61	59	59
query38	3090	2932	2907	2907
query39	1356	1326	1308	1308
query40	196	89	90	89
query41	34	36	33	33
query42	92	89	88	88
query43	628	708	724	708
query44	1100	719	715	715
query45	235	233	226	226
query46	1236	963	958	958
query47	1931	1742	1955	1742
query48	971	672	648	648
query49	613	355	361	355
query50	869	593	639	593
query51	5577	5454	5421	5421
query52	80	85	73	73
query53	437	316	304	304
query54	2659	2439	2463	2439
query55	89	75	93	75
query56	212	203	203	203
query57	1197	1114	1080	1080
query58	208	213	198	198
query59	3591	3258	3332	3258
query60	206	192	195	192
query61	80	82	80	80
query62	817	474	477	474
query63	471	327	329	327
query64	2309	1479	1470	1470
query65	3649	3580	3596	3580
query66	766	368	363	363
query67	17822	15844	15169	15169
query68	9520	651	664	651
query69	570	348	350	348
query70	1894	1784	1740	1740
query71	418	305	302	302
query72	6497	3444	3431	3431
query73	721	323	314	314
query74	6372	5877	5874	5874
query75	5389	3692	3792	3692
query76	5877	1126	1216	1126
query77	1017	251	256	251
query78	33849	51470	45679	45679
query79	16795	657	630	630
query80	5515	384	380	380
query81	563	228	235	228
query82	1344	97	96	96
query83	418	138	125	125
query84	263	67	68	67
query85	2638	280	266	266
query86	475	377	375	375
query87	3284	2986	2990	2986
query88	7242	2307	2299	2299
query89	493	300	298	298
query90	2512	213	210	210
query91	158	116	114	114
query92	62	49	47	47
query93	7343	536	575	536
query94	1854	213	206	206
query95	1112	1072	1063	1063
query96	654	338	320	320
query97	6409	6296	6465	6296
query98	180	176	172	172
query99	3957	873	887	873
Total cold run time: 358561 ms
Total hot run time: 236920 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca29622f5b0f2ff13ac8e24d3819ba0384e1b5a5, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.04	0.05
query4	1.79	0.06	0.07
query5	0.54	0.53	0.52
query6	1.23	0.59	0.61
query7	0.02	0.01	0.02
query8	0.04	0.02	0.02
query9	0.52	0.48	0.49
query10	0.53	0.54	0.53
query11	0.11	0.08	0.09
query12	0.11	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.78	0.80
query15	0.78	0.75	0.76
query16	0.36	0.36	0.36
query17	1.00	1.03	1.00
query18	0.22	0.25	0.26
query19	1.90	1.86	1.87
query20	0.01	0.01	0.01
query21	15.45	0.56	0.56
query22	2.10	2.18	1.10
query23	17.27	0.96	0.96
query24	8.07	1.73	1.05
query25	1.58	0.13	0.10
query26	0.45	0.14	0.14
query27	0.11	0.10	0.11
query28	5.05	0.75	0.76
query29	12.58	2.39	2.41
query30	0.60	0.52	0.51
query31	2.80	0.38	0.39
query32	3.37	0.49	0.49
query33	3.08	3.07	3.11
query34	15.27	4.80	4.84
query35	4.86	4.85	4.85
query36	1.07	1.03	1.03
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.17	0.13	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 105 s
Total hot run time: 30.6 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ca29622f5b0f2ff13ac8e24d3819ba0384e1b5a5 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@xiaokang xiaokang merged commit 8952d6a into apache:branch-2.0 Feb 19, 2024
@zxealous zxealous deleted the fix-outfile-empty-data-2.0 branch April 28, 2024 09:30
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants