Skip to content

[fix](file-writer) avoid empty file for segment writer #31355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Feb 25, 2024
Merged

[fix](file-writer) avoid empty file for segment writer #31355

morningman merged 2 commits into from
Feb 25, 2024

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Feb 23, 2024

bp #31169

@doris-robot
Copy link

doris-robot commented Feb 23, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

github-actions[bot]
github-actions bot reviewed Feb 23, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/io/fs/file_writer_options.h
@@ -26,6 +26,8 @@ struct FileWriterOptions {
bool is_cold_data = false;
bool sync_file_data = true; // Whether flush data into storage system
int64_t file_cache_expiration = 0; // Absolute time
Copy link
Contributor

@github-actions github-actions bot Feb 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: unknown type name 'int64_t' [clang-diagnostic-error]

    int64_t file_cache_expiration = 0; // Absolute time
    ^

@morningman
Copy link
Contributor Author

morningman commented Feb 23, 2024

run buildall

@morningman
Copy link
Contributor Author

morningman commented Feb 23, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Feb 23, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8035/21255)
Line Coverage: 29.48% (65533/222267)
Region Coverage: 28.94% (33709/116471)
Branch Coverage: 24.80% (17301/69766)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c3748fd7ff8c75d51657280656436fa9158cc74a_c3748fd7ff8c75d51657280656436fa9158cc74a/report/index.html

@doris-robot
Copy link

doris-robot commented Feb 23, 2024

TPC-H: Total hot run time: 50125 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3748fd7ff8c75d51657280656436fa9158cc74a, data reload: false

------ Round 1 ----------------------------------
q1	18044	4381	4345	4345
q2	2032	163	150	150
q3	10261	1903	1923	1903
q4	10127	1243	1347	1243
q5	8472	4028	4048	4028
q6	231	129	122	122
q7	2030	1628	1613	1613
q8	9298	2713	2748	2713
q9	11053	10632	10464	10464
q10	8680	3510	3516	3510
q11	437	239	239	239
q12	462	295	307	295
q13	18348	3975	3991	3975
q14	341	328	323	323
q15	512	447	458	447
q16	703	591	594	591
q17	1141	998	958	958
q18	7414	6925	6828	6828
q19	1664	1606	1557	1557
q20	509	306	317	306
q21	4497	4225	4117	4117
q22	512	398	400	398
Total cold run time: 116768 ms
Total hot run time: 50125 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4293	4313	4284	4284
q2	322	223	213	213
q3	4163	4141	4152	4141
q4	2794	2735	2747	2735
q5	7306	7194	7204	7194
q6	241	116	116	116
q7	3241	2852	2853	2852
q8	4364	4471	4513	4471
q9	17114	17023	17077	17023
q10	4244	4268	4304	4268
q11	764	678	690	678
q12	1020	855	848	848
q13	7668	3707	3728	3707
q14	471	427	415	415
q15	504	463	446	446
q16	765	718	698	698
q17	3866	3848	3887	3848
q18	8843	8752	9004	8752
q19	1723	1734	1679	1679
q20	2382	2187	2107	2107
q21	8561	8474	8601	8474
q22	1048	962	946	946
Total cold run time: 85697 ms
Total hot run time: 79895 ms

@doris-robot
Copy link

doris-robot commented Feb 23, 2024

TPC-DS: Total hot run time: 202047 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3748fd7ff8c75d51657280656436fa9158cc74a, data reload: false

query1	918	397	387	387
query2	6537	2224	1976	1976
query3	6940	212	207	207
query4	20294	17967	18010	17967
query5	19716	6526	6556	6526
query6	296	212	235	212
query7	4151	298	294	294
query8	250	238	227	227
query9	3135	2728	2655	2655
query10	424	291	309	291
query11	11300	10794	10768	10768
query12	121	79	72	72
query13	5564	683	621	621
query14	18034	13718	13625	13625
query15	362	250	228	228
query16	6437	267	264	264
query17	1713	1462	858	858
query18	2323	398	414	398
query19	200	145	141	141
query20	77	80	74	74
query21	186	101	93	93
query22	5282	5125	5033	5033
query23	32645	31919	32102	31919
query24	6868	6528	6490	6490
query25	510	425	411	411
query26	515	161	153	153
query27	1788	293	296	293
query28	6077	2249	2215	2215
query29	2840	2695	2782	2695
query30	240	159	166	159
query31	899	699	768	699
query32	58	60	58	58
query33	397	257	259	257
query34	863	465	487	465
query35	1128	933	880	880
query36	1571	1533	1360	1360
query37	90	61	58	58
query38	3094	2973	2899	2899
query39	1389	1323	1318	1318
query40	197	95	91	91
query41	37	32	36	32
query42	95	86	83	83
query43	590	568	561	561
query44	1106	698	710	698
query45	240	230	229	229
query46	1254	1011	970	970
query47	1740	1793	1831	1793
query48	966	699	677	677
query49	610	368	365	365
query50	863	607	680	607
query51	4760	4653	4695	4653
query52	87	78	71	71
query53	442	319	310	310
query54	2677	2458	2460	2458
query55	95	81	84	81
query56	220	196	205	196
query57	1177	1058	1059	1058
query58	214	205	175	175
query59	3385	3434	3153	3153
query60	216	189	189	189
query61	85	86	89	86
query62	887	522	498	498
query63	483	332	329	329
query64	2445	1440	1330	1330
query65	3626	3547	3567	3547
query66	760	376	360	360
query67	15841	15009	15288	15009
query68	8586	672	654	654
query69	572	341	351	341
query70	1788	1800	1667	1667
query71	391	290	308	290
query72	6554	3432	3425	3425
query73	728	321	318	318
query74	6287	5962	5836	5836
query75	5182	3708	3740	3708
query76	5193	1156	1244	1156
query77	884	255	244	244
query78	12633	11819	11785	11785
query79	7032	633	632	632
query80	983	395	390	390
query81	493	231	229	229
query82	1551	101	96	96
query83	160	130	129	129
query84	254	69	70	69
query85	846	282	277	277
query86	392	369	420	369
query87	3192	3035	2996	2996
query88	4842	2362	2338	2338
query89	371	282	276	276
query90	1961	209	199	199
query91	158	117	118	117
query92	54	52	50	50
query93	5211	610	585	585
query94	710	207	204	204
query95	1152	1077	1092	1077
query96	651	330	328	328
query97	6431	6371	6372	6371
query98	179	181	173	173
query99	2980	914	863	863
Total cold run time: 308731 ms
Total hot run time: 202047 ms

@doris-robot
Copy link

doris-robot commented Feb 23, 2024

ClickBench: Total hot run time: 31.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3748fd7ff8c75d51657280656436fa9158cc74a, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.02	0.02
query3	0.24	0.05	0.04
query4	1.81	0.06	0.06
query5	0.53	0.52	0.52
query6	1.23	0.61	0.62
query7	0.02	0.01	0.00
query8	0.03	0.03	0.03
query9	0.52	0.47	0.47
query10	0.54	0.55	0.54
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.62	0.62	0.61
query14	0.78	0.79	0.79
query15	0.79	0.77	0.76
query16	0.38	0.37	0.37
query17	1.03	0.99	1.00
query18	0.24	0.26	0.24
query19	1.93	1.77	1.90
query20	0.01	0.01	0.01
query21	15.46	0.58	0.58
query22	1.81	2.12	1.34
query23	16.48	1.09	0.94
query24	6.40	3.65	1.69
query25	1.54	0.12	0.11
query26	0.44	0.15	0.14
query27	0.11	0.11	0.11
query28	4.18	0.78	0.78
query29	12.76	2.18	2.35
query30	0.58	0.53	0.51
query31	2.80	0.38	0.38
query32	3.39	0.50	0.49
query33	3.07	3.09	3.06
query34	15.26	4.80	4.82
query35	4.84	4.85	4.84
query36	1.06	1.03	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.15	0.15	0.14
query41	0.06	0.02	0.01
query42	0.01	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 101.55 s
Total hot run time: 31.22 s

@doris-robot
Copy link

doris-robot commented Feb 23, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c3748fd7ff8c75d51657280656436fa9158cc74a with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@morningman morningman merged commit 65a939c into apache:branch-2.0 Feb 25, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morningman
[fix](file-writer) avoid empty file for segment writer (apache#31355)
eeac979 
bp apache#31169
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@morningman @doris-robot