Skip to content

Revert "[fix](outfile) Fix unable to export empty data (#30703)" #31148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
kaijchen wants to merge 1 commit into from
Closed

Revert "[fix](outfile) Fix unable to export empty data (#30703)" #31148

kaijchen wants to merge 1 commit into from

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Feb 20, 2024 •
edited
Loading

Proposed changes

This reverts commit 99c3b5f.

Because it causes empty segment file been written:

    2024-02-19 17:51:36.103 ERROR [suite-thread-3] (ScriptContext.groovy:122) - Run load_test_query_db in /home/work/unlimit_teamcity/TeamCity/Agents/20240219160058agent_172.16.0.186_1/work/60183217f6ee2a9c/regression-test/suites/load_p0/insert/load.groovy failed
    java.sql.SQLException: errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]failed to initialize storage reader. tablet=10153, res=[CORRUPTION]Bad segment file s3://doris-build-1308700295/ci/data/10153/02000000000000134448e90cff392f294f7c5ac2f0143090_0.dat: file size 0 < 12

http://43.132.222.7:8111/test/8007482201306640353?currentProjectId=Doris_DorisCloudRegression&expandTestHistoryChartSection=true

image

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Feb 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@kaijchen
Copy link
Member Author

kaijchen commented Feb 20, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Feb 20, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Feb 20, 2024

TeamCity be ut coverage result:
Function Coverage: 35.78% (8558/23921)
Line Coverage: 27.72% (69413/250372)
Region Coverage: 26.85% (36026/134200)
Branch Coverage: 23.65% (18424/77902)
Coverage Report: http://coverage.selectdb-in.cc/coverage/85a15bef1937c3f67a15a0f2925b64046678d2d3_85a15bef1937c3f67a15a0f2925b64046678d2d3/report/index.html

@doris-robot
Copy link

doris-robot commented Feb 20, 2024

TPC-H: Total hot run time: 41588 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85a15bef1937c3f67a15a0f2925b64046678d2d3, data reload: false

------ Round 1 ----------------------------------
q1	17628	4969	4889	4889
q2	2031	139	134	134
q3	10609	1051	998	998
q4	4751	962	978	962
q5	7745	3293	3291	3291
q6	196	134	134	134
q7	1274	779	758	758
q8	9365	2107	2083	2083
q9	7620	6724	6722	6722
q10	8320	2650	2651	2650
q11	418	229	201	201
q12	732	328	330	328
q13	18005	3706	3730	3706
q14	301	256	266	256
q15	585	516	507	507
q16	487	433	416	416
q17	931	866	819	819
q18	7562	6600	6653	6600
q19	1569	1498	1493	1493
q20	642	358	335	335
q21	6984	3979	3971	3971
q22	875	354	335	335
Total cold run time: 108630 ms
Total hot run time: 41588 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4889	4836	4894	4836
q2	294	191	202	191
q3	3598	3583	3578	3578
q4	2566	2524	2534	2524
q5	5736	5748	5750	5748
q6	221	124	128	124
q7	2268	1676	1658	1658
q8	3020	3118	3122	3118
q9	8707	8737	8715	8715
q10	6812	4255	4265	4255
q11	523	430	402	402
q12	781	550	554	550
q13	4320	3435	3406	3406
q14	283	250	236	236
q15	632	509	529	509
q16	516	453	440	440
q17	1672	1630	1624	1624
q18	8292	7604	7578	7578
q19	1648	1646	1643	1643
q20	2106	1843	1814	1814
q21	6663	6275	6244	6244
q22	568	507	531	507
Total cold run time: 66115 ms
Total hot run time: 59700 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2024

TPC-DS: Total hot run time: 186036 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85a15bef1937c3f67a15a0f2925b64046678d2d3, data reload: false

query1	924	359	343	343
query2	6524	1822	1812	1812
query3	6708	212	205	205
query4	23200	21195	21133	21133
query5	4195	378	380	378
query6	261	174	173	173
query7	4601	300	301	300
query8	269	193	217	193
query9	8465	2805	2775	2775
query10	424	227	231	227
query11	15021	14521	14651	14521
query12	140	83	88	83
query13	1696	425	429	425
query14	9155	7827	7849	7827
query15	215	193	194	193
query16	7250	260	259	259
query17	1411	567	545	545
query18	2003	267	268	267
query19	196	152	149	149
query20	91	79	81	79
query21	192	124	120	120
query22	4892	4782	4722	4722
query23	32502	31563	31506	31506
query24	12677	3416	3358	3358
query25	648	367	368	367
query26	1857	159	160	159
query27	3034	319	318	318
query28	6580	1801	1788	1788
query29	1196	646	621	621
query30	287	134	139	134
query31	949	752	764	752
query32	96	61	56	56
query33	733	238	234	234
query34	1072	499	512	499
query35	939	832	834	832
query36	989	919	849	849
query37	161	62	62	62
query38	3270	3204	3169	3169
query39	1400	1301	1303	1301
query40	289	109	108	108
query41	37	35	35	35
query42	105	105	99	99
query43	490	454	449	449
query44	1062	688	691	688
query45	204	182	180	180
query46	1042	774	755	755
query47	1607	1570	1560	1560
query48	416	337	348	337
query49	1198	308	306	306
query50	775	374	378	374
query51	5367	5182	5207	5182
query52	113	92	96	92
query53	392	306	311	306
query54	295	218	226	218
query55	86	82	85	82
query56	225	205	205	205
query57	1012	943	945	943
query58	220	204	206	204
query59	2374	2192	2207	2192
query60	246	218	214	214
query61	85	85	90	85
query62	595	386	386	386
query63	343	293	303	293
query64	6330	3061	3131	3061
query65	3293	3242	3236	3236
query66	1376	342	328	328
query67	14802	14175	14200	14175
query68	5095	562	555	555
query69	517	366	370	366
query70	1322	1256	1226	1226
query71	380	265	262	262
query72	6365	2795	2623	2623
query73	731	322	318	318
query74	6858	6499	6416	6416
query75	3164	2582	2561	2561
query76	3207	1107	1203	1107
query77	343	246	238	238
query78	9430	8879	8832	8832
query79	966	519	505	505
query80	537	357	346	346
query81	438	205	206	205
query82	159	87	87	87
query83	139	120	124	120
query84	227	81	80	80
query85	996	355	337	337
query86	307	316	310	310
query87	3483	3288	3260	3260
query88	2782	2337	2338	2337
query89	447	364	360	360
query90	1995	164	169	164
query91	156	133	125	125
query92	55	50	48	48
query93	995	496	471	471
query94	1163	190	177	177
query95	8678	8600	8342	8342
query96	585	271	275	271
query97	4451	4254	4312	4254
query98	228	203	199	199
query99	1018	707	742	707
Total cold run time: 277819 ms
Total hot run time: 186036 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2024

ClickBench: Total hot run time: 30.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85a15bef1937c3f67a15a0f2925b64046678d2d3, data reload: false

query1	0.02	0.03	0.02
query2	0.06	0.02	0.02
query3	0.22	0.07	0.08
query4	1.65	0.09	0.09
query5	0.49	0.48	0.49
query6	1.41	0.61	0.61
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.45	0.45
query10	0.50	0.50	0.48
query11	0.12	0.09	0.10
query12	0.12	0.10	0.10
query13	0.59	0.59	0.59
query14	0.75	0.78	0.79
query15	0.82	0.80	0.80
query16	0.33	0.33	0.35
query17	0.92	0.93	0.92
query18	0.18	0.17	0.19
query19	1.81	1.71	1.68
query20	0.01	0.01	0.01
query21	15.42	0.63	0.58
query22	2.98	3.39	1.59
query23	17.50	0.94	1.08
query24	1.99	0.43	0.36
query25	0.63	0.07	0.06
query26	0.16	0.14	0.13
query27	0.06	0.05	0.04
query28	12.14	0.84	0.83
query29	12.63	3.29	3.36
query30	0.54	0.50	0.48
query31	2.78	0.36	0.36
query32	3.36	0.48	0.47
query33	3.14	3.12	3.13
query34	15.36	4.49	4.51
query35	4.49	4.48	4.48
query36	1.10	0.96	0.96
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.14	0.15
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.3 s
Total hot run time: 30.18 s

@doris-robot
Copy link

doris-robot commented Feb 20, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 85a15bef1937c3f67a15a0f2925b64046678d2d3 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@kaijchen kaijchen marked this pull request as ready for review February 20, 2024 09:27
@kaijchen kaijchen mentioned this pull request Feb 20, 2024
Merged
@kaijchen kaijchen marked this pull request as draft February 20, 2024 09:41
@kaijchen
Copy link
Member Author

kaijchen commented Feb 20, 2024

#31169 could solve this issue.

@morningman morningman closed this Feb 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou will be requested when the pull request is marked ready for review gavinchou is a code owner

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring will be requested when the pull request is marked ready for review dataroaring is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaijchen @doris-robot @xiaokang @morningman @wm1581066