[Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values. #36668

BePPPower · 2024-06-21T08:12:55Z

Proposed changes

Issue Number: close #xxx

Previously, empty rows in CSV files were ignored by Doris.
Now, add a session variable to control whether empty rows in CSV files are read as NULL values.

doris-robot · 2024-06-21T08:13:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-06-21T08:19:13Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2024-06-21T09:14:47Z

run buildall

BePPPower · 2024-06-21T09:19:18Z

run buildall

github-actions · 2024-06-21T09:24:44Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-21T10:48:15Z

TeamCity be ut coverage result:
Function Coverage: 36.46% (9006/24701)
Line Coverage: 28.01% (73867/263759)
Region Coverage: 27.48% (38367/139610)
Branch Coverage: 24.18% (19556/80876)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fe4fe65c2b10e5a0f4e92179f7360d067744c3d1_fe4fe65c2b10e5a0f4e92179f7360d067744c3d1/report/index.html

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java

BePPPower · 2024-06-24T03:19:58Z

run buildall

github-actions · 2024-06-24T03:25:32Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2024-06-24T03:36:48Z

run buildall

BePPPower · 2024-06-24T03:44:19Z

run buildall

BePPPower · 2024-06-24T03:47:05Z

run buildall

github-actions · 2024-06-24T03:52:54Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2024-06-24T05:07:15Z

run buildall

BePPPower · 2024-06-24T05:12:15Z

run buildall

BePPPower · 2024-06-24T05:16:18Z

run buildall

github-actions · 2024-06-24T05:22:14Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-24T06:35:58Z

TPC-H: Total hot run time: 40895 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dfe28b37b65d2ed7243e308a6457d014cf23aeec, data reload: false

------ Round 1 ----------------------------------
q1	18285	4727	4446	4446
q2	2593	196	182	182
q3	12193	1181	1190	1181
q4	10619	764	932	764
q5	7524	2778	2714	2714
q6	224	138	140	138
q7	959	615	619	615
q8	9219	2091	2086	2086
q9	9078	6532	6440	6440
q10	8936	3707	3738	3707
q11	456	244	242	242
q12	422	240	237	237
q13	17751	2977	3011	2977
q14	267	228	218	218
q15	528	493	492	492
q16	492	379	372	372
q17	978	713	703	703
q18	8103	7464	7372	7372
q19	7674	1463	1589	1463
q20	668	320	325	320
q21	4999	3876	4059	3876
q22	413	350	350	350
Total cold run time: 122381 ms
Total hot run time: 40895 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4366	4257	4207	4207
q2	384	280	277	277
q3	2955	2783	2744	2744
q4	1917	1617	1615	1615
q5	5290	5338	5304	5304
q6	215	130	131	130
q7	2170	1730	1694	1694
q8	3209	3340	3349	3340
q9	8318	8340	8338	8338
q10	3857	3694	3573	3573
q11	570	499	484	484
q12	770	607	601	601
q13	17418	2952	3013	2952
q14	286	260	266	260
q15	523	470	483	470
q16	500	425	407	407
q17	1778	1499	1471	1471
q18	7595	7501	7383	7383
q19	1710	1464	1547	1464
q20	1997	1797	1769	1769
q21	4853	4655	4844	4655
q22	640	563	539	539
Total cold run time: 71321 ms
Total hot run time: 53677 ms

doris-robot · 2024-06-24T06:46:56Z

TPC-DS: Total hot run time: 173315 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dfe28b37b65d2ed7243e308a6457d014cf23aeec, data reload: false

query1	908	383	394	383
query2	6456	2316	2336	2316
query3	6646	210	215	210
query4	19215	17317	17228	17228
query5	4166	477	477	477
query6	262	166	167	166
query7	4610	296	293	293
query8	326	298	287	287
query9	8677	2356	2341	2341
query10	584	304	276	276
query11	10391	10044	9865	9865
query12	126	84	90	84
query13	1624	366	369	366
query14	9352	7632	7889	7632
query15	252	193	189	189
query16	7818	267	263	263
query17	1913	547	521	521
query18	1481	278	265	265
query19	199	153	155	153
query20	90	86	87	86
query21	208	128	127	127
query22	4362	4054	4097	4054
query23	33726	32859	32914	32859
query24	11863	2927	2891	2891
query25	646	348	357	348
query26	1609	153	151	151
query27	2765	313	322	313
query28	7019	2031	2018	2018
query29	946	643	604	604
query30	288	151	147	147
query31	960	733	734	733
query32	94	55	59	55
query33	758	278	281	278
query34	948	465	484	465
query35	730	628	634	628
query36	1082	910	916	910
query37	168	70	72	70
query38	2884	2749	2743	2743
query39	838	810	811	810
query40	278	130	122	122
query41	57	51	51	51
query42	117	95	101	95
query43	578	552	559	552
query44	1152	734	735	734
query45	202	161	163	161
query46	1082	727	710	710
query47	1871	1756	1755	1755
query48	363	309	304	304
query49	1165	402	400	400
query50	765	383	388	383
query51	6964	6829	6780	6780
query52	111	93	92	92
query53	356	296	288	288
query54	938	459	431	431
query55	77	75	75	75
query56	279	258	254	254
query57	1144	1052	1056	1052
query58	262	238	255	238
query59	3404	3010	3042	3010
query60	289	274	297	274
query61	89	91	94	91
query62	646	445	457	445
query63	320	292	296	292
query64	9871	2250	1729	1729
query65	3197	3151	3119	3119
query66	1245	365	329	329
query67	15378	14973	14978	14973
query68	4581	530	543	530
query69	521	430	329	329
query70	1193	1176	1154	1154
query71	403	270	275	270
query72	7655	5537	5549	5537
query73	757	320	326	320
query74	5856	5528	5404	5404
query75	3360	2668	2698	2668
query76	2780	996	898	898
query77	444	312	294	294
query78	10349	9844	9777	9777
query79	1987	515	522	515
query80	833	473	483	473
query81	589	217	223	217
query82	967	105	100	100
query83	263	169	167	167
query84	233	86	93	86
query85	1224	271	323	271
query86	436	319	330	319
query87	3369	3117	3088	3088
query88	3425	2460	2438	2438
query89	473	399	397	397
query90	1822	192	188	188
query91	125	101	102	101
query92	71	49	50	49
query93	1717	509	503	503
query94	1247	195	184	184
query95	408	312	306	306
query96	601	277	269	269
query97	3204	3082	3093	3082
query98	212	201	190	190
query99	1265	847	855	847
Total cold run time: 271560 ms
Total hot run time: 173315 ms

doris-robot · 2024-06-24T06:52:26Z

ClickBench: Total hot run time: 31.73 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dfe28b37b65d2ed7243e308a6457d014cf23aeec, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.22	0.05	0.06
query4	1.69	0.08	0.08
query5	0.50	0.49	0.48
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.49	0.50
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.58	0.58	0.58
query14	0.76	0.80	0.80
query15	0.85	0.80	0.81
query16	0.36	0.36	0.37
query17	1.04	1.01	1.02
query18	0.21	0.24	0.26
query19	1.84	1.78	1.77
query20	0.01	0.01	0.01
query21	15.41	0.65	0.65
query22	3.49	6.72	2.88
query23	18.29	1.32	1.30
query24	2.15	0.23	0.23
query25	0.15	0.09	0.08
query26	0.26	0.17	0.17
query27	0.07	0.07	0.08
query28	13.17	1.02	0.99
query29	12.65	3.36	3.38
query30	0.26	0.06	0.06
query31	2.86	0.40	0.39
query32	3.25	0.47	0.47
query33	2.88	2.97	2.91
query34	17.02	4.38	4.46
query35	4.44	4.48	4.44
query36	0.66	0.48	0.46
query37	0.20	0.16	0.15
query38	0.16	0.16	0.15
query39	0.05	0.04	0.04
query40	0.16	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 108.59 s
Total hot run time: 31.73 s

AshinGau · 2024-06-25T03:00:58Z

LGTM

github-actions · 2024-06-25T03:01:33Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-25T03:01:35Z

PR approved by anyone and no changes requested.

…s in CSV files are read as NULL values. (#36668) Previously, empty rows in CSV files were ignored by Doris. Now, add a session variable to control whether empty rows in CSV files are read as NULL values.

…s in CSV files are read as NULL values. (apache#36668) Previously, empty rows in CSV files were ignored by Doris. Now, add a session variable to control whether empty rows in CSV files are read as NULL values.

…s in CSV files are read as NULL values (#37153) bp: #36668

…s in CSV files are read as NULL values (apache#37153) bp: apache#36668

github-actions bot added the meta-change label Jun 21, 2024

BePPPower force-pushed the fixReadCsvEmptyLine branch from 4d07e5e to fe4fe65 Compare June 21, 2024 09:18

morningman reviewed Jun 21, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java Outdated Show resolved Hide resolved

morningman added the dev/2.1.x label Jun 21, 2024

BePPPower added 3 commits June 24, 2024 13:03

fix 1

55cdec6

fix 2

ec32494

fix 3

dc1077b

BePPPower force-pushed the fixReadCsvEmptyLine branch from 7a12a72 to dc1077b Compare June 24, 2024 05:06

fix 4

dfe28b3

AshinGau approved these changes Jun 25, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 25, 2024

github-actions bot added the reviewed label Jun 25, 2024

morningman approved these changes Jun 25, 2024

View reviewed changes

morningman merged commit 7287fdd into apache:master Jun 25, 2024

BePPPower mentioned this pull request Jul 2, 2024

[Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values #37153

Merged

morningman pushed a commit that referenced this pull request Jul 2, 2024

[Fix](csv_reader) Add a session variable to control whether empty row…

bd24a8b

…s in CSV files are read as NULL values (#37153) bp: #36668

morningman added dev/2.1.5-merged and removed dev/2.1.x labels Jul 2, 2024

morningman pushed a commit to morningman/doris that referenced this pull request Jul 9, 2024

[Fix](csv_reader) Add a session variable to control whether empty row…

caf21d9

…s in CSV files are read as NULL values (apache#37153) bp: apache#36668

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

[Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values. #36668

[Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values. #36668

Uh oh!

Conversation

BePPPower commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jun 21, 2024

Uh oh!

github-actions bot commented Jun 21, 2024

Uh oh!

BePPPower commented Jun 21, 2024

Uh oh!

BePPPower commented Jun 21, 2024

Uh oh!

github-actions bot commented Jun 21, 2024

Uh oh!

doris-robot commented Jun 21, 2024

Uh oh!

Uh oh!

BePPPower commented Jun 24, 2024

Uh oh!

github-actions bot commented Jun 24, 2024

Uh oh!

BePPPower commented Jun 24, 2024

Uh oh!

BePPPower commented Jun 24, 2024

Uh oh!

BePPPower commented Jun 24, 2024

Uh oh!

github-actions bot commented Jun 24, 2024

Uh oh!

BePPPower commented Jun 24, 2024

Uh oh!

BePPPower commented Jun 24, 2024

Uh oh!

BePPPower commented Jun 24, 2024

Uh oh!

github-actions bot commented Jun 24, 2024

Uh oh!

doris-robot commented Jun 24, 2024

Uh oh!

doris-robot commented Jun 24, 2024

Uh oh!

doris-robot commented Jun 24, 2024

Uh oh!

AshinGau commented Jun 25, 2024

Uh oh!

github-actions bot commented Jun 25, 2024

Uh oh!

github-actions bot commented Jun 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

BePPPower commented Jun 21, 2024 •

edited

Loading