[fix](Tvf) return empty set when tvf queries an empty file or an error uri #25280

BePPPower · 2023-10-11T02:39:07Z

Proposed changes

Issue Number:

Before:

return errors when tvf queries an empty file or an error uri:

get parsed schema failed, empty csv file
Can not get first file, please check uri.

Now:

we just return empty set when tvf queries an empty file or an error uri.

mysql> select * from s3( 
"uri" = "https://error_uri/exp_1.csv", 
"s3.access_key"= "xx", 
"s3.secret_key" = "yy", 
"format" = "csv") limit 10;

Empty set (1.29 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-10-11T02:47:13Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2023-10-11T03:48:55Z

run buildall

github-actions · 2023-10-11T03:56:26Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-11T04:25:09Z

TeamCity be ut coverage result:
Function Coverage: 36.24% (8149/22486)
Line Coverage: 28.35% (65162/229840)
Region Coverage: 27.28% (33768/123775)
Branch Coverage: 23.98% (17205/71760)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9d21faf52898085daf9204c6ce1bdb8bac883c6c_9d21faf52898085daf9204c6ce1bdb8bac883c6c/report/index.html

doris-robot · 2023-10-11T04:29:08Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.47 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162315204 Bytes

morningman · 2023-10-11T14:39:31Z

be/src/vec/exec/format/csv/csv_reader.cpp

    bool is_parse_name = false;
    RETURN_IF_ERROR(_prepare_parse(&read_line, &is_parse_name));

+    if (_file_reader->size() == 0) {


Need to add comment to explain why return OK when file size is 0

morningman · 2023-10-11T14:39:38Z

be/src/vec/exec/format/csv/csv_reader.cpp

    RETURN_IF_ERROR(_line_reader->read_line(&ptr, &size, &_line_reader_eof, _io_ctx));
    if (size == 0) {
-        return Status::InternalError("The first line is empty, can not parse column names");
+        return Status::OK();


morningman · 2023-10-11T14:43:58Z

fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java

        TNetworkAddress address = new TNetworkAddress(be.getHost(), be.getBrpcPort());
        try {
            PFetchTableSchemaRequest request = getFetchTableStructureRequest();
+            if (request == null) {


unify it to fillColumns

BePPPower · 2023-10-12T05:15:25Z

run buildall

doris-robot · 2023-10-12T05:55:22Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.86 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162376826 Bytes

BePPPower · 2023-10-12T07:02:30Z

run buildall

BePPPower · 2023-10-12T07:04:28Z

run buildall

doris-robot · 2023-10-12T07:42:06Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.9 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17161998075 Bytes

doris-robot · 2023-10-12T07:42:17Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.98 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162202182 Bytes

BePPPower · 2023-10-12T08:20:17Z

run buildall

doris-robot · 2023-10-12T09:45:32Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162410602 Bytes

BePPPower · 2023-10-12T14:05:06Z

run feut

BePPPower · 2023-10-13T03:01:07Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/tablefunction/LocalTableValuedFunction.java

doris-robot · 2023-10-13T03:45:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.03 seconds
stream load tsv: 550 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162093248 Bytes

morningman

LGTM

github-actions · 2023-10-13T07:44:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-13T07:44:39Z

PR approved by anyone and no changes requested.

morningman · 2023-10-13T07:49:02Z

run buildall

doris-robot · 2023-10-13T08:44:17Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.32 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162014325 Bytes

BePPPower · 2023-10-13T10:32:11Z

run p0

BePPPower · 2023-10-13T10:39:28Z

run buildall

doris-robot · 2023-10-13T11:11:41Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.97 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162064322 Bytes

yiguolei

lgtm

github-actions · 2023-10-14T14:40:50Z

PR approved by at least one committer and no changes requested.

morningman

LGTM

…r uri (apache#25280) ### Before: return errors when tvf queries an empty file or an error uri: 1. get parsed schema failed, empty csv file 2. Can not get first file, please check uri. ### Now: we just return empty set when tvf queries an empty file or an error uri. ```sql mysql> select * from s3( "uri" = "https://error_uri/exp_1.csv", "s3.access_key"= "xx", "s3.secret_key" = "yy", "format" = "csv") limit 10; Empty set (1.29 sec) ```

…r uri (apache#25280) return errors when tvf queries an empty file or an error uri: 1. get parsed schema failed, empty csv file 2. Can not get first file, please check uri. we just return empty set when tvf queries an empty file or an error uri. ```sql mysql> select * from s3( "uri" = "https://error_uri/exp_1.csv", "s3.access_key"= "xx", "s3.secret_key" = "yy", "format" = "csv") limit 10; Empty set (1.29 sec) ```

…r uri (#25280) (#31110) return errors when tvf queries an empty file or an error uri: 1. get parsed schema failed, empty csv file 2. Can not get first file, please check uri. we just return empty set when tvf queries an empty file or an error uri. ```sql mysql> select * from s3( "uri" = "https://error_uri/exp_1.csv", "s3.access_key"= "xx", "s3.secret_key" = "yy", "format" = "csv") limit 10; Empty set (1.29 sec) ``` Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>

…r uri (apache#25280) (apache#31110) return errors when tvf queries an empty file or an error uri: 1. get parsed schema failed, empty csv file 2. Can not get first file, please check uri. we just return empty set when tvf queries an empty file or an error uri. ```sql mysql> select * from s3( "uri" = "https://error_uri/exp_1.csv", "s3.access_key"= "xx", "s3.secret_key" = "yy", "format" = "csv") limit 10; Empty set (1.29 sec) ``` Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>

xiaokang added the dev/2.0.3 label Oct 11, 2023

morningman reviewed Oct 11, 2023

View reviewed changes

BePPPower force-pushed the fixTvfEmptyFile branch from eada360 to a51b15e Compare October 13, 2023 03:00

morningman reviewed Oct 13, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/tablefunction/LocalTableValuedFunction.java Outdated Show resolved Hide resolved

BePPPower force-pushed the fixTvfEmptyFile branch from a51b15e to df2fd0f Compare October 13, 2023 06:16

BePPPower added 5 commits October 13, 2023 14:50

fix

b9b7c9b

add test

9a5ff4b

fix code

3e8dbed

modify docs and add tests

d612d43

add hdfs test

94d168b

BePPPower force-pushed the fixTvfEmptyFile branch from df2fd0f to 94d168b Compare October 13, 2023 06:51

add docs

abb9bbc

morningman previously approved these changes Oct 13, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 13, 2023

github-actions bot added the reviewed label Oct 13, 2023

add en docs

52f04d1

BePPPower dismissed morningman’s stale review via 52f04d1 October 13, 2023 10:39

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 13, 2023

yiguolei approved these changes Oct 14, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 14, 2023

morningman approved these changes Oct 16, 2023

View reviewed changes

morningman removed the dev/2.0.3 label Oct 16, 2023

morningman merged commit 85b8497 into apache:master Oct 17, 2023

zxealous mentioned this pull request Feb 19, 2024

[fix](tvf) Fix tvf read empty file fail #31110

Merged

morningman added the dev/2.0.5-merged label Feb 19, 2024

[fix](Tvf) return empty set when tvf queries an empty file or an error uri #25280

[fix](Tvf) return empty set when tvf queries an empty file or an error uri #25280

Uh oh!

Conversation

BePPPower commented Oct 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Before:

Now:

Further comments

Uh oh!

github-actions bot commented Oct 11, 2023

Uh oh!

BePPPower commented Oct 11, 2023

Uh oh!

github-actions bot commented Oct 11, 2023

Uh oh!

doris-robot commented Oct 11, 2023

Uh oh!

doris-robot commented Oct 11, 2023

Uh oh!

morningman Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

morningman Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

morningman Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

doris-robot commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

doris-robot commented Oct 12, 2023

Uh oh!

doris-robot commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

doris-robot commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 13, 2023

Uh oh!

Uh oh!

doris-robot commented Oct 13, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 13, 2023

Uh oh!

github-actions bot commented Oct 13, 2023

Uh oh!

morningman commented Oct 13, 2023

Uh oh!

doris-robot commented Oct 13, 2023

Uh oh!

BePPPower commented Oct 13, 2023

Uh oh!

BePPPower commented Oct 13, 2023

Uh oh!

doris-robot commented Oct 13, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 14, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

BePPPower commented Oct 11, 2023 •

edited

Loading