[Enhancement](tvf) Table value function support reading local file #17404

gitccl · 2023-03-03T12:33:07Z

Proposed changes

Issue Number: close #17308

Problem summary

I tested the local tvf with tpch queries. First, generate lineitem datasets with 6001215 rows, and load it into lineitem table by:

insert into lineitem select c11, c1, c4, c2, c3, c5, c6, c7, c8, c9, c10, c12, c13, c14, c15, c16 
from local(
        "file_path" = "tools/tpch-tools/bin/tpch-data/lineitem.tbl.1", 
        "backend_id" = "10003", 
        "format" = "csv", 
        "column_separator" = "|"
);

Then, run q1 and q16 tpch queries, the query result is correct.

It can also analyze the BE's log directly like:

mysql> select * from local(
        "file_path" = "log/be.out",
        "backend_id" = "10006",
        "format" = "csv")
       where c1 like "%start_time%" limit 10;
+--------------------------------------------------------+
| c1                                                     |
+--------------------------------------------------------+
| start time: 2023年 08月 07日 星期一 23:20:32 CST       |
| start time: 2023年 08月 07日 星期一 23:32:10 CST       |
| start time: 2023年 08月 08日 星期二 00:20:50 CST       |
| start time: 2023年 08月 08日 星期二 00:29:15 CST       |
+--------------------------------------------------------+

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java

fe/fe-core/src/main/java/org/apache/doris/planner/external/ExternalFileScanNode.java

fe/fe-core/src/main/java/org/apache/doris/tablefunction/LocalTableValuedFunction.java

github-actions · 2023-03-07T05:12:55Z

clang-tidy review says "All clean, LGTM! 👍"

gitccl · 2023-05-21T03:27:34Z

run buildall

github-actions · 2023-05-21T03:29:40Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-05-21T06:02:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-05-21T06:06:36Z

clang-tidy review says "All clean, LGTM! 👍"

gitccl · 2023-05-21T06:12:52Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/planner/external/TVFScanNode.java

github-actions · 2023-05-28T14:29:12Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

LGTM

morningman · 2023-05-28T14:32:00Z

run buildall

github-actions · 2023-05-28T14:32:40Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-05-28T14:32:43Z

PR approved by anyone and no changes requested.

gitccl · 2023-05-28T14:35:18Z

run buildall

morningman

LGTM

morningman · 2023-08-08T10:19:55Z

run buildall

github-actions · 2023-08-08T10:24:02Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-08-08T11:28:27Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.34 seconds
stream load tsv: 516 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162135191 Bytes

morningman

LGTM

github-actions · 2023-08-08T13:18:55Z

PR approved by at least one committer and no changes requested.

BePPPower · 2023-08-08T14:08:39Z

be/src/io/fs/local_file_system.cpp

+    glob_t glob_result;
+    memset(&glob_result, 0, sizeof(glob_result));
+
+    int rc = glob(pattern.c_str(), GLOB_TILDE, NULL, &glob_result);


Suggested change

int rc = glob(pattern.c_str(), GLOB_TILDE, NULL, &glob_result);

int rc = glob(pattern.c_str(), GLOB_TILDE, nullptr, &glob_result);

BePPPower · 2023-08-08T14:29:51Z

fe/fe-core/src/main/java/org/apache/doris/planner/external/TVFScanNode.java

+
    protected String getFsName(FileSplit split) {
        return tableValuedFunction.getFsName();
    }


Suggested change

protected String getFsName(FileSplit split) {

return tableValuedFunction.getFsName();

}

@Override

protected String getFsName(FileSplit split) {

return tableValuedFunction.getFsName();

}

morningman · 2023-08-09T12:51:02Z

run buildall

github-actions · 2023-08-09T12:57:34Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-08-09T13:18:53Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-08-09T14:14:57Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.16 seconds
stream load tsv: 510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162481757 Bytes

morningman · 2023-08-09T15:37:45Z

run buildall

github-actions · 2023-08-09T15:44:25Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-08-09T16:06:07Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.58 seconds
stream load tsv: 510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162138241 Bytes

morningman

LGTM

github-actions · 2023-08-10T05:43:46Z

PR approved by at least one committer and no changes requested.

BePPPower

LGTM

…22915) cherry pick #17404

github-actions bot added the area/planner Issues or PRs related to the query planner label Mar 3, 2023

gitccl force-pushed the feat_17308 branch from a4a2a10 to 87e5998 Compare March 3, 2023 13:11

morningman reviewed Mar 5, 2023

View reviewed changes

github-actions bot added the area/vectorization label Mar 7, 2023

morningman self-assigned this Mar 24, 2023

gitccl force-pushed the feat_17308 branch from 88eab5d to 99ec8f1 Compare May 21, 2023 03:26

gitccl marked this pull request as ready for review May 21, 2023 03:26

github-actions bot added the kind/docs Categorizes issue or PR as related to documentation. label May 21, 2023

gitccl requested a review from morningman May 24, 2023 01:28

morningman reviewed May 28, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/planner/external/TVFScanNode.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/planner/external/TVFScanNode.java Outdated Show resolved Hide resolved

gitccl force-pushed the feat_17308 branch from df19c9c to 62ec080 Compare May 28, 2023 14:25

morningman previously approved these changes May 28, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 28, 2023

github-actions bot added the reviewed label May 28, 2023

gitccl dismissed morningman’s stale review via 3da03f9 May 28, 2023 14:33

gitccl force-pushed the feat_17308 branch from 62ec080 to 3da03f9 Compare May 28, 2023 14:33

github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 28, 2023

morningman previously approved these changes May 28, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 28, 2023

morningman force-pushed the feat_17308 branch from 78e5489 to 6a71eca Compare August 8, 2023 10:16

morningman previously approved these changes Aug 8, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 8, 2023

BePPPower reviewed Aug 8, 2023

View reviewed changes

morningman dismissed their stale review via 67f6ba8 August 9, 2023 12:50

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 9, 2023

morningman force-pushed the feat_17308 branch from 67f6ba8 to 6f7d12c Compare August 9, 2023 13:12

local tvf

7c7fd2c

morningman force-pushed the feat_17308 branch from 6f7d12c to 7c7fd2c Compare August 9, 2023 15:37

morningman approved these changes Aug 10, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 10, 2023

BePPPower approved these changes Aug 10, 2023

View reviewed changes

morningman merged commit 71807ce into apache:master Aug 10, 2023

xiaokang added the merge_conflict label Aug 11, 2023

morningman mentioned this pull request Aug 13, 2023

[Enhancement](tvf) Table value function support reading local file #22915

Merged

morningman added a commit that referenced this pull request Aug 13, 2023

[Enhancement](tvf) Table value function support reading local file (#…

1adb4cf

…22915) cherry pick #17404

morningman added dev/2.0.1-merged and removed dev/2.0.1 labels Aug 13, 2023

	int rc = glob(pattern.c_str(), GLOB_TILDE, NULL, &glob_result);
	int rc = glob(pattern.c_str(), GLOB_TILDE, nullptr, &glob_result);

[Enhancement](tvf) Table value function support reading local file #17404

[Enhancement](tvf) Table value function support reading local file #17404

Uh oh!

Conversation

gitccl commented Mar 3, 2023 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 7, 2023

Uh oh!

gitccl commented May 21, 2023

Uh oh!

github-actions bot commented May 21, 2023

Uh oh!

github-actions bot commented May 21, 2023

Uh oh!

github-actions bot commented May 21, 2023

Uh oh!

gitccl commented May 21, 2023

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 28, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented May 28, 2023

Uh oh!

github-actions bot commented May 28, 2023

Uh oh!

github-actions bot commented May 28, 2023

Uh oh!

gitccl commented May 28, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Aug 8, 2023

Uh oh!

github-actions bot commented Aug 8, 2023

Uh oh!

hello-stephen commented Aug 8, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 8, 2023

Uh oh!

BePPPower Aug 8, 2023

Choose a reason for hiding this comment

Uh oh!

BePPPower Aug 8, 2023

Choose a reason for hiding this comment

Uh oh!

morningman commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Uh oh!

hello-stephen commented Aug 9, 2023

Uh oh!

morningman commented Aug 9, 2023

Uh oh!

github-actions bot commented Aug 9, 2023

Uh oh!

hello-stephen commented Aug 9, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 10, 2023

Uh oh!

gitccl commented Mar 3, 2023 •

edited by morningman

Loading