[Enhancement](Load) Stream Load using SQL #22509

zzzzzzzs · 2023-08-02T10:07:11Z

Proposed changes

This PR was originally #16940 , but it has not been updated for a long time due to the original author @Cai-Yao . At present, we will merge some of the code into the master first.

thanks @Cai-Yao @yiguolei

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Co-authored-by: Cai-Yao <729673078@qq.com> Co-authored-by: yiguolei <676222867@qq.com>

github-actions · 2023-08-02T10:14:32Z

clang-tidy review says "All clean, LGTM! 👍"

zzzzzzzs · 2023-08-02T10:28:47Z

run buildall

freemandealer · 2023-08-03T12:07:23Z

docs/zh-CN/docs/data-operate/import/import-way/stream-load-manual.md

+示例：
+
+```
+curl  --location-trusted -u root: -T test.csv  -H "sql:insert into demo.example_tbl_1(user_id, age, cost) select c1, c4, c7 * 2 from stream("format" = "CSV", "column_separator" = "," ) where age >= 30"  http://127.0.0.1:28030/api/demo/example_tbl_1/_stream_load_with_sql


转义是怎么处理的？

freemandealer · 2023-08-03T12:09:19Z

docs/zh-CN/docs/data-operate/import/import-way/stream-load-manual.md

+
+```
+curl  --location-trusted -u root: -T test.csv  -H "sql:insert into demo.example_tbl_1(user_id, age, cost) select c1, c4, c7 * 2 from stream("format" = "CSV", "column_separator" = "," ) where age >= 30"  http://127.0.0.1:28030/api/demo/example_tbl_1/_stream_load_with_sql
+```


是不是再加一个直接请求 be 的 example? since they have different URI pattern.

这里的文档应该是错的，我们先把代码合入，然后慢慢改吧。我怕得反复的rebase

freemandealer · 2023-08-03T12:14:28Z

be/src/vec/exec/format/csv/csv_reader.cpp

-        _params.file_type == TFileType::FILE_BROKER) {
+    if (_params.file_type == TFileType::FILE_BROKER) {
        return Status::InternalError(
                "Getting parsered schema from csv file do not support stream load and broker "


Now that we support stream load, we should also update the err msg.

yiguolei · 2023-08-04T16:34:15Z

run buildall

yiguolei · 2023-08-04T16:35:40Z

rename to http stream???

Conflicts: fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java

zzzzzzzs · 2023-08-04T20:27:41Z

rename to http stream???

I will rename it as I refine it

github-actions · 2023-08-04T20:32:48Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2023-08-05T02:21:51Z

run buildall

github-actions · 2023-08-05T14:51:02Z

clang-tidy review says "All clean, LGTM! 👍"

zzzzzzzs · 2023-08-05T14:52:07Z

run buildall

hello-stephen · 2023-08-05T16:01:40Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.26 seconds
stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162483780 Bytes

yiguolei · 2023-08-06T02:22:14Z

run buildall

hello-stephen · 2023-08-06T03:01:27Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.4 seconds
stream load tsv: 510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.7 seconds inserted 10000000 Rows, about 336K ops/s
storage size: 17162068440 Bytes

github-actions · 2023-08-06T08:20:19Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2023-08-06T08:55:34Z

run buildall

hello-stephen · 2023-08-06T09:40:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 40.36 seconds
stream load tsv: 512 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162074473 Bytes

yiguolei

LGTM

github-actions · 2023-08-06T15:16:19Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-08-06T15:16:21Z

PR approved by anyone and no changes requested.

be/src/common/config.cpp

be/src/http/action/stream_load_with_sql.cpp

fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java

fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java

fe/fe-core/src/main/java/org/apache/doris/tablefunction/StreamTableValuedFunction.java

gensrc/thrift/FrontendService.thrift

github-actions · 2023-08-07T10:35:38Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-08-07T13:18:46Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2023-08-07T13:25:22Z

run buildall

hello-stephen · 2023-08-07T14:42:20Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.13 seconds
stream load tsv: 524 seconds loaded 74807831229 Bytes, about 136 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162372176 Bytes

morningman

LGTM

github-actions · 2023-08-08T01:51:52Z

PR approved by at least one committer and no changes requested.

yiguolei

LGTM

Cai-Yao and others added 18 commits July 29, 2023 22:58

init

c2c3f14

fix context is null

9bab338

add setFileAttributes and clean unused code

c6f64e8

add stream load result and some regression test

b5d7f3f

fix some bugs

a47ab61

add stream load regression test

1bec888

add docs

65e6dec

fix bug and remove fe check privileges

1579dd1

add line delimiter and column separator test

773e04f

add reportStreamLoadStatus thread in FE and fix

bce76f4

Update be/src/http/action/stream_load_with_sql.cpp

a4586d7

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

add stream load with local file

83c8e79

rebase

0929bae

Update be/src/http/action/stream_load_with_sql.cpp

db9573e

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Merge branch 'master' into stream_load

c083718

rebase

7c6043c

Comment out the unit test for stream load SQL.

9248708

code format

706d30e

Co-authored-by: Cai-Yao <729673078@qq.com> Co-authored-by: yiguolei <676222867@qq.com>

freemandealer reviewed Aug 3, 2023

View reviewed changes

Merge branch 'master' into zs_stream_load

cb33d15

Conflicts: fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java

delete useless code

83462bd

resolve memory leak

70a4003

yiguolei previously approved these changes Aug 6, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 6, 2023

github-actions bot added the reviewed label Aug 6, 2023

morningman reviewed Aug 6, 2023

View reviewed changes

Removed some temporary code and added some TODO.

1e3c314

zzzzzzzs dismissed yiguolei’s stale review via 1e3c314 August 7, 2023 10:27

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 7, 2023

Changing the Variable Type in the Thrift File

3200045

morningman approved these changes Aug 8, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 8, 2023

yiguolei approved these changes Aug 8, 2023

View reviewed changes

yiguolei merged commit 66784ce into apache:master Aug 8, 2023

zzzzzzzs deleted the zs_stream_load branch September 6, 2023 05:25

[Enhancement](Load) Stream Load using SQL #22509

[Enhancement](Load) Stream Load using SQL #22509

Uh oh!

Conversation

zzzzzzzs commented Aug 2, 2023

Proposed changes

Further comments

Uh oh!

github-actions bot commented Aug 2, 2023

Uh oh!

zzzzzzzs commented Aug 2, 2023

Uh oh!

freemandealer Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

freemandealer Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

yiguolei Aug 6, 2023

Choose a reason for hiding this comment

Uh oh!

freemandealer Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

yiguolei commented Aug 4, 2023

Uh oh!

yiguolei commented Aug 4, 2023

Uh oh!

zzzzzzzs commented Aug 4, 2023

Uh oh!

github-actions bot commented Aug 4, 2023

Uh oh!

yiguolei commented Aug 5, 2023

Uh oh!

github-actions bot commented Aug 5, 2023

Uh oh!

zzzzzzzs commented Aug 5, 2023

Uh oh!

hello-stephen commented Aug 5, 2023

Uh oh!

yiguolei commented Aug 6, 2023

Uh oh!

hello-stephen commented Aug 6, 2023

Uh oh!

github-actions bot commented Aug 6, 2023

Uh oh!

yiguolei commented Aug 6, 2023

Uh oh!

hello-stephen commented Aug 6, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 6, 2023

Uh oh!

github-actions bot commented Aug 6, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Aug 7, 2023

Uh oh!

github-actions bot commented Aug 7, 2023

Uh oh!

yiguolei commented Aug 7, 2023

Uh oh!

hello-stephen commented Aug 7, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 8, 2023

Uh oh!