[Audit][Stream Load] Support audit function for stream load by weizuo93 · Pull Request #5452 · apache/doris

weizuo93 · 2021-03-03T13:04:31Z

Proposed changes

Record finished stream load job (both successful job and failed job) into audit log so that we can see when the stream load job was executed and check the details of stream load jobs.

2021-03-09 14:11:51,821 [stream_load] |Label=1615270311|Db=test_db_1|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8926|StartTime=2021-03-09 14:11:51.692|FinishTime=2021-03-09 14:11:51.819
2021-03-09 14:11:52,969 [stream_load] |Label=1615270312|Db=test_db_2|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8926|StartTime=2021-03-09 14:11:52.848278|FinishTime=2021-03-09 14:11:52.968169
2021-03-09 14:11:54,122 [stream_load] |Label=1615270313|Db=test_db_3|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8926|StartTime=2021-03-09 14:11:53.995|FinishTime=2021-03-09 14:11:54.120
2021-03-09 14:12:00,494 [stream_load] |Label=1615270320|Db=test_db_0|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8947|StartTime=2021-03-09 14:12:00.314894|FinishTime=2021-03-09 14:12:00.493533
2021-03-09 14:12:01,648 [stream_load] |Label=1615270321|Db=test_db_1|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8947|StartTime=2021-03-09 14:12:01.525|FinishTime=2021-03-09 14:12:01.649
2021-03-09 14:12:02,809 [stream_load] |Label=1615270322|Db=test_db_2|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8947|StartTime=2021-03-09 14:12:02.675|FinishTime=2021-03-09 14:12:02.803
2021-03-09 14:12:03,957 [stream_load] |Label=1615270323|Db=test_db_3|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8947|StartTime=2021-03-09 14:12:03.831452|FinishTime=2021-03-09 14:12:03.955965
2021-03-09 14:12:10,333 [stream_load] |Label=1615270330|Db=test_db_0|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8928|StartTime=2021-03-09 14:12:10.209635|FinishTime=2021-03-09 14:12:10.332249
2021-03-09 14:12:11,548 [stream_load] |Label=1615270331|Db=test_db_1|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8928|StartTime=2021-03-09 14:12:11.363|FinishTime=2021-03-09 14:12:11.549
2021-03-09 14:12:12,702 [stream_load] |Label=1615270332|Db=test_db_2|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8928|StartTime=2021-03-09 14:12:12.574945|FinishTime=2021-03-09 14:12:12.700693
2021-03-09 14:12:13,914 [stream_load] |Label=1615270333|Db=test_db_3|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8928|StartTime=2021-03-09 14:12:13.731|FinishTime=2021-03-09 14:12:13.915
2021-03-09 14:12:20,237 [stream_load] |Label=1615270340|Db=test_db_0|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8933|StartTime=2021-03-09 14:12:20.110|FinishTime=2021-03-09 14:12:20.237
2021-03-09 14:12:21,391 [stream_load] |Label=1615270341|Db=test_db_1|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8933|StartTime=2021-03-09 14:12:21.264|FinishTime=2021-03-09 14:12:21.389
2021-03-09 14:12:22,557 [stream_load] |Label=1615270342|Db=test_db_2|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8933|StartTime=2021-03-09 14:12:22.418|FinishTime=2021-03-09 14:12:22.554
2021-03-09 14:12:23,720 [stream_load] |Label=1615270343|Db=test_db_3|Table=table_test|User=root|ClientIp=xxx|Status=Success|Message=OK|Url=N/A|TotalRows=200|LoadedRows=200|FilteredRows=0|UnselectedRows=0|LoadBytes=8933|StartTime=2021-03-09 14:12:23.587|FinishTime=2021-03-09 14:12:23.718

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

[] Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
[] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[] Documentation Update (if none of the other choices apply)
Code refactor (Modify the code structure, format the code, etc...)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
Compiling and unit tests pass locally with my changes
[] I have added tests that prove my fix is effective or that my feature works
[] If these changes need document changes, I have updated the document
Any dependent changes have been merged

be/src/common/config.h

be/src/http/action/stream_load.cpp

be/src/olap/storage_engine.cpp

be/src/runtime/stream_load/stream_load_context.cpp

be/src/runtime/stream_load/stream_load_record.cpp

fe/fe-core/src/main/java/org/apache/doris/load/StreamLoadRecordMgr.java

fe/fe-core/src/main/java/org/apache/doris/system/Backend.java

fe/fe-core/src/main/java/org/apache/doris/load/StreamLoadRecordMgr.java

morningman · 2021-03-21T14:52:43Z

And I pulled and run your PR, submit several stream load.
I can see log put stream_load_record rocksdb successfully. label: f0130520-835a-4d05-a38d-95f67f5b8c7b in be.INFO.
And I can see log finished to pull stream load records of all backend in fe.log.

But there is nothing got from BE. I don't know why, maybe you need to check it.

be/src/common/config.h

be/src/runtime/stream_load/stream_load_record.cpp

weizuo93 · 2021-03-26T02:49:36Z

And I pulled and run your PR, submit several stream load.
I can see log put stream_load_record rocksdb successfully. label: f0130520-835a-4d05-a38d-95f67f5b8c7b in be.INFO.
And I can see log finished to pull stream load records of all backend in fe.log.

But there is nothing got from BE. I don't know why, maybe you need to check it.

I test again. There should be nothing wrong.

be/src/common/config.h

be/src/runtime/stream_load/stream_load_record.h

gensrc/thrift/BackendService.thrift

be/src/http/action/stream_load.cpp

be/src/runtime/CMakeLists.txt

be/src/runtime/stream_load/stream_load_recorder.h

be/src/runtime/stream_load/stream_load_recorder.cpp

be/src/service/backend_service.cpp

morningman

LGTM

) Record finished stream load job (both successful job and failed job) into audit log so that we can see when the stream load job was executed and check the details of stream load jobs. Change-Id: I67f806a82f7b4f9d0ee28be5e9ad52e8c2a93aff

weizuo93 force-pushed the stream-load-audit branch from 3980b76 to a3ebfd4 Compare March 16, 2021 06:13

morningman self-assigned this Mar 21, 2021

morningman added the kind/feature Categorizes issue or PR as related to a new feature. label Mar 21, 2021

morningman reviewed Mar 21, 2021

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/load/StreamLoadRecordMgr.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/load/StreamLoadRecordMgr.java Outdated Show resolved Hide resolved

acelyc111 reviewed Mar 24, 2021

View reviewed changes

be/src/common/config.h Outdated Show resolved Hide resolved

be/src/runtime/stream_load/stream_load_record.cpp Outdated Show resolved Hide resolved

weizuo93 mentioned this pull request Mar 30, 2021

[Feature][Stream Load] Add "show stream load" to show stream load record #5488

Merged

4 tasks

morningman reviewed Apr 4, 2021

View reviewed changes

be/src/common/config.h Outdated Show resolved Hide resolved

be/src/runtime/stream_load/stream_load_record.h Outdated Show resolved Hide resolved

gensrc/thrift/BackendService.thrift Outdated Show resolved Hide resolved

weizuo added 2 commits April 6, 2021 11:59

store stream load record in rocksdb

ef96fc2

add default path for stream_load_recorder

30eae4b

weizuo93 force-pushed the stream-load-audit branch from d087f5f to 30eae4b Compare April 6, 2021 06:01

weizuo added 2 commits April 6, 2021 17:59

add default path for stream_load_recorder

ffe44b9

transfer timestamp as long type by RPC

d9069f4

acelyc111 reviewed Apr 6, 2021

View reviewed changes

weizuo added 2 commits April 7, 2021 20:43

transfer last stream load timestamp as int64 type from FE to BE by RPC

492aa67

transfer last stream load timestamp as int64 type from FE to BE by RPC

1c54c56

morningman reviewed Apr 7, 2021

View reviewed changes

be/src/service/backend_service.cpp Show resolved Hide resolved

morningman approved these changes Apr 20, 2021

View reviewed changes

morningman added the approved Indicates a PR has been approved by one committer. label Apr 20, 2021

morningman merged commit a4f8194 into apache:master Apr 21, 2021

morningman mentioned this pull request Oct 10, 2021

Release note 0.15.0 #6806

Closed

Conversation

weizuo93 commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Types of changes

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman commented Mar 21, 2021

Uh oh!

Uh oh!

Uh oh!

weizuo93 commented Mar 26, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

weizuo93 commented Mar 3, 2021 •

edited

Loading