-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[improvement](fe) support specify broker usage for Multi-catalog #23024
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
run buildall |
|
cc @morningman |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
new pr #24830 ,close this pr |
Proposed changes
Issue Number: close #xxx
I want to use Doris Multi-catalog to accelerate HMS query. My organization has custom distributed file system, and we think wrapping the fs access difference into broker (listLocatedFiles, openReader..) would be a elegant approach.
This pr introduce HMS catalog conf
bind.broker.name. If we set this conf, file split, query scan operation will send to broker.usage:
create a hms catalog with broker usage
When we try to query from this catalog, file split and query scan request will send to broker
hdfs_broker.More details about this pr:
bind.broker.nameto specify broker name to do remote path work. Whenbind.broker.nameis set,enable.self.splittermust betrueto ensure file splitting process is executed in FeTBrokerIsSplittableResponse isSplittable(1: TBrokerIsSplittableRequest request), helps to invoke input formatisSplitableinterface.TBrokerListResponse listLocatedFiles(1: TBrokerListPathRequest request), helps to dolistFilesorlistLocatedStatusfor remote file systemisSplittablelistLocatedFilesof table / partition locations.OpenReaderfor specified file splits.Further comments