[debug](rpc) debug rpc time consumption problem #39852
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
jacktengg
force-pushed
the
2.1-rpc-debug
branch
from
a114052 to
c1387a3
Compare
…lacklist to skip some backend during query
jacktengg
force-pushed
the
2.1-rpc-debug
branch
from
c1387a3 to
fc8bc74
Compare
Contributor
Author
|
run buildall |
jacktengg
force-pushed
the
2.1-rpc-debug
branch
from
ce2ce33 to
b268b08
Compare
yiguolei
reviewed
Aug 24, 2024
| template <typename Parent> | ||
| void ExchangeSinkBuffer<Parent>::register_sink(TUniqueId fragment_instance_id) { | ||
| void ExchangeSinkBuffer<Parent>::register_sink(TUniqueId fragment_instance_id, | ||
| vectorized::PipChannel<Parent>* channel) { |
| template <typename Parent> | ||
| Status ExchangeSinkBuffer<Parent>::_send_rpc(InstanceLoId id) { | ||
| if (_parent) { | ||
| SCOPED_TIMER(_parent->brpc_send_timer()); |
Contributor
There was a problem hiding this comment.
这么写没意义， scoped 就是在这个 {} 有用
| out << "Instance " << std::hex << stats->inst_lo_id; | ||
| auto stats_str = fmt::format( | ||
| "Count: {}, MaxTime: {}, MinTime: {}, AvgTime: {}, SumTime: {}", | ||
| stats->rpc_count, PrettyPrinter::print(stats->max_time, TUnit::TIME_NS), |
Contributor
There was a problem hiding this comment.
CloseTime: avg 11.310us, max 11.310us, min 11.310us
- ExecTime: avg 150.158us, max 150.158us, min 150.158us
- InitTime: avg 72.193us, max 72.193us, min 72.193us
| [](const auto& a, const auto& b) { return a->max_time > b->max_time; }); | ||
| auto count = std::min((size_t)max_count, _instance_to_rpc_stats_vec.size()); | ||
| int i = 0; | ||
| auto* detail_profile = profile->create_child("RpcInstanceDetails", true, true); |
Contributor
There was a problem hiding this comment.
这个profile 对象得检查下是否为nullptr
jacktengg
force-pushed
the
2.1-rpc-debug
branch
from
b268b08 to
51145fb
Compare
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
Contributor
Author
|
run buildall |
16 tasks
Gabriel39
pushed a commit
that referenced
this pull request
Nov 12, 2024
…n problem (#43284) Picked from the 2.1 branch, only the RPC profile-related code was selected. #39852 #40117 ``` DATA_STREAM_SINK_OPERATOR (id=2,dst_id=2): - RpcCount: sum 16, avg 4, max 4, min 4 - RpcMaxTime: avg 1.15ms, max 1.163ms, min 818.493us - RpcAvgTime: 11.850ms - RpcCount: 10 - RpcMaxTime: 86.891ms - RpcMinTime: 15.200ms - RpcSumTime: 118.503ms - SerializeBatchTime: 13.517ms - SplitBlockDistributeByChannelTime: 38.923ms - SplitBlockHashComputeTime: 2.659ms - UncompressedRowBatchSize: 135.19 KB - WaitForDependencyTime: 0ns - WaitForRpcBufferQueue: 0ns RpcInstanceDetails: - Instance 85d4f75b72a9ea61: Count: 4, MaxTime: 36.238ms, MinTime: 12.107ms, AvgTime: 21.722ms, SumTime: 86.891ms - Instance 85d4f75b72a9ea91: Count: 3, MaxTime: 11.107ms, MinTime: 2.431ms, AvgTime: 5.470ms, SumTime: 16.412ms - Instance 85d4f75b72a9eac1: Count: 3, MaxTime: 7.554ms, MinTime: 3.160ms, AvgTime: 5.066ms, SumTime: 15.200m ```
zzzxl1993
pushed a commit
to zzzxl1993/doris
that referenced
this pull request
Nov 12, 2024
…n problem (apache#43284) Picked from the 2.1 branch, only the RPC profile-related code was selected. apache#39852 apache#40117 ``` DATA_STREAM_SINK_OPERATOR (id=2,dst_id=2): - RpcCount: sum 16, avg 4, max 4, min 4 - RpcMaxTime: avg 1.15ms, max 1.163ms, min 818.493us - RpcAvgTime: 11.850ms - RpcCount: 10 - RpcMaxTime: 86.891ms - RpcMinTime: 15.200ms - RpcSumTime: 118.503ms - SerializeBatchTime: 13.517ms - SplitBlockDistributeByChannelTime: 38.923ms - SplitBlockHashComputeTime: 2.659ms - UncompressedRowBatchSize: 135.19 KB - WaitForDependencyTime: 0ns - WaitForRpcBufferQueue: 0ns RpcInstanceDetails: - Instance 85d4f75b72a9ea61: Count: 4, MaxTime: 36.238ms, MinTime: 12.107ms, AvgTime: 21.722ms, SumTime: 86.891ms - Instance 85d4f75b72a9ea91: Count: 3, MaxTime: 11.107ms, MinTime: 2.431ms, AvgTime: 5.470ms, SumTime: 16.412ms - Instance 85d4f75b72a9eac1: Count: 3, MaxTime: 7.554ms, MinTime: 3.160ms, AvgTime: 5.066ms, SumTime: 15.200m ```
py023
pushed a commit
to py023/doris
that referenced
this pull request
Nov 13, 2024
…n problem (apache#43284) Picked from the 2.1 branch, only the RPC profile-related code was selected. apache#39852 apache#40117 ``` DATA_STREAM_SINK_OPERATOR (id=2,dst_id=2): - RpcCount: sum 16, avg 4, max 4, min 4 - RpcMaxTime: avg 1.15ms, max 1.163ms, min 818.493us - RpcAvgTime: 11.850ms - RpcCount: 10 - RpcMaxTime: 86.891ms - RpcMinTime: 15.200ms - RpcSumTime: 118.503ms - SerializeBatchTime: 13.517ms - SplitBlockDistributeByChannelTime: 38.923ms - SplitBlockHashComputeTime: 2.659ms - UncompressedRowBatchSize: 135.19 KB - WaitForDependencyTime: 0ns - WaitForRpcBufferQueue: 0ns RpcInstanceDetails: - Instance 85d4f75b72a9ea61: Count: 4, MaxTime: 36.238ms, MinTime: 12.107ms, AvgTime: 21.722ms, SumTime: 86.891ms - Instance 85d4f75b72a9ea91: Count: 3, MaxTime: 11.107ms, MinTime: 2.431ms, AvgTime: 5.470ms, SumTime: 16.412ms - Instance 85d4f75b72a9eac1: Count: 3, MaxTime: 7.554ms, MinTime: 3.160ms, AvgTime: 5.066ms, SumTime: 15.200m ```
Mryange
added a commit
to Mryange/doris
that referenced
this pull request
Dec 4, 2024
…n problem (apache#43284) Picked from the 2.1 branch, only the RPC profile-related code was selected. apache#39852 apache#40117 ``` DATA_STREAM_SINK_OPERATOR (id=2,dst_id=2): - RpcCount: sum 16, avg 4, max 4, min 4 - RpcMaxTime: avg 1.15ms, max 1.163ms, min 818.493us - RpcAvgTime: 11.850ms - RpcCount: 10 - RpcMaxTime: 86.891ms - RpcMinTime: 15.200ms - RpcSumTime: 118.503ms - SerializeBatchTime: 13.517ms - SplitBlockDistributeByChannelTime: 38.923ms - SplitBlockHashComputeTime: 2.659ms - UncompressedRowBatchSize: 135.19 KB - WaitForDependencyTime: 0ns - WaitForRpcBufferQueue: 0ns RpcInstanceDetails: - Instance 85d4f75b72a9ea61: Count: 4, MaxTime: 36.238ms, MinTime: 12.107ms, AvgTime: 21.722ms, SumTime: 86.891ms - Instance 85d4f75b72a9ea91: Count: 3, MaxTime: 11.107ms, MinTime: 2.431ms, AvgTime: 5.470ms, SumTime: 16.412ms - Instance 85d4f75b72a9eac1: Count: 3, MaxTime: 7.554ms, MinTime: 3.160ms, AvgTime: 5.066ms, SumTime: 15.200m ```
16 tasks
Mryange
added a commit
to Mryange/doris
that referenced
this pull request
Jan 10, 2025
…n problem (apache#43284) Picked from the 2.1 branch, only the RPC profile-related code was selected. apache#39852 apache#40117 ``` DATA_STREAM_SINK_OPERATOR (id=2,dst_id=2): - RpcCount: sum 16, avg 4, max 4, min 4 - RpcMaxTime: avg 1.15ms, max 1.163ms, min 818.493us - RpcAvgTime: 11.850ms - RpcCount: 10 - RpcMaxTime: 86.891ms - RpcMinTime: 15.200ms - RpcSumTime: 118.503ms - SerializeBatchTime: 13.517ms - SplitBlockDistributeByChannelTime: 38.923ms - SplitBlockHashComputeTime: 2.659ms - UncompressedRowBatchSize: 135.19 KB - WaitForDependencyTime: 0ns - WaitForRpcBufferQueue: 0ns RpcInstanceDetails: - Instance 85d4f75b72a9ea61: Count: 4, MaxTime: 36.238ms, MinTime: 12.107ms, AvgTime: 21.722ms, SumTime: 86.891ms - Instance 85d4f75b72a9ea91: Count: 3, MaxTime: 11.107ms, MinTime: 2.431ms, AvgTime: 5.470ms, SumTime: 16.412ms - Instance 85d4f75b72a9eac1: Count: 3, MaxTime: 7.554ms, MinTime: 3.160ms, AvgTime: 5.066ms, SumTime: 15.200m ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx
Add detail RPC time info for each channel, sorted by max rpc time of channels: