Skip to content

[profile](rpc) add RpcInstanceDetails profile to debug rpc consumption problem #43284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Gabriel39 merged 3 commits into from
Nov 12, 2024
Merged

[profile](rpc) add RpcInstanceDetails profile to debug rpc consumption problem #43284

Gabriel39 merged 3 commits into from
Nov 12, 2024

Conversation

@Mryange
Copy link
Contributor

@Mryange Mryange commented Nov 5, 2024 •
edited by Gabriel39
Loading

What problem does this PR solve?

Picked from the 2.1 branch, only the RPC profile-related code was selected.
#39852
#40117

                      DATA_STREAM_SINK_OPERATOR  (id=2,dst_id=2):
                            -  RpcCount:  sum  16,  avg  4,  max  4,  min  4
                            -  RpcMaxTime:  avg  1.15ms,  max  1.163ms,  min  818.493us
               
 -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

doris-robot commented Nov 5, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Mryange
Copy link
Contributor Author

Mryange commented Nov 5, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Mryange
Copy link
Contributor Author

Mryange commented Nov 6, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

TeamCity be ut coverage result:
Function Coverage: 37.85% (9836/25986)
Line Coverage: 29.02% (81811/281909)
Region Coverage: 28.25% (42148/149181)
Branch Coverage: 24.84% (21391/86116)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8797d4171878d3b97522d304a5b898672b5dbdc9_8797d4171878d3b97522d304a5b898672b5dbdc9/report/index.html

@Mryange
Copy link
Contributor Author

Mryange commented Nov 6, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

TeamCity be ut coverage result:
Function Coverage: 37.85% (9835/25986)
Line Coverage: 29.02% (81809/281909)
Region Coverage: 28.26% (42154/149181)
Branch Coverage: 24.83% (21384/86116)
Coverage Report: http://coverage.selectdb-in.cc/coverage/db64af31e200cb8a3cd816f76a4bfc0f8ed93113_db64af31e200cb8a3cd816f76a4bfc0f8ed93113/report/index.html

HappenLee
HappenLee reviewed Nov 6, 2024
View reviewed changes
be/src/pipeline/exec/exchange_sink_buffer.cpp Outdated
// attach task for memory tracker and query id when core
SCOPED_ATTACH_TASK(_state);
set_rpc_time(id, start_rpc_time, result.receive_time());
{
Copy link
Contributor

@HappenLee HappenLee Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need the {}

HappenLee
HappenLee previously approved these changes Nov 6, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 6, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by anyone and no changes requested.

@Mryange
Copy link
Contributor Author

Mryange commented Nov 6, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 6, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

TeamCity be ut coverage result:
Function Coverage: 37.87% (9848/26002)
Line Coverage: 29.03% (81883/282062)
Region Coverage: 28.25% (42167/149253)
Branch Coverage: 24.83% (21394/86150)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e595f7ace16526356f753cae5cbbf2ebb451dd54_e595f7ace16526356f753cae5cbbf2ebb451dd54/report/index.html

@github-actions
Copy link
Contributor

github-actions bot commented Nov 12, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 12, 2024
@Gabriel39 Gabriel39 merged commit 736f1ad into apache:master Nov 12, 2024
zzzxl1993 pushed a commit to zzzxl1993/doris that referenced this pull request Nov 12, 2024
@Mryange @zzzxl1993
[profile](rpc) add RpcInstanceDetails profile to debug rpc consumptio…
79e3297 
…n problem (apache#43284)

Picked from the 2.1 branch, only the RPC profile-related code was
selected.
apache#39852
apache#40117

```
                      DATA_STREAM_SINK_OPERATOR  (id=2,dst_id=2):
                            -  RpcCount:  sum  16,  avg  4,  max  4,  min  4
                            -  RpcMaxTime:  avg  1.15ms,  max  1.163ms,  min  818.493us
               
 -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m
```
py023 pushed a commit to py023/doris that referenced this pull request Nov 13, 2024
@Mryange @py023
[profile](rpc) add RpcInstanceDetails profile to debug rpc consumptio…
a31a941 
…n problem (apache#43284)

Picked from the 2.1 branch, only the RPC profile-related code was
selected.
apache#39852
apache#40117

```
                      DATA_STREAM_SINK_OPERATOR  (id=2,dst_id=2):
                            -  RpcCount:  sum  16,  avg  4,  max  4,  min  4
                            -  RpcMaxTime:  avg  1.15ms,  max  1.163ms,  min  818.493us
               
 -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m
```
Mryange added a commit to Mryange/doris that referenced this pull request Dec 4, 2024
@Mryange
[profile](rpc) add RpcInstanceDetails profile to debug rpc consumptio…
f12669d 
…n problem (apache#43284)

Picked from the 2.1 branch, only the RPC profile-related code was
selected.
apache#39852
apache#40117

```
                      DATA_STREAM_SINK_OPERATOR  (id=2,dst_id=2):
                            -  RpcCount:  sum  16,  avg  4,  max  4,  min  4
                            -  RpcMaxTime:  avg  1.15ms,  max  1.163ms,  min  818.493us

 -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m
```
@Mryange Mryange mentioned this pull request Dec 4, 2024
Closed
16 tasks
Mryange added a commit to Mryange/doris that referenced this pull request Jan 10, 2025
@Mryange
[profile](rpc) add RpcInstanceDetails profile to debug rpc consumptio…
f770be1 
…n problem (apache#43284)

Picked from the 2.1 branch, only the RPC profile-related code was
selected.
apache#39852
apache#40117

```
                      DATA_STREAM_SINK_OPERATOR  (id=2,dst_id=2):
                            -  RpcCount:  sum  16,  avg  4,  max  4,  min  4
                            -  RpcMaxTime:  avg  1.15ms,  max  1.163ms,  min  818.493us

 -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m
```
@Mryange Mryange mentioned this pull request Jan 10, 2025
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jacktengg jacktengg jacktengg approved these changes

@Gabriel39 Gabriel39 Gabriel39 approved these changes

@HappenLee HappenLee HappenLee left review comments

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Mryange @doris-robot @HappenLee @jacktengg @Gabriel39 @yiguolei