[fix][broker]fix the publish latency spike issue with large number of producers #20607
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
/pulsarbot run-failure-checks |
nodece
approved these changes
Jun 19, 2023
lifepuzzlefun
approved these changes
Jun 19, 2023
Contributor
Author
|
/pulsarbot run-failure-checks |
Codecov Report
@@ Coverage Diff @@
## master #20607 +/- ##
============================================
+ Coverage 72.29% 73.05% +0.76%
+ Complexity 31951 31889 -62
============================================
Files 1854 1867 +13
Lines 138462 138663 +201
Branches 15221 15237 +16
============================================
+ Hits 100095 101302 +1207
+ Misses 30384 29330 -1054
- Partials 7983 8031 +48
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Technoboy-
approved these changes
Jun 20, 2023
poorbarcode
approved these changes
Jun 20, 2023
mattisonchao
approved these changes
Jun 20, 2023
Technoboy-
pushed a commit
that referenced
this pull request
Jun 26, 2023
codelipenghui
added
release/3.0.1
cherry-picked/branch-3.0
and removed
release/3.0.2
labels
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jul 3, 2023
… producers (apache#20607) (cherry picked from commit 084d2ab) (cherry picked from commit 9fa56fc)
15 tasks
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The issue should only happen on the topic with many producers, and the max producer limitation is enabled. When a new producer is added, the broker will check all the existing producers and filter out the remote producer (from the replicator), which will run out of the CPU resource for a short period of time.
profiler.html.txt
Modifications
A counter will be much more efficient instead of going through all the producers. So a new userCreatedProducerCount is added to the topic.
Verifying this change
The existing tests already covered the max producer limitation. The change will not affect the correctness.
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
Documentation
docdoc-requireddoc-not-neededdoc-complete