Avoid potentially blocking calls to metadata on critical threads#12339
Merged
merlimat merged 6 commits intoapache:masterfrom Oct 14, 2021
Merged
Avoid potentially blocking calls to metadata on critical threads#12339merlimat merged 6 commits intoapache:masterfrom
merlimat merged 6 commits intoapache:masterfrom
Conversation
merlimat
requested review from
cckellogg,
codelipenghui,
hangc0276,
jerrypeng and
rdhabalia
cckellogg
approved these changes
Oct 12, 2021
michaeljmarshall
requested changes
Oct 12, 2021
Member
michaeljmarshall
left a comment
michaeljmarshall
left a comment
There was a problem hiding this comment.
Looks good. Left some minor comments.
merlimat
added a commit
to merlimat/pulsar
that referenced
this pull request
Oct 12, 2021
merlimat
added a commit
to merlimat/pulsar
that referenced
this pull request
Oct 12, 2021
codelipenghui
approved these changes
Oct 13, 2021
Contributor
|
@michaeljmarshall Please help review again |
merlimat
added a commit
to streamnative/pulsar-archived
that referenced
this pull request
Oct 13, 2021
merlimat
added a commit
that referenced
this pull request
Oct 13, 2021
… threads (#12339) (#12340) * Avoid potentially blocking calls to metadata on critical threads (#12339) * Fixed NPE * Addressed comments * Fixed issue with mocked tests * Fixed behavior in BacklogQuotaManager to be like before * Fixed AuthorizationProducerConsumerTest * Fixed PersistentTopicTest * Fixed PersistentTopicTest
eolivelli
approved these changes
Oct 13, 2021
michaeljmarshall
approved these changes
Oct 13, 2021
Member
michaeljmarshall
left a comment
michaeljmarshall
left a comment
There was a problem hiding this comment.
LGTM. Left one question about exception handling.
Comment on lines
+785
to
+787
| policies = brokerService.pulsar().getPulsarResources().getNamespaceResources().getPoliciesIfCached( | ||
| TopicName.get(topic).getNamespaceObject()) | ||
| .orElseGet(() -> new Policies()); |
Member
There was a problem hiding this comment.
I see this was added in the most recent commit to help with a mocked test. This block doesn't throw any checked exceptions. Do we expect RuntimeExceptions from it? If we do, should we wrap other similar code blocks in this PR with try and catch? If we don't, perhaps we should update the test.
Contributor
Author
There was a problem hiding this comment.
Many tests are using mocked versions which would throw exception... then the try/catch was the thing that was making all these tests to work..
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Oct 27, 2021
…che#12339) * Avoid potentially blocking calls to metadata on critical threads * Fixed log arguments order * Addressed comments * Fixed mock in PersistentSubscriptionTest * Fixed issue in mocked tests * Fixed test that was force policies modification under the hood (cherry picked from commit c0e87c0)
bharanic-dev
pushed a commit
to bharanic-dev/pulsar
that referenced
this pull request
Mar 18, 2022
…che#12339) * Avoid potentially blocking calls to metadata on critical threads * Fixed log arguments order * Addressed comments * Fixed mock in PersistentSubscriptionTest * Fixed issue in mocked tests * Fixed test that was force policies modification under the hood
Jason918
pushed a commit
to Jason918/pulsar
that referenced
this pull request
Jul 31, 2022
…ause org.apache.pulsar.broker.admin.AdminApiTest#testTopicStatsLastExpireTimestampForSubscription failing.
2 tasks
Jason918
pushed a commit
to Jason918/pulsar
that referenced
this pull request
Jul 31, 2022
Jason918
added a commit
that referenced
this pull request
Jul 31, 2022
* Revert "[fix][proxy] Fix client service url (#16834)" This reverts commit 10b4e99. * Revert "[Build] Use grpc-bom to align grpc library versions (#15234)" This reverts commit 99c93d2. * Revert "upgrade aircompressor to 0.20 (#11790)" This reverts commit 5ad16b6. * Revert "[Branch-2.7] Fixed deadlock on metadata cache missing while doing checkReplication (#12484)" This reverts commit 32fe228. * Revert changes of PersistentTopic#getMessageTTL in #12339. Co-authored-by: JiangHaiting <janghaiting@apache.org>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
We have several places that are accessing the namespace policies in a synchronous way from the critical IO threads of the broker. These accesses are almost always non-blocking calls since the policies are already cached (given that the topic was just loaded and the policies were checked then).
There are few corner cases though in which we would incur in cache misses and that could have a big impact, up to result in a complete deadlock, if we exhaust all the threads in the ordered executors.
For example this can happen when there are thousands of producers/consumers connecting and there is a change in the policies that triggers a cache invalidation.
We must have no potentially blocking calls in the critical path.