[WIP][Issue 6437][broker] Prevent marker messages from accumulating in backlog of replicated subscription #6592
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
@jiazhai @codelipenghui @merlimat can you review this pull request? |
Comment on lines
+126
to
+137
| try { | ||
| ReplicatedSubscriptionsSnapshot snapshot = Markers.instantiateReplicatedSubscriptionsSnapshot(snapshotId, | ||
| controller.localCluster(), p.getLedgerId(), p.getEntryId(), responses); | ||
| controller.topic().getSubscriptions().forEach((subName, sub) -> { | ||
| if (sub != null) { | ||
| sub.processReplicatedSubscriptionSnapshot(snapshot); | ||
| } | ||
| }); | ||
| } catch (Throwable t) { | ||
| log.warn("[{}] Failed to process replicated subscription snapshot {} -- {}", controller.topic().getName(), | ||
| snapshotId, t.getMessage()); | ||
| } |
Contributor
There was a problem hiding this comment.
I don’t understand a bit here. If we don't store snapshots into the topic, how can we recover snapshots after broker restarts?
Contributor
Author
There was a problem hiding this comment.
I don't think that snapshots need to be persisted because they are constantly being updated with new ones, is that wrong? What issues do you think would arise if we don't persist ReplicatedSubscriptionsSnapshot messages?
massakam
changed the title
Contributor
|
move to 2.7.0, feel free to move it back if needed. |
added 2 commits
June 9, 2020 13:35
massakam
force-pushed
the
repl-sub-backlog
branch
from
b402e99 to
6815fad
Compare
Contributor
Author
|
Close this PR once. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #6437
Motivation
In a replicated subscription with no consumers connected, the number of marker messages in the backlog will continue to increase. If at least one consumer is connected, the marker messages will be acknowledged and deleted by the dispatcher.
pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java
Lines 80 to 92 in 5fc4a90
However, if no consumers are connected, the dispatcher does not exist or has stopped reading entries. As a result, the marker messages accumulate in the backlog without being acknoledged by anyone.
Modifications
There are four types of marker messages:
Of these, three messages, except
ReplicatedSubscriptionsSnapshot, are not used in the local cluster. They are published to local topics for sending to remote clusters. So, modified theReplicatedSubscriptionsControllerclass to acknowledge these marker messages on all subscriptions immediately after publishing the messages to a local topic.On the other hand,
ReplicatedSubscriptionsSnapshotis used only in the local cluster and does not need to be sent to remote clusters. So stopped publishingReplicatedSubscriptionsSnapshotto topics.In addition, marker messages sent from remote clusters are now acknowledged by the replicator on all subscriptions.
With these changes, marker messages no longer continue to accumulate in the replicated subscription backlog.