Skip to content

Fix system topic replicate issue #14605

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
codelipenghui merged 3 commits into from
Mar 10, 2022
Merged

Fix system topic replicate issue #14605

codelipenghui merged 3 commits into from
Mar 10, 2022

Conversation

@Technoboy-
Copy link
Contributor

@Technoboy- Technoboy- commented Mar 8, 2022

Motivation

PIP 92 has introduced topic policies across clusters. But after #12517, if the policy is not global, it set the replicate cluster to an empty set.

PulsarEvent.PulsarEventBuilder builder = PulsarEvent.builder();
 if (policies == null || !policies.isGlobalPolicies()) {
     // we don't need to replicate local policies to remote cluster, so set `replicateTo` to empty.
     builder.replicateTo(new HashSet<>());
}

It should set the replicateTo with the local cluster, not an empty set.

Otherwise, it will cause the system event to be replicated. Details are here :

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java

Lines 319 to 328 in d4c2e61

if (msg.hasReplicateTo() && !msg.getReplicateTo().contains(remoteCluster)) {
if (log.isDebugEnabled()) {
log.debug("[{}][{} -> {}] Skipping message at position {}, replicateTo {}", topicName,
localCluster, remoteCluster, entry.getPosition(), msg.getReplicateTo());
}
cursor.asyncDelete(entry.getPosition(), this, entry.getPosition());
entry.release();
msg.recycle();
continue;
}

Documentation

  • no-need-doc

@Technoboy- Technoboy- self-assigned this Mar 8, 2022
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 8, 2022
@codelipenghui codelipenghui added this to the 2.11.0 milestone Mar 9, 2022
@codelipenghui codelipenghui requested a review from hangc0276 March 9, 2022 05:32
Jason918
Jason918 approved these changes Mar 9, 2022
View reviewed changes
Copy link
Contributor

@Jason918 Jason918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Technoboy- Technoboy- mentioned this pull request Mar 10, 2022
Merged
1 task
hangc0276
hangc0276 approved these changes Mar 10, 2022
View reviewed changes
Copy link
Contributor

@hangc0276 hangc0276 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!

@codelipenghui codelipenghui merged commit e470de5 into apache:master Mar 10, 2022
codelipenghui pushed a commit that referenced this pull request Mar 12, 2022
@Technoboy- @codelipenghui
Fix system topic replicate issue (#14605)
e75f0a8 
### Motivation
PIP 92 has introduced topic policies across clusters. But after #12517, if the policy is not global, it set the replicate cluster to an empty set.
```
PulsarEvent.PulsarEventBuilder builder = PulsarEvent.builder();
 if (policies == null || !policies.isGlobalPolicies()) {
     // we don't need to replicate local policies to remote cluster, so set `replicateTo` to empty.
     builder.replicateTo(new HashSet<>());
}
```
It should set the `replicateTo` with the local cluster, not an empty set.

Otherwise,  it will cause the system event to be replicated. Details are here :
https://github.com/apache/pulsar/blob/d4c2e613d305f8f785b5ef357b7cbe2ccc271043/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java#L319-L328

(cherry picked from commit e470de5)
@codelipenghui codelipenghui modified the milestones: 2.11.0, 2.10.0 Mar 12, 2022
Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Apr 20, 2022
@Technoboy-
Fix system topic replicate issue (apache#14605)
26fa691 
### Motivation
PIP 92 has introduced topic policies across clusters. But after apache#12517, if the policy is not global, it set the replicate cluster to an empty set.
```
PulsarEvent.PulsarEventBuilder builder = PulsarEvent.builder();
 if (policies == null || !policies.isGlobalPolicies()) {
     // we don't need to replicate local policies to remote cluster, so set `replicateTo` to empty.
     builder.replicateTo(new HashSet<>());
}
```
It should set the `replicateTo` with the local cluster, not an empty set.

Otherwise,  it will cause the system event to be replicated. Details are here :
https://github.com/apache/pulsar/blob/d4c2e613d305f8f785b5ef357b7cbe2ccc271043/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java#L319-L328
@Technoboy- Technoboy- deleted the fix-system-topic-replicator-1 branch August 10, 2022 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jason918 Jason918 Jason918 approved these changes

@hangc0276 hangc0276 hangc0276 approved these changes

@315157973 315157973 315157973 approved these changes

@codelipenghui codelipenghui codelipenghui approved these changes

@nodece nodece nodece approved these changes

Assignees

@Technoboy- Technoboy-

Labels

area/broker doc-not-needed Your PR changes do not impact docs

Projects

None yet

Milestone

2.10.0

Development

Successfully merging this pull request may close these issues.

6 participants

@Technoboy- @Jason918 @hangc0276 @315157973 @codelipenghui @nodece