[fix][broker] Avoid infinite bundle unloading #20822

Demogorgon314 · 2023-07-17T10:03:34Z

Motivation

The bundle ownership assignment logic doesn't know the previous unloaded broker when unloading happens. It might assign the bundle to the same broker. In this case, it might cause an infinite bundle unloading loop.

To resolve this issue, we can check the destination broker when doing the doLoadShedding stage. When the destination broker is the same as the current owner broker, we can skip this unload, and if it is different, we set the new owner in this stage.

Modifications

Transfer the bundle to the new owner when needed to unload.

Verifying this change

See the ModularLoadManagerImplTest#testLoadShedding unit test.

Documentation

doc
doc-required
doc-not-needed
doc-complete

codelipenghui

The PR title should be a fix for the infinite bundle unloading.

If I understand correctly, a breaking change will be introduced to the behavior of loadBalancerDistributeBundlesEvenlyEnabled because the current solution just filters out the previous owner, then the bundle can be moved to other brokers. I think the correct solution should be to skip the bundle unloading if the next owner is the same one as the current owner.

IMO, the main point is we don't care about the next owner broker when performing the bundle unloading. But the bundle unloading and the owner selection are based on the same load data. So is it worth checking the next owner at the unloading stage? @Demogorgon314 @heesung-sn

codelipenghui

BTW, I noticed no one put data to LoadData.brokerData and LoadData.bundleData except for the test. Is it still useful? or maybe I missed something.

Demogorgon314 · 2023-07-19T01:48:00Z

BTW, I noticed no one put data to LoadData.brokerData and LoadData.bundleData except for the test. Is it still useful? or maybe I missed something.

@codelipenghui Yes, it is still in use. The ModularLoadManagerImpl#updateAllBrokerData method puts data to LoadData.brokerData, and the ModularLoadManagerImpl#updateBundleData method puts data to LoadData.bundleData

Demogorgon314 · 2023-07-19T07:39:32Z

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

+                        pulsar.getAdminClient().namespaces()
+                                .unloadNamespaceBundle(namespaceName, bundleRange, destBroker.get());


The destinationBroker parameter is added in this PR: #18663

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

codelipenghui · 2023-07-19T07:58:38Z

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

                    try {
-                        pulsar.getAdminClient().namespaces().unloadNamespaceBundle(namespaceName, bundleRange);
+                        pulsar.getAdminClient().namespaces()
+                                .unloadNamespaceBundle(namespaceName, bundleRange, destBroker.get());


Just a reminder, The destBroker is only available after 2.11.0. We are not able to cherry-pick to branch-2.10

Or maybe we don't need to ensure the bundle will actually go to the dest broker? We just want to avoid the infinite loop as possible

When we have many brokers, it still has a chance to select the same broker if we don't pass the destBroker parameter. I wonder if this is acceptable.

But it will only sometimes go to the same broker, not always. In 2.10, we don't have this API to support a strict owner assignment at the loading stage.

Ok, I see. After this PR is merged, I will push a PR for branch 2.10.

For 2.10 we don't have a good way for that. What will happen if the same broker is still selected after unloading? Will it retry?

@BewareMyPower It will be selected to unload again. And hope next time it will be skipped.

Ok, I see. After this PR is merged, I will push a PR for branch 2.10.

To be clear, how are you going to push this fix to 2.10? According to @codelipenghui, the destBroker option in the unload admin is only available since 2.11.

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

…ce/impl/ModularLoadManagerImpl.java Co-authored-by: Yunze Xu <xyzinfernity@163.com>

codecov-commenter · 2023-07-21T10:47:21Z

Codecov Report

Attention: Patch coverage is 77.33333% with 17 lines in your changes missing coverage. Please review.

Project coverage is 72.64%. Comparing base (57fbee4) to head (531e964).
Report is 1754 commits behind head on master.

Files with missing lines	Patch %	Lines
...roker/loadbalance/impl/ModularLoadManagerImpl.java	77.33%	13 Missing and 4 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #20822      +/-   ##
============================================
- Coverage     72.97%   72.64%   -0.34%     
- Complexity    32157    32227      +70     
============================================
  Files          1868     1856      -12     
  Lines        139164   139139      -25     
  Branches      15314    15329      +15     
============================================
- Hits         101555   101077     -478     
- Misses        29562    29999     +437     
- Partials       8047     8063      +16

Flag	Coverage Δ
inttests	`24.24% <48.00%> (?)`
systests	`∅ <ø> (∅)`
unittests	`72.40% <76.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...roker/loadbalance/impl/ModularLoadManagerImpl.java	`82.75% <77.33%> (+0.27%)`	⬆️

... and 168 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codelipenghui · 2023-07-25T14:34:23Z

@Demogorgon314 Could you please help cherry-pick the PR into release branches?

(cherry picked from commit 3f63768)

heesung-sohn · 2023-07-27T20:29:53Z

Though this is another great improvement in ModularLoadManager, this is a behavior change too.
For this behavior change,

Do we need to update any doc?
Do we need to put this logic under any config, as this is a behavior change?

It is great that this logic can block unloading bundles to the same broker.
e.g) Unloading to the same broker
A->A

However, could infinite unloading still happen if unloading bounces to multiple brokers?
e.g) Unloading bounces between two brokers
A->B->A->B

Demogorgon314 · 2023-07-28T00:52:12Z

@heesung-sn

Do we need to update any doc?

Yes, It's good to add a doc to notify the users.

Do we need to put this logic under any config, as this is a behavior change?

I think we can leave config as is, since unloading to the same broker is not make sense.

However, could infinite unloading still happen if unloading bounces to multiple brokers?

Yes, this can happen, not sure if we have a good way to prevent it.

…20877)

shibd · 2023-10-21T09:22:59Z

cherry-picked branch-2.11 by #20878

hanmz · 2024-02-21T11:39:43Z

Why not call selectBrokerForAssignment directly, but add a new selectBroker method?The only difference I see is that it is not added to preallocated . What is the impact if added to preallocated ?

BewareMyPower · 2024-08-23T17:32:23Z

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

+        final ConcurrentOpenHashMap<String, ConcurrentOpenHashSet<String>> namespaceToBundleRange =
+                brokerToNamespaceToBundleRange
+                        .computeIfAbsent(broker,
+                                k -> ConcurrentOpenHashMap.<String,
+                                        ConcurrentOpenHashSet<String>>newBuilder()
+                                        .build());
+        synchronized (namespaceToBundleRange) {
+            namespaceToBundleRange.computeIfAbsent(namespaceName,
+                    k -> ConcurrentOpenHashSet.<String>newBuilder().build())
+                    .add(bundleRange);
+        }


The synchronized block here seems meaningless?

Yes... not sure why used synchronized block here before.

Demogorgon314 added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages area/broker labels Jul 17, 2023

Demogorgon314 requested a review from codelipenghui July 17, 2023 10:03

Demogorgon314 self-assigned this Jul 17, 2023

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jul 17, 2023

Demogorgon314 force-pushed the Demogorgon314/Avoid-assign-bundles-to-previous-unloaded-broker branch from f953f89 to bcbc1af Compare July 17, 2023 10:04

Demogorgon314 marked this pull request as ready for review July 18, 2023 02:04

codelipenghui requested a review from hangc0276 July 18, 2023 10:31

codelipenghui reviewed Jul 19, 2023

View reviewed changes

Demogorgon314 changed the title ~~[improve][broker] Avoid assigning bundle to the previous unloaded broker~~ [fix][broker] Avoid infinite bundle unloading Jul 19, 2023

Demogorgon314 marked this pull request as draft July 19, 2023 04:49

Transfer the bundle to selected broker in doLoadShedding stage

bb2ed0d

Demogorgon314 force-pushed the Demogorgon314/Avoid-assign-bundles-to-previous-unloaded-broker branch from ef68011 to bb2ed0d Compare July 19, 2023 07:17

streamnativebot mentioned this pull request Jul 19, 2023

PR-20822: [fix][broker] Avoid infinite bundle unloading streamnative/pulsar-archived#5838

Open

4 tasks

Demogorgon314 marked this pull request as ready for review July 19, 2023 07:26

streamnativebot mentioned this pull request Jul 19, 2023

PR-20822: [fix][broker] Avoid infinite bundle unloading streamnative/pulsar-archived#5839

Open

4 tasks

Demogorgon314 commented Jul 19, 2023

View reviewed changes

streamnativebot mentioned this pull request Jul 19, 2023

PR-20822: [fix][broker] Avoid infinite bundle unloading streamnative/pulsar-archived#5841

Closed

4 tasks

codelipenghui reviewed Jul 19, 2023

View reviewed changes

codelipenghui added type/bug The PR fixed a bug or issue reported a bug and removed type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages labels Jul 19, 2023

Addressed comments

aaa4498

codelipenghui added this to the 3.1.0 milestone Jul 19, 2023

codelipenghui approved these changes Jul 19, 2023

View reviewed changes

codelipenghui added release/3.0.2 release/2.11.3 labels Jul 19, 2023

mattisonchao self-requested a review July 19, 2023 08:45

codelipenghui added the release/2.10.6 label Jul 19, 2023

Demogorgon314 added the ready-to-test label Jul 19, 2023

BewareMyPower approved these changes Jul 19, 2023

View reviewed changes

BewareMyPower reviewed Jul 19, 2023

View reviewed changes

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java Outdated Show resolved Hide resolved

Update pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalan…

531e964

…ce/impl/ModularLoadManagerImpl.java Co-authored-by: Yunze Xu <xyzinfernity@163.com>

Demogorgon314 closed this Jul 21, 2023

Demogorgon314 reopened this Jul 21, 2023

codelipenghui merged commit 3f63768 into apache:master Jul 25, 2023

Demogorgon314 added a commit to Demogorgon314/pulsar that referenced this pull request Jul 26, 2023

[fix][broker] Avoid infinite bundle unloading (apache#20822)

a0499bd

(cherry picked from commit 3f63768)

Demogorgon314 added a commit to Demogorgon314/pulsar that referenced this pull request Jul 26, 2023

[fix][broker] Avoid infinite bundle unloading (apache#20822)

19ae109

(cherry picked from commit 3f63768)

Demogorgon314 added a commit that referenced this pull request Jul 26, 2023

[fix][broker] Avoid infinite bundle unloading (#20822)

f674ad1

(cherry picked from commit 3f63768)

Demogorgon314 added the cherry-picked/branch-3.0 label Jul 26, 2023

RobertIndie added release/3.0.1 and removed release/3.0.2 labels Jul 26, 2023

Technoboy- pushed a commit that referenced this pull request Aug 8, 2023

[branch-2.10][fix][broker] Avoid infinite bundle unloading (#20822) (#…

31d59cf

…20877)

shibd added the cherry-picked/branch-2.10 label Oct 21, 2023

shibd added the cherry-picked/branch-2.11 label Oct 21, 2023

BewareMyPower reviewed Aug 23, 2024

View reviewed changes

		pulsar.getAdminClient().namespaces()
		.unloadNamespaceBundle(namespaceName, bundleRange, destBroker.get());

[fix][broker] Avoid infinite bundle unloading #20822

[fix][broker] Avoid infinite bundle unloading #20822

Uh oh!

Conversation

Demogorgon314 commented Jul 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Documentation

Uh oh!

codelipenghui left a comment

Choose a reason for hiding this comment

Uh oh!

codelipenghui left a comment

Choose a reason for hiding this comment

Uh oh!

Demogorgon314 commented Jul 19, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Jul 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codelipenghui commented Jul 25, 2023

Uh oh!

heesung-sohn commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Demogorgon314 commented Jul 28, 2023

Uh oh!

shibd commented Oct 21, 2023

Uh oh!

hanmz commented Feb 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Demogorgon314 commented Jul 17, 2023 •

edited

Loading

codecov-commenter commented Jul 21, 2023 •

edited

Loading

heesung-sohn commented Jul 27, 2023 •

edited

Loading

hanmz commented Feb 21, 2024 •

edited

Loading