support broker level dispatch rate limiter #11325

wangjialing218 · 2021-07-15T10:02:45Z

Motivation

Fixes #7720

Current we support broker level publish rate limiter.
This PR will add broker level dispatch rate limiter.

Modifications

Add broker level dispatch rate limiter for msgs and bytes.
Refact calculateToRead() to reduce duplicate code.

Verifying this change

Add testBrokerBytesRateLimitingReceiveAllMessagesAfterThrottling()

wangjialing218 · 2021-07-15T10:06:34Z

@codelipenghui @315157973 please have a look

...er/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java

Anonymitaet · 2021-07-16T00:34:36Z

Thanks for your contribution.

The PR template contains info about doc, which helps others know more about the changes. Can you provide doc info in future PR descriptions? Thanks

wangjialing218 · 2021-07-16T04:04:19Z

/pulsarbot run-failure-checks

wangjialing218 · 2021-07-19T01:52:01Z

/pulsarbot run-failure-checks

wangjialing218 · 2021-10-08T03:04:25Z

@315157973 @codelipenghui @hangc0276 PTAL when you have time

315157973 · 2021-10-08T07:07:02Z

The current reading process is like this:

Determine whether the threshold is exceeded
Enter the pending queue and prepare to read
The entry is successfully read, and the threshold count is deducted

There is a time difference between 1 and 3. When the Broker just restarts, if it is read concurrently, the problem in that PR will still exists

wangjialing218 · 2021-10-08T09:21:07Z

The current reading process is like this:

Determine whether the threshold is exceeded

Enter the pending queue and prepare to read

The entry is successfully read, and the threshold count is deducted

There is a time difference between 1 and 3. When the Broker just restarts, if it is read concurrently, the problem in that PR will still exists

The problem you mentioned may be solved in this PR #8611

315157973 · 2021-10-08T12:26:41Z

The current reading process is like this:

Determine whether the threshold is exceeded

Enter the pending queue and prepare to read

The entry is successfully read, and the threshold count is deducted

There is a time difference between 1 and 3. When the Broker just restarts, if it is read concurrently, the problem in that PR will still exists

The problem you mentioned may be solved in this PR #8611

This PR will only prevent the message from being sent to the consumer for a long time later, but it cannot solve the memory surge during concurrent reading when broker restart.

315157973 · 2021-10-08T12:30:01Z

We can support broker level dispatcher rate limiter, but this PR can not fix #7720

wangjialing218 · 2021-10-09T01:26:18Z

This PR will only prevent the message from being sent to the consumer for a long time later, but it cannot solve the memory surge during concurrent reading when broker restart.

As current reading process, it's not easy to completely prevent the sudden increase entry read traffic when mutliply comsumers starting at same time.
With PR #8611 and this PR, the increase traffic will last only one second. This could reduce the influence of write traffic in case of #7720

...ar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/DispatchRateLimiter.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java

.../java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java

...er/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java

315157973 · 2021-10-10T12:21:04Z

...er/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java

+
+        long start = System.currentTimeMillis();
+        // Asynchronously produce messages
+        for (int i = 0; i < numProducedMessagesEachTopic; i++) {


I suggest to stop the thread pool that refreshes the quota in rateLimiter at first, otherwise this unit test will often fail because we cannot predict the running of the thread

Could you point out the thread which need to stop?

315157973 · 2021-10-10T12:22:16Z

...er/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java

+        consumer2.close();
+        producer1.close();
+        producer2.close();
+        log.info("-- Exiting {} test --", methodName);


Please add a unit test that test the priority of policies at different levels

For topic dispatch rate limiter, we can configure from namespace level policies and topic level policies, and topic level policies will have a high priority. That's because the enforcement is done at the topic level.
For broker level, topic level and subscription level dispatch rate limiter, the enforcement is done at different level, so they have same priority and take effect together.
I have add a test, when all these dispatch rate limiter are configured, the minimum dispatch rate limiter should take effect in case of one consumer for one topic.

wangjialing218 · 2021-10-11T06:59:41Z

@315157973 could you have a look again

wangjialing218 · 2021-10-11T09:20:01Z

/pulsarbot run-failure-checks

wangjialing218 · 2021-10-12T01:17:15Z

/pulsarbot run-failure-checks

315157973 · 2021-10-12T06:56:17Z

Some unit tests seem to fail

…riptionMessageDispatchThrottlingTest.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

…jialing218/pulsar into branch-broker-dispatch-rate-limiter

wangjialing218 · 2021-10-25T01:31:33Z

/pulsarbot run-failure-checks

codelipenghui · 2021-11-10T12:38:45Z

conf/broker.conf


+# Max Rate(in 1 seconds) of Message allowed to dispatch from a broker if broker dispatch rate limiting enabled
+# (Disable message rate limit with value 0)
+brokerDispatchThrottlingMaxMessageRate=0


Suggested change

brokerDispatchThrottlingMaxMessageRate=0

dispatchThrottlingRateInMsg=0

conf/broker.conf

…tch-rate-limiter

wangjialing218 · 2022-01-18T07:10:58Z

/pulsarbot run-failure-checks

Anonymitaet · 2022-01-19T02:53:15Z

As discussed w/ @wangjialing218, he will add the two parameters and corresponding descriptions to the configuration - standalone page.

Anonymitaet reviewed Jul 16, 2021

View reviewed changes

...er/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java Outdated Show resolved Hide resolved

...er/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java Outdated Show resolved Hide resolved

Anonymitaet added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Jul 16, 2021

sijie requested review from 315157973, addisonj, codelipenghui, hangc0276 and merlimat July 23, 2021 20:58

sijie assigned wangjialing218 Jul 23, 2021

315157973 reviewed Oct 10, 2021

View reviewed changes

wangjialing and others added 6 commits October 12, 2021 15:11

support broker level dispatch rate limiter

ef98a13

correct debug log

940c67e

Update pulsar-broker/src/test/java/org/apache/pulsar/client/api/Subsc…

ca48966

…riptionMessageDispatchThrottlingTest.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

Update pulsar-broker/src/test/java/org/apache/pulsar/client/api/Subsc…

cc935e1

…riptionMessageDispatchThrottlingTest.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

correct member name

0fb920d

update after review

9d51e98

wangjialing218 force-pushed the branch-broker-dispatch-rate-limiter branch from 544c69b to 9d51e98 Compare October 12, 2021 07:11

wangjialing and others added 6 commits October 15, 2021 10:29

correct debug log

3eb0739

Update pulsar-broker/src/test/java/org/apache/pulsar/client/api/Subsc…

90f4a87

…riptionMessageDispatchThrottlingTest.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

Update pulsar-broker/src/test/java/org/apache/pulsar/client/api/Subsc…

e6e02ac

…riptionMessageDispatchThrottlingTest.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

correct member name

43102e1

update after review

920f365

Merge branch 'master' into branch-broker-dispatch-rate-limiter

0dc4c10

wangjialing218 force-pushed the branch-broker-dispatch-rate-limiter branch from 5f758c9 to b7be33e Compare October 15, 2021 04:50

wangjialing and others added 3 commits October 15, 2021 12:50

Merge branch 'branch-broker-dispatch-rate-limiter' of github.com:wang…

b7be33e

…jialing218/pulsar into branch-broker-dispatch-rate-limiter

Merge branch 'master' into branch-broker-dispatch-rate-limiter

02ba0a9

rebase code for apache#12294

99a776f

codelipenghui added this to the 2.10.0 milestone Nov 10, 2021

codelipenghui reviewed Nov 10, 2021

View reviewed changes

rename configurations

a975212

wangjialing218 requested review from Anonymitaet and codelipenghui December 7, 2021 01:53

codelipenghui approved these changes Dec 7, 2021

View reviewed changes

Merge remote-tracking branch 'apache/master' into branch-broker-dispa…

14673a1

…tch-rate-limiter

codelipenghui added doc-required Your PR changes impact docs and you will update later. and removed doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. labels Jan 18, 2022

fix compile error after rebase master

fa1c04b

codelipenghui merged commit 497e911 into apache:master Jan 18, 2022

wangjialing218 mentioned this pull request Jan 19, 2022

[Doc] add descriptions for dispatchThrottlingRate related configuration #13829

Merged

Anonymitaet added doc-complete Your PR changes impact docs and the related docs have been already added. and removed doc-required Your PR changes impact docs and you will update later. labels Jan 19, 2022

wangjialing218 mentioned this pull request Jan 26, 2022

[Broker] Fix no value present when updating dispatch rate #13900

Merged

3 tasks

heesung-sohn mentioned this pull request Nov 16, 2022

[Doc] Improve Pulsar Pub/Sub Throttling Doc #18506

Closed

2 tasks

sijie mentioned this pull request Nov 16, 2022

ISSUE-18506: [Doc] Improve Pulsar Pub/Sub Throttling Doc streamnative/pulsar-archived#5129

Open

2 tasks

	brokerDispatchThrottlingMaxMessageRate=0
	dispatchThrottlingRateInMsg=0

support broker level dispatch rate limiter #11325

support broker level dispatch rate limiter #11325

Uh oh!

Conversation

wangjialing218 commented Jul 15, 2021

Motivation

Modifications

Verifying this change

Uh oh!

wangjialing218 commented Jul 15, 2021

Uh oh!

Uh oh!

Uh oh!

Anonymitaet commented Jul 16, 2021

Uh oh!

wangjialing218 commented Jul 16, 2021

Uh oh!

wangjialing218 commented Jul 19, 2021

Uh oh!

wangjialing218 commented Oct 8, 2021

Uh oh!

315157973 commented Oct 8, 2021

Uh oh!

wangjialing218 commented Oct 8, 2021

Uh oh!

315157973 commented Oct 8, 2021

Uh oh!

315157973 commented Oct 8, 2021

Uh oh!

wangjialing218 commented Oct 9, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

315157973 Oct 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangjialing218 Oct 11, 2021

Choose a reason for hiding this comment

Uh oh!

315157973 Oct 10, 2021

Choose a reason for hiding this comment

Uh oh!

wangjialing218 Oct 11, 2021

Choose a reason for hiding this comment

Uh oh!

wangjialing218 commented Oct 11, 2021

Uh oh!

wangjialing218 commented Oct 11, 2021

Uh oh!

wangjialing218 commented Oct 12, 2021

Uh oh!

315157973 commented Oct 12, 2021

Uh oh!

wangjialing218 commented Oct 25, 2021

Uh oh!

codelipenghui Nov 10, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wangjialing218 commented Jan 18, 2022

Uh oh!

Anonymitaet commented Jan 19, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

315157973 Oct 10, 2021 •

edited

Loading