[improve] [broker] Improve CPU resources usege of TopicName Cache by poorbarcode · Pull Request #23052 · apache/pulsar

poorbarcode · 2024-07-18T17:14:40Z

Motivation

LoadableCache.get cost too many CPU resources

broker_cpu.html.txt

pulsar-io-4-20" #135 prio=5 os_prio=0 cpu=79326083.91ms elapsed=1465985.25s tid=0x00007f5bc40740b0 nid=0xe9 runnable  [0x00007f5ba51fd000]
   java.lang.Thread.State: RUNNABLE
	at java.util.concurrent.ConcurrentLinkedQueue.offer(java.base@17.0.10/ConcurrentLinkedQueue.java:380)
	at java.util.concurrent.ConcurrentLinkedQueue.add(java.base@17.0.10/ConcurrentLinkedQueue.java:283)
	at com.google.common.cache.LocalCache$Segment.recordRead(LocalCache.java:2546)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2068)
	at com.google.common.cache.LocalCache.get(LocalCache.java:4012)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4035)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5011)
	at org.apache.pulsar.common.naming.TopicName.get(TopicName.java:81)
	at org.apache.pulsar.common.naming.SystemTopicNames.isEventSystemTopic(SystemTopicNames.java:62)
	at org.apache.pulsar.common.naming.SystemTopicNames.isSystemTopic(SystemTopicNames.java:86)
	at org.apache.pulsar.common.topics.TopicList.lambda$filterSystemTopic$1(TopicList.java:65)
	at org.apache.pulsar.common.topics.TopicList$$Lambda$2132/0x00007f5d049a4800.test(Unknown Source)
	at java.util.stream.ReferencePipeline$2$1.accept(java.base@17.0.10/ReferencePipeline.java:178)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(java.base@17.0.10/ArrayList.java:1625)
	at java.util.stream.AbstractPipeline.copyInto(java.base@17.0.10/AbstractPipeline.java:509)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@17.0.10/AbstractPipeline.java:499)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java.base@17.0.10/ReduceOps.java:921)
	at java.util.stream.AbstractPipeline.evaluate(java.base@17.0.10/AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(java.base@17.0.10/ReferencePipeline.java:682)
	at org.apache.pulsar.common.topics.TopicList.filterSystemTopic(TopicList.java:66)
	at org.apache.pulsar.broker.service.ServerCnx.lambda$handleGetTopicsOfNamespace$47(ServerCnx.java:2126)
	at org.apache.pulsar.broker.service.ServerCnx$$Lambda$2129/0x00007f5d049b1c80.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniAcceptNow(java.base@17.0.10/CompletableFuture.java:757)
	at java.util.concurrent.CompletableFuture.uniAcceptStage(java.base@17.0.10/CompletableFuture.java:735)
	at java.util.concurrent.CompletableFuture.thenAccept(java.base@17.0.10/CompletableFuture.java:2182)
	at org.apache.pulsar.broker.service.ServerCnx.lambda$handleGetTopicsOfNamespace$49(ServerCnx.java:2123)
	at org.apache.pulsar.broker.service.ServerCnx$$Lambda$2128/0x00007f5d049b1a38.apply(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniApplyNow(java.base@17.0.10/CompletableFuture.java:684)
	at java.util.concurrent.CompletableFuture.uniApplyStage(java.base@17.0.10/CompletableFuture.java:662)
	at java.util.concurrent.CompletableFuture.thenApply(java.base@17.0.10/CompletableFuture.java:2168)
	at org.apache.pulsar.broker.service.ServerCnx.handleGetTopicsOfNamespace(ServerCnx.java:2120)
	at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:315)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:202)
	at io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:164)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17.0.10/Thread.java:840)

Modifications

Because there is no need for a strict invalidate time, change the implementation to ConcurrentHashMap.
after_improvement.html

Performance review

Env

Server: master
- topic count: 11000 topics. 110 partitioned topic, each partitioned topic contains 100 partitions
  - topic name public/default/{uuid}
- enableBrokerSideSubscriptionPatternEvaluation : false
Client: 2.11.4
- 1 client with 220 pattern consumers
  - without any change, 220 consumers can put pressure make the broker’s CPU usage to keep 90%
- initialize 16 timer task for 220 consumers ( in default, there is only a single timer for each client )
- change the time-unit of patternAutoDiscoveryPeriod to seconds , and set it to 1
- connectionsPerBroker: 100

Test cases:

Without any change
With Caffiene Cache
With PR [improve] [broker] Improve CPU resources usege of TopicName Cache #23052
With PR [improve] [broker] high CPU usage caused by list topics under namespace #23049

perf_result.zip

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: x

poorbarcode · 2024-07-18T17:17:58Z

/pulsarbot rerun-failure-checks

codecov-commenter · 2024-07-18T19:47:34Z

Codecov Report

Attention: Patch coverage is 89.47368% with 2 lines in your changes missing coverage. Please review.

Project coverage is 73.45%. Comparing base (bbc6224) to head (9fe8dad).
Report is 460 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23052      +/-   ##
============================================
- Coverage     73.57%   73.45%   -0.12%     
- Complexity    32624    33503     +879     
============================================
  Files          1877     1915      +38     
  Lines        139502   143956    +4454     
  Branches      15299    15727     +428     
============================================
+ Hits         102638   105743    +3105     
- Misses        28908    30113    +1205     
- Partials       7956     8100     +144

Flag	Coverage Δ
inttests	`27.74% <68.42%> (+3.15%)`	⬆️
systests	`24.74% <68.42%> (+0.41%)`	⬆️
unittests	`72.52% <89.47%> (-0.33%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...org/apache/pulsar/broker/ServiceConfiguration.java	`99.02% <100.00%> (-0.37%)`	⬇️
...rg/apache/pulsar/broker/service/BrokerService.java	`82.18% <100.00%> (+1.40%)`	⬆️
...ava/org/apache/pulsar/common/naming/TopicName.java	`94.69% <80.00%> (-1.37%)`	⬇️

... and 503 files with indirect coverage changes

BewareMyPower · 2024-07-20T10:35:00Z

Here is the benchmark's source code: https://github.com/BewareMyPower/CacheBenchmark

A typical output:

CacheReadBenchmark.testCaffine             thrpt    5   7432187.879 ±  746705.561  ops/s
CacheReadBenchmark.testGuava               thrpt    5   6962764.989 ±  222299.106  ops/s
CacheReadBenchmark.testMapComputeIfAbsent  thrpt    5  34219723.889 ±  686448.613  ops/s
CacheReadBenchmark.testMapGet              thrpt    5  72643406.351 ± 1312709.392  ops/s

The overhead for expiration seems significant. @ben-manes @lhotari Please help review if there is something wrong with it.

The motivation is that the time cost of parsing a TopicName (with an efficient implementation, the current Pulsar implementation is inefficient) is just similar to getting the cached object from Guava LoadingCache.

ben-manes · 2024-07-20T10:53:01Z

Thanks! I’ll look later today (it’s late my time). If you are running this on Mac/Windows then reading the current time is a system call, iirc. That’s much slower than a user space call like Linux, so that can artificially skew your results by not being matching the deployment destination. The ticker can be defined if truly necessary to optimize the clock reads.

Co-authored-by: Zixuan Liu <nodeces@gmail.com>

The concern is not important, we can impove the code in ther future

poorbarcode · 2024-07-22T09:40:23Z

Since @ben-manes and @lhotari did not answer, let me merge it first. We can improve the code continuously in the future.

…3052) Co-authored-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 81aed6c)

hangc0276 · 2024-07-22T09:56:00Z

Good job! LGTM.

…ache#23052) Co-authored-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 81aed6c) (cherry picked from commit 5a83958)

…ache#23052) Co-authored-by: Zixuan Liu <nodeces@gmail.com>

…ache (apache#23052)" This reverts commit 81aed6c.

github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Jul 18, 2024

poorbarcode self-assigned this Jul 18, 2024

poorbarcode added release/2.11.5 release/3.3.1 release/3.0.6 release/3.2.4 ready-to-test labels Jul 18, 2024

liangyepianzhou reviewed Jul 19, 2024

View reviewed changes

Comment thread pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java

BewareMyPower reviewed Jul 19, 2024

View reviewed changes

Comment thread conf/broker.conf Outdated

poorbarcode mentioned this pull request Jul 19, 2024

[improve] [broker] high CPU usage caused by list topics under namespace #23049

Merged

4 tasks

merlimat approved these changes Jul 19, 2024

View reviewed changes

Comment thread conf/broker.conf Outdated

Comment thread pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java Outdated

poorbarcode requested a review from liangyepianzhou July 19, 2024 07:00

codelipenghui reviewed Jul 19, 2024

View reviewed changes

Comment thread conf/broker.conf Outdated

Comment thread pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java

poorbarcode requested a review from codelipenghui July 19, 2024 07:58

codelipenghui approved these changes Jul 19, 2024

View reviewed changes

lhotari requested changes Jul 19, 2024

View reviewed changes

Comment thread pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java

poorbarcode requested review from BewareMyPower and lhotari July 19, 2024 09:16

Technoboy- approved these changes Jul 19, 2024

View reviewed changes

dao-jun approved these changes Jul 19, 2024

View reviewed changes

lhotari previously requested changes Jul 19, 2024

View reviewed changes

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java

Comment thread pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamespaceName.java Outdated

poorbarcode requested review from lhotari and removed request for lhotari July 19, 2024 10:11

poorbarcode force-pushed the improve/topic_name_cache branch from b96dbbd to e794b67 Compare July 19, 2024 10:49

ben-manes reviewed Jul 20, 2024

View reviewed changes

Comment thread pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java

rebase master

61a7595

poorbarcode force-pushed the improve/topic_name_cache branch from 3f19821 to 61a7595 Compare July 21, 2024 17:52

nodece reviewed Jul 22, 2024

View reviewed changes

Comment thread conf/broker.conf Outdated

nodece reviewed Jul 22, 2024

View reviewed changes

Comment thread pulsar-broker/src/test/java/org/apache/pulsar/broker/PulsarServiceTest.java

Update conf/broker.conf

bd60644

Co-authored-by: Zixuan Liu <nodeces@gmail.com>

poorbarcode requested a review from nodece July 22, 2024 03:33

fix typo

9fe8dad

poorbarcode merged commit 81aed6c into apache:master Jul 22, 2024

poorbarcode deleted the improve/topic_name_cache branch July 22, 2024 09:41

poorbarcode added a commit that referenced this pull request Jul 22, 2024

[improve] [broker] Improve CPU resources usege of TopicName Cache (#2…

5f7ec53

…3052) Co-authored-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 81aed6c)

poorbarcode added a commit that referenced this pull request Jul 22, 2024

[improve] [broker] Improve CPU resources usege of TopicName Cache (#2…

5a83958

…3052) Co-authored-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 81aed6c)

poorbarcode added cherry-picked/branch-3.0 cherry-picked/branch-3.3 and removed release/3.2.4 labels Jul 22, 2024

poorbarcode added a commit that referenced this pull request Jul 22, 2024

[improve] [broker] Improve CPU resources usege of TopicName Cache (#2…

7f736e2

…3052) Co-authored-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 81aed6c)

poorbarcode added the cherry-picked/branch-2.11 label Jul 22, 2024

poorbarcode mentioned this pull request Sep 6, 2024

[improve][broker] Add limits for Key_Shared Subscription look ahead in dispatching #23231

Merged

4 tasks

lhotari added this to the 4.0.0 milestone Oct 14, 2024

hanmz pushed a commit to hanmz/pulsar that referenced this pull request Feb 12, 2025

[improve] [broker] Improve CPU resources usege of TopicName Cache (ap…

0d0327e

…ache#23052) Co-authored-by: Zixuan Liu <nodeces@gmail.com>

lhotari added a commit to lhotari/pulsar that referenced this pull request Jun 24, 2025

Revert "[improve] [broker] Improve CPU resources usege of TopicName C…

f91475d

…ache (apache#23052)" This reverts commit 81aed6c.

lhotari mentioned this pull request Jun 24, 2025

[fix] Fix memory leak in TopicName cache and reduce heap memory usage through deduplication #24457

Closed

4 tasks

poorbarcode mentioned this pull request Jun 24, 2025

[fix][proxy]Pulsar Proxy OOMs because it forgets to clear the topic name cache #24458

Closed

4 tasks

BewareMyPower mentioned this pull request Jun 26, 2025

[Bug] The TopicName cache will not shrink if a topic is created and deleted #24466

Open

3 tasks

BewareMyPower mentioned this pull request Jul 25, 2025

[improve][broker] Improve the performance of TopicName constructor #24463

Merged

4 tasks

lhotari mentioned this pull request Mar 20, 2026

[improve][common] Optimize TopicName.get() to reduce lock contention on cache lookup #25367

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improve] [broker] Improve CPU resources usege of TopicName Cache#23052

[improve] [broker] Improve CPU resources usege of TopicName Cache#23052
poorbarcode merged 4 commits intoapache:masterfrom
poorbarcode:improve/topic_name_cache

poorbarcode commented Jul 18, 2024 •

edited

Loading

Uh oh!

poorbarcode commented Jul 18, 2024

Uh oh!

codecov-commenter commented Jul 18, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BewareMyPower commented Jul 20, 2024

Uh oh!

ben-manes commented Jul 20, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

poorbarcode commented Jul 22, 2024

Uh oh!

hangc0276 commented Jul 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Conversation

poorbarcode commented Jul 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Performance review

Documentation

Matching PR in forked repository

Uh oh!

poorbarcode commented Jul 18, 2024

Uh oh!

codecov-commenter commented Jul 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BewareMyPower commented Jul 20, 2024

Uh oh!

ben-manes commented Jul 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

poorbarcode commented Jul 22, 2024

Uh oh!

hangc0276 commented Jul 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

poorbarcode commented Jul 18, 2024 •

edited

Loading

codecov-commenter commented Jul 18, 2024 •

edited

Loading

ben-manes commented Jul 20, 2024 •

edited

Loading