Fix Huge Number of Watches in ZooKeeper by GWphua · Pull Request #17482 · apache/druid

GWphua · 2024-11-15T08:21:43Z

Description

This PR is built upon #6683 and #9172 and aims to reduce the number of ZooKeeper watch counts.

Fixed Huge Number of Watches in ZooKeeper

The current Announcer.java leverages on Apache Curator's PathChildrenCache. In its present form, the announcement mechanism watches the immediate parent of the specified path. This results in all child nodes under the parent path being monitored by the ZooKeeper ensemble, including sibling nodes and children of the specified path. This causes an unnecessarily large number of ZooKeeper watches to be produced.

The new NodeAnnouncer.java class is simply Announcer.java but leverages on NodeCache instead to watch a single node during announcement. By eliminating the watches on child nodes, this approach significantly reduces the total number of watch counts in ZooKeeper. Users can opt-in to use the new NodeAnnouncer by setting toggling the feature flag druid.zk.service.pathChildrenCacheStrategy=false.

Tests conducted on the production server also indicate a decrease in watch counts resulting from this change.

Note:
The use of the two different announcer classes simultaneously may result in a KeeperException.NotEmptyException. This happens when two nodes are sharing the same parent, and since both announcers do not have a full picture of the nodes it is watching, the exception will be thrown when the following occurs:

PathChildrenAnnouncer removes all of its tracked children nodes.
Thinking that after all the removal the parent node has no children anymore, PathChildrenAnnouncer tries to remove the parent node.
If NodeAnnouncer is still watching one or more child node, the attempt by PathChildrenAnnouncer in removing the parent node will result in the exception.

Documentation

Remove humor in error logs.
Add JavaDocs and comments within code to better describe the process.
Add documentations on how to enable NodeAnnouncer.

Refactoring

Rename Announcer to PathChildrenAnnouncer
Shift Announceable class out of PathChildrenAnnouncer.
Add ServiceAnnouncer interface to facilitate dependency injection for different flavours of caching strategies.
Refactor long methods by creating helper functions.
Add ZKPathsUtils.java to abstract the retrieval of ZooKeeper path and ZooKeeper node.

Release note

New: A new opt-in caching strategy is provided that uses a much smaller number of ZooKeeper watches for service announcement.

Key changed/added classes in this PR

Announcer.java -> PathChildrenAnnouncer.java
ServiceAnnouncer.java
NodeAnnouncer.java
Announceable.java
AnnouncerModule.java
CuratorConfig.java
DirectExecutorAnnouncer & SingleThreadedAnnouncer annotations for Guice.
docs/configuration/index.md & .spelling for docs.
Related test files.

This PR has:

been self-reviewed.
- using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
a release note entry in the PR description.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
been tested in a test Druid cluster.

@GuardedBy

* Tear down nodeAnnouncer * Remove useless Logger and ExecutorService * Init CuratorListener by lambda * Improve explicit type * Using CuratorMultiTransaction instead of CuratorTransaction * Add @GuardedBy("toAnnounce") for toUpdate field * Improve docs

GWphua · 2025-03-17T10:40:33Z

Hi @kgyrtkirk,

I have made changes according to your suggestions.
I also took the suggestion of @cryptoe and created a feature flag to allow users to choose between the two flavours of announcers currently available. The default is set to using the old PathChildrenCache Announcer, and users can opt in for the new NodeCache Announcer should they face problems with huge ZooKeeper watch count.

The PR description is edited accordingly.

kgyrtkirk · 2025-04-11T08:50:39Z

sorry for forgotting about this ; I'll reply back today

kgyrtkirk

looks good to me

kgyrtkirk · 2025-04-11T10:48:15Z


  @LifecycleStart
+  @Override
  public void start()


NodeAnnouncer and PathChildrenAnnouncer share a lot of common pieces ; I wonder if it could be placed into a common abstract - or the old approach should be removed after the cache is proven to work correctly?

I do not mind either choices, I can try and make an abstract by next week.

Hi @kgyrtkirk, I have taken a look at making an abstract -- It is true that there's alot of shared methods, but it is not trivial to make the changes. I am considering this instead

We give some time to use the NodeAnnouncer instead, if it proves to perform better, we can simply delete the PathChildrenCacheAnnouncer.

If using NodeCache provides a trade-off of CPU for Memory, a PR can further work towards replacing the deprecated PathChildrenAnnouncer using CuratorCache. We can decide to make an Abstract class then, as some of the complexities (such as listeners having type of ConcurrentMap<String, PathChildrenCache> will be changed to ConcurrentMap<String, CuratorCache>, which can be shared with NodeAnnouncer). This seems to be the direction where Apache Curator is trying to go with their caches.

kgyrtkirk · 2025-04-11T10:56:25Z

 |`druid.zk.service.connectionTimeoutMs`|ZooKeeper connection timeout, in milliseconds.|`15000`|
 |`druid.zk.service.compress`|Boolean flag for whether or not created Znodes should be compressed.|`true`|
 |`druid.zk.service.acl`|Boolean flag for whether or not to enable ACL security for ZooKeeper. If ACL is enabled, zNode creators will have all permissions.|`false`|
+|`druid.zk.service.pathChildrenCacheStrategy`|Dictates the underlying caching strategy for service announcements. Set true to let announcers to use Apache Curator's PathChildrenCache strategy, otherwise NodeCache strategy. Consider using NodeCache strategy when you are dealing with huge number of ZooKeeper watches in your cluster.|`true`|


I wonder why not make the NodeCache approach the default as that should work better - but retain the old approach if some issue happens?

cc: @cryptoe

Copilot

Pull Request Overview

This PR reduces the number of ZooKeeper watches by introducing a new NodeAnnouncer that leverages Curator’s NodeCache instead of the PathChildrenCache, along with refactoring the announcement classes and their dependency injections.

Replaces Announcer with NodeAnnouncer (and conditionally with PathChildrenAnnouncer based on a config flag).
Introduces a new ServiceAnnouncer interface, refactors tests and updates configuration accordingly.
Updates related documentation and dependency injection modules.

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
server/src/test/java/org/apache/druid/server/coordination/coordination/BatchDataSegmentAnnouncerTest.java	Updated tests to use NodeAnnouncer and reformatted multi-line JOINER calls.
server/src/test/java/org/apache/druid/curator/discovery/CuratorDruidNodeAnnouncerAndDiscoveryTest.java	Replaced Announcer instantiation with NodeAnnouncer.
server/src/test/java/org/apache/druid/curator/announcement/PathChildrenAnnouncerTest.java	Renamed test cases and updated instantiation to PathChildrenAnnouncer.
server/src/test/java/org/apache/druid/curator/announcement/NodeAnnouncerTest.java	Added new tests for NodeAnnouncer feature including update and session kill cases.
server/src/test/java/org/apache/druid/client/client/BatchServerInventoryViewTest.java	Updated to use NodeAnnouncer and refined thread pool executor creation.
server/src/main/java/org/apache/druid/server/coordination/CuratorDataSegmentServerAnnouncer.java	Changed injection from Announcer to ServiceAnnouncer.
server/src/main/java/org/apache/druid/server/coordination/BatchDataSegmentAnnouncer.java	Updated type references and dependency injection for the new ServiceAnnouncer.
server/src/main/java/org/apache/druid/guice/AnnouncerModule.java	Provided alternative bindings for single-threaded and direct executor announcers.
server/src/main/java/org/apache/druid/curator/discovery/CuratorDruidNodeAnnouncer.java	Updated constructor injection and logging to reflect the new ServiceAnnouncer.
server/src/main/java/org/apache/druid/curator/announcement/ServiceAnnouncer.java	Added new interface to abstract announcer behavior.
server/src/main/java/org/apache/druid/curator/announcement/PathChildrenAnnouncer.java	Refactored implementation and logging to support the PathChildrenAnnouncer functionality.
server/src/main/java/org/apache/druid/curator/announcement/Announceable.java	Moved the Announceable class out of the original Announcer to support reuse.
server/src/main/java/org/apache/druid/curator/CuratorConfig.java	Added new configuration property to switch between caching strategies.
docs/configuration/index.md & docs/api-reference/tasks-api.md	Updated documentation to explain and reflect the new caching strategy option.
Various indexing-service test files	Updated tests to use NodeAnnouncer and adjusted thread configuration for executor creation.

kgyrtkirk · 2025-05-05T08:31:21Z

@GWphua you've a conflict with master; I think we could merge it after that's addressed!

GWphua · 2025-05-06T02:28:21Z

@kgyrtkirk The conflicts have been addressed. Thanks for the heads-up!

kgyrtkirk · 2025-05-08T06:53:31Z

thank you @GWphua for improving on this!

maytasm · 2025-08-20T20:25:59Z

@GWphua What version of zk are you running?

GWphua · 2025-08-22T02:15:52Z

We are using 3.5.9

jtuglu1 · 2025-08-22T06:09:33Z

3.5.9

Thanks @GWphua. What about Curator version? Same as OSS? Did you have to make any ZK/Curator-related upgrades/config changes to get things to work here?
We've seen some issues with the following combination:

Curator version = 5.8.0 (Druid v34 OSS version)
ZK server version = 3.5.8
ZK client version = 3.8.4 (Druid v34 OSS version)

GWphua · 2025-08-22T06:44:50Z

Maybe the Curator/ZK version i'm using is too old:

<apache.curator.version>4.3.0</apache.curator.version>
<zookeeper.version>3.5.9</zookeeper.version>

What are the issues you are facing + Did you experience problems in both PathChildrenCache + NodeCache?

maytasm · 2025-08-22T15:46:47Z

Maybe the Curator/ZK version i'm using is too old:
<apache.curator.version>4.3.0</apache.curator.version>
<zookeeper.version>3.5.9</zookeeper.version>
What are the issues you are facing + Did you experience problems in both PathChildrenCache + NodeCache?

Yes, we are experiencing issues with Curator versions greater than 5.0.0, which are used in the latest OSS (v34) Druid. We have encountered problems with both versions 5.5.0 and 5.8.0. This appears to be related to CURATOR-549, which was introduced in Curator 5.0.0 and later. Curator is attempting to use a new feature that is only available in ZooKeeper server version 3.6 or higher. Since we are running ZooKeeper server version 3.5.8, this causes Druid to fail to connect to ZooKeeper. As a result, we are seeing continuous SUSPEND/RECONNECT events, with the ZooKeeper server closing the connection.

Note that this issue only occurs with NodeCache, as that is where Curator attempts to use the new feature. PathChildrenCache does not have this problem because it does not use the new ZooKeeper feature code path.

We suspect that this new feature Curator is trying to use is related to persistent watchers/CuratorCache. Seems related to https://lists.apache.org/thread/nl7zrzgyfp2b5wxdkrovk0yhqfto9yl7

So, I think to use NodeCache, you would either have to be on Curator <5.0.0 or running ZK Server >=3.6. Although, I think Curator has some critical fixes in 5.x.x too.

maytasm · 2025-08-22T15:56:39Z

@GWphua Btw....related to Huge Number of Watches, have you try setting druid.announcer.skipSegmentAnnouncementOnZk to true and use http for segment discovery (druid.serverview.type=http). Http for segment discovery has been the default since Druid v25 (#13592 (comment))

GWphua · 2025-08-25T06:46:46Z

Hey @maytasm, we did not configure this setting in our clusters. However, we can take a look at whether it helps us with our use-case. Thanks 😄

GWphua · 2025-08-25T06:55:02Z

Note that this issue only occurs with NodeCache, as that is where Curator attempts to use the new feature. PathChildrenCache does not have this problem because it does not use the new ZooKeeper feature code path.

We suspect that this new feature Curator is trying to use is related to persistent watchers/CuratorCache. Seems related to https://lists.apache.org/thread/nl7zrzgyfp2b5wxdkrovk0yhqfto9yl7

Not too confident about this, but I feel the problem may be because of upgrading the deprecated NodeCache to CuratorCache. Should Curator remove the deprecated PathChildrenCache in the future, we may be forced to:

Stay at a lower version
Upgrade PathChildrenCache to CuratorCache (Which may cause similar problems?)
Move away from ZooKeeper

asdf2014 and others added 16 commits January 13, 2020 14:33

Patch comments

8420d73

Add license

da8538b

Patch comments

67b5064

Merge branch 'master' into tmp

23cd4d8

Merge reduce ZK

ce64445

Update to use CloseableUtils instead of CloseQuietly.

da2f212

Add Javadocs for Announceable class

a5d0669

Javadocs on Announcers

f3b21a4

Refactor announce method

1053042

Refactor NodeAnnouncer

903db88

Add Unit Tests

954fa53

Remove humor from logs

25de238

Add unit test for NodeAnnouncer announcing same path different payload.

a7733e3

Merge branch 'apache:master' into zk-fix

31f1d3d

Tweak import and change Javadocs

f541b84

github-actions Bot added the Area - Ingestion label Nov 15, 2024

GWphua added 3 commits November 18, 2024 09:54

Update Druid unannouncement logging to make sense

20cfb0f

Replace deprecated inTransaction() with transaction()

2a1f803

Refactor buildParentPath

a857bfc

GWphua marked this pull request as draft November 21, 2024 01:24

GWphua added 9 commits November 21, 2024 14:11

Merge branch 'github' into zk-fix

1cbfa99

Merge branch 'github' into zk-fix

e50e2aa

Add executorService for NodeCache

582691e

Try replace all Announcer

a305748

Add JavaDocs

438bfd9

Refactor Test classes

c6bfbd6

Add JavaDocs and logging messages

1aea321

Fix Checkstyle

6e262f8

Add debug logs for update

b412415

GWphua added 2 commits March 17, 2025 17:29

Feature flag for announcers

7a9e09f

Checkstyle

5e67ed2

github-actions Bot added the Area - Documentation label Mar 17, 2025

GWphua added 3 commits March 18, 2025 10:28

Dependency injection for Announcers

c0fc475

Rename announcer to serviceAnnouncer

5c0845a

Fix checkstyle for import order

ea5d905

GWphua requested a review from kgyrtkirk April 10, 2025 02:55

kgyrtkirk approved these changes Apr 11, 2025

View reviewed changes

Merge branch 'apache:master' into zk-fix

4090566

FrankChen021 requested a review from Copilot April 21, 2025 02:16

Copilot AI reviewed Apr 21, 2025

View reviewed changes

Merge branch 'github' into zk-fix

195b9e0

kgyrtkirk merged commit 228304e into apache:master May 8, 2025
74 checks passed

GWphua deleted the zk-fix branch May 13, 2025 06:23

capistrant added this to the 34.0.0 milestone Jul 22, 2025

This was referenced Aug 25, 2025

Document missing property - druid.announcer.skipSegmentAnnouncementOnZk #12891

Merged

Set druid.announcer.skipSegmentAnnouncementOnZk to true by default #18445

Closed

Conversation

GWphua commented Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fixed Huge Number of Watches in ZooKeeper

Documentation

Refactoring

Release note

Key changed/added classes in this PR

Uh oh!

GWphua commented Mar 17, 2025

Uh oh!

kgyrtkirk commented Apr 11, 2025

Uh oh!

kgyrtkirk left a comment

Choose a reason for hiding this comment

Uh oh!

kgyrtkirk Apr 11, 2025

Choose a reason for hiding this comment

Uh oh!

GWphua Apr 15, 2025

Choose a reason for hiding this comment

Uh oh!

GWphua Apr 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kgyrtkirk Apr 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

kgyrtkirk commented May 5, 2025

Uh oh!

GWphua commented May 6, 2025

Uh oh!

Uh oh!

kgyrtkirk commented May 8, 2025

Uh oh!

maytasm commented Aug 20, 2025

Uh oh!

GWphua commented Aug 22, 2025

Uh oh!

jtuglu1 commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GWphua commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maytasm commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maytasm commented Aug 22, 2025

Uh oh!

GWphua commented Aug 25, 2025

Uh oh!

GWphua commented Aug 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

GWphua commented Nov 15, 2024 •

edited

Loading

GWphua Apr 20, 2025 •

edited

Loading

jtuglu1 commented Aug 22, 2025 •

edited

Loading

GWphua commented Aug 22, 2025 •

edited

Loading

maytasm commented Aug 22, 2025 •

edited

Loading