huge number of watch in zookeeper cause zookeeper full gc

[kaijianding]

I noticed there are about 100M watches in zookeeper in my product environment, and once I restart some of realtime tasks, zookeeper may full gc even it is configured to 32GB heap and druid is configured to use http server view.

After investigation, this issue is caused by the Announcer.java implementation.
The Announcer is trying best to avoid temporary disconnecting to zookeeper server and watch every child path under the specified base path, but for base path like /druid/prod/announcements and /druid/prod/listeners/lookups/__default/, there are plenty of hosts as child path under them.

If there are 10000 realtime tasks(not single realtime job, but like 50 jobs), then each task will create 2 * 10000 watches, finally we will get 200M watches!!

I'm not familiar with curator, can curator only watch a particular leaf path? Thus we can implement a simpler Announcer for these two pathes to reduce watch number in zookeeper to avoid zk OOM. @gianm

[jihoonson]

I'm not familiar with curator, can curator only watch a particular leaf path? Thus we can implement a simpler Announcer for these two paths to reduce watch number in zookeeper to avoid zk OOM.

It sounds possible, but I wonder what your solution is. Would you elaborate more details? What is the simpler implementation of Announcer you think?

[kaijianding]

My thought is that only watch the leaf path instead of watching all the child paths under the base path.
Rather than use childEvent, watch the path itself, I don't know whether it is possible, I'm not familiar with curator.

[love1693294577]

I also encountered this problem, how did you solve it?

[love1693294577]

What I want is to delete the directory under discovery and delete it after the task is finished.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.