Realtime Index tasks need to know when the segments is handed over to the historical nodes.
To get this information realtime index task keeps a FilteredBrokerserverView and gets data from zookeeper for all the segments in the cluster. In case of ZK disconnects, each task will reconnect to zookeeper and starts reading the distribution of segments in the cluster which puts a high load on zookeeper as the number of tasks starts to grow.
One solution to handle this is to make the overlord responsible for keeping the state segments in cluster and the realtime nodes will interact with the overlord to know when a segment has been handed off to another historical node.
Realtime Index tasks need to know when the segments is handed over to the historical nodes.
To get this information realtime index task keeps a FilteredBrokerserverView and gets data from zookeeper for all the segments in the cluster. In case of ZK disconnects, each task will reconnect to zookeeper and starts reading the distribution of segments in the cluster which puts a high load on zookeeper as the number of tasks starts to grow.
One solution to handle this is to make the overlord responsible for keeping the state segments in cluster and the realtime nodes will interact with the overlord to know when a segment has been handed off to another historical node.