It looks like under certain conditions the coordinator can announce itself twice in service discovery. The indexing overlord may have a similar issue, since its announcement process is similar.
We observed something like:
1- Coordinator #1 logged ZK "SUSPENDED" followed by "Unannouncing service"
2- Coordinator #2 became leader and logged "Announcing service"
3- Coordinator #1 logged ZK "LOST" followed by "RECONNECTED"
Then some time later:
4- Coordinator #2 logged ZK "SUSPENDED" followed by "Unannouncing service" followed by ZK "RECONNECTED"
5- Coordinator #1 became leader and logged "Announcing service"
At this point Coordinator #1 was leader and was registered twice in service discovery. It appears that it probably did not successfully unannounce its service in step 1.
It looks like under certain conditions the coordinator can announce itself twice in service discovery. The indexing overlord may have a similar issue, since its announcement process is similar.
We observed something like:
1- Coordinator #1 logged ZK "SUSPENDED" followed by "Unannouncing service"
2- Coordinator #2 became leader and logged "Announcing service"
3- Coordinator #1 logged ZK "LOST" followed by "RECONNECTED"
Then some time later:
4- Coordinator #2 logged ZK "SUSPENDED" followed by "Unannouncing service" followed by ZK "RECONNECTED"
5- Coordinator #1 became leader and logged "Announcing service"
At this point Coordinator #1 was leader and was registered twice in service discovery. It appears that it probably did not successfully unannounce its service in step 1.