Coordinator Election -- Dual Coordinator Election After ZK Crash

[razinbouzar]

Affected Version

28.0.1

Description

The issue occurs as follows:

• ZK failure, and a leader re-elected. coordinator-0 was leader at the time of the ZK failure.
• After failure, coordinator-0 loses leader status (observed in a log line saying "I am no longer the leader.").
• coordinator-1 becomes leader (observed in a log line "I am the leader of the coordinators, all must bow!").
• Seemingly simultaneously a third coordinator, coordinator-2, thinks it is the leader and begins killing tasks started by coordinator-1 and vice versa. There are no log lines that indicate it was elected the leader as was seen in an earlier issue.
• Killing coordinator-2 restores the cluster health and tasks resume normally.

[razinbouzar]

This condition seems to be similar to what is reported in https://issues.apache.org/jira/browse/CURATOR-696, and addressed in #16617.

[gianm]

Thanks for the reminder. Since Curator 5.8.0 is out now, I raised #17857 to update to that instead of Curator 5.7.1.