Overlord should backoff on leadership changes

[drcrallen]

When an overlord looses leadership, it should have a backoff strategy for trying to acquire leadership again.

Mentioned in Dev Sync on 2016-09-06

[jihoonson]

Hi, I would like to take this ticket, and am wondering the back-off strategy means any exact implementation of a back-off strategy algorithm.

I believe that this proposal is to make the nodes which are frequently disconnected from the cluster have the lower priority for leader election. One possible solution is to provide an executor service to the leader selector which is capable of backing off participating leader election for the given amount of time.

Meanwhile, the leader latch provides a kind of back-off strategy inherently. With the leader latch, a leader is elected by the order of written ZK paths. That is, the node which wrote its name first becomes the leader. Thus, since a node rewrites its name whenever it is restored from disconnection, it will have the lowest priority.

I think that the second approach will work well even though there is only one overlord node in a cluster. What do you think?

[gianm]

@jihoonson, the goal here is mainly that the overlord shouldn't rapidly gain and lose leadership if something is wrong. Currently, it does that if the leadership startup throws an exception. So IMO this issues could be solved by waiting for some delay if leadership startup throws an exception.

One thing to keep in mind is making sure we don't block up any threads/executor services that Curator or anything else is using for any other purposes.

[gianm]

btw, I think you should be able to reproduce this rapid leadership cycling situation by just doing a throw new RuntimeException() in the TaskMaster leadership startup code.

[jihoonson]

Oh, I totally misunderstood the purpose of this issue. Thanks @gianm for your help.
I'm going to further investigate Curator, but until now, it seems difficult to do using LeaderSelector because takeLeadership() must return immediately when a master loses the leadership to release an inter-process mutex which is used in Curator. It means, blocking in takeLeadership() seems to block other overlord nodes from taking the leadership. See the below code.

    try
    {
        mutex.acquire();

        hasLeadership = true;
        try
        {
            ...
            listener.takeLeadership(client);
        }
        ...
    }
    ...
    finally
    {
        if ( hasLeadership )
        {
            hasLeadership = false;
            try
            {
                mutex.release();
            }
            ...
        }
    }

[drcrallen]

Resolved by #4699