[backport 17.06] Adding a recovery mechanism for a split gossip cluster

[fcrisciani]

Backport bugfix #2134

Signed-off-by: Dani Louca dani.louca@docker.com
(cherry picked from commit 744334d)

[fcrisciani]

Still WIP the backport was not clean, just checked that compiles at the moment

[fcrisciani]

@dani-docker can you take a look too? I did the backport of your fix on 17.06, there were a bunch of conflicts so better to have another pair of eyes on this

[dani-docker]

@fcrisciani
The changes here
https://github.com/docker/libnetwork/blob/744334d441587278813cddbb3f79eb12a48fad1c/networkdb/delegate.go#L41-L46
are not ported to https://github.com/fcrisciani/libnetwork/blob/46c52cdcb67137eeaeabe69a3143d317268a56a7/networkdb/delegate.go#L98-L105

[fcrisciani]

@dani-docker cannot backport that part here is why:
In the event_delegate.go we have on a join:

func (e *eventDelegate) NotifyJoin(mn *memberlist.Node) {            
  logrus.Infof("Node %s/%s, joined gossip cluster", mn.Name, mn.Addr)
  e.broadcastNodeEvent(mn.Addr, opCreate)                            
  e.nDB.Lock()                                                       
  // In case the node is rejoining after a failure or leave,         
  // wait until an explicit join message arrives before adding       
  // it to the nodes just to make sure this is not a stale           
  // join. If you don't know about this node add it immediately.     
  _, fOk := e.nDB.failedNodes[mn.Name]                               
  _, lOk := e.nDB.leftNodes[mn.Name]                                 
  if fOk || lOk {                                                    
    e.nDB.Unlock()                                                   
    return                                                           
  }                                                                  

meaning that in this code base the rejoin of a node that lose connectivity but do no change identity has to happen through the logic of delegate.go. If I embed that change, this will break and a node that temporary gets disconnected won't be able to join back.

[dani-docker]

LGTM

[tiborvass]

To test this PR:

1. Bring up 3 managers 2 workers
2. docker network create -d overlay --attachable net1
3. docker service create --name test --network net1 --replicas 10 busybox sleep 10000
4. Bring down 3 managers
5. Bring up 3 managers
6. docker run -dit --network net1 --name foo busybox sleep 10000
7. ssh to worker and run: docker exec -it $someTaskContainer nslookup foo

The last step should fail without this PR, and pass with this PR.

Another way to test is:

Everything the same till step 5
6. wait 5 min
7. cat docker.log | grep "NetworkDB stats"  # look for netPeers
8. ssh to worker and run: cat docker.log | grep "NetworkDB stats" # look for netPeers

The two netPeers should not match with this PR, and match with this PR.

[tiborvass]

@fcrisciani I tested this PR, and nslookup still fails with this PR.

[tiborvass]

@fcrisciani I now managed to confirm this patch fixes the original issue.

[tiborvass]

LGTM (IANAM)

[tiborvass]

Ping @abhi

[abhi]

LGTM