Cannot connect to a container in an overlay network from a different swarm node: `could not resolve peer \"<nil>\": timed out resolving peer by querying the cluster`

[paulcadman]

Description of problem:

Very rarely (observed twice after using 1000s of containers) we start a new container
into an overlay network in a docker swarm. Existing containers in the overlay network that
are on different nodes cannot connect to the new container. However containers in the
overlay network on the same node as the new container are able to connect.

The new container receives an IP address in the overlay network subnet, but this does not
seem to work correctly when resolved from a different node.

The second time this happened we fixed the problem by stopping and starting the new
container.

We haven't found a way to reliably reproduce this problem. Is there any other debugging
I can provide that would help diagnose this issue?

The error message is the same as the one reported on #617.

docker version:

Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 19:04:33 2016
 OS/Arch:      linux/amd64

Server:
 Version:      swarm/1.1.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   a0fd82b
 Built:        Thu Feb  4 08:55:18 UTC 2016
 OS/Arch:      linux/amd64

docker info:

Containers: 102
 Running: 53
 Paused: 0
 Stopped: 49
Images: 372
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 3
 glera.int.corefiling.com: 10.0.0.57:2375
  └ Status: Healthy
  └ Containers: 32
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 32.94 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.2.6-201.fc22.x86_64, operatingsystem=Fedora 22 (Twenty Two), storagedriver=devicemapper
  └ Error: (none)
  └ UpdatedAt: 2016-02-22T11:20:16Z
 kafue.int.corefiling.com: 10.0.0.17:2375
  └ Status: Healthy
  └ Containers: 36
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 16.4 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.2.6-201.fc22.x86_64, operatingsystem=Fedora 22 (Twenty Two), storagedriver=devicemapper
  └ Error: (none)
  └ UpdatedAt: 2016-02-22T11:20:20Z
 paar.int.corefiling.com: 10.0.1.1:2375
  └ Status: Healthy
  └ Containers: 34
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 16.44 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.2.6-201.fc22.x86_64, operatingsystem=Fedora 22 (Twenty Two), storagedriver=devicemapper
  └ Error: (none)
  └ UpdatedAt: 2016-02-22T11:20:31Z
Plugins:
 Volume:
 Network:
Kernel Version: 4.2.6-201.fc22.x86_64
Operating System: linux
Architecture: amd64
CPUs: 12
Total Memory: 65.77 GiB
Name: 9dd94ffb6aea

uname -a:

Linux glera.int.corefiling.com 4.2.6-201.fc22.x86_64 #1 SMP Tue Nov 24 18:42:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

Physical - docker swarm cluster.

How reproducible:

Rare - happened 2 times after creating/starting 1000s of containers.

Steps to Reproduce:

1. Create/start a container in an overlay network
2. In the same overlay network create/start a container on a different host in the swarm.
   A process in the container is listening on port 80 and this port is exposed to the overlay network.
3. Try to connect to the container of step 2 from within the container of step 1 with http client.

Actual Results:

Get a connection timeout. For example with the golang http client:

 http: proxy error: dial tcp 10.158.0.60:80: i/o timeout

10.158.0.60 is the address of the container in step 2 in the overlay network subnet.

The docker logs on the swarm node that launched the container in step 2 contain (from journalctl -u docker):

level=error msg="could not resolve peer \"<nil>\": timed out resolving peer by querying the cluster".

We see a line like this for each failed request between the containers.

When we make the same request from a container in the overlay network on the same swarm node as the
container running the http server the expected connection is established and a response is received.

Expected Results:

The http client receieves a response from the container its trying to connect to.

Additional info:

The second time this occurred we fixed the problem by stopping and starting the container running
the http server.

We are using Consul as the KV store of the overlay network and swarm.

When removing the container that cannot be connected to, docker logs (journalctl -u docker) contain the line:

error msg="Peer delete failed in the driver: could not delete fdb entry into the sandbox: could not delete neighbor entry: no such file or directory\n"

The docker log lines are emitted by https://github.com/docker/libnetwork/blob/master/drivers/overlay/ov_serf.go#L180. I can't find an existing issue tracking this.

[byrnedo]

I'm seeing the same thing: moby/moby#22144

[jsumali]

Hit this issue as well, restarting the container seems to fix it, but isn't ideal.

We're using:

Client:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 19:36:04 2016
 OS/Arch:      linux/amd64

Server:
 Version:      swarm/1.2.3
 API version:  1.22
 Go version:   go1.5.4
 Git commit:   eaa53c7
 Built:        Fri May 27 17:25:03 UTC 2016
 OS/Arch:      linux/amd64

[longkt90]

I have a same problem. Just comment to see if there is any update on this.

My env:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

[dvenza]

Another me too.
I can add that for me it is sufficient to ping from the container that cannot do the connect any other container on the same network to fix the issue, the first ping may take slightly longer, but then everything starts to work as intended. I am using Zookeeper.

Some weirdness with ARP?
Perhaps relates to this: #338 ?

[dvenza]

No, ARP seems to be fine, at the container level. I have in front of me a "blocked" container and arp -a reports the correct mac address for the server container it is trying to connect to. So there is something wrong with IP.

I can reproduce it quite often by running Spark clusters. I would say that one cluster out of five comes out bad with some worker unable to talk to the master. Since we do a lot of automated tests and performance measurements, we get many outliers and basically makes Swarm with overlay networks too unreliable.

[dvenza]

There is something fishy in the overlay network namespace. Here is the vxlan fdb on the host on which the "blocked" container is running:

$ sudo nsenter --net=/var/run/docker/netns/1-07e76e17d6 bridge fdb show dev vxlan1
6e:ed:5e:b7:60:b1 vlan 1 permanent
6e:ed:5e:b7:60:b1 permanent
02:42:0a:00:04:11
33:33:00:00:00:01 self permanent
01:00:5e:00:00:01 self permanent
33:33:ff:b7:60:b1 self permanent
02:42:0a:00:04:10 dst 192.168.47.12 self permanent
02:42:0a:00:04:03 dst 192.168.47.19 self permanent
02:42:0a:00:04:0f dst 192.168.47.12 self permanent
02:42:0a:00:04:0e dst 192.168.47.20 self permanent
02:42:0a:00:04:0d dst 192.168.47.20 self permanent
02:42:0a:00:04:0c dst 192.168.47.13 self permanent
02:42:0a:00:04:0b dst 192.168.47.17 self permanent
02:42:0a:00:04:09 dst 192.168.47.15 self permanent
02:42:0a:00:04:08 dst 192.168.47.16 self permanent
02:42:0a:00:04:07 dst 192.168.47.16 self permanent
02:42:0a:00:04:13 dst 192.168.47.21 self permanent
02:42:0a:00:04:06 dst 192.168.47.21 self permanent
02:42:0a:00:04:12 dst 192.168.47.18 self permanent
02:42:0a:00:04:05 dst 192.168.47.18 self permanent

02:42:0a:00:04:11 is the MAC address of the container with which the "blocked" container cannot talk. The other MAC addresses have an IP address, but that one does not.

Any idea, @mavenugo ?

[mavenugo]

Thanks @dvenza. i will page @mrjana to see if he has any ideas.

[adamlc]

I'm also getting this issue too. Unfortunately I know virtually nothing about networking. My setup is pretty much identical to @paulcadman.

I've actually had the issue for some time over the last few docker releases. It got so bad at one point I recreated the whole overlay network and containers and its been good for a few weeks, but its happening again :(

If there's anything I can debug let me know. We're running on EC2 is that helps.

Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Wed Apr 27 00:34:42 2016
 OS/Arch:      linux/amd64

Server:
 Version:      swarm/1.2.0
 API version:  1.22
 Go version:   go1.5.4
 Git commit:   a6c1f14
 Built:        Wed Apr 13 05:58:31 UTC 2016
 OS/Arch:      linux/amd64

Containers: 33
 Running: 26
 Paused: 0
 Stopped: 7
Images: 45
Server Version: swarm/1.2.0
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 3
 swarm-node-eu-west1-1a-2: 10.0.1.193:2376
  └ Status: Healthy
  └ Containers: 10
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 1.322 GiB / 3.624 GiB
  └ Labels: executiondriver=, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=overlay
  └ Error: (none)
  └ UpdatedAt: 2016-08-11T08:13:38Z
  └ ServerVersion: 1.11.1
 swarm-node-eu-west1-1b-2: 10.0.0.150:2376
  └ Status: Healthy
  └ Containers: 12
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 1.322 GiB / 3.624 GiB
  └ Labels: executiondriver=, kernelversion=3.10.0-327.10.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=overlay
  └ Error: (none)
  └ UpdatedAt: 2016-08-11T08:13:30Z
  └ ServerVersion: 1.11.1
 swarm-node-eu-west1-1c-2: 10.0.2.227:2376
  └ Status: Healthy
  └ Containers: 11
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 330 MiB / 3.624 GiB
  └ Labels: executiondriver=, kernelversion=3.10.0-327.10.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=overlay
  └ Error: (none)
  └ UpdatedAt: 2016-08-11T08:13:43Z
  └ ServerVersion: 1.11.1
Plugins:
 Volume:
 Network:
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: linux
Architecture: amd64
CPUs: 6
Total Memory: 10.87 GiB
Name: fcf67f74802a
Docker Root Dir:
Debug mode (client): false
Debug mode (server): false
WARNING: No kernel memory limit support

Linux support-eu-west1-1a-1 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[dvenza]

On a new deployment with 25 machines this problem blocks network communication for about half of the containers that I create, making it completely worthless.

Any news?

[lamhaison]

I have face to with same issue. I think overlay network is unreliable in ubuntu.

Server Version: swarm/1.2.3

Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:47:50 2016
OS/Arch: linux/amd64

Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:47:50 2016
OS/Arch: linux/amd64

[hdanniel]

We have the same issue in a docker swarm with thousands of containers. In all the docker nodes (1.10.3) we have an nginx containers which can communicate with different application containers in the overlay network (using consul). Sometimes one of the nginx containers cannot connect to an app container in a different node, receiving the same message in the log:

ERRO[5758943] could not resolve peer "<nil>": timed out resolving peer by querying the cluster

Additionally we are seeing that the app containers which fail always have the same IP addresses. Restarting the app container doesn't work if the container received the same IP address. What work for us is:

• Stop docker (where the nginx container is running), clean iptables, start docker
• Clean the network namespaces in the docker node where the nginx container is running. This work but not inmediatly
• After some hours the IP address start to answer without any apparent reason.

We tried to debug where the packets are going without success. There is not activity in the remote node so we think the packet is not leaving the node where the nginx container is running.

[antoinetran]

We have the exact same issue, with a different environment:

• Swarm (classic image mode 1.2.6)
• docker-engine CE 17.03.1
  Have to reboot machines to have a working Swarm overlay network again.