docker crashing

[oppianmatt]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

Actual behavior

Docker is crashing.

Steps to reproduce the behaviour

Unknown. It's just crashing randomly sometimes.

Output of docker version:

docker version
Client:
 Version:       17.12.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    c97c6d6
 Built: Wed Dec 27 20:11:19 2017
 OS/Arch:       linux/amd64
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Output of docker info:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon runni

Additional environment details (AWS, VirtualBox, physical, etc.)

Physical box running Ubuntu 16.04 part of a swarm cluster.

Entries in journalctl when it happens:

eb 21 16:10:39 pu-frontend2.pitchup.com kernel: device veth649888b left promiscuous mode
Feb 21 16:10:39 pu-frontend2.pitchup.com kernel: docker_gwbridge: port 2(veth649888b) entered disabled state
Feb 21 16:10:39 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:39.377379295Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:10:39 pu-frontend2.pitchup.com kernel: br0: port 2(veth1) entered disabled state
Feb 21 16:10:39 pu-frontend2.pitchup.com kernel: device veth1 left promiscuous mode
Feb 21 16:10:39 pu-frontend2.pitchup.com kernel: br0: port 2(veth1) entered disabled state
Feb 21 16:10:39 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:39.955298637Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Feb 21 16:10:40 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:40.192849465Z" level=error msg="68b82a6800b10f6cb11d57fda68524527a3a61dcba9aaea14574fe725c57ef4f cleanup: failed to delete container from containerd: no such container"
Feb 21 16:10:40 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:40.214787578Z" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:296: starting container process caused \"process_linux.go:398: container init caused \\\"process_linux.go:381: running prestart hook 0 caused \\\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f09974db428 m=5 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\n\\\\\\\\ngoroutine 1 [GC assist wait, locked to thread]:\\\\\\\\ngithub.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/user.ParsePasswdFilter(0x388b020, 0xc42000e690, 0xc4204d9b50, 0x388b020, 0xc42000e690, 0x390af48, 0x7f0998416140, 0xc4204d9b90)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/user/user.go:120 +0x1a5\\\\\\\\ngithub.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/user.lookupUser(0xc4204d9b50, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/user/lookup.go:24 +0xed\\\\\\\\ngithub.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/user.LookupUid(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/user/lookup.go:51 +0x9d\\\\\\\\ngithub.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/user.CurrentUser(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/opencontainers/runc/libcontainer/user/lookup_unix.go:38 +0x85\\\\\\\\ngithub.com/docker/docker/pkg/homedir.Get(0x0, 0x38ff9e0)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/pkg/homedir/homedir_unix.go:23 +0xab\\\\\\\\ngithub.com/docker/docker/cli/config.init.0()\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/
Feb 21 16:10:40 pu-frontend2.pitchup.com dockerd[7229]: docker/cli/config/configdir.go:23 +0x3c\\\\\\\\ngithub.com/docker/docker/cli/config.init()\\\\\\\\n\\\\\\\\t<autogenerated>:1 +0x95\\\\\\\\nmain.init()\\\\\\\\n\\\\\\\\t<autogenerated>:1 +0x195\\\\\\\\n\\\\\\\\ngoroutine 18 [syscall]:\\\\\\\\nos/signal.signal_recv(0x14a1e10)\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/sigqueue.go:131 +0xa6\\\\\\\\nos/signal.loop()\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:22 +0x22\\\\\\\\ncreated by os/signal.init.0\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:28 +0x41\\\\\\\\n\\\\\\\\nrax    0x0\\\\\\\\nrbx    0x7f099786b700\\\\\\\\nrcx    0x7f09974db428\\\\\\\\nrdx    0x6\\\\\\\\nrdi    0x67fa\\\\\\\\nrsi    0x70d0\\\\\\\\nrbp    0x2c4c8b6\\\\\\\\nrsp    0x7f0994be7838\\\\\\\\nr8     0x7f099786c770\\\\\\\\nr9     0x7f0994be8700\\\\\\\\nr10    0x8\\\\\\\\nr11    0x206\\\\\\\\nr12    0x7f09880008c0\\\\\\\\nr13    0xf1\\\\\\\\nr14    0x11\\\\\\\\nr15    0x0\\\\\\\\nrip    0x7f09974db428\\\\\\\\nrflags 0x206\\\\\\\\ncs     0x33\\\\\\\\nfs     0x0\\\\\\\\ngs     0x0\\\\\\\\n\\\\\\\"\\\"\": unknown" module=node/agent/taskmanager node.id=povnl4sbu3gfn261rjmd42a3e service.id=9tvbwgyqqrt3d283ztbazneo1 task.id=t92jzduwcfohzn934jgprsm7e
Feb 21 16:10:40 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:40.223525643Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:10:40 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:40.674692595Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Feb 21 16:10:40 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:40.740526321Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:10:41 pu-frontend2.pitchup.com dd.dogstatsd[11456]: INFO (dogstatsd.py:276): Flush #912: flushed 20 metrics, 0 events, and 0 service check runs
Feb 21 16:10:41 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:41.674720710Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Feb 21 16:10:42 pu-frontend2.pitchup.com kernel: eth1: renamed from veth06e0b22
Feb 21 16:10:42 pu-frontend2.pitchup.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth89e1fe1: link becomes ready
Feb 21 16:10:42 pu-frontend2.pitchup.com kernel: docker_gwbridge: port 1(veth89e1fe1) entered forwarding state
Feb 21 16:10:42 pu-frontend2.pitchup.com kernel: docker_gwbridge: port 1(veth89e1fe1) entered forwarding state
Feb 21 16:10:42 pu-frontend2.pitchup.com kernel: IPVS: Creating netns size=2192 id=66594
Feb 21 16:10:42 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:42.715667126Z" level=warning msg="unknown container" container=b1e2afdc516aac8a7973b229f52fe425032fed4ab84202abfc061b2f2133a88e module=libcontainerd namespace=plugins.moby
Feb 21 16:10:42 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:42.741516655Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Feb 21 16:10:42 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:42.749225312Z" level=warning msg="unknown container" container=b1e2afdc516aac8a7973b229f52fe425032fed4ab84202abfc061b2f2133a88e module=libcontainerd namespace=plugins.moby
Feb 21 16:10:42 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:42.850595405Z" level=warning msg="unknown container" container=b1e2afdc516aac8a7973b229f52fe425032fed4ab84202abfc061b2f2133a88e module=libcontainerd namespace=plugins.moby
Feb 21 16:10:42 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:42Z" level=info msg="shim reaped" id=b1e2afdc516aac8a7973b229f52fe425032fed4ab84202abfc061b2f2133a88e module="containerd/tasks"
Feb 21 16:10:42 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:42.926776234Z" level=info msg="ignoring event" module=libcontainerd namespace=plugins.moby topic=/tasks/delete type="*events.TaskDelete"
Feb 21 16:10:42 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:42.930982875Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 21 16:10:42 pu-frontend2.pitchup.com kernel: br0: port 3(veth0) entered disabled state
Feb 21 16:10:42 pu-frontend2.pitchup.com kernel: br0: port 1(vxlan0) entered disabled state
Feb 21 16:10:43 pu-frontend2.pitchup.com kernel: ov-001001-wymg7: renamed from br0
Feb 21 16:10:43 pu-frontend2.pitchup.com kernel: device veth0 left promiscuous mode
Feb 21 16:10:43 pu-frontend2.pitchup.com kernel: ov-001001-wymg7: port 3(veth0) entered disabled state
Feb 21 16:10:43 pu-frontend2.pitchup.com kernel: device vxlan0 left promiscuous mode
Feb 21 16:10:43 pu-frontend2.pitchup.com kernel: ov-001001-wymg7: port 1(vxlan0) entered disabled state
Feb 21 16:10:43 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:10:43.914521365Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Feb 21 16:10:43 pu-frontend2.pitchup.com kernel: vx-001001-wymg7: renamed from vxlan0

What's annoying is the constant filling of log files with unknown container and link not found errors that just come all the time. Hard to actually trust the logs if it's constantly spewing info.

Plenty more similar errors, and then errors about not able to remove tasks, lots of messages about virtual interfaces left in promiscuous mode.

eb 21 16:11:03 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:03.758784014Z" level=error msg="5d602d4081dccaa84642e5a3d57c69204334d4cb096606c4b501f6c71221bb1c cleanup: failed to delete container from containerd: no such container"
Feb 21 16:11:03 pu-frontend2.pitchup.com kernel: device vethd8d2b7b left promiscuous mode
Feb 21 16:11:03 pu-frontend2.pitchup.com kernel: docker_gwbridge: port 1(vethd8d2b7b) entered disabled state
Feb 21 16:11:04 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:04.028308569Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Feb 21 16:11:04 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:04Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/f87b58eb302a4019698f5706e6bd12852baaf4696bedf2c998a83d780c41f972/shim.sock" debug=false module="containerd/tasks" pid=20165
Feb 21 16:11:04 pu-frontend2.pitchup.com systemd-udevd[12152]: Could not generate persistent MAC address for vx-001001-wymg7: No such file or directory
Feb 21 16:11:04 pu-frontend2.pitchup.com kernel: vetha708174: renamed from veth0
Feb 21 16:11:04 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:04.435270885Z" level=error msg="Handler for GET /v1.35/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Feb 21 16:11:04 pu-frontend2.pitchup.com kernel: veth02a1f06: renamed from veth1
Feb 21 16:11:04 pu-frontend2.pitchup.com dockerd[7229]: http: multiple response.WriteHeader calls
Feb 21 16:11:04 pu-frontend2.pitchup.com systemd-udevd[31804]: Could not generate persistent MAC address for vetha708174: No such file or directory
Feb 21 16:11:04 pu-frontend2.pitchup.com systemd-udevd[32512]: Could not generate persistent MAC address for veth02a1f06: No such file or directory
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05.037321867Z" level=warning msg="unknown container" container=f87b58eb302a4019698f5706e6bd12852baaf4696bedf2c998a83d780c41f972 module=libcontainerd namespace=plugins.moby
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05.077524141Z" level=warning msg="unknown container" container=f87b58eb302a4019698f5706e6bd12852baaf4696bedf2c998a83d780c41f972 module=libcontainerd namespace=plugins.moby
Feb 21 16:11:05 pu-frontend2.pitchup.com kernel: veth77fe895: renamed from eth0
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05.154617663Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05.183537979Z" level=warning msg="unknown container" container=f87b58eb302a4019698f5706e6bd12852baaf4696bedf2c998a83d780c41f972 module=libcontainerd namespace=plugins.moby
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05.200996525Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05Z" level=info msg="shim reaped" id=f87b58eb302a4019698f5706e6bd12852baaf4696bedf2c998a83d780c41f972 module="containerd/tasks"
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05.278725422Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05.278828675Z" level=info msg="ignoring event" module=libcontainerd namespace=plugins.moby topic=/tasks/delete type="*events.TaskDelete"
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05.544413570Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05.718953579Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Feb 21 16:11:05 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:05Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3edbd01d5eea3a5d9c24ccb0b5adb5ef363b2901f45b11cf26418013b063384b/shim.sock" debug=false module="containerd/tasks" pid=18136
Feb 21 16:11:05 pu-frontend2.pitchup.com kernel: docker_gwbridge: port 2(vethd03ce33) entered disabled state
Feb 21 16:11:05 pu-frontend2.pitchup.com kernel: vethbe71dfe: renamed from eth1
Feb 21 16:11:06 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:06.180265988Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Feb 21 16:11:06 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:06.181893881Z" level=error msg="fatal task error" error="task: non-zero exit (1)" module=node/agent/taskmanager node.id=povnl4sbu3gfn261rjmd42a3e service.id=9tvbwgyqqrt3d283ztbazneo1 task.id=vya5wjdxquc9njuh8iee1fxyt
Feb 21 16:11:06 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:06.243332255Z" level=warning msg="unknown container" container=3edbd01d5eea3a5d9c24ccb0b5adb5ef363b2901f45b11cf26418013b063384b module=libcontainerd namespace=plugins.moby
Feb 21 16:11:06 pu-frontend2.pitchup.com kernel: IPVS: Creating netns size=2192 id=66600
Feb 21 16:11:06 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:06.284942827Z" level=warning msg="unknown container" container=3edbd01d5eea3a5d9c24ccb0b5adb5ef363b2901f45b11cf26418013b063384b module=libcontainerd namespace=plugins.moby
Feb 21 16:11:06 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:06.661031772Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:11:06 pu-frontend2.pitchup.com kernel: docker_gwbridge: port 2(vethd03ce33) entered disabled state
Feb 21 16:11:06 pu-frontend2.pitchup.com kernel: device vethd03ce33 left promiscuous mode
Feb 21 16:11:06 pu-frontend2.pitchup.com kernel: docker_gwbridge: port 2(vethd03ce33) entered disabled state
Feb 21 16:11:07 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:07 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:07 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:07 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:07 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:07 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:07 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:07 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:07 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:07.572216802Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Feb 21 16:11:07 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:07.573764068Z" level=error msg="fatal task error" error="task: non-zero exit (1)" module=node/agent/taskmanager node.id=povnl4sbu3gfn261rjmd42a3e service.id=i36g6t2qva2ptlzvtoi854nnv task.id=omil683nx1voncp73qiky0tdn
Feb 21 16:11:07 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:07.574121010Z" level=error msg="Handler for GET /containers/7024e83300455ea20df8aff1290ae80fce424d57c1e1802ee8c0d6d3f2fdeb2b/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Feb 21 16:11:07 pu-frontend2.pitchup.com dockerd[7229]: http: multiple response.WriteHeader calls
Feb 21 16:11:08 pu-frontend2.pitchup.com kernel: IPVS: Creating netns size=2192 id=66601
Feb 21 16:11:08 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:08.248626705Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:11:08 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:08.746520725Z" level=error msg="Failed to deserialize netlink ndmsg: invalid argument"
Feb 21 16:11:08 pu-frontend2.pitchup.com kernel: br0: renamed from ov-001007-b2498
Feb 21 16:11:09 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:09.018613764Z" level=error msg="c9c4ba04f110b330642827324544e69e1df5272df86a1cdb06100981868313f5 cleanup: failed to delete container from containerd: no such container"
Feb 21 16:11:09 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:09.018662987Z" level=error msg="fatal task error" error="starting container failed: subnet sandbox join failed for \"10.0.6.0/24\": error creating vxlan interface: file exists" module=node/agent/taskmanager node.id=povnl4sbu3gfn261rjmd42a3e service.id=ztdtdxadv5z3lk3tefa8927vy task.id=nkl6zwkdsq8akdv6kpa4t9bre
Feb 21 16:11:09 pu-frontend2.pitchup.com kernel: br0: renamed from ov-001005-alhed
Feb 21 16:11:09 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:09.362814481Z" level=error msg="5d602d4081dccaa84642e5a3d57c69204334d4cb096606c4b501f6c71221bb1c cleanup: failed to delete container from containerd: no such container"
Feb 21 16:11:09 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:09.362866252Z" level=error msg="fatal task error" error="starting container failed: subnet sandbox join failed for \"10.0.4.0/24\": error creating vxlan interface: file exists" module=node/agent/taskmanager node.id=povnl4sbu3gfn261rjmd42a3e service.id=xgbqcfjay7h0jhap8pyksu6c4 task.id=kv1ztd0bw7kf3b3kclq92mziv
Feb 21 16:11:09 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:09.363004798Z" level=error msg="cee1ef61ca9dd693cafaa7d80afee4103388a456963c9623ae5b7b58399cc205 cleanup: failed to delete container from containerd: no such container"
Feb 21 16:11:09 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:09.363054654Z" level=error msg="fatal task error" error="starting container failed: subnet sandbox join failed for \"10.0.4.0/24\": error creating vxlan interface: file exists" module=node/agent/taskmanager node.id=povnl4sbu3gfn261rjmd42a3e service.id=wkruiix8ixedfhsblfo7mbqyb task.id=jcmhqmcamr9gjbq8zq8c1rp54
Feb 21 16:11:09 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:09.494521387Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:11:09 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:09 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:09 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:09 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:09 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:09 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:09 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter
Feb 21 16:11:09 pu-frontend2.pitchup.com kernel: IPVS: __ip_vs_del_service: enter

skipping some:

Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.637212160Z" level=error msg="network pitchup_varnishnet remove failed: network pitchup_varnishnet not found" module=node/agent node.id=povnl4sbu3gfn261rjmd42a3e
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.637263721Z" level=error msg="remove task failed" error="network pitchup_varnishnet not found" module=node/agent node.id=povnl4sbu3gfn261rjmd42a3e task.id=kv1ztd0bw7kf3b3kclq92mziv
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.640105160Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_gunicorn 5a8a694f90051b488ff51914715a5fc9a1ee38d5be6c7da5ccb40fab829c6cf6 aborted lb.backEnds[eid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.640958690Z" level=warning msg="rmServiceBinding handleEpTableEvent pitchup_gunicorn eff5b98dc1deb7a4ceb9b54a9bfd69acf9bb1c64227331e4b1eb8e4223d6b963 aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641123420Z" level=error msg="failed removing service binding for 0a4b3c349e4374c948ddfe083030329e1eb24d17dd78f2f88d5ea2978cdaa106 epRec:{pitchup_varnish_agent.klomxd7t0kxy16vm27icx2ejf.6jjfroocm9vcm4mq0y4px3r5p pitchup_varnish_agent xgbqcfjay7h0jhap8pyksu6c4 10.0.4.10 10.0.4.11 [] [varnish_agent] [b30ae9791b7a]} err:network alhed1qdnnpirydno503carxc not found"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641166761Z" level=error msg="failed removing service binding for 1e8e17f8585054d62bcadb08c596236e3effcec141da9a04e5c43529c8d9348e epRec:{pitchup_varnish_plus.klomxd7t0kxy16vm27icx2ejf.wriu42aafzvcg3fcwyjmpx32c pitchup_varnish_plus ztdtdxadv5z3lk3tefa8927vy 10.0.4.8 10.0.4.210 [] [varnish_plus] [4a48d358bdf3]} err:network alhed1qdnnpirydno503carxc not found"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641201363Z" level=error msg="failed removing service binding for a57880ea382be11acea5ad8b3e9ba04b3afc90229d67b7443b86d881aec9d1c0 epRec:{pitchup_nginx.klomxd7t0kxy16vm27icx2ejf.nca27zil468myaldqwg444fn9 pitchup_nginx wkruiix8ixedfhsblfo7mbqyb 10.0.4.88 10.0.4.65 [] [nginx] [59549b7985ce]} err:network alhed1qdnnpirydno503carxc not found"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641200850Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_gunicorn cc82348c4d6946b1d5f9ce9b470e19134f1ccdb357f9ea9aa5c69b6f981411a6 aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641231748Z" level=error msg="failed removing service binding for eb7cb9f1431005548431fe186245a59fb934ef4172a1841a514a22ea2dc20af0 epRec:{pitchup_nginx.6q0y1qws9hgh41qu13i8sam4p.19qha5mhkco30irb81v1t2gzf pitchup_nginx wkruiix8ixedfhsblfo7mbqyb 10.0.4.88 10.0.4.73 [] [nginx] [ced1a3a64372]} err:network alhed1qdnnpirydno503carxc not found"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641254762Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_static e267a6f5c7a5e135dc97b48d079028887301c0803b097d2b7e613290fd2ac6e4 aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641285317Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_static f8ea7b3f994b0768602da0fe81e3ed2c782cb1e237f70816ba608970d3d74709 aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641312684Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_varnish_plus e499168a43f46f7c49c1ec99e94151de26a1ef49bef5d155fed8781d20d0cf84 aborted c.serviceBindings[skey] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641339568Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_nonadc d2f83262dd3366445aeab1f422a2199ea633097d18d43c1d1a318da0212f4f6d aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641365733Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_nonadc fad598a1f0d3ff13e872a65222d04db9249021c9e854de2dbd9d82e511773a99 aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641391843Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_nonadc 66e77d79bfae14577c07ed1c5856f4aadffee76d048ef66e25eab420cb5f5241 aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641417226Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_nonadc c40a81880b20a27322e869e9e6ca16442c0503b14fdcfd1f274340971a7800ff aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641444349Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_nonadc cd8fa91b9ed1383136d686a6758651d4cf79483ed49a6d9f8e663f5bb827d365 aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641466275Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_nonadc 6ac12963aadc75801a32e41e8e334d12aaf9f5e719844ca83b4d8df8a7665e17 aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641499197Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_nonadc 841944233f5af9df4b9e11a7efad736e8d7a29d70e01f066fd2e6be67d437c58 aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641532865Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_nonadc a02a693184ee246577ea2a159e2e6fff3a87b3014e1dbb7d9b0cff0a11b520ff aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641559863Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_nonadc a9c7abe17640b5b4794648a38e1a5ceda0b1c72f6a159152e9e8619f593a28ff aborted s.loadBalancers[nid] !ok"
Feb 21 16:11:12 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:12.641589573Z" level=warning msg="rmServiceBinding cleanupServiceBindings pitchup_bots 7184e191eea84cf1f9b8b3587d011078ef6b7e7ca69b70954e7c682c0386892d aborted s.loadBalancers[nid] !ok"

then it has even more errors and finally a stack dump with dockerd dying:

Feb 21 16:11:18 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:18.227609338Z" level=error msg="Failed to deserialize netlink ndmsg: invalid argument"
Feb 21 16:11:18 pu-frontend2.pitchup.com kernel: br0: renamed from ov-001005-alhed
Feb 21 16:11:18 pu-frontend2.pitchup.com kernel: IPVS: Creating netns size=2192 id=66605
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:19.144752870Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:19.262782339Z" level=error msg="36fe91a21ff9dd755539555dce4a0d3554980dbef5422e1963ed167d8367237c
cleanup: failed to delete container from containerd: no such container"
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:19.262839630Z" level=error msg="fatal task error" error="starting container failed: subnet sandbo
x join failed for \"10.0.4.0/24\": error creating vxlan interface: file exists" module=node/agent/taskmanager node.id=povnl4sbu3gfn261rjmd42a3e service.id=wkruiix8ixedfhsblfo7m
bqyb task.id=2zmpthg33j5s0jfasv040uyiq
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:19.342564060Z" level=error msg="62aaa14896705ecf9bcdfac697c11420b95d2672e504c5a5277ec75784cd34a5
cleanup: failed to delete container from containerd: no such container"
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:19.342624970Z" level=error msg="fatal task error" error="starting container failed: subnet sandbo
x join failed for \"10.0.4.0/24\": error creating vxlan interface: file exists" module=node/agent/taskmanager node.id=povnl4sbu3gfn261rjmd42a3e service.id=xgbqcfjay7h0jhap8pyks
u6c4 task.id=e674l7gf1o4rotykjcawqtv55
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: runtime/cgo: pthread_create failed: Resource temporarily unavailable
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: SIGABRT: abort
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: PC=0x7f0b3cc02428 m=3 sigcode=18446744073709551610
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: goroutine 0 [idle]:
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: goroutine 1 [chan receive, locked to thread]:
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*lexer).nextItem(...)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/lex.go:196
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*Tree).next(...)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:64
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*Tree).peekNonSpace(0xc42014b700, 0x0, 0x0, 0x0, 0x0, 0x0)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:113 +0x1e2
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*Tree).itemList(0xc42014b700, 0x2a4e479, 0x5, 0xc4202c8e60)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:329 +0xda
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*Tree).parseControl(0xc42014b700, 0x1302a00, 0x2a4e479, 0x5, 0x0, 0x0, 0xc4202c8e60, 0x0, 0x0)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:456 +0xf0
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*Tree).rangeControl(0xc42014b700, 0x1c, 0x69)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:498 +0x5a
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*Tree).action(0xc42014b700, 0x9, 0x67)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:371 +0x4f6
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*Tree).textOrAction(0xc42014b700, 0x1c, 0x69)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:348 +0x22c
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*Tree).parse(0xc42014b700)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:294 +0x445
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*Tree).Parse(0xc42014b700, 0x2ade77a, 0x123, 0x0, 0x0, 0x0, 0x0, 0xc4203f75f0, 0xc4203ee4b0, 0x2, ...)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:233 +0x244
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.Parse(0x2a4e05f, 0x5, 0x2ade77a, 0x123, 0x0, 0x0, 0x0, 0x0, 0xc4203ee4b0, 0x2, ...)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/parse.go:55 +0x144
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template.(*Template).Parse(0xc4203289c0, 0x2ade77a, 0x123, 0xc4203289d8, 0xc420217e88, 0x1186888)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/template.go:198 +0x125
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: html/template.(*Template).Parse(0xc4203f75c0, 0x2ade77a, 0x123, 0x0, 0x0, 0x0)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/html/template/template.go:189 +0xa6
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: net/http/pprof.init()
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/net/http/pprof/pprof.go:259 +0xc8
eb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: github.com/docker/docker/api/server/router/debug.init()
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         <autogenerated>:1 +0x4e
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: github.com/docker/docker/api/server.init()
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         <autogenerated>:1 +0x7a
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: main.init()
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         <autogenerated>:1 +0x96
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: goroutine 18 [syscall]:
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: os/signal.signal_recv(0x14a1e10)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/runtime/sigqueue.go:131 +0xa6
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: os/signal.loop()
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: created by os/signal.init.0
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: goroutine 30 [runnable]:
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*lexer).emit(0xc420358500, 0xe)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/lex.go:156 +0xc5
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.lexRightDelim(0xc420358500, 0x2aec2d8)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/lex.go:348 +0x94
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: text/template/parse.(*lexer).run(0xc420358500)
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/lex.go:231 +0x5f
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: created by text/template/parse.lex
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]:         /usr/local/go/src/text/template/parse/lex.go:224 +0x15f
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rax    0x0
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rbx    0x7f0b3cf92700
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rcx    0x7f0b3cc02428
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rdx    0x6
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rdi    0x46d8
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rsi    0x4f60
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rbp    0x2c4c8b6
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rsp    0x7f0b3b3107f8
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: r8     0x7f0b3cf93770
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: r9     0x7f0b3b311700
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: r10    0x8
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: r11    0x206
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: r12    0x7f0b2c0008c0
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: r13    0xf1
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: r14    0x11
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: r15    0x0
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rip    0x7f0b3cc02428
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: rflags 0x206
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: cs     0x33
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: fs     0x0
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: gs     0x0
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:19.453980298Z" level=error msg="Failed to receive from netlink: interrupted system call "
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:19.464379550Z" level=warning msg="Failed to disable IPv6 on all interfaces on network namespace \"/var/run/docker/netns/43c5e740f84a\": reexec to set IPv6 failed: exit status 2"
Feb 21 16:11:19 pu-frontend2.pitchup.com dockerd[7229]: time="2018-02-21T16:11:19Z" level=warning msg="Running modprobe nf_nat failed with message: ``, error: fork/exec /sbin/modprobe: resource temporarily unavailable"

[oppianmatt]

docker info after a reboot and a system prune:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 6
Server Version: 17.12.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: pending
 NodeID: povnl4sbu3gfn261rjmd42a3e
 Is Manager: false
 Node Address: 213.138.116.237
 Manager Addresses:
  213.138.116.235:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-81-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 125.8GiB
Name: pu-frontend2.pitchup.com
ID: T3MJ:7HUM:M74V:AKLX:WRSK:JGB7:XU2T:VW35:CN4A:CI4Q:JN65:KEST
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[thaJeztah]

ping @ddebroy PTAL

[ddebroy]

This is most likely addressed by moby/libnetwork#2051 from @fcrisciani

[thaJeztah]

Thanks! I see that fix is scheduled to be included in docker 17.12.1 (through moby/libnetwork#2052, and docker-archive/docker-ce#393), and, I think should already be in Docker 18.02 and up (but those are on the "edge" channel)

[thaJeztah]

17.12.1 was released with the fix that was mentioned, so I go ahead and close this, but comment if you're still able to reproduce on 17.12.1

[daniloradenovic]

Before upgrading to 17.12.1 I was experiencing this issue several times a week or so (at random). After upgrading to 17.12.1, about a month ago, I haven't seen this issue happening.

However, today I got a similar error (maybe it's a different issue). Here's an excerpt from the logs:

level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/c1624b9bccf5439abce51dfbb7f07dc2fa695049e90c5f800586011cae248a88/shim.sock" debug=false module="containerd/tasks" pid=18468
level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/f89574a07c2ae6415fa37259ef439649b746d1226c0370dc36c8f938194ffff3/shim.sock" debug=false module="containerd/tasks" pid=18528
level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/38371c4f33eab75fb4bea8dcc84906db4a6d43214df787f213705c8a239a18fb/shim.sock" debug=false module="containerd/tasks" pid=18748
level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/4025c5e592a8db94f0d1accd681f188ea7ce63d8681428e43727a14772e2e49d/shim.sock" debug=false module="containerd/tasks" pid=18812
level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/6cd99696b79e9e13b3db5ee3c3643bb45726163acc701dc8d2a3ac9f0c9ec8ee/shim.sock" debug=false module="containerd/tasks" pid=18918
level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/2d8cc9657df52ba209924af61060231b048edff91808a20ee233956ee3b08518/shim.sock" debug=false module="containerd/tasks" pid=19094
level=warning msg="Could not rollback container connection to network bridge"
level=info msg="shim reaped" id=2d8cc9657df52ba209924af61060231b048edff91808a20ee233956ee3b08518 module="containerd/tasks"
level=error msg="stream copy error: reading from a closed fifo"
level=error msg="stream copy error: reading from a closed fifo"
level=error msg="93595d254df0d5aa2962fd5e595d567d6444ccd8fbc749982606435967f765aa cleanup: failed to delete container from containerd: no such container"
level=error msg="Handler for POST /containers/93595d254df0d5aa2962fd5e595d567d6444ccd8fbc749982606435967f765aa/start returned error: failed to update the store state of sandbox: failed to update store for object type *libnetwork.sbState: timeout"
level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/fb18cb4ea6873b971695d61ca4cfb9598161c2f90569e9b32e4d74c0d3481da5/shim.sock" debug=false module="containerd/tasks" pid=19191
aufs au_opts_verify:1597:dockerd[15989]: dirperm1 breaks the protection by the permission bits on the lower branch
level=info msg="shim reaped" id=fb18cb4ea6873b971695d61ca4cfb9598161c2f90569e9b32e4d74c0d3481da5 module="containerd/tasks"
level=error msg="stream copy error: reading from a closed fifo"
level=error msg="stream copy error: reading from a closed fifo"
level=error msg="93595d254df0d5aa2962fd5e595d567d6444ccd8fbc749982606435967f765aa cleanup: failed to delete container from containerd: no such container"
level=info msg="shim reaped" id=38371c4f33eab75fb4bea8dcc84906db4a6d43214df787f213705c8a239a18fb module="containerd/tasks"
level=info msg="shim reaped" id=6cd99696b79e9e13b3db5ee3c3643bb45726163acc701dc8d2a3ac9f0c9ec8ee module="containerd/tasks"
level=info msg="shim reaped" id=f89574a07c2ae6415fa37259ef439649b746d1226c0370dc36c8f938194ffff3 module="containerd/tasks"
level=info msg="shim reaped" id=c1624b9bccf5439abce51dfbb7f07dc2fa695049e90c5f800586011cae248a88 module="containerd/tasks"
level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
level=error msg="2d8cc9657df52ba209924af61060231b048edff91808a20ee233956ee3b08518 cleanup: failed to delete container from containerd: no such container"
level=error msg="Handler for POST /containers/2d8cc9657df52ba209924af61060231b048edff91808a20ee233956ee3b08518/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-04-18T01:31:45Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
level=error msg="fb18cb4ea6873b971695d61ca4cfb9598161c2f90569e9b32e4d74c0d3481da5 cleanup: failed to delete container from containerd: no such container"
level=error msg="Handler for POST /containers/fb18cb4ea6873b971695d61ca4cfb9598161c2f90569e9b32e4d74c0d3481da5/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:385: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\\\\\\\\\"2018-04-18T01:34:26Z\\\\\\\\\\\\\\\" level=fatal msg=\\\\\\\\\\\\\\\"failed to update store for object type *libnetwork.sbState: timeout\\\\\\\\\\\\\\\"\\\\\\\\n\\\\\\\"\\\"\": unknown"
aufs au_opts_verify:1597:dockerd[3946]: dirperm1 breaks the protection by the permission bits on the lower branch
aufs au_opts_verify:1597:dockerd[16012]: dirperm1 breaks the protection by the permission bits on the lower branch

docker info

Containers: 32
 Running: 1
 Paused: 0
 Stopped: 31
Images: 7
Server Version: 17.12.1-ce
Storage Driver: aufs
 Root Dir: /mnt/docker/aufs
 Backing Filesystem: extfs
 Dirs: 152
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
******
Kernel Version: 4.4.0-1052-aws
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 29.44GiB
Name: ********
ID: ********
Docker Root Dir: /mnt/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

It's an AWS box and I could not find any suspicious load on the machine - CPU, RAM, disk were all on moderate levels without some peaks.

UPDATE: This looked like the same issue at a first glance, but on a second thought, it might be unrelated. I've opened another issue #300 . If it happens to be a duplicate, please close it.