Docker-in-Docker build is fragile to loopback exhaustion

[smarterclayton]

I'm seeing this quite commonly - builds with DinD fail because they can't get a loopback device. Is there some obvious cleanup step we are not doing, or is this path just really fragile?

$ docker logs 235fb847e05c
Attempting to create /dev/loop4 for docker ...
Attempting to create /dev/loop5 for docker ...
2014/09/18 08:49:51 docker daemon: 1.1.2 d84a070/1.1.2; execdriver: native; graphdriver: 
[4eab6c59] +job serveapi(unix:///var/run/docker.sock)
[4eab6c59] +job initserver()
[4eab6c59.initserver()] Creating server
[error] attach_loopback.go:42 There are no more loopback devices available.
loopback mounting failed
[4eab6c59] -job initserver() = ERR (1)
2014/09/18 08:49:51 loopback mounting failed
Docker-in-Docker daemon not accessible

[smarterclayton]

@ncdc as wel

[csrwng]

@smarterclayton @ncdc This sounds like the issue that was fixed in upstream libcontainer that was causing pod containers not get the shared network device from the net container.

[smarterclayton]

The upstream issue had to do with filesystem device mapper loopback mounts? That was (afaik) a race on setup of networks, not the raw disk access

On Sep 18, 2014, at 5:22 AM, Cesar Wong notifications@github.com wrote:

@smarterclayton @ncdc This sounds like the issue that was fixed in upstream libcontainer that was causing pod containers not get the shared network device from the net container.

—
Reply to this email directly or view it on GitHub.

[ncdc]

@smarterclayton @csrwng no, this is not the libcontainer issue. There is something holding the dind devmapper mounts open when docker is trying to stop and remove the container. It doesn't happen all the time, but it does happen fairly frequently. The best thing I've found that seems to have possibly relevant info is moby/moby#2714 (comment), but that's a really old issue so I'm not sure if the root cause is still the same.

lsof doesn't show anything related to the loops in question. The best I've come up with is grep [container id or dm id] /proc/*/mounts, but I still don't know why the loopbacks are "busy" and can't be unmounted.

[ironcladlou]

I think one problem is that signals aren't making it to and being trapped by the dind script, preventing graceful shutdown. I resolved this in my branch for running origin with docker-in-docker.

However, I'm proposing we ditch DinD altogether: #111

[ironcladlou]

Another note on fragility: moby/moby#7285

[smarterclayton]

Closed by #111