docker-ce package makes existing KVM guests unusable

[TheCasualObserver]

• This is a bug report
• This is a feature request
• I searched existing issues for the keyword 'KVM' and 'QEMU' before opening this one

Expected behavior

The act of installing the package 'docker-ce' should not interfere with existing KVM-guests and make them unable to access the network on Debian Buster.

Actual behavior

Installing the package 'docker-ce' causes, that KVM guests can no longer access the network.
In some cases a KVM guest can immediately after the installation no longer access the networt. After rebooting any KVM-guest, this KVM guest has no longer access to the network.

After deinstallation of the package 'docker-ce' and rebooting the system, all KVM guests can access the networt again.

Steps to reproduce the behavior

Prerequisite: Debian Buster 64 bit host with KVM-QEMU debian guests

1. Follow the chapter "Install using the repository" on https://docs.docker.com/install/linux/docker-ce/debian/ and install the packages 'docker-ce-cli' and 'containerd.io' (after this step the KVM guests work without problems.)
2. install 'docker-ce' (after this step KVM guests can no longer access the network)

Output of docker version:

:~# docker version
Client: Docker Engine - Community
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.16
 Git commit:        369ce74a3c
 Built:             Thu Feb 13 01:27:58 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.16
  Git commit:       369ce74a3c
  Built:            Thu Feb 13 01:26:32 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

:~# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-8-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.13GiB
 Name: REDACTED
 ID: FKUI:EV5T:GBFV:LJYK:UNRV:GDKL:I5DO:3BAF:KDRI:L3XY:NWVN:EBLN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

The host is a physical 64-bit system.

:~# kvm --version
QEMU emulator version 3.1.0 (Debian 1:3.1+dfsg-8+deb10u4)
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers

Thank you for having a look at this issue.

[cpuguy83]

It sounds likely that the docker bridge network is conflicting with your kvm net? Docker tries to pick an unused address space. I'm not sure I've seen it pick one that was in use before only error out because it can't find one.

What you might want to do is either:

1. Specify a custom subnet for the bridge (daemon option --bip=<bridge ip>/<subnet>)
2. Specify a custom bridge (daemon option --bridge=myBridge)
3. Disable the bridge (daemon option --bridge=none)... you may also want to disable iptables management if you are going to do this --iptables=false

[TheCasualObserver]

Thank you for taking the time to compile these hints. However they do not help in my situation, because the problem happens during the installation of the 'docker-ce' package - long before I have an opportunity to configure the docker daemon options.

Today I found a workaround which allows me to install 'docker-ce' without breaking my KVM guests.
I changed /etc/sysctl.conf and added net.ipv4.ip_forward=1.

[cpuguy83]

Ah yes... libnetwork messes with that...
It used to enable it explicitly... then it disabled it explicitly... I don't know why it even touches it.

[cpuguy83]

@arkodg WDYT? Can libnetwork just not change net.ipv4.ip_forward?

[arkodg]

@cpuguy83 I don't understand how libnetwork get called when you install docker-ce
In 19.03.6 when dockerd is spawned this https://github.com/moby/libnetwork/blob/74191015cdfdfae1c1774c07e1c46e590d4adc13/drivers/bridge/setup_ip_forwarding.go#L24 gets called, and it looks like it does the following

1. if Ipv4 forwarding is disabled, it enables it
2. if --iptables=false it bails out
3. else it attempts to add a iptables default drop policy for the forward chain
4. if that fails it disables Ipv4 forwarding

Step 4 might not be ideal, we might want to reset it the original value that was read

Is this what is happening ?

@TheCasualObserver
do you see this error (https://github.com/moby/libnetwork/blob/5eb06eb42dec3cb0586ea7ce39d7829580e712f2/iptables/iptables.go#L520) in your dockerd logs

[cpuguy83]

Because, as I recall, dockerd is started on install.

I think actually in the past we used to set the forward policy to accept, and now we set it to drop.

[thaJeztah]

I think that was changed, related to moby/moby#14041 (in moby/libnetwork#1526)

[arkodg]

@cpuguy83 that change you are referring to is only on master (moby/libnetwork#2450), not in 19.03 :)

[wyphan]

Sorry for bumping up an old issue, but I can confirm that this also happens with Ubuntu 20.04.1 LTS (focal) amd64 host. And the workaround described above still works.

[nazar-pc]

I have the same issue after starting Docker at system boot in Alpine Linux (docker package from their official repos), but not when starting after.

Interestingly, default policy for FORWARD changes in that case 🤔

When booting without Docker installed, iptables rules look like this:

Details

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_INP  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_FWX  all  --  anywhere             anywhere            
LIBVIRT_FWI  all  --  anywhere             anywhere            
LIBVIRT_FWO  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_OUT  all  --  anywhere             anywhere            

Chain LIBVIRT_FWI (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target     prot opt source               destination         
ACCEPT     all  --  192.168.122.0/24     anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain LIBVIRT_INP (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:68

If I start Docker afterwards, it is updated to the following:

Details

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_INP  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
LIBVIRT_FWX  all  --  anywhere             anywhere            
LIBVIRT_FWI  all  --  anywhere             anywhere            
LIBVIRT_FWO  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_OUT  all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain LIBVIRT_FWI (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target     prot opt source               destination         
ACCEPT     all  --  192.168.122.0/24     anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain LIBVIRT_INP (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:68

However, if Docker was enabled from the start (and before libvirt because it is alphabetically first on Alpine Linux), then I get this and libvirt is broken:

Details

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_INP  all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         
LIBVIRT_FWX  all  --  anywhere             anywhere            
LIBVIRT_FWI  all  --  anywhere             anywhere            
LIBVIRT_FWO  all  --  anywhere             anywhere            
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
LIBVIRT_OUT  all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain LIBVIRT_FWI (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target     prot opt source               destination         
ACCEPT     all  --  192.168.122.0/24     anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain LIBVIRT_INP (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:68

I'm wondering why this happens and whether it works properly with default policy ACCEPT as well? I do not know much about networking, but would be surprised if it did.