test-containerized fails on ipv6 tests when running with docker V17.06 or later.

[djlwilder]

Here is the failure.. The failure is not seen with older version of docker.

<....>
• Failure [2.991 seconds]
Libnetwork Tests
/go/src/github.com/projectcalico/libnetwork-plugin/tests/default_environment/libnetwork_test.go:346
  docker run ipv6
  /go/src/github.com/projectcalico/libnetwork-plugin/tests/default_environment/libnetwork_test.go:343
    creates a container on a network  and checks all assertions [It]
    /go/src/github.com/projectcalico/libnetwork-plugin/tests/default_environment/libnetwork_test.go:342

    Expected
        <string>: 2001:db8:7f30:3859:b73a:3e8b:9882:1bc0 dev cali0  metric 256 
        fe80::/64 dev cali0  metric 256 
        unreachable default dev lo  metric -1  error -101
        ff00::/8 dev cali0  metric 256 
        unreachable default dev lo  metric -1  error -101
    to match regular expression
        <string>: default via fe80::.* dev cali0  metric 1024
<....>

As of Docker 17.06 the Docker server configures the container network for IPv4 only. Ipv6 is disabled by default. When docker creates a new network name-space the default setting for ipv6 is "disabled".

$ docker run ppc64le/busybox sysctl -a | grep disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1   
net.ipv6.conf.default.disable_ipv6 = 1   <<<<<<<
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0

Current Behavior

How is this affecting Calico test code?
The libnetwork test runs in a docker-in-docker container thus ipv6 is disabled by default in the dind container. This example show the results seen after the test has failed.

$ docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS                    NAMES
a2e43b48e3f5        ppc64le/docker:dind                  "dockerd-entrypoin..."   31 seconds ago      Up 30 seconds       0.0.0.0:5375->2375/tcp   dind
8257f40d8c03        quay.io/coreos/etcd:v3.2.5-ppc64le   "etcd --advertise-..."   32 seconds ago      Up 31 seconds                                calico-etcd

ipv6 is disabled for the caliXXX interface.

$ docker exec dind sysctl -a | grep disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.cali7eea7e5f232.disable_ipv6 = 1  <<<<<<
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.docker0.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0

An Ipv6 link-local address has not be created for caliXXXX.

$ docker exec dind ifconfig cali7eea7e5f232
cali7eea7e5f232 Link encap:Ethernet  HWaddr C6:36:C8:0F:B2:6F  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:728 (728.0 B)  TX bytes:0 (0.0 B)

If ipv6 is enabled in the dind container then the caliXXX interface will be assigned a ipv6 link-local address.

$ docker exec dind sysctl -w net.ipv6.conf.cali7eea7e5f232.disable_ipv6=0
net.ipv6.conf.cali7eea7e5f232.disable_ipv6 = 0

$ docker exec dind ifconfig cali7eea7e5f232
cali7eea7e5f232 Link encap:Ethernet  HWaddr C6:36:C8:0F:B2:6F  
          inet6 addr: fe80::c436:c8ff:fe0f:b26f/64 Scope:Link  <<<<<<<<<
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:728 (728.0 B)  TX bytes:508 (508.0 B)

The test code can be fixed by adding a line to the run-plugin target to enable ipv6 as was shown above.

However, is this a test bug or a libnetwork-plugin issue?
In normal operation I would not expect to see this issue, typically net.ipv6.conf.default.disable_ipv6 is 0 on the host, adding the extra dind layer in the test uncovers the issue. If the user disables ipv6 on the host then the problem will return. Is that expected behavior? I looked at how the bridge driver handled this.

If the dind container is started with --ipv6 the docker0 interface will have ipv6 enabled.

$ docker exec dind sysctl -a | grep disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.calidc49bd71b45.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.docker0.disable_ipv6 = 0  <<<<<
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0

I found code in the bridge driver that enables ipv6 on the docker0 interface when the container is started with --ipv6. Should calico do the same? The bridge driver only need to enable ipv6 on intrinsic bridge interfaces, calico would need to do this for every ipv6 enabled host side interface.

Possible Solution

enable ipv6 in the dind container ?

diff --git a/Makefile b/Makefile
index 709e518..df6386f 100644
--- a/Makefile
+++ b/Makefile
@@ -125,6 +125,7 @@ run-plugin: run-etcd dist/libnetwork-plugin-$(ARCH)
                -v $(PLUGIN_LOCATION):/libnetwork-plugin \
                $(DIND_IMAGE) --cluster-store=etcd://$(LOCAL_IP_ENV):2379
        # View the logs by running 'docker exec dind cat plugin.log'
+       docker exec -tid --privileged dind sh -c 'sysctl -w net.ipv6.conf.default.disable_ipv6=0'
        docker exec -tid --privileged dind sh -c '/libnetwork-plugin 2>>/plugin.log'
        # To speak to this docker:
        # export DOCKER_HOST=localhost:5375

[caseydavenport]

@djlwilder FWIW, the CNI plugin has code which enables ipv6 on the veths it creates whenever an ipv6 address has been assigned: https://github.com/projectcalico/cni-plugin/blob/bcd2d3f825002c218bfaa872206f9bf42753c7e3/utils/network.go#L117-L127

Perhaps the libnetwork plugin should be doing the same?

[djlwilder]

I believe the code you pointed to is enabling ipv6 in the container's namespace. The issue with the test failing is because ipv6 disabled in the host's namespace. In this case the "host's" namespace is the namespace of the dind container (were are running the libnetwork-plugin in a dind container).

Docker made a changes at some point to set net.ipv6.conf.all.disable_ipv6 = 1 in the container's namespace. This results in the calico interface created in the dind container to have ipv6 disabled as well (net.ipv6.conf.calidc49bd71b45.disable_ipv6 = 1) as the setting was inherited form the default value..

An analogy is if we set disable ipv6 on the host (net.ipv6.conf.all.disable_ipv6 = 1) and then started a container that uses a calico network. I would expect ipv6 to fail in this case. But is that expected as ipv6 is disabled.

[caseydavenport]

@djlwilder ah right, I overlooked that. Thanks for the details.

[djlwilder]

We could enable ipv6 for every cali interface in the host. This would allow ipv6 to work (in the normal case, not the dind environment) even if the administration has disabled ipv6 on the host. But is that the correct behaviour?

[caseydavenport]

If I understand the problem correctly (which is a big assumption!) then IPv6 will still work if properly enabled on the host, and IPv4 will still work even if IPv6 is not enabled on the host. Is that right?

If so, then I'm happy to say this is just a bug in the way the CI is stood up. (Since the tests assert IPv6 addresses are present, but don't enable IPv6 on the system).

[djlwilder]

yes that is correct.

[caseydavenport]

OK thanks for bearing with me on that.

In that case, I think we can close this since I merged your fix here: #163