Connecting IPv4 network to container with IPv6 network resets IPv6 Gateway

[freva]

I'm trying to connect containers on different hosts with each other and the outside world using the macvlan driver over IPv6. Additionally, the containers need to reach some IPv4 services, so I connected them through docker's default bridge network. The issue is the IPv4 network resets the IPv6 gateway. The two networks are:

[
 {
        "Name": "bridge",
        "Id": "1eb235a94875b7451aa9b259e01c08d7d2771844d279df733214059f7db6067d",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.42.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    },
    {
        "Name": "macvlan",
        "Id": "6f4191fe93879b6200719a16ab2723e195601a8875f35f9e432676c10c5d3c39",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": true,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1/16"
                },
                {
                    "Subnet": "w:x:y:z::/64",
                    "Gateway": "w:x:y:z::1"
                }
            ]
        },
        "Internal": false,
        "Containers": {},
        "Options": {
            "macvlan_mode": "bridge",
            "parent": "ens15"
        },
        "Labels": {}
    }
]

Each of them works correctly individually, but adding both through docker network connect always resets the IPv6 gateway. for example:

freva@docker:~$ sudo docker network create --ipv6 -d macvlan --subnet w:x:y:z::/64 --gateway w:x:y:z::1 -o parent=ens15 -o macvlan_mode=bridge macvlan
6f4191fe93879b6200719a16ab2723e195601a8875f35f9e432676c10c5d3c39
freva@docker:~$ sudo docker run -tid --name=macvlanbridge --net=macvlan --ip6=w:x:y:z::1078 ubuntu:14.10 bash
bb5b2ea440335379e83676d7968f921fde7307e66a318b3244f8f092ed98c570
freva@docker:~$ sudo docker inspect macvlan
[
    ...
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "c100356b5ba1ae5e369ae8d99cd8ba3bff5751869e8d39e25ad6c59929a16375",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/c100356b5ba1",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "macvlan": {
                    "IPAMConfig": {
                        "IPv6Address": "w:x:y:z::1078"
                    },
                    "Links": null,
                    "Aliases": [
                        "bb5b2ea44033"
                    ],
                    "NetworkID": "6f4191fe93879b6200719a16ab2723e195601a8875f35f9e432676c10c5d3c39",
                    "EndpointID": "eb5bfc0a7f2f37203cd831282a577d1f7050e170c02f8b36e305c6a0455f439f",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "w:x:y:z::1",
                    "GlobalIPv6Address": "w:x:y:z::1078",
                    "GlobalIPv6PrefixLen": 64,
                    "MacAddress": "02:42:ac:12:00:02"
                }
            }
        }
    }
]
freva@docker:~$ sudo docker stop macvlanbridge && sudo docker network connect bridge macvlanbridge && sudo docker start macvlanbridge
macvlanbridge
macvlanbridge
freva@docker:~$ sudo docker inspect macvlanbridge
[
    ...
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "bca5cbd08b2a8e8b5d4f96130ab98e180ed6498f9deafc993d24d633987da09d",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/bca5cbd08b2a",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "e46410c0e9f8104dd5190bbec5e2026ec1a91b31f8d679fa34562118e6c1c3cd",
            "Gateway": "172.17.42.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.1",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:01",
            "Networks": {
                "bridge": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [],
                    "NetworkID": "1eb235a94875b7451aa9b259e01c08d7d2771844d279df733214059f7db6067d",
                    "EndpointID": "e46410c0e9f8104dd5190bbec5e2026ec1a91b31f8d679fa34562118e6c1c3cd",
                    "Gateway": "172.17.42.1",
                    "IPAddress": "172.17.0.1",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:01"
                },
                "macvlan": {
                    "IPAMConfig": {
                        "IPv6Address": "w:x:y:z::1078"
                    },
                    "Links": null,
                    "Aliases": [
                        "bb5b2ea44033"
                    ],
                    "NetworkID": "6f4191fe93879b6200719a16ab2723e195601a8875f35f9e432676c10c5d3c39",
                    "EndpointID": "2af12e0af0593ffa388d11f5f15708743a5b136b21b46c25cff68a530176bc98",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "w:x:y:z::1",
                    "GlobalIPv6Address": "w:x:y:z::1078",
                    "GlobalIPv6PrefixLen": 64,
                    "MacAddress": "02:42:ac:12:00:02"
                }
            }
        }
    }
]
freva@docker:~$ sudo docker exec -ti macvlanbridge bash
root@bb5b2ea44033:/# route -A inet6 -n
Kernel IPv6 routing table
Destination                    Next Hop                   Flag Met Ref Use If
w:x:y:z::/64                   ::                         U    256 0     0 eth1
fe80::/64                      ::                         U    256 0     0 eth0
fe80::/64                      ::                         U    256 0     0 eth1
::/0                           ::                         !n   -1  1     1 lo
::1/128                        ::                         Un   0   1     2 lo
w:x:y:z::1078/128              ::                         Un   0   1     0 lo
fe80::42:acff:fe11:1/128       ::                         Un   0   1     0 lo
fe80::42:acff:fe12:2/128       ::                         Un   0   1     0 lo
ff00::/8                       ::                         U    256 0     0 eth0
ff00::/8                       ::                         U    256 0     0 eth1
::/0                           ::                         !n   -1  1     1 lo
root@bb5b2ea44033:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.17.42.1     0.0.0.0         UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth1

After this, the IPv4 network works correctly, but the IPv6 is unreachable because no gateway is set. Simply running
sudo route -A inet6 add default gw w:x:y:z::1 eth1
will fix the macvlan net as well.

Is this related to #1141/#1142 or is the IPv6Gateway set to "" because bridge IPv6Gateway is not set and bridge comes before macvlan lexicographically? The expected outcome was that it should assume macvlan's IPv6 default gateway since it is the only IPv6 gateway that is actually set.

[freva]

Forgot to add:

freva@docker:~$ sudo docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   d0db6a0
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   d0db6a0
 Built:        
 OS/Arch:      linux/amd64

freva@docker:~$ sudo docker info
Containers: 3
 Running: 3
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-253:2-394188-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 21.47 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 6.196 GB
 Data Space Total: 107.4 GB
 Data Space Available: 101.2 GB
 Metadata Space Used: 6.914 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.141 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /home/docker/data/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /home/docker/data/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge null macvlan overlay host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0.x86_64
Operating System: Red Hat Enterprise Linux Server 7.1 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 30.82 GiB
Name: docker.xxx.yyy
ID: LM2N:E45S:J3XM:QQKO:VLV6:WUKY:6ZZQ:PRK5:DLTS:LMKL:35KT:HRL4
Docker Root Dir: /home/docker/data
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 45
 Goroutines: 80
 System Time: 2016-09-13T10:08:29.057656196Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

[aboch]

Is this related to #1141/#1142 or is the IPv6Gateway set to "" because bridge IPv6Gateway is not set and bridge comes before macvlan lexicographically? The expected outcome was that it should assume macvlan's IPv6 default gateway since it is the only IPv6 gateway that is actually set.

You are correct, that is the issue along with #1141 (comment)

As a workaround, instead of connecting your container to the default bridge network, could you try to create a bridge network which names comes after macvlan and connect your container to that instead ?

[freva]

The problem is that as far as I know, there is no way to disable IPv4, so my pure IPv6 macvlan network still has an IPv4 subnet and gateway, then, since macvlan would come before whatever the new name of that other network would be, the IPv4 config would be wrong.

So what I need is that it takes IPv4 subnet+gateway from one network and IPv6 subnet+gateway from the other. Then, there no way to name these so that both of these hold.

However, if it simply checked if the IPv6 subnet+gateway are not set for current network, it should simply ignore it, then my current setup should work:

• Both bridge and macvlan have IPv4 subnet+gateway, so it applies bridge's values since it comes before macvlan.
• Only macvlan has IPv6 subnet+gateway, so it should apply its values even though bridge comes before macvlan.

[aboch]

The problem is that as far as I know, there is no way to disable IPv4

True, there is no way at the moment.
Sometime back I proposed a --ipv6-only flag on network creation moby/moby#18607 (comment). We have not had time to look into this yet.

[freva]

But is it possible to at least ignore IPv6 subnet+gateway of bridge since it is not set? Under current implementation (lexical order) there is no way to have subnet+gateway from two different networks. I understand that the general case is hard and maybe even would have to be re-implemented, but having two different networks for IPv4 and IPv6 should be a common case and easy to separate.

My suggestion is to continue setting subnet+gateway the way you described it in #1141, but also include an exception if the gateway+subnet is not actually set. That is,
When a container is connected to multiple networks, its external connectivity is provided via the first non-internal network with subnet/gateway set, in lexical order.

If this is evaluated individually for IPv4 and IPV6, then it is possible to have different subnet+gateway for IPv4 and IPv6 by having the IPv4 network be lexically first and not specifying any IPv6 values on it.

[freva]

Any chance this will be fixed soon? The network setup does all of the connecting correctly, but performs an unnecessary overwrite:

• First connect the container to a dual-stack network --> This sets IPv4 and IPv6 subnet, gateway, etc.
• Then connect the container to an IPv4-only network --> This network comes alphabetically before the previous network, so it overwrites all the IPv4 settings (so far everything is great), but then it also unsets IPv6 detault gateway, why? This network has EnableIPv6 flag to false, so it should just ignore all IPv6 values?