IPAM driver proposal

[aboch]

Proposal

There have been many requests from end-users about having some control of the ip address assignment for containers. In order to satisfy this requirement in a common way across the different network drivers, we need to make some changes to the way ip addresses are managed today in the CNM model.

We think the address management should be deferred to a new component in libnetwork separate from the network drivers. CNM would interact with this component via a well known API during the network/endpoint creation/deletion based on the input received from user.

We are proposing to add an infrastructure and API for a new pluggable IPAM driver decoupled from the network driver.

CNM will ship with a default IPAM component, which database is strongly consistent across the cluster. User will be able to plug their own IPAM implementation and instruct CNM to use that instead of or along to the default one. CNM will provide a plugin mechanism for IPAMs similarly to the one offered for remote network drivers.

During the Network/Endpoint life cycle, libnetwork will interact with the ipam driver and allocate the minimally (for now) necessary address pool and addresses based on the user directives, if passed, or on some preallocated info otherwise. When libnetwork will invoke CreateNetwork() on network driver, it will also pass the network's ipam operational data (see IpamData below)

Initial definition of the IPAM API/contract will favor simplicity over comprehensiveness. A rough definition can be found below.

package ipam

type Config interface {
    RequestPool(addressSpace, pool, subPool string, options map[string]string, v6 bool) (string, *net.IPNet, map[string]string, error)
    ReleasePool(poolID string) error
}

type Allocator interface {
    RequestAddress(poolID string, preferredAddress net.IP, options map[string]string) (net.IP, map[string]string, error)
    ReleaseAddress(poolID string, address net.IP) error
}

An address space is a set of non-overlapping address pools.
The subPool may or may not be supported by the driver, but idea is that it is a way to split a pool in smaller non-overlapping pools (think of the docker `FixedCIDR` use case). If subPool is specified, the poolID returned by `RequestPool() identifies the subPool`
The options passed to `RequestAddress()` may contain, for example, the endpoint's mac address.

// IPData represents the per-network or per-endpoint ipam operational
// information libnetwork will send to the network driver during
// CreateNetwork() and CreateEndpoint() calls
type IPData struct {
    Pool *net.IPNet
    Meta map[string]string
}

The CNM/IPAM flow follows (some behaviour listed may be specfic to the default IPAM driver):

1. NewNetwork() is called. CNM will associate one or more subnet and gateway address to the new network. The subnet and gateway are optional parameters and can be specified by the caller of NewNetwork(). In case the subnet is not passed, its upto the ipam driver implementation to handle the case appropriately. The default/in-built IPAM driver will return the first un-allocated subnet from a predefined list along with a self designated address as gw. ipam plugin should return an id for the allocated subnet which will be associated with the network.
   (The above sequence will repeat for as many subnets as the user passes for the network creation).
2. CNM invoke the CreateNetwork() on the network driver and passes along the subnet and gateway address.
3. CreateEndpoint() is called. CNM will request ipam an address and store it in the endpoint.
   The address is an optional parameter and can be specified by the caller of CreateEndpoint().
   CNM will try to allocate an (or the preferred) address with ipam on the subnet id for this network.
   (In case of multiple subnets configured per Network, user is expected to also pass the subnet during endpoint creation call, otherwise libnetwork will request from each stored subnet until succeds)
4. CNM invokes the CreateEndpoint() on the network driver and passes along the interface address.
5. On endpoint Delete() CNM will release the address from the subnet id
6. On network Delete() CNM will release the subnet id

Open items

• Regarding multiple interfaces for Endpoint: can this requirement be removed ? Otherwise CNM would need to query the network driver to find out how many interfaces are needed per endpoint.
• At this point we are not clear yet if CNM should work with only one global IPAM driver or with a per-network driver. Current thought is to allow a per-network IPAM driver.

[Updated on 9/3] As per @tomdee suggestion, added options (labels) support in API and made it more explicit that multiple subnets per network is allowed ()

[Updated on 9/14] As per @lxpollitt and @tomdee comments, replaced "subnet" with "address pool" and have request address api return a CIDR (*net.IPNet) instead of a simple ip (net.IP).
Also replaced default gateway (net.IP) with ipam metadata (map[string]string) where, for example, and ipam driver can return a default gateway in form of a (key,value) pair at time of pool request or at the time of address request.
This metadata should be useful to address the case where ipam driver is backed by a DHCP.

[mavenugo]

ping @shykes @crosbymichael @icecrime @squaremo @tomdee @shettyg @thockin @erikh

IP addressing of services and containers have been one of the most requested feature in docker. So far, it was defined within the drivers, with minimal configuration (such as bridge-ip & fixed-cidr). With the introduction of Network plugins and ability to place containers on multiple networks, it is essential to provide a flexible IPAM that provides consistent addressing across the cluster. With the ability to plug an IPAM driver of choice, the user can potentially plug-n-play and network driver and still get consistent ip-addressing policy.

[thockin]

Can you sketch how something like DHCP would work?

[shettyg]

Creating a Network

NewNetwork() is called. CNM will associate a subnet and a gateway address to the new network.
The subnet and gateway are optional parameters and can be specified by the caller of NewNetwork().

If I have to pass a subnet to docker daemon while calling "docker network create -d openvswitch foo", how do you see it being done? i.e. what would a hypothetical user facing command look like?

If subnet is passed by the user and not gateway, what would happen? Will .1 in the subnet be chosen?

Also in the above command "openvswitch" refers to my plugin. Will the IPAM commands sent to the same plugin (good enough for me)? If not, how does the user provide that plugin information?

In case the subnet is not passed, its upto the ipam driver implementation to handle the case
appropriately.

So in this case, RequestSubnet() will be called to the remote ipam driver. Will this RequestSubnet() be broadcasted across the cluster?

The default/in-built IPAM driver will return the first un-allocated subnet from a predefined
list along with a self designated address as gw.

What will be the logic for choosing "self designated address as gw"? Just .1 in that subnet?

Also when will the default IPAM driver kick in? When the remote driver returns NULL? I ask this because, if ipam driver is per network, how do you decide which network driver wants to have a remote ipam and which network driver wants to have inbuilt ipam.

ipam plugin should return an id for the allocated subnet which will be associated with the network.

So the remote ipam plugin returns back an uuid? And that uuid will be provided back whenever that subnet is referenced?

[aboch]

@shettyg

If subnet is passed by the user and not gateway, what would happen? Will .1 in the subnet be chosen?

IPAM will chose itself an IP from that subnet for your gateway.

Also in the above command "openvswitch" refers to my plugin. Will the IPAM commands sent to the same plugin (good enough for me)? If not, how does the user provide that plugin information?

We think the user will specify the IPAM driver in the UI as he does for the network driver today

So in this case, RequestSubnet() will be called to the remote ipam driver. Will this RequestSubnet() be broadcasted across the cluster?

No. The remote IPAM driver will have to manage its own database.

What will be the logic for choosing "self designated address as gw"? Just .1 in that subnet?

I am thinking the self designated gateway address will be the last in the subnet range

Also when will the default IPAM driver kick in? When the remote driver returns NULL? I ask this because, if ipam driver is per network, how do you decide which network driver wants to have a remote ipam and which network driver wants to have inbuilt ipam.

When a network is configured and no ipam driver is specified in the UI, the default IPAM driver will be associated to that network by CNM.

So the remote ipam plugin returns back an uuid? And that uuid will be provided back whenever that subnet is referenced?

It will not be a uuid, just an identifier IPAM chooses for that subnet. Yes further requests to IPAM will have to specify that id.

[mavenugo]

@shettg @aboch

If I have to pass a subnet to docker daemon while calling "docker network create -d openvswitch foo", how do you see it being done? i.e. what would a hypothetical user facing command look like?

From a UX standpoint, we could use the labels for such operations (or) make it first-class option for the network command.

If subnet is passed by the user and not gateway, what would happen? Will .1 in the subnet be chosen?

Its entirely upto the ipam driver implementation. Maybe it is something pre-configured/determined in a customer network and say dhcp could be a backend driver giving out these config.

Also in the above command "openvswitch" refers to my plugin. Will the IPAM commands sent to the same plugin (good enough for me)? If not, how does the user provide that plugin information?

We are thinking of adding a --ipam-driver option. Should we make it per-network ipam-driver or a global ipam-driver that all networks will share ? My personal preference is to make it a global ipam-driver (since in reality the users will be interested in ip-address management in a more centralized way.

[jainvipin]

+1 for dhcp support - will help many use cases of migration, use of existing tools, etc.

[mavenugo]

@jainvipin yes. dhcp backend will be extremely useful. Will you be interested in contributing a dhcp based ipam-driver ? that would be awesome !

[aboch]

@thockin ATM we are not planning for a implement a DHCP plugin as part of libnetwork. In the case you want to use a DHCP server for your IP address management, I am thinking you would write a remote IPAM driver which would interface with DHCP server.

If you hit any road blocks during the design/implementation of such a plugin, please let us know which changes are needed to the IPAM driver API in order to facilitate this.

[thockin]

I think IPAM is really per-network driver

On Thu, Sep 3, 2015 at 9:22 AM, Madhu Venugopal notifications@github.com
wrote:

@jainvipin https://github.com/jainvipin yes. dhcp backend will be
extremely useful. Will you be interested in contributing a dhcp based
ipam-driver ? that would be awesome !

—
Reply to this email directly or view it on GitHub
#489 (comment).

[thockin]

I just want a sketch of how you think it would work, to demonstrate how
viable this proposal is

On Thu, Sep 3, 2015 at 9:39 AM, aboch notifications@github.com wrote:

@thockin https://github.com/thockin ATM we are not planning for a
implement a DHCP plugin as part of libnetwork. In the case you want to use
a DHCP server for your IP address management, I am thinking you would write
a remote IPAM driver which would interface with DHCP server.

If you hit any road blocks during the design/implementation of such a
plugin, please let us know which changes are needed to the IPAM driver API
in order to facilitate this.

—
Reply to this email directly or view it on GitHub
#489 (comment).

[mavenugo]

@thockin if it is a per-network driver, how can we provide any guarantees of ip-address reuse in different network and a container happen to belong to these multiple networks ?

[erikh]

Aside, it would be really nice to have configuration files for this
stuff.

On 3 Sep 2015, at 9:13, Madhu Venugopal wrote:

@shettg @aboch

If I have to pass a subnet to docker daemon while calling "docker
network create -d openvswitch foo", how do you see it being done?
i.e. what would a hypothetical user facing command look like?

From a UX standpoint, we could use the labels for such operations (or)
make it first-class option for the network command.

If subnet is passed by the user and not gateway, what would happen?
Will .1 in the subnet be chosen?

Its entirely upto the ipam driver implementation. Maybe it is
something pre-configured/determined in a customer network and say dhcp
could be a backend driver giving out these config.

Also in the above command "openvswitch" refers to my plugin. Will
the IPAM commands sent to the same plugin (good enough for me)? If
not, how does the user provide that plugin information?

We are thinking of adding a --ipam-driver option. Should we make it
per-network ipam-driver or a global ipam-driver that all networks will
share ? My personal preference is to make it a global ipam-driver
(since in reality the users will be interested in ip-address
management in a more centralized way.

---

Reply to this email directly or view it on GitHub:
#489 (comment)

[erikh]

I’m going to see what I can do this weekend.

On 3 Sep 2015, at 9:22, Madhu Venugopal wrote:

@jainvipin yes. dhcp backend will be extremely useful. Will you be
interested in contributing a dhcp based ipam-driver ? that would be
awesome !

---

Reply to this email directly or view it on GitHub:
#489 (comment)

[mavenugo]

@erikh 👍

[mavenugo]

Aside, it would be really nice to have configuration files for this stuff

@erikh you mean to configure the ipam driver ? That is entirely upto the plugin to depend and make use of any configuration file.