Support for swarm mode in Docker 1.12

[alantrrs]

I'm trying to use nvidia-docker with the swarm functionality introduced in the new Docker 1.12.

I tried to create a service docker service create .. with nvidia-docker (nvidia-docker service create ..) and didn't work. I haven't seen any way to pass devices to docker service create so I'm wondering if it's even supported on docker's side.

Any thoughts?

[flx42]

This question was asked on the docker GitHub a few hours ago:
moby/moby#24750

Currently, nvidia-docker doesn't support Docker Swarm and thus service create will be simply pass-through to the docker CLI.
You are right that there doesn't seem to be a way to pass devices to service create, so far we can only mount the volume:

docker service create --mount type=volume,source=nvidia_driver_367.35,target=/usr/local/nvidia,volume-driver=nvidia-docker [...]

But that's not enough, we can't get around the device cgroup.

Even if we could, in a cluster environment with Swarm there will also be a problem if different machines have a different number of GPUs.

@3XX0 thoughts?

[josef-hak]

+1 to add nvidia-docker support for docker swarm.

[davad]

@flx42 @3XX0 any movement on this? I looked over the related issues and don't see anything recent. I'm itching to orchestrate CUDA jobs via docker across multiple machines 😄

[el3ment]

Any progress on this?

[anaderi]

would be cool, no?

[3XX0]

This is basically what we need for basic GPU support:

moby/swarmkit#2090

[mjp0]

@3XX0 It got merged today! I've been managing nvidia-docker containers manually via docker-compose so having swarmkit and all the v3 deploy things work would be absolutely fantastic.

Is there any potential merge conflicts that has to be dealt with before this can be merged to nvidia-docker?

[cheyang]

@0fork , can you share any docs about how you played with it? So we can also try this cool feature. Thanks.

[3XX0]

Yes, this is big step forward to get GPU working within Swarm. However, we're not quite there yet, we still need to add support in Docker itself and we are still missing some pieces which should come with nvidia-docker 2.0.

Stay tuned ;)

[luiscborbon]

+1

[erbas]

@3XX0 What's the timeframe for this to come together?