Using nvidia-docker from third-party tools

[hannes-brt]

It's very easy to use nvidia-docker when running individual containers, but is there a way to run nvidia-docker instead of docker from other Docker tools like docker-compose, Tutum, Rancher, etc?

I am assuming one would just need to specify the nvidia-docker volume to be mounted in the container, but I couldn't find any documentation on the correct syntax.

[flx42]

If the tool supports overriding the docker command, then you should use that to plug in nvidia-docker. For example, we provide this option ourselves with environment variable NV_DOCKER.

If it's not possible, you can query the Docker CLI arguments to the plugin:

$ curl -s localhost:3476/docker/cli
--device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia0 --device=/dev/nvidia1 --volume-driver=nvidia-docker --volume=nvidia_driver_352.68:/usr/local/nvidia:ro

Of course you will need to transform this into YAML format for docker-compose (for example).
We were wondering what to do in this case and we couldn't find a clean solution. Would that help if we add a REST endpoint that returns the CLI above as YAML? Or as JSON?

Thank you!

[ruffsl]

If a sorf-of multiline variable substitution was possible in compose files, I guess you could add a YAML format option for the for the REST endpoint. But I think the environment variables are queried after the YAML syntax is parsed, resulting in "invalid mode" error from the improper structure. This still might be useful for just a 'copy and paste' approach for compose files though.

Is there a way we could use the volume-driver option with say set label format to convey the card IDs to export?

[3XX0]

@ruffsl I like the idea of using the variable substitution.
Can't we leverage the one-line YAML syntax? If the Rest, output something like:

NV_DEVICES="['/dev/nvidia0', '/dev/nvidia1']"

We could easily do something like (the same way docker-machine does it):

eval "$(curl -s localhost:3476/docker/env)"

given the following docker-compose.yml, it would work right?

devices: ${NV_DEVICES}

[ruffsl]

I'm just testing a foo bar example:

test:
  image: ubuntu
  volumes: ${FOO_BAR}
  command: ping 127.0.0.1

and am seeing this:

$ mkdir /tmp/foo
$ mkdir /tmp/bar
$ export FOO_BAR="['/tmp/bar:/bar', '/tmp/foo:/foo']"
$ docker-compose up
ERROR: Validation failed in file './docker-compose.yml', reason(s):
Service 'test' configuration key 'volumes' contains an invalid type, it should be an array

Is this the correct one-line YAML syntax? I've never seen it before.
Also, how could you keep other volumes or device lists defined alongside (stuff you'd like to keep in the compose file)?

[3XX0]

Not sure if docker-compose supports it but it's part of the YAML spec:
http://yaml.org/spec/1.2/spec.html#id2759963

For other volumes/devices if docker-compose supports multiple volumes or devices keywords it would work, otherwise by omitting the brackets I suppose.

[flx42]

It seems to work without variable substitution:

test:
  image: ubuntu
  devices: ['/dev/nvidiactl', '/dev/nvidia-uvm', '/dev/nvidia0']
  command: ls /dev/

Variable substitution might be designed to generate values, and not YAML.
But let's wait for an official response from the docker-compose developers.

@ruffsl: I didn't understand the following, could you explain it?

Is there a way we could use the volume-driver option with say set label format to convey the card IDs to export?

[ruffsl]

@flx42 This was an example of what I was thinking

test:
  image: ubuntu
  volume_driver: nvidia-docker-driver
  labels:
    nvidia.gpu: "0,1"
  command: nvidia-smi

I'm not sure how feasible this would be, but I think its attractive in its simplicity from a user perspective. You've developed a plugin, could a custom driver be extend to parse the label metadata? This then could be a clean way to define nvidia containers with compose.

You could also use the variable substitution with this as well: export NV_GPU='0,1'

labels:
    nvidia.gpu: ${NV_GPU}

[flx42]

A volume plugin will not be able to mount devices, and actually a plugin can't even inspect the starting image AFAIK.

[flx42]

Good progress on docker/compose#2750
There is a pending PR to solve this use-case (haven't tested it).
But it will require a bleeding edge version of docker-compose if it's accepted.

[3XX0]

@flx42 is correct, currently the only supported plugin is VolumeDriver and it only deals with volume names (code).
I think a JSON endpoint could be useful here, however, it means you would need to write small wrappers around docker-compose and other tools to do the conversion (e.g. JSON -> YAML config)