[RFC] GPU support in CLI

[flx42]

The goal of this RFC is to discuss how we can expose the GPU support that was recently added to containerd by @crosbymichael in this PR: containerd/containerd#2330

Since there are multiple ways of implementing this, I think it makes sense to start with the user experience, i.e. the CLI. If we can agree on the format, the next step will be that I write a PR for API changes in the github.com/docker/docker/api/types package. My rationale being that if we decide to not allow some configuration options in the CLI, it would be useless to have api/types expose it. As a concrete example, if we don't allow passing options on a per-GPU basis, it shouldn't be part of the API.

After discussing with @crosbymichael, he advised to limit the support to GPUs. We are excluding other types of specialized hardware such as FPGAs, InfiniBand, ASICs... from this RFC.

This RFC introduces some vocabulary, then presents the approach I find is the most compelling.

GPU Vendors

• While the containerd PR is only for NVIDIA GPUs, the CLI should be able to support multiple vendors.
• For simplicity, we should enforce the GPU vendor to be a lowercase string with no special characters.
  e.g. nvidia, intel or amd.
• Reverse-DNS notation (e.g. com.nvidia) is too verbose and should probably be avoided for a CLI.
• The list of support GPU vendors could be hardcoded in the CLI and checked, since it's unlikely to change quickly, but it is not a requirement.

GPU identifiers

• GPU support should not be a boolean flag like --with-gpus, users should be able to select a subset of all GPUs, similarly to --cpuset-cpus: nvidia=0,1,2,3
• We should not assume a numerical identifier for GPUs (unlike --cpuset-cpus), GPUs can have a string representation, for instance a UUID: nvidia=GPU-fef8089b-4820-abfc-e83e-94318197576e
• The CLI can only verify the basic format of GPU identifiers, it can't detect if they are ultimately valid (e.g. if the GPU exists or not), this will be done later and will be vendor-specific.

GPU options

• In addition to selecting GPUs, the user should be to add options to modify the runtime behavior. For the containerd PR it would mean having a way to modify this struct from the CLI: https://github.com/containerd/containerd/blob/master/contrib/nvidia/nvidia.go#L88-L96
• The baseline would be to have options that apply to all GPUs/to the system: nvidia.capabilities = compute.
• An extension would be to have options per-GPU, for instance to enforce that GPU 0 can only use 2 GB of memory, whereas GPU 1 can use 6 GB: limit_mem[0] = 2GB limit_mem[1] = 6GB.
• Same as for identifiers, the CLI can't perform actual validation of the options passed by the user, it's vendor-specific.

Suggested approach

Add two CLI options:

• --gpus for specifying a list of GPUs (like --cpuset-cpus)
• --gpus-opt for specifying a list of GPU options (like --storage-opt)

docker run --gpus nvidia=0,1,2,3 --gpus nvidia=GPU-abcd --gpus-opt nvidia.capabilities=compute --gpus-opt nvidia.kmods=true

The important element being the vendor name, which serves as a key for specifying options and identifiers.
I think this is easy to read since you don't have to pack everything in a single argument.

If we want per-GPU options, we can introduce some sort of indexing in --gpus-opt:

--gpus-opt nvidia.0.limit_mem=2GB
--gpus-opt nvidia[0].limit_mem=2GB

Discarded approaches

I think the approach above is the best, but here are some thoughts on other possibilities.

Single CSV argument

Similar to --mount, add a new argument --gpus which expects a CSV value.

docker run --gpus vendor=nvidia,options=capabilities=compute,options=kmods=true,0,1,2,3,GPU-abcd

This is explicit and everything is contained in one option, but it's also bizarre since we have nested equal signs, and we have to expand the list of GPU identifiers as multiple arguments, to avoid needing complicated quoting from the shell.

A follow-up question would then be: can you specify multiple times the same vendor? If yes, what if the listed options don't match?

Similar to --device

Add a new argument --gpus which expects 2 or 3 strings separated by :

docker run --gpus nvidia:0,1,2,3,GPU-abcd:capabilities=compute,kmods=true

Less problematic than approach above, but hard to read.

NVIDIA specific

Same than the suggested approach, but without being generic:

docker run --nvidia-gpus 0,1,2,3 --nvidia-gpus GPU-abcd --nvidia-opt capabilities=compute --nvidia-opt kmods=true

It removes the need to use the vendor name as the key, and is fine for now, but is likely to be removed in the future if we add more vendors. Also, this doesn't match the approach used by the containerd PR.

[flx42]

cc @3XX0 @crosbymichael

[cpuguy83]

First approach SGTM

[cwgem]

• What's the recommended method of centralizing these options for mass deployment? Something like docker compose? Given how lengthy this makes the CLI arguments (even more so if per GPU properties are supported) I feel like someone would rather dump all the config info in a JSON file or so and have the CLI read off that instead.
• What's the support matrix look like? Not sure if "whatever docker runs on" would be specific enough (OSX for example). I'm assuming OS package availability and kernel version would play into this.

[cpuguy83]

What's the recommended method of centralizing these options for mass deployment?

alias nvidia-docker="docker run --gpus=nvidia:0,1,2,3 --gpus-opt="..."

Really, if I just had a standard set of things I'd want to include, I'd put it into an variable or a shell function that I can use to inject it into my docker run with docker run ${my_gpu_opts}

What's the support matrix look like?

The docker client doesn't need to care about this. If the system that the engine is running on doesn't support it (e.g., likely that docker4mac doesn't unless they end up injecting a GPU passthrough or something?) then it would error out from the daemon side, likely at runtime.

[cwgem]

I thought about responding with additional feedback but not even worth it at this point. If your response to someone actually commenting on an RFC (Request For Comments) is some kind of low level trolling plus affirming that's okay via a laughing face from the original RFC author why even bother?

[cpuguy83]

I made a lighthearted joke at no ones expense and gave some real feedback. This isn't my proposal.

[crosbymichael]

@flx42 Ya, I like the suggested approach and namespaced keys for the opts. I'm guessing it would all people to get a quick default, docker run --gpus 0 but then expand on the options if needed.

[crosbymichael]

Maybe a compose section looks something like:

- gpus:
    - type: "nvidia"
    - devices: 
         - 0
         - "guid"
    - opts:
         - "..."

[cpuguy83]

Yeah, but not sure how we are reconciling this with what's supported on Swarm.

[cpuguy83]

The alternative would be to shove this into the "mounts" API, though we can still keep the somewhat specialized CLI for ease of use.

[flx42]

I thought about responding with additional feedback but not even worth it at this point. If your response to someone actually commenting on an RFC (Request For Comments) is some kind of low level trolling plus affirming that's okay via a laughing face from the original RFC author why even bother?

I laughed because it's true, that's something we might need to do, and we will call that backwards-compatibility :). I apologize if it appeared dismissive of your reply.

I think, as mentioned above, we need a sensible default, therefore we need to make sure --gpus nvidia=0,1 is sufficient for most (all?) use-cases. If you are doing advanced use cases such as resource limitation or security tuning, you will indeed need additional CLI arguments, but in this case is it that different from specifying all the cgroups, mounts and security options on the CLI today?

[cpuguy83]

@flx42 Would it make sense to take that a step further and support something like --gpus=nvidia and just include everything?

[flx42]

@cpuguy83 I was also wondering if we should include a special value to select all the GPUs:

--gpus nvidia # your proposal
--gpus nvidia=* # might be problematic with shell globbing, but --device-cgroup-rule is similar
--gpus nvidia=all # ambiguous with GPU ID

I'm fine with your option.

[cpuguy83]

I think --gpus nvidia should be good, we may want to make sure that --gpus nvidia= (no value after the =) is invalid, in terms of flag parsing.

@cwgem does this work in terms of simplifying the experience?

---

Separate topic: Where should this fit into the API?