Running nvidia-container-runtime with podman is blowing up.

[rhatdan]

1. Issue or feature description
   rootless and rootful podman does not work with the nvidia plugin

2. Steps to reproduce the issue
   Install the nvidia plugin, configure it to run with podman
   execute the podman command and check if the devices is configured correctly.

3. Information to attach (optional if deemed irrelevant)

   Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info
   Kernel version from uname -a
   Fedora 30 and later
   Any relevant kernel output lines from dmesg
   Driver information from nvidia-smi -a
   Docker version from docker version
   NVIDIA packages version from dpkg -l 'nvidia' or rpm -qa 'nvidia'
   NVIDIA container library version from nvidia-container-cli -V
   NVIDIA container library logs (see troubleshooting)
   Docker command, image and tag used

I am reporting this based on other users complaining. This is what they said.

We discovered that the ubuntu 18.04 machine needed a configuration change to get rootless working with nvidia:
"no-cgroups = true" was set in /etc/nvidia-container-runtime/config.toml
Unfortunately this config change did not work on Centos 7, but it did change the rootless error to:
nvidia-container-cli: initialization error: cuda error: unknown error\\n\"""

This config change breaks podman running from root, with the error:
Failed to initialize NVML: Unknown Error

Interestingly, root on ubuntu gets the same error even though rootless works.

[rhatdan]

The Podman team would like to work with you guys to get this to work well in both root full and rootless containers if possible. But we need someone to work with.

[rhatdan]

@mheon @baude FYI

[zvonkok]

@sjug FYI

[RenaudWasTaken]

Hello!

@rhatdan do you mind filling the following issue template: https://github.com/NVIDIA/nvidia-docker/blob/master/.github/ISSUE_TEMPLATE.md

Thanks!

[nvjmayo]

I can work with the podman team.

[rhatdan]

@hholst80 FYI

[rhatdan]

containers/podman#3659

[eaepstein]

@nvjmayo Thanks for the suggestions. Some good news and less good.

This works rootless:
podman run --rm --hooks-dir /usr/share/containers/oci/hooks.d nvcr.io/nvidia/cuda nvidia-smi
The same command continues to fail with the image: docker.io/nvidia/cuda

In fact rootless works with or without /usr/share/containers/oci/hooks.d/01-nvhook.json installed using the image: nvcr.io/nvidia/cuda

Running as root continues to fail when no-cgroups = true for either container, returning:
Failed to initialize NVML: Unknown Error

[rhatdan]

Strange I would not expect podman to run a hook that did not have a json file describing the hook.

[nvjmayo]

@eaepstein I'm still struggling to reproduce the issue you see. Using docker.io/nvidia/cuda also works for me with the hooks dir.

$ podman run --rm --hooks-dir /usr/share/containers/oci/hooks.d/ docker.io/nvidia/cuda nvidia-smi
Tue Oct 22 21:35:44 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 710      Off  | 00000000:65:00.0 N/A |                  N/A |
| 50%   38C    P0    N/A /  N/A |      0MiB /  2001MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0                    Not Supported                                       |
+-----------------------------------------------------------------------------+

without the hook I would expect to see a failure roughly like:

Error: time="2019-10-22T14:35:14-07:00" level=error msg="container_linux.go:346: starting container process caused \"exec: \\\"nvidia-smi\\\": executable file not found in $PATH\""
container_linux.go:346: starting container process caused "exec: \"nvidia-smi\": executable file not found in $PATH": OCI runtime command not found error

This is because the libraries and tools get installed by the hook in order to match the host drivers. (an unfortunate limitation of tightly coupled driver+library releases)

I think there is a configuration issue and not an issue of the container image (docker.io/nvidia/cuda vs nvcr.io/nvidia/cuda).

Reviewing my earlier posts, I recommend changing my 01-nvhook.json and remove the NVIDIA_REQUIRE_CUDA=cuda>=10.1 from it. My assumption is everyone has the latest CUDA install, which was kind of a silly assumption on my part. The CUDA version doesn't have to be specified, and you can leave this environment variable out of your set up. It was an artifact of my earlier experiments.

[eaepstein]

@nvjmayo we started from scratch with a new machine (CentOS Linux release 7.7.1908) and both docker.io and nvcr.io images are working for us now too. And --hooks-dir must now be specified for both to work. Thanks for the help!

[eaepstein]

@rhatdan @nvjmayo Turns out that getting rootless podman working with nvidia on centos 7 is a bit more complicated, at least for us.

Here is our scenario on brand new centos 7.7 machine

1. run nvidia-smi with rootless podman
   1.result: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused "process_linux.go:413: running prestart hook 0 caused \"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: cuda error: unknown error\\n\""

2. run podman with user=root
   2.result: nvidia-smi works

3. run podman rootless
   3.result: nvidia-smi works!

4. reboot machine, run podman rootless
   4.result: fails again with same error as Add README image nvidia-docker#1

Conclusion: running nvidia container with podman as root changes the environment for rootless to work. Environment cleared on reboot.

One other comment: podman as root and rootless podman cannot run with the same /etc/nvidia-container-runtime/config.toml - no-cgroups must =false for root and =true for rootless

[rhatdan]

If the nvidia hook is doing any privileged operations like modifying /dev and adding devicenodes, then this will not work with rootless. (In rootless all processes are running with the Users UID. Probably when you run rootfull, it is doing the privileged operations, so the next time you run rootless, those activities do not need to be done.

I would suggest for rootless systems, that the /dev and nvidia ops be done as a systemd unit file, so the system is preconfigured and then the rootless jobs will work fine.

[eaepstein]

After running nvidia/cuda with rootfull podman, the following exist:
crw-rw-rw-. 1 root root 195, 254 Oct 25 09:11 nvidia-modeset
crw-rw-rw-. 1 root root 195, 255 Oct 25 09:11 nvidiactl
crw-rw-rw-. 1 root root 195, 0 Oct 25 09:11 nvidia0
crw-rw-rw-. 1 root root 241, 1 Oct 25 09:11 nvidia-uvm-tools
crw-rw-rw-. 1 root root 241, 0 Oct 25 09:11 nvidia-uvm

None of these devices exist after boot. Running nvidia-smi rootless (no podman) creates:
crw-rw-rw-. 1 root root 195, 0 Oct 25 13:40 nvidia0
crw-rw-rw-. 1 root root 195, 255 Oct 25 13:40 nvidiactl

I created the other three entries using "sudo mknod -m 666 etc..." but that is not enough to run rootless. Something else is needed in the environment.

Running nvidia/cuda with rootfull podman at boot would work, but not pretty.

Thanks for the suggestion