Load nvidia kernel modules using kata configuration file failed

[RyanAoh]

Description of problem

I used kata 1.9.3 with ‘kernel_modules=["nvidia", "nvidia-uvm", "nvidia-modeset"]’ configed，those module files(.ko) have been added to the rootfs image.
Then I run the following command to add a nvidia GPU vfio device into VM:

docker run -it --rm  --device /dev/vfio/45 -v /dev:/dev centos:7 bash

Expected result

The container runs normally, and modules("nvidia", "nvidia-uvm", "nvidia-modeset") should be loaded automaticly

Actual result

The container run failed. The error message is as follows:

root@zs-GPU-kata-1:~# docker run -it --rm  --device /dev/vfio/45 -v /dev:/dev centos:7 bash
docker: Error response from daemon: OCI runtime create failed: rpc error: code = Unknown desc = could not load module: exit status 1: modprobe: ERROR: could not insert 'nvidia': No such device
insmod /lib/modules/4.19.75/kernel/drivers/video/nvidia.ko nvidia-uvm nvidia-modeset: unknown.

---

However, without ‘kernel_modules=["nvidia", "nvidia-uvm", "nvidia-modeset"]’ configed, I can run the container successfully, and modprobe -a nvidia nvidia_uvm nvidia_modeset can be exected to load modules successfully by connecting to the virtual machine using the debug console.

bash-4.2# lsmod
Module                  Size  Used by
bash-4.2# modprobe -a nvidia nvidia_uvm nvidia_modeset
bash-4.2# lsmod
Module                  Size  Used by
nvidia_modeset       1085440  0
nvidia_uvm            823296  0
nvidia              18141184  2 nvidia_uvm,nvidia_modeset
bash-4.2#

[RyanAoh]

I have read the code in both kata-runtime and kata-agent. I found loadKernelModule is called in function CreateSandbox in grpc.go file, however, a gpu vfio device invoked with '--device' will be added by addDevices in function CreateContainer. CreateContainer is requested after CreateSandbox. So the error 'No such device' makes sense, because the gpu vfio device had not been added to the vm when loadKernelModule was called. Am I right？
Some code in grpc.go:

...
func (a *agentGRPC) CreateSandbox(ctx context.Context, req *pb.CreateSandboxRequest) (*gpb.Empty, error) {
	if a.sandbox.running {
		return emptyResp, grpcStatus.Error(codes.AlreadyExists, "Sandbox already started, impossible to start again")
	}

	a.sandbox.hostname = req.Hostname
	a.sandbox.containers = make(map[string]*container)
	a.sandbox.network.ifaces = make(map[string]*types.Interface)
	a.sandbox.network.dns = req.Dns
	a.sandbox.running = true
	a.sandbox.sandboxPidNs = req.SandboxPidns
	a.sandbox.storages = make(map[string]*sandboxStorage)
	a.sandbox.guestHooks = &specs.Hooks{}
	a.sandbox.guestHooksPresent = false

	for _, m := range req.KernelModules {
		if err := loadKernelModule(m); err != nil {
			return emptyResp, err
		}
	}


...


func (a *agentGRPC) CreateContainer(ctx context.Context, req *pb.CreateContainerRequest) (resp *gpb.Empty, err error) {
	if err := a.createContainerChecks(req); err != nil {
		return emptyResp, err
	}

	// re-scan PCI bus
	// looking for hidden devices
	if err = rescanPciBus(); err != nil {
		agentLog.WithError(err).Warn("Could not rescan PCI bus")
	}

	// Some devices need some extra processing (the ones invoked with
	// --device for instance), and that's what this call is doing. It
	// updates the devices listed in the OCI spec, so that they actually
	// match real devices inside the VM. This step is necessary since we
	// cannot predict everything from the caller.
	if err = addDevices(ctx, req.Devices, req.OCI, a.sandbox); err != nil {
		return emptyResp, err
	}

...	

[RyanAoh]

@devimc @bergwolf

[devimc]

@RyanAoh yeah, seems like I didn't consider the case where a driver can't be loaded when there is no device to drive (not all drivers behave this way) 😓

[RyanAoh]

@devimc So, do you have any good idea to solve this problem？ Maybe we can use annotations to state which modules need to be loaded for a specific container when using kata with kubernetes, but how about docker? Or, maybe we could ignore the 'No such device' error, and then load modules every time creating container or sandbox, uh...sounds not very good.

[devimc]

@RyanAoh

do you have any good idea to solve this problem？

not yet

Maybe we can use annotations to state which modules need to be loaded for a specific container when using kata with kubernetes

annotations per container? is that possible?

Or, maybe we could ignore the 'No such device' error, and then load modules every time creating container or sandbox

A better approach could be to save the list of modules that were not loaded at CreateSandbox and try to load them after CreateContainer, what do you think?

[amshinde]

@devimc I think its worth looking at if we can add support for annotations per container to handle this case. I remember reading about containerd supporting annotations per container. I'll try to look it up.

[RyanAoh]

@devimc
Yeah, saving the list of modules that were not loaded at CreateSandbox looks like a better way, but, another question, how to check whether all modules are loaded after creating all containers finished? Modules should be loaded after addDevice but before the container being started.

However, adding support for annotations per container may be a long term way.
For the short term, I have an another idea, we could add an container_kernel_modules=[] option in the configuration file for the case where a kata vm just contains only one container, like docker. And for multiple container in a kata vm, like kubernetes, we could use an annotation like this:

annotations:
  io.katacontainers.config.agent.container_kernel_modules: "{container-1:[nvidia nvidia-uvm nvidia-modeset],container-2:[...],...}"

This configuration or annotation could be added into the spc.annotations for every container before creating a grpc request. And then, if anyone adds support for annotations per container, we can also reuse these code.

[devimc]

@RyanAoh

io.katacontainers.config.agent.container_kernel_modules: "{container-1:[nvidia nvidia-uvm nvidia-modeset],container-2:[...],...}"

but here we are relying in how the container runtime (containerd/cri-o) creates the sandbox, I think this is prone to errors, for example we have 2 containers: cA and cB, where the first container is cA, if the runtime is unable to setup cA (i.e couldn't pull image, etc) then it'll be skipped and kata won't know that cA was the first container, for kata cB is the first container.

[RyanAoh]

@devimc
Sorry I may not have made it clear. I mean "container-n" is the name of container. According to the name we can decode the annotation and format it to every container's own spec.annotation. when CreateContainer being called, container's spec.annotation could be checked to decide whether modules need to be loaded. There is no need to care about the order of containers.

[devimc]

@RyanAoh afaik container name is container engine (docker, kubernetes, etc) specific, runtimes like kata don't receive such information, the only way that we have to identify a container is using IDs but they are generated randomly, users don't know such information

[ericwadams]

I don't know if this is related but with Kata 1.9.3 I get an error on both a NUC and WolfPass server when I try to pass -v /dev:/dev to a kata container. This used to previously work and started failing at version 1.9.0-rc0 I noted this in Issue 891 a few days ago.

Kata 1.9.3

$ docker run -it --runtime=kata-qemu -v /dev:/dev  clearlinux bash
docker: Error response from daemon: OCI runtime create failed: rpc error: code = Unknown desc = open /run/kata-containers/sandbox/resolv.conf: no such file or directory: unknown.

Kata 1.8.2

$ docker run -it --runtime=kata-qemu -v /dev:/dev  clearlinux bash
root@a4b4b2ce1737/ #

[devimc]

@eadamsintel docker run -it --runtime=kata-qemu -v /dev:/dev clearlinux bash works for me
have you tried with the latest 1.10 ?

[ericwadams]

@devimc I just used kata-deploy to try with 1.10.0-rc0 and it worked. I guess just the 1.9.x versions it fails on but is now fixed for 1.10? That is strange. I saw the OP was using 1.9.3 and using that -v /dev:/dev option which has given me issues on 3 separate systems (2 Clear Linux and 1 Fedora 30). However, my error was complaining about a missing resolv.conf file.

[devimc]

@eadamsintel yeah, may be we forgot to backport a patch, fortunately now it's working

[grahamwhaley]

@devimc @eadamsintel - can we add a note to kata-containers/packaging#891 saying which backport fixed that? thx.