Intel gvt-g with nvidia prime render offload causes several issues

[grazzolini]

This comment was edited

I have started using intel gvt-g a few months ago and, at the time, I was also using nvidia prime render offload method for offloading to the nvidia card. I had issues getting gvt-g to work on any recent kernel (> 5.5), but I had success using the 5.4 lts branch on Arch Linux.

Very recently, however, with the advent of the version 5.4.51 of the lts kernel, as well the mesa 20.1.3 update, my vm with gvt-g would load normally and when X would get started, it would hang, causing 100% cpu usage on one of the hosts CPU's with a blank screen.

Reverting things to kernel 5.4.50 and previous mesa, didn't seem to work. This is where a recent nvidia update, 450.57 comes in. It was the responsible for this new behavior of making my VM to hang.

During my investigations, I've found out that, if using nvidia prime render offload, the VM using the virtualized intel card would work, up until the previously mentioned versions of the software, but, it would not display the mouse pointer properly and also, I would only be able to see the boot from early userspace onward, no POST, no bootloader, nothing.

So, I've done a few experiments with using bbswitch to keep the nvidia card not only powered off, but also to prevent nvidia modules to be loaded, making sure prime render offload is not used. That gave me garbled graphics as in #152. So, I have added the <qemu:env name="MESA_LOADER_DRIVER_OVERRIDE" value="i965"/> workaround. Not only the screen is not garbled anymore, but I can also see the POST messages.

Also, I'm able to run with gvt-g on the latest mainline kernel, 5.7.8, without any issues. As long as I'm not using the prime render offload method, everything works. I have no idea why this interferes with the virtualized card, but my bet is on some weird mesa/xorg issue.

[grazzolini]

I have done a new round of testing in light of recent mesa updates and also new 5.8 kernel. Here are the results:

When using nvidia prime render offload:

linux 5.8.0:
The VM boots, no BIOS is displayed (with or without the MESA_LOADER_DRIVER_OVERRIDE workaround), only the output from the early userspace onward is shown. When trying to starting X though, qemu will segfault and the machine will be stopped.

I've tested both kernel 5.8 and 5.4 on the guest and it seems the kernel that is running on the guest doesn't matter for reproducibility of this issue.

linux 5.4.57:
The VM boots, no BIOS can be seen (with or without the MESA_LOADER_DRIVER_OVERRIDE workaround) , and X starts, but then the machine hangs on 100% CPU and needs to be destroyed. Once again, the kernel running on the guest seems to play no part in what happens.

When using bbswitch + bumblebee:

linux 5.8.0:
The VM boots and X can be started normally. Even the BIOS can be seen (with MESA_LOADER_DRIVER_OVERRIDE workaround). But, if the VM is stopped and the host is suspended, the VM cannot be started again, it stays stuck right after loading the initramfs (when it loads the i915 module). Again, the kernel being ran on the guest plays no part on this. Tried both 5.8.0 and 5.4.57.

linux 5.4.57:
It's the only combination that works flawlessly. For not getting garbled graphics, the MESA_LOADER_DRIVER_OVERRIDE workaround is needed. But, the host can be suspended, after the VM is stopped, and when resuming the host, the VM can be started again.

Suspending the host while the VM is powered on never worked properly for me regardless of kernel version, or if using prime render offload or not. The VM would either crash when the host would be suspended or X would crash on the guest VM.

[rnd-ash]

I can confirm issues with my i7 8750hq and GTX 1060 Maxq setup on my Razer blade 2018 (Kernel 5.8.14-arch1-1):

With NVIDIA GPU enable:
Starting up the VM my laptop screen goes black, and all the external monitors change their positions. Also, all UI elements on both the VM and host become messed up when the mouse hovers over them.

[grazzolini]

I can confirm issues with my i7 8750hq and GTX 1060 Maxq setup on my Razer blade 2018 (Kernel 5.8.14-arch1-1):

With NVIDIA GPU enable:
Starting up the VM my laptop screen goes black, and all the external monitors change their positions. Also, all UI elements on both the VM and host become messed up when the mouse hovers over them.

I never got this kind of issue, the VM wasn't able to affect the host. Then again, I didn't try again with newer kernels. It's harder to troubleshoot this, since the nvidia driver is a blob.

[rnd-ash]

Here is a link to it actually messing up with the VM turning on:
https://streamable.com/75f91k

No segfaults or anything being reported in DMESG

Its even stranger....once the display crashes, the mouse can move my entire Xorg desktop to the left or right...?????

[TinaZhangZW]

The dma-buf display provided by gvt-g aims to provide a way for igfx or cpu to access the guest framebuffer w/o any copy. It seems that this issue is cased by letting other vendor's gpu consume gvt-g vGPU's framebuffer, which isn't supported by gvt-g.

[grazzolini]

@TinaZhangZW
So, this is happening because that's precisely what the nvidia driver does in order to be able to display things using the iGPU's (intel) framebuffer and, for some reason, it's also trying to use the vGPU one, causing the corruption? In that case, I think the fix should be done on the nvidia's side. But, also, there isn't any way to prevent this from happening from the vGPU side?

[TinaZhangZW]

Yes, the information of the gvt-g vGPU's framebuffer can only be understood correctly by i915, as there're some vendor specific formats which are only supported by i915. Since guest takes vGPU as a full functional iGFX, there's no way to stop guest from using those i915 specific formats.

I think the solution might be in host user space. If the vGPU's framebuffer consumer in host user space (e.g. QEMU UI) can first copy the vGPU's framebuffer into another framebuffer which has general format (e.g. something like RGBA8888) and can be recognized by any GPU, then the problem could be solved. However, this will introduce more buffer copy.

[rugubara]

I've posted a note on the nvidia dev forum about this issue:
https://forums.developer.nvidia.com/t/issues-with-igvt-g-and-virtual-machines/157923. Everybody's welcome to reply to let nvidia know I'm not a lone freak suffering from this.

[rugubara]

Recently I stumbled upon this issue with the nouveau drivers managing my optimus Quadro 2000 card. As long as nouveau module is loaded, the behaviour is exactly the same. No boot animation, guest crashes as soon as the mouse pointer enters the window if the guest has initialized the intel igfx driver.

[grazzolini]

I haven't tested with nouveau. But, if this is also happening with it, then the issue isn't on the nvidia proprietary driver. There might be some common element to both that interact badly with the intel gvt-g framebuffer. My guess, from my investigations so far, is that, if X is using two cards (I wouldn't be surprised if this manifested as well with an AMD card as dGPU), and you create vGPU and start a guest, the VM framebuffer is accessing/writing to a region of memory it shouldn't. We could try to get a full stacktrace from qemu, because on the more recent kernel, the guest still segfaults immediately.

[evelikov]

@grazzolini sounds like there may be multiple bugs here:

Userspace:
Visual changes when using MESA_LOADER_DRIVER_OVERRIDE - indicates a regression in the "iris" driver relative to the "i965" one. Capture an apitrace of the issue and report it to mesa

Kernel:
5.8.0 vs 5.4.57 regression (under BB) - consider bisecting the kernel to the commits which caused the breakage

nvidia and/or X stack:
Please try nouveau, as you mentioned. Additionally having a) crash dump and b) gdb output for the 5.8.0 and 5.4.57 cases (respectively) will give devs some ideas what is happening.

Aside:
bbswitch + bumblebee - for the sake of simplicity, ignore bbswitch and consider using only the rendering part of bumblebee (virtualgl or primus) while experimenting with both rendering options