Error causing all kvm gvt guests to freeze

[reedog117]

This error happens sporadically but seems to happen more often with more VGPU load or multiple VMs running gvt.

This seems like an error with ppgtt_populate_spt

Jun  2 11:42:04 virt-slc-90 kernel: [  943.161714] WARNING: CPU: 0 PID: 27
42 at drivers/gpu/drm/i915/gvt/gtt.c:683 ppgtt_populate_spt+0x1f1/0x3a0 [i
915]

Proxmox 6.2-4 - kernel 5.4.41-1-pve
Coffee Lake ER - Xeon E-2276G
Windows 10 Guest - Newest available DCH drivers but have tried the last 3 versions

From the kernel log:

Jun  2 11:42:04 virt-slc-90 kernel: [  943.157228] gvt: guest page write error, gpa 13e8cacf8
Jun  2 11:42:04 virt-slc-90 kernel: [  943.158203] gvt: vgpu 2: fail: shadow page 000000008eed785e guest entry 0x7d3408cc24763883 type 9.
Jun  2 11:42:04 virt-slc-90 kernel: [  943.158204] gvt: guest page write error, gpa 1409c9bb0
Jun  2 11:42:04 virt-slc-90 kernel: [  943.158656] gvt: vgpu 1: fail: shadow page 000000001ee6d741 guest entry 0xffffffffffffffff type 9
Jun  2 11:42:04 virt-slc-90 kernel: [  943.159188] ------------[ cut here ]------------
Jun  2 11:42:04 virt-slc-90 kernel: [  943.159702] gvt: vgpu 1: fail: spt 
00000000dd87fe57 guest entry 0xffffffffffffffff type 9
Jun  2 11:42:04 virt-slc-90 kernel: [  943.159703] gvt: vgpu 1: fail: shadow page 00000000dd87fe57 guest entry 0xffffffffffffffff type 9.
Jun  2 11:42:04 virt-slc-90 kernel: [  943.159704] gvt: guest page write error, gpa 13e8cad00
Jun  2 11:42:04 virt-slc-90 kernel: [  943.160713] invalid entry type
Jun  2 11:42:04 virt-slc-90 kernel: [  943.161207] gvt: vgpu 1: fail: shadow page 000000001ee6d741 guest entry 0xffffffffffffffff type 9
Jun  2 11:42:04 virt-slc-90 kernel: [  943.161714] WARNING: CPU: 0 PID: 2742 at drivers/gpu/drm/i915/gvt/gtt.c:683 ppgtt_populate_spt+0x1f1/0x3a0 [i915]
Jun  2 11:42:04 virt-slc-90 kernel: [  943.162152] gvt: vgpu 1: fail: spt 00000000dd87fe57 guest entry 0xffffffffffffffff type 9
Jun  2 11:42:04 virt-slc-90 kernel: [  943.162153] gvt: vgpu 1: fail: shadow page 00000000dd87fe57 guest entry 0xffffffffffffffff type 9.
Jun  2 11:42:04 virt-slc-90 kernel: [  943.162598] Modules linked in: xt_multiport(E) ipt_REJECT(E) nf_reject_ipv4(E) ebtable_filter(E) ebtables(E) 
ip_set(E) ip6table_raw(E) iptable_raw(E) ip6table_filter(E) ip6_tables(E) 
sctp(E) iptable_filter(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) iptable_nat(E) xt_MASQUERADE(E) bpfilter(E) bonding(E) openvswitch(E) nsh(E) nf_c
onncount(E) nf_nat(E) softdog(E) nfnetlink_log(E) nfnetlink(E) zfs(POE) zunicode(POE) zlua(POE) zavl(POE) icp(POE) ipmi_ssif(E) zcommon(POE) znvpair(POE) spl(OE) vhost_net(E) vhost(E) tap(E) nf_conntrack_pptp(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) ib_iser(E) rdma_cm(E) iw_cm(E) ib_cm(E) ib_core(E) iscsi_tcp(E) libiscsi_tcp(E) libiscsi(E) scsi_transport_iscsi(E) vfio_pci(E) vfio_virqfd(E) kvmgt(E) intel_rapl_msr(E) intel_rapl_common(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) aesni_int
el(E) crypto_simd(E) cdc_ether(E) cryptd(E) glue_helper(E) ast(E) joydev(E)
Jun  2 11:42:04 virt-slc-90 kernel: [  943.162613]  usbnet(E)
Jun  2 11:42:04 virt-slc-90 kernel: [  943.163561] gvt: guest page write error, gpa 13e8cad08
Jun  2 11:42:04 virt-slc-90 kernel: [  943.164533]  intel_cstate(E) input_leds(E) drm_vram_helper(E) mii(E) intel_rapl_perf(E) wmi_bmof(E) pcspkr(E)
 ttm(E) 8250_dw(E) mei_me(E) mei(E) intel_pch_thermal(E) ie31200_edac(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) mac_hid(E) acpi_power_meter(E) acpi_pad(E) acpi_tad(E) i915(E) vfio_mdev(E) mdev(E) vfio_iommu_type1(E) vfio(E) drm_kms_helper(E) drm(E) fb_sys_fops(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) kvm(E) irqbypass(E) sunrpc(E) ip_tables(E) x_tables(E) autofs4(E) hid_generic(E) usbkbd(E) usbmouse(E) usbhid(E) hid(E) btrfs(E) xor(E) zstd_compress(E) uas(E) usb_storage(E) raid6_pq(E) libcrc32c(E) igb(E) i2c_i801(E) ahci(E) intel_lpss_pci(E) xhci_pci(E) i2c_algo_bit(E) intel
_lpss(E) dca(E) libahci(E) idma64(E) xhci_hcd(E) virt_dma(E) wmi(E) video(E) pinctrl_cannonlake(E) pinctrl_intel(E)
Jun  2 11:42:04 virt-slc-90 kernel: [  943.164605] CPU: 0 PID: 2742 Comm: 
kvm Tainted: P           OE     5.4.41-1-pve #1
Jun  2 11:42:04 virt-slc-90 kernel: [  943.164606] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./E3C246D4U, BIOS P2.30 12/25/2019
Jun  2 11:42:04 virt-slc-90 kernel: [  943.165680] gvt: vgpu 1: fail: shadow page 000000001ee6d741 guest entry 0xffffffffffffffff type 9
Jun  2 11:42:04 virt-slc-90 kernel: [  943.166623] RIP: 0010:ppgtt_populate_spt+0x1f1/0x3a0 [i915]

[reedog117]

Seems to occur in other distros too:

https://bugzilla.redhat.com/show_bug.cgi?id=1790109
https://bugs.centos.org/view.php?id=16611

[reedog117]

Is there any way to disable ppgtt to avoid this error until it's fixed? the enable_ppgtt option used to exist in previous kernels but not these current ones.

[reedog117]

@zhenyw @coxuintel What's the best avenue to have Intel people take a look at this bug and propose a fix? This seems to be common among Coffee Lake chips across multiple distributions - (see linked issues as well as similar RedHat and CentOS bug reports)

[reedog117]

@TerrenceXu am i taking the right path in reporting this? Wasn't sure who else to tag since this seems to be a pretty big issue and is being reported via multiple avenues. Just looking for acknowledgement and an estimated version where this is fixed.

[Josua-SR]

++

I encounter a similar problem on my laptop with UHD620 (i7-8550U)
openSUSE Tumbleweed - Linux 5.7.1 :(

[reedog117]

I wish they would convey an official place to report issues that is continuously monitored. I have an entire server farm worth of logs that I can make available to them.

[werman]

Have the same issue with Arch Linux + UHD630 + Linux 5.8.0-rc2 + Windows 10 Guest

[tombohler]

I had similar issues and disabling above 4g decoding in the bios helped a lot. This limits aperture memory to 1gb, but prevents complete hangs of guest and host.
I still have regular crashes when using an ipad with rdp to connect to a vm using gvt-g. It happens when switching between rdp client on ipad and different apps. Ipad freezes rdp process when switching to a different app. windows guest hangs in this setup.

[reedog117]

Tried a much newer kernel and the problem is still there - Linux 5.7.2-1 #1 SMP 5.7.2-1

[reedog117]

issues still occur with kernel 5.7.7-1

[reedog117]

And still occurring with 5.7.8-1

[reedog117]

And even weirder - all of my crash issues went away switching from Win10 guests to Windows Server 2019 guests. Using the exact same kernels and iGPU driver versions too.