Why SEV-SNP breaks cross-device P2P?

[aakarshcool15-crypto]

Host Configuration:

dmesg | grep -i sev

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.16.0-snp-host-68799c0277b2+ root=/dev/mapper/vg00-rootvol ro printk.time=1 nomodeset gfxpayload=text fb=false mem_encrypt=on kvm_amd.sev=1 kvm_amd.sev_es=1 kvm_amd.sev_snp=1 amd_iommu=on
[ 0.000000] SEV-SNP: RMP table physical range [0x0000018279300000 - 0x00000183fd3fffff]
[ 0.027006] SEV-SNP: Reserving start/end of RMP table on a 2MB boundary [0x0000018279200000]
[ 1.147647] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.16.0-snp-host-68799c0277b2+ root=/dev/mapper/vg00-rootvol ro printk.time=1 nomodeset gfxpayload=text fb=false mem_encrypt=on kvm_amd.sev=1 kvm_amd.sev_es=1 kvm_amd.sev_snp=1 amd_iommu=on
[ 38.256904] ccp 0000:03:00.5: sev enabled
[ 61.424502] ccp 0000:03:00.5: SEV-SNP API:1.55 build:42
[ 61.497361] ccp 0000:03:00.5: SEV API:1.55 build:42
[ 61.499835] kvm_amd: SEV enabled (ASIDs 2 - 1006)
[ 61.499844] kvm_amd: SEV-ES enabled (ASIDs 1 - 1)
[ 61.499848] kvm_amd: SEV-SNP enabled (ASIDs 1 - 1)

I have two PCIe devices located under different PCIe switches, and I need to perform peer-to-peer (P2P) data transfers between them. ACS (Access Control Services) is disabled, so P2P traffic should route through the root complex and will involve IOMMU translations. So now every device’s DMA must go through the IOMMU with translation enabled.
When I run this setup inside a confidential VM (SEV-SNP enabled), P2P operations fail with the error:

Failed to access P2P device

However, when I run the same configuration inside a normal VM (without SEV-SNP), P2P works correctly across both devices.(host kernel also wihout SEV features)
The PCIe topology and configuration are identical in both cases. I am trying to understand why P2P succeeds in a normal VM but fails in an SEV-SNP confidential VM.

So is this expected or I am doing something wrong or there is some limitation?

Thanks in advance!!

[tlendacky]

This could be related to the encryption bit being set on the I/O page table entries? Try booting the host without mem_encrypt=on and see if that resolves the issue. If you want host encrypted/protected memory, I recommend using TSME which can be configured in your BIOS.

See https://lore.kernel.org/lkml/SEZPR01MB4399D06A044791DEA8BE43CBDCAAA@SEZPR01MB4399.apcprd01.prod.exchangelabs.com/

[aakarshcool15-crypto]

Thanks after removing mem_encrypt=on from cmdline P2P works :)

I have some follow up questions.

With mem_encrypt=on and TSME enabled at BIOS, I see below logs

#dmesg | grep -i SME
[ 3.085879] Memory Encryption Features active: AMD SME
[ 38.341871] ccp 0000:03:00.5: psp: Both TSME and SME are active, SME is unnecessary when TSME is active.
[ 38.353836] ccp 0000:88:00.5: psp: Both TSME and SME are active, SME is unnecessary when TSME is active.

After removing mem_encrypt=on from command line and enabling TSME at BIOS, I see below logs now.

#dmesg | grep -i SME
[ 38.137129] ccp 0000:03:00.5: psp: TSME enabled
[ 38.156049] ccp 0000:88:00.5: psp: TSME enabled

how come disabling mem_encrypt=on does not break SEV-SNP?
With SME disabled and TSME enabled, guest memory and PTEs in IOMMU page tables are encrypted?

Also, I see the patch link you provided, these changes will be applied on latest 6.19 kernel only. But I am using
6.16 kernel. The patch files are missing in my kernel.

Since AMD SME marks individual memory pages as encrypted by setting the C-bit in all page table entries.
I think removing mem_encrypt=on will clear C-bit for all PTEs in IOMMU page tables instead clearing only C-bit for MMIO-backed mappings as mentioned in the patch.

On my current host kernel 6.16, I wanted to keep mem_encrypt=on and clear C-bit in PTEs for only MMIO backed mappings. Similar to what this patch does.

Do we have any provision to do this or TSME takes care of this?

Thanks a lot in advance :)

[tlendacky]

how come disabling mem_encrypt=on does not break SEV-SNP?

SME is not required for SEV. SEV uses the encryption bit in the guest page tables to encrypt guest memory.

With SME disabled and TSME enabled, guest memory and PTEs in IOMMU page tables are encrypted?

SEV guest memory is always encrypted if the guest page table has the encryption bit set. IOMMU page tables are host I/O page tables. The encryption bit in the I/O page table affects I/O to the host/hypervisor. All I/O would be to host memory through the IOMMU and then guest access to that memory is through the guest page tables (which are using SWIOTLB which is non-encrypted guest memory), so access would depend on the nested page table PTE entry. With SME, the I/O page table encryption bit matches the NPT encryption bit so access is consistent. With TSME, anything going through the memory controller (including DMA) is encrypted/decrypted using the key established by TSME or the SEV guest key.

Also, I see the patch link you provided, these changes will be applied on latest 6.19 kernel only. But I am using
6.16 kernel. The patch files are missing in my kernel.

It was just to show you the why you were experiencing the issue. If you use TSME you don't need the patch.

On my current host kernel 6.16, I wanted to keep mem_encrypt=on and clear C-bit in PTEs for only MMIO backed mappings. Similar to what this patch does.

With TSME you don't need to worry about anything and all data will be encrypted in memory - either with the TSME key or with the SEV guest key.

TSME is recommended over SME for host memory encryption.

[aakarshcool15-crypto]

Thanks a lot for detailed explanation.

SEV guest memory is always encrypted if the guest page table has the encryption bit set

Just curious to know, where to check this encryption bit enabled or not in guest page table?

[tlendacky]

Thanks a lot for detailed explanation.

SEV guest memory is always encrypted if the guest page table has the encryption bit set

Just curious to know, where to check this encryption bit enabled or not in guest page table?

You would need to be in the guest in kernel mode to examine the page tables being used by the guest. The guest kernel will, for the most part, mark all memory as private (encrypted - meaning the encryption bit is set in the page table entry for a virtual address) unless absolutely necessary to make shared (unencrypted - SWIOTLB is an example of shared memory).

[aakarshcool15-crypto]

Thanks for the clarification!!

I’m seeing these kernel warnings after the P2P test completes:

[ 7293.744434] Wrong/unhandled opcode bytes: 0x8b, exit_code: 0x404, rIP: 0x76c1b45a08c0
[ 7293.745730] SEV: Unsupported exit-code 0x404 in #VC exception (IP: 0x76c1b45a08c0)

The P2P test passes successfully, but I’d like to understand what might be causing these warnings.

[tlendacky]

This looks to be a userspace test. I'm not sure what the test is doing, but an 0x404 code is the error code that is issued when the memory being referenced has the encryption-bit set and the page hasn't been validated. Userspace page tables always use the encryption bit, there isn't a way to create a userspace mapping without it.