Consider backporting PMU support for Alder Lake

[trympet]

Is your feature request related to a problem? Please describe.
There is no PMU driver for the Intel Alder Lake platform. Hardware event sampling does not work with VTune or perf.

vtune -collect hotspots -knob sampling-mode=hw -knob sampling-interval=0.5 /home/trym/source/stud/tdt4186/practical2/build/release/webserver /tmp 8889 12 24
vtune: Error: Unable to perform driverless collection on this platform.
vtune: Error: Cannot enable event-based sampling collection: Architectural Performance Monitoring version is 0. Make sure the vPMU feature is enabled in your hypervisor.

root@DESKTOP-CMKEO60:~# dmesg | grep -i pmu
[    0.177428] Performance Events: unsupported p6 CPU model 151 no PMU driver, software events only.

Describe the solution you'd like
Backport the driver or provide an alternate solution.

Additional context
AFAIK, the only workarounds are to use Hyper-V or dual boot.

https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.13-Perf-Alder-Lake
https://lore.kernel.org/lkml/20210311114814.GA5829@zn.tnic/T/

cpuid.txt

[Trass3r]

Reportedly a newer kernel alone is not enough: #4678 (comment)
Your cpuid report does not list the features either:

   Architecture Performance Monitoring Features (0xa):
      version ID                               = 0x0 (0)
      number of counters per logical processor = 0x0 (0)
      bit width of counter                     = 0x0 (0)
      length of EBX bit vector                 = 0x0 (0)
      core cycle event not available           = false
      instruction retired event not available  = false
      reference cycles event not available     = false
      last-level cache ref event not available = false
      last-level cache miss event not avail    = false
      branch inst retired event not available  = false
      branch mispred retired event not avail   = false
      fixed counter  0 supported               = false
...
      fixed counter 31 supported               = false
      number of fixed counters                 = 0x0 (0)
      bit width of fixed counters              = 0x0 (0)
      anythread deprecation                    = false

[clemenswasser]

@benhillis
are you aware of this issue and is there maybe even a fix coming?
It's very frustrating not to have PMU support for Alderlake…
(Especially since Hyper-V supports it, as @trympet pointed out)

[clemenswasser]

I've hit this today again, since I wanted to profile something under Linux.
Why is this not being fixed?!? "Plain" Hyper-V already supports PMU's with Alderlake!
I hope you can understand that this is really frustrating as a user 😞...

[topolarity]

I'm interested in this feature, as well.

WSL2 has been a great experience for me, but it's a real shame not to be able to get HW perf counters on newer CPUs.

[tyler274]

This is also an issue for me as well.

[samlihaha]

This is also an issue for me as well +1.

[clemenswasser]

I've also asked on Twitter: https://twitter.com/clemenswasser/status/1669265762991714304
Seems like we're just being ghosted 💀, which is extremely disappointing since many require performance counters support and they already work when using Hyper-V...
@benhillis @craigloewen-msft ping, are you working on this?

[clemenswasser]

I've once again looked into this and this still hasn't been fixed. In the old issue, I noticed this comment: #4678 (comment)
Which seems to have documented the root of the issue pretty well. The problem is that the WSL VM hasn't activated Perfmon (the arch_perfmon feature is missing), which seems to be a hard requirement for newer Intel CPUs for performance counters to work on Linux. Instructions for enabling Perfmon are in the Hyper-V documentation. I could validate this by running the following command:

$ cpuid | grep 'performance monitor'
      performance monitor support available    = false
      performance monitor support available    = false
[...]

Sadly, we can't just call Set-VMProcessor MyVMName -Perfmon @("ipt", "pmu", "lbr", "pebs") on the WSL VM, as it seems to be hidden. I only managed to list the vm by running hcsdiag list, but it seems to absent for all hyper-v commands.

@benhillis @craigloewen-msft
Since we now know what is missing, could you please activate all Perfmon features for the WSL VMs so that perf and other software which use performance counters now finally work on newer CPUs?

[espkk]

Any update?

[clemenswasser]

Wow, three years already, impressive! ;)
Now that WSL is open source, we get to do the job of Microsoft employees who are paid to work full-time on WSL and debug this ourselves 🥳 (fun, isn’t it?).
The relevant code can be found here:

WSL/src/windows/service/exe/WslCoreVm.cpp

Lines 1640 to 1651 in 4b5cb64

	#ifdef _AMD64_
	// Enable hardware performance counters if they are supported.
	if (m_vmConfig.EnableHardwarePerformanceCounters)
	{
	HV_X64_HYPERVISOR_HARDWARE_FEATURES hardwareFeatures{};
	__cpuid(reinterpret_cast<int*>(&hardwareFeatures), HvCpuIdFunctionMsHvHardwareFeatures);
	vmSettings.ComputeTopology.Processor.EnablePerfmonPmu = hardwareFeatures.ChildPerfmonPmuSupported != 0;
	vmSettings.ComputeTopology.Processor.EnablePerfmonLbr = hardwareFeatures.ChildPerfmonLbrSupported != 0;
	}
	#endif

To enable the perfmon features, WSL first checks the CPUID for support for emulation from the host CPU.
On my system (i7-12700K, Win11 Pro 27783), these bits are all false, so the perfmon features aren’t activated for the WSL VM:

> .\cpuid.exe
ChildPerfmonPmuSupported: 0
ChildPerfmonLbrSupported: 0
ChildPerfmonIptSupported: 0

It would be great to see data from other systems, if these bits are set for you, please check with the script I attached.
The check seems to be incorrect, or at least prone to false negatives. We’ve already confirmed that Hyper-V can actually enable (and virtualize?) perfmon features even when the CPUID bits are off (Alderlake and above).
cpuid.c:

// clang -o cpuid.exe cpuid.c
#include <intrin.h>
#include <stdint.h>
#include <stdio.h>


typedef struct _HV_X64_HYPERVISOR_HARDWARE_FEATURES {
  //
  // Eax
  //
  uint32_t ApicOverlayAssistInUse : 1;
  uint32_t MsrBitmapsInUse : 1;
  uint32_t ArchitecturalPerformanceCountersInUse : 1;
  uint32_t SecondLevelAddressTranslationInUse : 1;
  uint32_t DmaRemappingInUse : 1;
  uint32_t InterruptRemappingInUse : 1;
  uint32_t MemoryPatrolScrubberPresent : 1;
  uint32_t DmaProtectionInUse : 1;
  uint32_t HpetRequested : 1;
  uint32_t SyntheticTimersVolatile : 1;
  uint32_t HypervisorLevel : 4;
  uint32_t PhysicalDestinationModeRequired : 1;
  uint32_t UseVmfuncForAliasMapSwitch : 1;
  uint32_t HvRegisterForMemoryZeroingSupported : 1;
  uint32_t UnrestrictedGuestSupported : 1;
  uint32_t RdtAFeaturesSupported : 1;
  uint32_t RdtMFeaturesSupported : 1;
  uint32_t ChildPerfmonPmuSupported : 1;
  uint32_t ChildPerfmonLbrSupported : 1;
  uint32_t ChildPerfmonIptSupported : 1;
  uint32_t ApicEmulationSupported : 1;
  uint32_t ChildX2ApicRecommended : 1;
  uint32_t HardwareWatchdogReserved : 1;
  uint32_t DeviceAccessTrackingSupported : 1;
  uint32_t Reserved : 5;

  //
  // Ebx
  //
  uint32_t DeviceDomainInputWidth : 8;
  uint32_t ReservedEbx : 24;

  //
  // Ecx
  //
  uint32_t ReservedEcx;

  //
  // Edx
  //
  uint32_t ReservedEdx;

} HV_X64_HYPERVISOR_HARDWARE_FEATURES, *PHV_X64_HYPERVISOR_HARDWARE_FEATURES;

#define HvCpuIdFunctionMsHvHardwareFeatures 0x40000006

int main() {
  HV_X64_HYPERVISOR_HARDWARE_FEATURES hardwareFeatures = {};
  __cpuid((int *)(&hardwareFeatures), HvCpuIdFunctionMsHvHardwareFeatures);
  printf("ChildPerfmonPmuSupported: %u\n",
         hardwareFeatures.ChildPerfmonPmuSupported);
  printf("ChildPerfmonLbrSupported: %u\n",
         hardwareFeatures.ChildPerfmonLbrSupported);
  printf("ChildPerfmonIptSupported: %u\n",
         hardwareFeatures.ChildPerfmonIptSupported);
  printf("ApicOverlayAssistInUse: %u\n",
         hardwareFeatures.ApicOverlayAssistInUse);
  printf("MsrBitmapsInUse: %u\n", hardwareFeatures.MsrBitmapsInUse);
  printf("ArchitecturalPerformanceCountersInUse: %u\n",
         hardwareFeatures.ArchitecturalPerformanceCountersInUse);
  printf("SecondLevelAddressTranslationInUse: %u\n",
         hardwareFeatures.SecondLevelAddressTranslationInUse);
  printf("DmaRemappingInUse: %u\n", hardwareFeatures.DmaRemappingInUse);
  printf("InterruptRemappingInUse: %u\n",
         hardwareFeatures.InterruptRemappingInUse);
  printf("MemoryPatrolScrubberPresent: %u\n",
         hardwareFeatures.MemoryPatrolScrubberPresent);
  printf("DmaProtectionInUse: %u\n", hardwareFeatures.DmaProtectionInUse);
  printf("HpetRequested: %u\n", hardwareFeatures.HpetRequested);
  printf("SyntheticTimersVolatile: %u\n",
         hardwareFeatures.SyntheticTimersVolatile);
  printf("HypervisorLevel: %u\n", hardwareFeatures.HypervisorLevel);
  printf("PhysicalDestinationModeRequired: %u\n",
         hardwareFeatures.PhysicalDestinationModeRequired);
  printf("UseVmfuncForAliasMapSwitch: %u\n",
         hardwareFeatures.UseVmfuncForAliasMapSwitch);
  printf("HvRegisterForMemoryZeroingSupported: %u\n",
         hardwareFeatures.HvRegisterForMemoryZeroingSupported);
  printf("UnrestrictedGuestSupported: %u\n",
         hardwareFeatures.UnrestrictedGuestSupported);
  printf("RdtAFeaturesSupported: %u\n", hardwareFeatures.RdtAFeaturesSupported);
  printf("RdtMFeaturesSupported: %u\n", hardwareFeatures.RdtMFeaturesSupported);
  printf("ApicEmulationSupported: %u\n",
         hardwareFeatures.ApicEmulationSupported);
  printf("ChildX2ApicRecommended: %u\n",
         hardwareFeatures.ChildX2ApicRecommended);
  printf("HardwareWatchdogReserved: %u\n",
         hardwareFeatures.HardwareWatchdogReserved);
  printf("DeviceAccessTrackingSupported: %u\n",
         hardwareFeatures.DeviceAccessTrackingSupported);
  printf("DeviceDomainInputWidth: %u\n",
         hardwareFeatures.DeviceDomainInputWidth);
  printf("ReservedEbx: %u\n", hardwareFeatures.ReservedEbx);
  printf("ReservedEcx: %u\n", hardwareFeatures.ReservedEcx);
  printf("ReservedEdx: %u\n", hardwareFeatures.ReservedEdx);
  return 0;
}

[samlihaha]

does that mean we can use intel pt in wsl ? @clemenswasser

[samlihaha]

Wow, three years already, impressive! ;) Now that WSL is open source, we get to do the job of Microsoft employees who are paid to work full-time on WSL and debug this ourselves 🥳 (fun, isn’t it?). The relevant code can be found here:

WSL/src/windows/service/exe/WslCoreVm.cpp

Lines 1640 to 1651 in 4b5cb64

#ifdef AMD64

 // Enable hardware performance counters if they are supported. 
 if (m_vmConfig.EnableHardwarePerformanceCounters) 
 { 
     HV_X64_HYPERVISOR_HARDWARE_FEATURES hardwareFeatures{}; 
     __cpuid(reinterpret_cast<int*>(&hardwareFeatures), HvCpuIdFunctionMsHvHardwareFeatures); 
     vmSettings.ComputeTopology.Processor.EnablePerfmonPmu = hardwareFeatures.ChildPerfmonPmuSupported != 0; 
     vmSettings.ComputeTopology.Processor.EnablePerfmonLbr = hardwareFeatures.ChildPerfmonLbrSupported != 0; 
 } 

#endif

To enable the perfmon features, WSL first checks the CPUID for support for emulation from the host CPU.
On my system (i7-12700K, Win11 Pro 27783), these bits are all false, so the perfmon features aren’t activated for the WSL VM:

.\cpuid.exe
ChildPerfmonPmuSupported: 0
ChildPerfmonLbrSupported: 0
ChildPerfmonIptSupported: 0
It would be great to see data from other systems, if these bits are set for you, please check with the script I attached. The check seems to be incorrect, or at least prone to false negatives. We’ve already confirmed that Hyper-V can actually enable (and virtualize?) perfmon features even when the CPUID bits are off (Alderlake and above). cpuid.c:

// clang -o cpuid.exe cpuid.c
#include <intrin.h>
#include <stdint.h>
#include <stdio.h>

typedef struct _HV_X64_HYPERVISOR_HARDWARE_FEATURES {
//
// Eax
//
uint32_t ApicOverlayAssistInUse : 1;
uint32_t MsrBitmapsInUse : 1;
uint32_t ArchitecturalPerformanceCountersInUse : 1;
uint32_t SecondLevelAddressTranslationInUse : 1;
uint32_t DmaRemappingInUse : 1;
uint32_t InterruptRemappingInUse : 1;
uint32_t MemoryPatrolScrubberPresent : 1;
uint32_t DmaProtectionInUse : 1;
uint32_t HpetRequested : 1;
uint32_t SyntheticTimersVolatile : 1;
uint32_t HypervisorLevel : 4;
uint32_t PhysicalDestinationModeRequired : 1;
uint32_t UseVmfuncForAliasMapSwitch : 1;
uint32_t HvRegisterForMemoryZeroingSupported : 1;
uint32_t UnrestrictedGuestSupported : 1;
uint32_t RdtAFeaturesSupported : 1;
uint32_t RdtMFeaturesSupported : 1;
uint32_t ChildPerfmonPmuSupported : 1;
uint32_t ChildPerfmonLbrSupported : 1;
uint32_t ChildPerfmonIptSupported : 1;
uint32_t ApicEmulationSupported : 1;
uint32_t ChildX2ApicRecommended : 1;
uint32_t HardwareWatchdogReserved : 1;
uint32_t DeviceAccessTrackingSupported : 1;
uint32_t Reserved : 5;

//
// Ebx
//
uint32_t DeviceDomainInputWidth : 8;
uint32_t ReservedEbx : 24;

//
// Ecx
//
uint32_t ReservedEcx;

//
// Edx
//
uint32_t ReservedEdx;

} HV_X64_HYPERVISOR_HARDWARE_FEATURES, *PHV_X64_HYPERVISOR_HARDWARE_FEATURES;

#define HvCpuIdFunctionMsHvHardwareFeatures 0x40000006

int main() {
HV_X64_HYPERVISOR_HARDWARE_FEATURES hardwareFeatures = {};
__cpuid((int *)(&hardwareFeatures), HvCpuIdFunctionMsHvHardwareFeatures);
printf("ChildPerfmonPmuSupported: %u\n",
hardwareFeatures.ChildPerfmonPmuSupported);
printf("ChildPerfmonLbrSupported: %u\n",
hardwareFeatures.ChildPerfmonLbrSupported);
printf("ChildPerfmonIptSupported: %u\n",
hardwareFeatures.ChildPerfmonIptSupported);
printf("ApicOverlayAssistInUse: %u\n",
hardwareFeatures.ApicOverlayAssistInUse);
printf("MsrBitmapsInUse: %u\n", hardwareFeatures.MsrBitmapsInUse);
printf("ArchitecturalPerformanceCountersInUse: %u\n",
hardwareFeatures.ArchitecturalPerformanceCountersInUse);
printf("SecondLevelAddressTranslationInUse: %u\n",
hardwareFeatures.SecondLevelAddressTranslationInUse);
printf("DmaRemappingInUse: %u\n", hardwareFeatures.DmaRemappingInUse);
printf("InterruptRemappingInUse: %u\n",
hardwareFeatures.InterruptRemappingInUse);
printf("MemoryPatrolScrubberPresent: %u\n",
hardwareFeatures.MemoryPatrolScrubberPresent);
printf("DmaProtectionInUse: %u\n", hardwareFeatures.DmaProtectionInUse);
printf("HpetRequested: %u\n", hardwareFeatures.HpetRequested);
printf("SyntheticTimersVolatile: %u\n",
hardwareFeatures.SyntheticTimersVolatile);
printf("HypervisorLevel: %u\n", hardwareFeatures.HypervisorLevel);
printf("PhysicalDestinationModeRequired: %u\n",
hardwareFeatures.PhysicalDestinationModeRequired);
printf("UseVmfuncForAliasMapSwitch: %u\n",
hardwareFeatures.UseVmfuncForAliasMapSwitch);
printf("HvRegisterForMemoryZeroingSupported: %u\n",
hardwareFeatures.HvRegisterForMemoryZeroingSupported);
printf("UnrestrictedGuestSupported: %u\n",
hardwareFeatures.UnrestrictedGuestSupported);
printf("RdtAFeaturesSupported: %u\n", hardwareFeatures.RdtAFeaturesSupported);
printf("RdtMFeaturesSupported: %u\n", hardwareFeatures.RdtMFeaturesSupported);
printf("ApicEmulationSupported: %u\n",
hardwareFeatures.ApicEmulationSupported);
printf("ChildX2ApicRecommended: %u\n",
hardwareFeatures.ChildX2ApicRecommended);
printf("HardwareWatchdogReserved: %u\n",
hardwareFeatures.HardwareWatchdogReserved);
printf("DeviceAccessTrackingSupported: %u\n",
hardwareFeatures.DeviceAccessTrackingSupported);
printf("DeviceDomainInputWidth: %u\n",
hardwareFeatures.DeviceDomainInputWidth);
printf("ReservedEbx: %u\n", hardwareFeatures.ReservedEbx);
printf("ReservedEcx: %u\n", hardwareFeatures.ReservedEcx);
printf("ReservedEdx: %u\n", hardwareFeatures.ReservedEdx);
return 0;
}

That is my result:
PS D:\workspace> .\check.exe
ChildPerfmonPmuSupported: 0
ChildPerfmonLbrSupported: 0
ChildPerfmonIptSupported: 0
ApicOverlayAssistInUse: 1
MsrBitmapsInUse: 1
ArchitecturalPerformanceCountersInUse: 1
SecondLevelAddressTranslationInUse: 1
DmaRemappingInUse: 0
InterruptRemappingInUse: 1
MemoryPatrolScrubberPresent: 0
DmaProtectionInUse: 1
HpetRequested: 0
SyntheticTimersVolatile: 0
HypervisorLevel: 0
PhysicalDestinationModeRequired: 0
UseVmfuncForAliasMapSwitch: 0
HvRegisterForMemoryZeroingSupported: 0
UnrestrictedGuestSupported: 1
RdtAFeaturesSupported: 0
RdtMFeaturesSupported: 0
ApicEmulationSupported: 1
ChildX2ApicRecommended: 1
HardwareWatchdogReserved: 0
DeviceAccessTrackingSupported: 0
DeviceDomainInputWidth: 39
ReservedEbx: 0
ReservedEcx: 0
ReservedEdx: 0

and my cpu is 12th Gen Intel(R) Core(TM) i7-12800HX

[samlihaha]

In fact, the wsl.exe run on Windows host, that means the __cpuid instruction just get the windows hardwareFeatures, however, windows does not use PerfmonPmu or PerfmonLbr yet, definitely is 0.

But wsl hyper-v vm can set it to 1 and use this hardwareFeatures, that is not a problem because hyper-v vm is seprate with the host.

WSL/src/windows/service/exe/WslCoreVm.cpp

Lines 1640 to 1651 in 4b5cb64

	#ifdef _AMD64_
	// Enable hardware performance counters if they are supported.
	if (m_vmConfig.EnableHardwarePerformanceCounters)
	{
	HV_X64_HYPERVISOR_HARDWARE_FEATURES hardwareFeatures{};
	__cpuid(reinterpret_cast<int*>(&hardwareFeatures), HvCpuIdFunctionMsHvHardwareFeatures);
	vmSettings.ComputeTopology.Processor.EnablePerfmonPmu = hardwareFeatures.ChildPerfmonPmuSupported != 0;
	vmSettings.ComputeTopology.Processor.EnablePerfmonLbr = hardwareFeatures.ChildPerfmonLbrSupported != 0;
	}
	#endif