Fix Thread.GetCurrentProcessorId for > 64 CPUs on Windows.

[VSadov]

GetCurrentProcessorNumber is capped to 64 on Windows and that results in unexpected sharing when having 64+ cores. In particular if the processor groups are in different NUMA nodes.
We need to use GetCurrentProcessorNumberEx.

GCToOSInterface::GetCurrentProcessorNumber has another implementation, which looks correct. This is basically a short version of that.

[sergiy-k]

Is it OK if ThreadNative::GetCurrentProcessorNumber might return an index that is larger than total number of active processors on the system?

[VSadov]

Yes. What we return to the user is technically “ID correlated with last core we ran on”. We may even default to ThreadID if OS API is not functional. (PAL may return -1).

Are process groups contiguous? (1,2,3, ...)

[VSadov]

To answer my question - yes, OS "packs" cores into as few process groups as possible and considers topology when assigning.
https://docs.microsoft.com/en-us/windows/win32/procthread/processor-groups

[sergiy-k]

/cc: @janvorli

[janvorli]

GetCurrentProcessorNumber is capped to 64 on Windows and that results in unexpected sharing when having 64+ cores.

It is not capped, it returns number of processor within the current processor group. Processor groups cannot have more than 64 CPUs.
I am not sure if returning a combined processor number the way you've done it (group * 64 + in_group_cpu_index) is something that we should make public, as Windows have no concept of CPU index larger than 64, and groups don't necessarily have to be completely filled. That means that the range of CPU indices would not be necessarily continuous.
In GC code, it is actually used as an internal encoding of the group / index, we always decode it back to group / index before using it.

[VSadov]

The goal here is to form an integer that user can use to softly affinitize data to cores.
This is not to describe topology. If interesting, we may need another API for that.

Returning proc number within a group is clearly wrong since it maps all cores into 0-64 range.

If there is a better way to produce a CoreID - how?

[jkotas]

Any difference in perf compared to what we have today?

[VSadov]

@jkotas - perf difference in API itself due to << and | or perf difference on 256core machine where every proc #N shares data with procs #N in 3 other NUMA nodes?

The latter really depends on the app.
The sharing is very unintuitive though. Even if we wanted to compress the ProcID range to 64, I'd prefer to share within nodes, not across.

[VSadov]

on machines with fast GetCurrentProcessorNumber this change adds 5-10% to the cost of the FCALL.

The following are measurements as reported by printf-instrumented #467 when it is forced to calibrate FCALL against standalone access to ThreadStatic
I picked just the best measurements, since there is some noise.

================= Main machine (12 logical cores, Coffeelake 4.2 GHz)

100ns tick (Stopwatch)
4096 iters

times (in ticks for one iteration)
=== ignoring cpu group (just calling GetCurrentProcessorNumber)
ID: 0.08984375 TLS: 0.04443359375

=== considering cpu group (calling and folding GetCurrentProcessorNumberEx)
ID: 0.095703125 TLS: 0.04443359375

adds 6.5% to ID call

================== Older machine (8 logical cores, Kabylake 4.0 GHz)
284ns tick
2048 iters

times (in ticks)
=== ignoring cpu group
ID: 0.03466796875 TLS: 0.015625

=== considering cpu group
ID: 0.0361328125 TLS: 0.0146484375

adds 4% to ID call

================== Rome (256 logical cores, Zen2, 2.4 GHz, has RDPID)
100ns tick
4096 iters

times (in ticks)
=== ignoring cpu group
ID: 0.0455322265625 TLS: 0.074462890625

=== considering cpu group
ID: 0.0498046875 TLS: 0.074462890625

adds 9% per ID call

[jkotas]

Sounds reasonable.