[7.0][garbage collector] Performance issue for non-concurrent server garbage collection

[Marluwe]

Description

The current preview .Net7 preview 7 shows a performance issue related to the garbage collector with ServerGarbageCollection enabled and ConcurrentGarbageCollection disabled.
The program starts with the expected good performance and suddenly drops by factor about 2.5-3 and stays at that.

We were able to narrow it down to the GC settings. We run a loop with 70 iterations. Each iteration calls the same single-threaded algorithm with the same data. The performance then drops about iteration 30.

Why turning off the concurrent garbage collection at all
In .Net6 and prior versions, the non-concurrent server GC has the highest performance for our optimization algorithms.

Reproduction Steps

I haven't been able to break it down to a minimal example.
All minimal examples show a stable performance.

I'm hoping for some ideas, how to approach this. Maybe one has to force a situation, where the section based gc performs better than the region based gc.

Expected behavior

The performance should be stable (or improve) over each iteration, see scenario V0

Actual behavior

This test is done using several configurations (see below).
The diagram below shows the runtimes over each iteration step.
As expected for an computational heavy program, using the ServerGC improves the performance.
The strange behavior can be seen in V1 and V4, where the performance drops about iteration 30.

Configuration of the different scenarios:

	Tiered compilation	using clrgc.dll	ConcurrentGC	ServerGC	Tiered PGO
V0 (base)	true	true	false	true	true
V1	true	false	false	true	false
V2	true	false	true	true	false
V3	false	false	true	true	false
V4	false	false	false	true	false
V5	false	true	false	true	false
V6	true	true	false	true	false
V7	true	false	true	false	false
V8	true	true	true	false	false

Regression?

In .Net6 the performance is stable over all iterations.

Known Workarounds

Tuning on the ConcurrentGC solves this particular issue, but performs worse when running the whole program - at least for our optimization algorithms.

Configuration

General information about the configuration and the system:

Parameter	Value
RuntimeIdentifier	win10-x64
PlatformTarget	x64
Configuration	Release
Debugger attached	No
OS	Microsoft Windows 10.0.19044, Platform: Win32NT (Microsoft Windows NT 10.0.19044.0)
GC latency mode	Interactive
CPU	Intel(R) Xeon(R) E-2146G CPU @ 3.50GHz   3.50 GHz
RAM	32 GB
.net7	.NET 7.0.0-preview.7.22375.6

Other information

No response

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

The current preview .Net7 preview 7 shows a performance issue related to the garbage collector with ServerGarbageCollection enabled and ConcurrentGarbageCollection disabled.
The program starts with the expected good performance and suddenly drops by factor about 2.5-3 and stays at that.

We were able to narrow it down to the GC settings. We run a loop with 70 iterations. Each iteration calls the same single-threaded algorithm with the same data. The performance then drops about iteration 30.

Why turning off the concurrent garbage collection at all
In .Net6 and prior versions, the non-concurrent server GC has the highest performance for our optimization algorithms.

Reproduction Steps

I haven't been able to break it down to a minimal example.
All minimal examples show a stable performance.

I'm hoping for some ideas, how to approach this. Maybe one has to force a situation, where the section based gc performance better than the region based gc.

Expected behavior

The performance should be stable (or improve) over each iteration, see scenario V0

Actual behavior

This test is done using several configurations (see below).
The diagram below shows the runtimes over each iteration step.
As expected for an computational heavy program, using the ServerGC improves the performance.
The strange behavior can be seen in V1 and V4, where the performance drops about iteration 30.

Configuration of the different scenarios:

	Tiered compilation	using clrgc.dll	ConcurrentGC	ServerGC	Tiered PGO
V0 (base)	true	true	false	true	true
V1	true	false	false	true	false
V2	true	false	true	true	false
V3	false	false	true	true	false
V4	false	false	false	true	false
V5	false	true	false	true	false
V6	true	true	false	true	false
V7	true	false	true	false	false
V8	true	true	true	false	false

Regression?

In .Net6 the performance is stable over all iterations.

Known Workarounds

Tuning on the ConcurrentGC solves this particular issue, but performs worse when running the whole program - at least for our optimization algorithms.

Configuration

General information about the configuration and the system:

Parameter	Value
RuntimeIdentifier	win10-x64
PlatformTarget	x64
Configuration	Release
Debugger attached	No
OS	Microsoft Windows 10.0.19044, Platform: Win32NT (Microsoft Windows NT 10.0.19044.0)
GC latency mode	Interactive
CPU	Intel(R) Xeon(R) E-2146G CPU @ 3.50GHz   3.50 GHz
RAM	32 GB
.net7	.NET 7.0.0-preview.7.22375.6

Other information

No response

Author:	Marluwe
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[Maoni0]

it would be helpful to indicate what performance this is - is it throughput? heap size? something else?
in rc1 I've disabled something that causes problems for non concurrent GC. would it be possible to try the rc1 build?
if that's not possible, or that doesn't solve the problem, could you please capture some top level GC traces so we can start investigating it?

How to collect top level GC metrics

GC emits lightweight Informational level events that you can collect (and leave on all the time if you want) that covers all top level GC metrics. Use this PerfView commandline to collect these events –

perfview /GCCollectOnly /AcceptEULA /nogui collect

after you are done, press s in the perfview cmd window to stop it.

This should be run long enough to capture enough GC activities, e.g., if you know problems occur at times, this should cover time that leads up to when problems happen (not only during problematic time). If you are not sure when the problem starts to occur, you can just have it on for a long time.

If you know how long to run the collection, use the following (this is used much more often actually) –

perfview /GCCollectOnly /AcceptEULA /nogui /MaxCollectSec:1800 collect

and replace 1800 (half an hour) with however many seconds you need. You can of course apply this arg to other commandline as well. This will result in a file called PerfViewGCCollectOnly.etl.zip. In PerfView terms, we call this a GCCollectOnly trace.

[Marluwe]

The issue is solved in RC1
Thank's for the information!

[Maoni0]

thanks for the confirmation!