AccessViolationException in HttpClient.SendAsync() on macOS arm64 in .NET 9 preview.4

[martincostello]

Description

I have an application I'm testing against .NET 9 daily builds. With the latest SDK and package versions I'm seeing System.AccessViolationException : Attempted to read or write protected memory. This is often an indication that other memory is corrupt. errors in my tests when running on GitHub Actions' macOS arm64 runners.

I have 9 applications I'm doing this with, but I'm only observing this issue with one of them.

Reproduction Steps

1. Clone martincostello/costellobot@18966b8
2. Run build.ps1 on a macOS arm64 machine.

Expected behavior

The tests all pass.

Actual behavior

Lots of test failures with a stack trace similar to the below:

System.AccessViolationException : Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at MartinCostello.Costellobot.Infrastructure.IntegrationTests`1.PostWebhookAsync(String event, Object value, String webhookSecret, String delivery) in /_/tests/Costellobot.Tests/Infrastructure/IntegrationTests`1.cs:line 176
   at MartinCostello.Costellobot.Handlers.DeploymentStatusHandlerTests.PostWebhookAsync(DeploymentStatusDriver driver, String action) in /_/tests/Costellobot.Tests/Handlers/DeploymentStatusHandlerTests.cs:line 667
   at MartinCostello.Costellobot.Handlers.DeploymentStatusHandlerTests.Deployment_Is_Approved_For_Trusted_User_And_Dependency_When_Penultimate_Build_Skipped() in /_/tests/Costellobot.Tests/Handlers/DeploymentStatusHandlerTests.cs:line 128
--- End of stack trace from previous location ---

That corresponds to this code, but it doesn't appear to have been recently changed:

runtime/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs

Lines 520 to 556 in 8f3c687

	async Task<HttpResponseMessage> Core(
	HttpRequestMessage request, HttpCompletionOption completionOption,
	CancellationTokenSource cts, bool disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
	{
	bool telemetryStarted = StartSend(request);
	bool responseContentTelemetryStarted = false;
	HttpResponseMessage? response = null;
	try
	{
	// Wait for the send request to complete, getting back the response.
	response = await base.SendAsync(request, cts.Token).ConfigureAwait(false);
	ThrowForNullResponse(response);
	// Buffer the response content if we've been asked to.
	if (ShouldBufferResponse(completionOption, request))
	{
	if (HttpTelemetry.Log.IsEnabled() && telemetryStarted)
	{
	HttpTelemetry.Log.ResponseContentStart();
	responseContentTelemetryStarted = true;
	}
	await response.Content.LoadIntoBufferAsync(_maxResponseContentBufferSize, cts.Token).ConfigureAwait(false);
	}
	return response;
	}
	catch (Exception e)
	{
	HandleFailure(e, telemetryStarted, response, cts, originalCancellationToken, pendingRequestsCts);
	throw;
	}
	finally
	{
	FinishSend(response, cts, disposeCts, telemetryStarted, responseContentTelemetryStarted);
	}
	}

Regression?

Yes.

Known Workarounds

None.

Configuration

.NET SDK 9.0.100-preview.4.24265.4

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[antonfirsov]

@martincostello any chance you can help us by collecting traces? This could give us some preliminary ideas before someone manages to jump on the issue.

dotnet-trace collect --providers System.Net.Http,Private.InternalDiagnostics.System.Net.Http,System.Net.Security,System.Net.Sockets,System.Net.NameResolution,System.Threading.Tasks.TplEventSource:0x80:4 --process-id <PID>

[antonfirsov]

Could this be the root cause?

[martincostello]

any chance you can help us by collecting traces?

Sure, I'll try and do this later today.

Could #101479 (comment) be the root cause?

I did spot that issue, but it seemed a bit too old to still be an issue in the nightlies?

[martincostello]

dotnet-trace collect --providers System.Net.Http,Private.InternalDiagnostics.System.Net.Http,System.Net.Security,System.Net.Sockets,System.Net.NameResolution,System.Threading.Tasks.TplEventSource:0x80:4 --process-id <PID>

Just looked at this - how do I do this exactly when I don't know what the test process' ID will be until after I start the tests?

[wfurt]

I know this annoying. I often add Console.Readline() to the repro and hit enter after tracing started. Other alternative is to add EventListener the the repro itself.

[wfurt]

BTW the linked issue blames libgit2sharp. Do you see same issue on any other platform @martincostello ?

[martincostello]

At the moment my repro is just "run dotnet test" - I don't have anything more specific at the moment.

I could maybe add something to the tests to have the tests start monitor itself (if that works?)

I'm only seeing this on macOS, not Windows or Linux, but only in this one project. I'm not specifically using lib2gitsharp anywhere, and if it is being used, I would assume it's from an SDK or something and would/should be present in my other repos I'm testing nightlies with.

[martincostello]

If it helps narrow the search, this has started happening at some point since the 24th April, but all I had to go on was "dotnet test crashes" at that point and I figured it would stabilise at some point. It's only in the last day or so I've noticed it's progressed to not killing the whole process and giving me an exception message and stack trace.

[martincostello]

But that probably correlates more with GitHub Actions runners moving from macOS 12 x64 to macOS 14 arm64, so it was probably broken before then and my build was just on a different arch so I didnt see it before...