Convert constructor invocations from the runtime to use UnmanagedCallersOnly

[jkoritzinsky]

This PR moves the class constructor and instance constructor invocations from within the runtime to use the UnmanagedCallersOnly pattern established with #123864 to invoke the constructors.

Contributes to #123864.

As this PR removes all usages of CATCH_HANDLER_FOUND_NOTIFICATION_CALLSITE and PREPARE_NONVIRTUAL_CALLSITE_USING_METHODDESC , these pieces of infrastructure are removed as well.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR continues the migration away from MethodDescCallSite / CallDescrWorker-style VM-to-managed invocation, switching class constructor and default instance constructor calls over to the UnmanagedCallersOnly reverse P/Invoke pattern (per #123864). It also removes the now-unused CATCH_HANDLER_FOUND_NOTIFICATION_CALLSITE / PREPARE_NONVIRTUAL_CALLSITE_USING_METHODDESC infrastructure.

Changes:

• Route MethodTable::RunClassInitEx (.cctor invocation) through a new InitHelpers.CallClassConstructor [UnmanagedCallersOnly] entrypoint and UnmanagedCallersOnlyCaller.
• Route default constructor invocations through a new RuntimeHelpers.CallDefaultConstructor [UnmanagedCallersOnly] entrypoint and UnmanagedCallersOnlyCaller.
• Remove debugger notification dispatch flag and wrapper logic from DispatchCallSimple/call helper macros.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/coreclr/vm/methodtable.cpp	Switches class constructor invocation to UnmanagedCallersOnlyCaller.
src/coreclr/vm/corelib.h	Adds CoreLib binder entries for the new UCO-managed helpers.
src/coreclr/vm/cominterfacemarshaler.cpp	Switches extensible RCW default ctor invocation to UCO path.
src/coreclr/vm/callhelpers.h	Removes catch-handler-found notification flag/macro; retains UCO caller helper.
src/coreclr/vm/callhelpers.cpp	Removes debugger wrapper dispatch path; switches default ctor helper to UCO path.
src/coreclr/System.Private.CoreLib/.../RuntimeHelpers.CoreCLR.cs	Adds [UnmanagedCallersOnly] method for default ctor invocation.
src/coreclr/System.Private.CoreLib/.../InitHelpers.cs	Adds [UnmanagedCallersOnly] method for class constructor invocation.

[jkotas]

I think the right solution here is to do the allocation in managed code as well.

We would have to use reflection (Activator.CreateInstance) for that. We may find surprising cycles between reflection and EEException::CreateThrowable where these constructors are called from in the VM. EEException::CreateThrowable is several layers lower than reflection. I think it is ok to leave it as is for this PR.

[AaronRobinsonMSFT]

I think the right solution here is to do the allocation in managed code as well.

We would have to use reflection (Activator.CreateInstance) for that. We may find surprising cycles between reflection and EEException::CreateThrowable where these constructors are called from in the VM. EEException::CreateThrowable is several layers lower than reflection. I think it is ok to leave it as is for this PR.

Okay.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

[jkoritzinsky]

@AaronRobinsonMSFT any idea why we'd be getting asserts for the incorrect GC mode on thread shutdown on some wasm tests with this change?

[AaronRobinsonMSFT]

@AaronRobinsonMSFT any idea why we'd be getting asserts for the incorrect GC mode on thread shutdown on some wasm tests with this change?

I would guess we're missing a transition call somewhere. Do you have a stack?

[jkoritzinsky]

@AaronRobinsonMSFT any idea why we'd be getting asserts for the incorrect GC mode on thread shutdown on some wasm tests with this change?

I would guess we're missing a transition call somewhere. Do you have a stack?

Only stacks I have are on the failures on CI, which would be the DetachThread call when the Wasm VM shuts down.

https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-124920-merge-ddd2d58e98b8440db1/Loader/1/console.8b7394fc.log?helixlogtype=result

Maybe the problem is some sort of crash on WASM only that takes down the VM and hits this assert before it can report anything useful about the actual test failure?

[AaronRobinsonMSFT]

Maybe the problem is some sort of crash on WASM only that takes down the VM and hits this assert before it can report anything useful about the actual test failure?

Quite possibly. This looks like something that you should be able to create a small repo and debug on node.js through corerun.

[jkoritzinsky]

Ok it looks like this failure is unrelated to this PR, but this PR exposes it.

Basically, when we get the exception back from the cctor, we try to throw it. As part of that, we try to get the stack trace. During that call in the failing test suites, we allocate enough memory to trigger the GC. The GC tries to trigger the finalizer. For some reason, there's nothing marking the runtime to be kept alive, so it gets shut down after the finalization tick. However, the finalizer has left the GC mode for the thread in the incorrect state (looking into why now).

I guess by going down COMPlusThrow instead of the other route for how exceptions were routed ends up allocating just enough to run the finalizer in a place where just enough is allocated to run the finalizer now.

I'll continue investigating.

[jkoritzinsky]

Yep, when we enter SystemJS_ExecuteFinalizationCallback, the GC mode in this case is MODE_COOPERATIVE. Since we don't have contracts enabled, we don't detect it.

[jkoritzinsky]

Looks like I was getting the async stack trace, so the issue is not that we're recursing into the finalizer tick while the GC is running, but instead we're triggering the next loop in the event loop from a thread in COOP mode.

I'll open a known issue if I can't figure this out in a reasonable time frame.

[jkoritzinsky]

Found it: Fix in f3405df

Without that fix, Emscripten will throw a MemoryError that we don't handle, which takes down the runtime in its current state (no C++ unwinding). Then, we were running the next tick which ran the finalizer (entering the runtime in a bad state).

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[am11]

@jkoritzinsky, has it covered these:

(or are you planning to convert them? otherwise, I can take a look)

[jkoritzinsky]

@am11 this didn't cover those. I've been looking at the next phase of work, not the items in that list.