Fix native memory leak in AsAnyMarshaler.ConvertLayoutToNative on exception

[Copilot]

Description

AsAnyMarshaler.ConvertLayoutToNative leaks the Marshal.AllocCoTaskMem-allocated buffer when LayoutTypeConvertToUnmanaged throws — the exception unwinds past the return pNativeHome with no cleanup.

Fix

Wrap the conversion call in a try/catch that frees on failure, matching the defensive pattern already used in sibling code in the same file:

IntPtr pNativeHome = Marshal.AllocCoTaskMem(allocSize);
try
{
    if (IsIn(dwFlags))
        StubHelpers.LayoutTypeConvertToUnmanaged(pManagedHome, (byte*)pNativeHome, ref cleanupWorkList);
}
catch
{
    Marshal.FreeCoTaskMem(pNativeHome);
    throw;
}

Changes

• src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs: Added try/catch around LayoutTypeConvertToUnmanaged call in ConvertLayoutToNative to free native memory on exception.

Testing

Risk

Low — strictly adds a cleanup path that was previously missing; no behavioral change on the happy path.

[Copilot]

Pull request overview

Fixes an exception-path leak in CoreCLR’s AsAnyMarshaler.ConvertLayoutToNative (in System.Private.CoreLib) by ensuring unmanaged memory allocated via Marshal.AllocCoTaskMem is freed if LayoutTypeConvertToUnmanaged throws during layout marshaling.

Changes:

• Wrap LayoutTypeConvertToUnmanaged in try/catch within ConvertLayoutToNative.
• Free the CoTaskMem buffer on exception before rethrowing.

[Copilot]

Pull request overview

Fixes an exception-path native memory leak in AsAnyMarshaler.ConvertLayoutToNative within CoreLib interop marshaling by ensuring native allocations (and any accumulated cleanup work) are released when LayoutTypeConvertToUnmanaged throws.

Changes:

• Wraps the LayoutTypeConvertToUnmanaged call in ConvertLayoutToNative with a try/catch.
• On exception, destroys the cleanupWorkList and frees the Marshal.AllocCoTaskMem buffer before rethrowing.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[AaronRobinsonMSFT]

Is the full clean-up always safe here? I'd assume that there are already clean-up places for this. The FreeCoTaskMem makes sense, but do we really need the StubHelpers.DestroyCleanupList() as well?

[jkoritzinsky]

We don't technically "need" the cleanup list cleanup as the only entries (SafeHandle instances) will eventually be finalized, but without this, they'd have a mismatched refcount until finalization.