P/Invoke calls terminate application after unmanaged library modification (macOS, Arm64)

[sbergen]

Description

When using either DllImport or LibraryImport to call an unmanaged method:

• if the application has been already run at least once, and
• the unmanaged library is modified,

when the application is run again, it will terminate with exit code 137 at the first unmanaged call.

Reproduction Steps

Note that I have only been able to reproduce this on macOS with Arm64 .NET.

Reproduction using small project and script:

1. Unzip the content of PInvokeIssue.zip to a directory
2. run the reproduce-issue.sh script in the directory created above
3. Observe the output (includes demonstration of using dotnet clean as a workaround)

Description of issue:

Using the following C# program:

using System.Runtime.InteropServices;

Console.WriteLine("Before P/Invoke");
NativeFunction();
Console.WriteLine("After P/Invoke");

[DllImport("native")]
static extern void NativeFunction();

an unmanaged library built from the following C code:

#include <stdio.h>

extern __attribute__((visibility("default"))) void NativeFunction()
{
    puts("in native code");
}

and the following build integration in the .csproj file:

    <ItemGroup>
        <None Include="libnative.dylib">
            <CopyToOutputDirectory>Always</CopyToOutputDirectory>
        </None>
    </ItemGroup>

    <Target Name="NativeBuild" BeforeTargets="BeforeBuild">
        <Exec Command="clang -dynamiclib -o libnative.dylib native.c" />
    </Target>

1. Run with dotnet run. Observe three lines of output
2. Modify the unmanaged code, by e.g. replacing "native" with "modified"
3. Run the application again

Expected behavior

You see three lines of output with the middle line modified and an exit code of 0:

Before P/Invoke
in modified code
After P/Invoke

Actual behavior

Only the first line of output is printed, and the application unexpectedly terminates with status code 137:

Before P/Invoke

Regression?

The issue exists also in .NET 6.

Known Workarounds

• Running dotnet with lldb seems avoid the issue
• Using dotnet clean and rebuilding clears the issue
• Reverting the unmanaged library back to its original state causes the application to run normally again.

Configuration

• dotnet version: 7.0.202
• macOS Monterey 12.6.1 (21G217)
• ARM64 (Apple M1 Pro)

The issues doesn't seem to reproduce on Windows or Intel macs.

Other information

With my minimal reproduction, I wasn't able to reproduce the issue by simply replacing the unmanaged library in the build output directory and running again with dotnet path/to/dll. However, with my actual project, even this causes the process to terminate.

[AaronRobinsonMSFT]

@sbergen This is likely unrelated to .NET but rather it is some macOS policy. This has been observed within the runtime too. In order to confirm I would use the Console application on macOS to see why the application is crashing/failing to load the modified shared library. It is likely the issue with be related to a macOS security policy.

This issue has been marked needs-author-action and may be missing some important information.

[sbergen]

@sbergen This is likely unrelated to .NET but rather it is some macOS policy. This has been observed within the runtime too. In order to confirm I would use the Console application on macOS to see why the application is crashing/failing to load the modified shared library. It is likely the issue with be related to a macOS security policy.

Thanks, this is indeed the case, and I was able to figure out a workaround, which is deleting the unmanaged library in the output directory before recompiling it.

After a lot of digging, I found this Apple forum post which explains the reason this has to be done:

Specifically, code signing information is hung off the vnode within the kernel, and modifying the file behind that cache will cause problems. You need a new vnode, which means a new file, that is, a new inode.

While this obviously is not a dotnet runtime issue, is there something that could be done on the SDK side to mitigate this? I think at least the P/Invoke documentation could contain some mention of this.

And while on the topic of documentation, I wasn't able to find any documentation that would have helped me get crash reports for this. I had some from a few days ago, but I do not know why I got them then, but wasn't able to get any crash reports now.

I tried adding

export DOTNET_EnableCrashReport=1
export DOTNET_DbgEnableMiniDump=1
export DOTNET_CreateDumpVerboseDiagnostics=1
ulimit -c unlimited

and used various ways to run the app (running the appHost, dotnet run, dotnet /path/to/dll), but none of them produced a crash report.

FWIW, the crash reports from a few days ago contained this info, which was helpful:

Exception Type:        EXC_CRASH (SIGKILL (Code Signature Invalid))
Exception Codes:       0x0000000000000001, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace CODESIGNING, Code 2 

If someone has concrete suggestions on the best way to take forward the potential improvements I'm hinting at, some pointers would be appreciated. But otherwise I believe this issue can be considered solved.

[sbergen]

I was working on a workaround and it got complicated, so I'll leave this here as a note to myself (if I report this to the SDK) and others having this issue:

If you have multiple projects in your solution, this issue propagates to all projects referencing the project that contains the unmanaged library in it. E.g. if SomeAssembly.Tests references SomeAssembly, and SomeAssembly contains the unmanaged library, then MSBuild will copy the library from the output directory of SomeAssembly to the output directory of SomeAssembly.Tests also. And since the file was never deleted, this will also trigger the OS to kill the tests when executing the tests.

[AaronRobinsonMSFT]

While this obviously is not a dotnet runtime issue, is there something that could be done on the SDK side to mitigate this?

Since this is an unmanaged binary being loaded the .NET SDK isn't involved so there isn't anything it can do. Additionally the runtimes is making a dlopen call and that causing a failing in the process by the OS - the runtime can't do much about that unfortunately.

I think at least the P/Invoke documentation could contain some mention of this.

Absolutely. We should update the doc to help users in this situation. I will leave this issue open to track that doc update - excellent suggestion.

/cc @dotnet/interop-contrib