`bool` return from `PInvoke` misinterpreted on Alipine

[swaroop-sridhar]

On linux-musl-release-x64 builds, the return value from a native function returning false is sometimes incorrectly interpreted as true because of the mismatch between the number of bytes expected for bool on native and managed sides.

Repro:
The host defines a native function bundle_probe with type

bool STDMETHODCALLTYPE bundle_probe(const char16_t* path, int64_t* offset, int64_t* size)

If this function is called from managed code using the following delegate:

 [UnmanagedFunctionPointer(CallingConvention.StdCall)]	        
 public delegate bool BundleProbeDelegate([MarshalAs(UnmanagedType.LPWStr)] string path, IntPtr size, IntPtr offset);

The invocations where bundle_probe return false, are interpreted as true. Gor example:

Probe(bundleProbeDelegate, "", isExpected: false);

Once this issue is fixed, this test asset should be updated to use the bool return type in the PInvoke delegate.

[stephentoub]

bool STDMETHODCALLTYPE bundle_probe(const char16_t* path, int64_t* offset, int64_t* size)

Isn't the issue here that this signature uses bool rather than, say, int32_t? By default, a bool in C# is marshaled as a 4-byte value.

[jkotas]

This behavior is by design. The test has mismatched signatures between managed and unmanaged sides that is causing the bug.

bool in C# does not match bool in C++. The default marshaling for C# bool is 4-byte BOOL.

It is best to avoid bool in the interop signatures, on both managed and unmanaged sides.

[swaroop-sridhar]

Thanks for the clarification @jkotas and @stephentoub.

The bundle_probe function is an internal API intended to be used from native code. It was used via PInvoke only for unit-testing, and I have fixed the test code now (#36327).