Struct-by-value return in P/Invoke causes crash on Linux

[tritao]

Hello.

In the process of porting CppSharp to .NET Core, we found a crasher that is blocking us.

I've managed to minimize the code to make a repro:

native.cpp:

namespace CppSharp { namespace CppParser { namespace AST {

struct VTableComponent
{
    VTableComponent();
    int Kind;
};

VTableComponent::VTableComponent() : Kind(0)
{

}

struct VTableLayout
{
    VTableLayout();
    VTableComponent getComponents(unsigned i);
    VTableComponent Components;
};

VTableLayout::VTableLayout()
{
    Components.Kind = 2;
}

VTableComponent VTableLayout::getComponents(unsigned i)
{
    return Components;
}

} } }

Native code can be compiled with: clang -fno-rtti -fno-exceptions -fPIC -shared -lstdc++ -olibnative.so native.cpp

Program.cs

using System;
using System.Runtime.InteropServices;

namespace crash
{
    public unsafe partial class VTableLayout
    {
        [StructLayout(LayoutKind.Explicit, Size = 0)]
        public partial struct __Internal
        {
            [DllImport("native", CallingConvention = System.Runtime.InteropServices.CallingConvention.Cdecl,
                EntryPoint="_ZN8CppSharp9CppParser3AST12VTableLayoutC2Ev")]
            internal static extern void ctor(System.IntPtr instance);

            [DllImport("native", CallingConvention = System.Runtime.InteropServices.CallingConvention.Cdecl,
                EntryPoint="_ZN8CppSharp9CppParser3AST12VTableLayout13getComponentsEj")]
            internal static extern VTableComponent.__Internal GetComponents(System.IntPtr instance, uint i);
        }

        public System.IntPtr __Instance { get; protected set; }

        public VTableLayout()
        {
            __Instance = Marshal.AllocHGlobal(sizeof(VTableLayout.__Internal));
            __Internal.ctor(__Instance);
        }

        public VTableComponent GetComponents(uint i)
        {
            var __ret = __Internal.GetComponents((__Instance), i);
            return new VTableComponent();
        }
    }

    public unsafe partial class VTableComponent
    {
        [StructLayout(LayoutKind.Explicit, Size = 4)]
        public partial struct __Internal
        {
            [FieldOffset(0)]
            internal int kind;
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            var layout = new VTableLayout();

            System.Console.WriteLine("Calling VTableLayout::GetComponents()");
            var ret = VTableLayout.__Internal.GetComponents(layout.__Instance, 0);
            System.Console.WriteLine("VTableComponent::Kind = {0}", ret.kind);
        }
    }
}

Before minimization, the code has 100% crash rate in the native side (due to calling into std::vector stuff).
After minimization, there is no crash, but the returned value is always different on each run.

joao@joao-HP:~/dev/CppSharp/build/crash/bin/Debug/netcoreapp2.2$ mono crash.dll && echo && dotnet crash.dll
Calling VTableLayout::GetComponents()
VTableComponent::Kind = 2

Calling VTableLayout::GetComponents()
VTableComponent::Kind = 1656532064

Mono works fine with the same native DLL.

This looks like a bug in System V AMD64 ABI handling.

I just saw dotnet/coreclr#22041, which looks like it maybe could be related to this?

[jkoritzinsky]

This is likely related to dotnet/coreclr#22041.

[jkoritzinsky]

Fixed by dotnet/coreclr#22041