Error in passing structure to native method on OSX

[alsemenn]

On OSX debug build (both native and managed) following occurs:
The number written by native part is not 10 (see attached source code).
But if struct layout changed to Sequential (in .cs file) everything works fine.

src.zip

[janvorli]

The reason is that CoreCLR interop on amd64 cannot pass structs with explicit layout by value in registers. Only sequential layout is supported.

[alsemenn]

Is there is a way to learn about from some documentation? Or is there a way to add warnings about the issue, so other developers doesn't fall in the same trap?

[janvorli]

@jkotas do you know if we have it documented somewhere?

[jkotas]

cc @yizhang82

@yizhang82 Do we have documentation somewhere about what to be careful about when writing managed views of unmanaged structs?

We have never treated structures with explicit layout as HFAs or HVAs. It started with Windows on arm that was the first platform with HFAs that CoreCLR was targeting and we have maintained the same design since then.

[yizhang82]

@jkotas Thanks. I didn't know about the HFA/HVA. This makes sense, but it is a rather subtle difference in how we handle HFA/HVA with regarding to C++ (C++ doesn't seem to care whether it is a struct vs union in this case, according to @tijoytom 's experiement in #5552 ).
The best doc I can find on related struct marshaling topic is this one: https://msdn.microsoft.com/en-us/library/eshywdt7(v=vs.110).aspx I'll create a doc bug to get this added.
However, I think the best way going forward is to create a tool that creates the right struct definition in the first place.

[yizhang82]

Thinking about this more, I'm a bit concerned that in the case of an C/C++ union, there might be no way to create a equivalent managed struct with explicit layout that works everywhere due to HFA/HVA.
@tijoytom Can you do some quick experiments and see what the experience is like?

[tijoytom-zz]

@yizhang82 @janvorli @jkotas

I did some experiments with it and the below scenarios wont work on all platform unless the native method signature can be changed.

Managed Definition

[StructLayout(LayoutKind.Explicit)]
public struct ExplStruct
{
[FieldOffset(0)]
public int a;
[FieldOffset(0)]
public int b;
public ExplStruct(int nm) {
b = 0;
a = nm;
}
}

Native:

typedef union {
int a;
int b;
}ExplUnion;
CdeclSimpleExplStruct(ExplUnion p) { }