Handle struct returns for COM interface methods across all marshalling modes #1536
+691
−121
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
COM interfaces with methods that return struct have never worked correctly -- see #167. With all the different marshalling modes that cswin32 supports this also gets fairly complex. This change applies fixes / workarounds to all the modes in order to get this to work:
1. In .NET Core + source generated COM
This just works, [GeneratedComInterface] takes care of this for us.
There is a risk that type def'd structs are improperly handled (see discussion in #1478). To handle this we add a custom marshaller to typedefs so that the native type is the pointer and it's wrapped in the struct during marshalling.
2. In .NET Core + blittable mode
a. We can change members with struct returns to use MemberFunction calling convention.
b. We apply this only to functions that have struct returns just so that we don't perturb the generated code too much. When the return type is non-struct, there should be no difference.
3. In .NET Framework + blittable mode
Sadly there is no MemberFunction calling convention we can use so we emulate it using the suggestion @tannergooding offered in #167:
Which is to say, change the return type to pointer and add a new 1st parameter which is a pointer to the buffer to receive the return value. This is relatively contained in the DeclareInterfaceAsStruct path where we can keep the public method the same and only change the function pointer signature and implementation of the method call.
4. Built-in COM
Built-in COM for both .NET Framework and .NET Core mis-handles this scenario as well. We can use the same technique as blittable mode without MemberFunction and modify the ABI methods of the COM interface to be the modified signature. When modifying the signature we also rename the ABI member with ("_StructReturn") so it's obvious what happened:
And then the friendly overloads help to smooth this over for callers:
All of these modes have test coverage for not just the codegen but samples that call the functions and validate the correct values are returned. Previously these tests cases either crashed or returned garbage results.
Fixes #167, Fixes #1479, Fixes #1478, Fixes #1436.