Windows exception interop support

[AndyAyersMS]

Jit codegen for inline pinvoke is currently more restricted than necessary for CoreCLR, since it is working around issues that arise with exception interop on windows.

We should remove the support and the restrictions on jit codegen.

See notes for dotnet/coreclr#8263.

[AndyAyersMS]

@janvorli I'd be willing to give this a try, but would need some guidance.

[janvorli]

@AndyAyersMS sure. I believe that for AMD64, ARM and ARM64 we need to replace the personality routine on TheUMEntryPrestub and UMThunkStub in all the .asm files listed below with a new personality routine that behaves like the UnhandledExceptionHandlerUnix on Unix, it means it calls the DefaultCatchHandler and fails fast after that.
Here are the files where it needs to be replaced:

• src\vm\arm\asmhelpers.asm
• src\vm\amd64\UMThunkStub.asm
• src\vm\arm64\asmhelpers.asm

As for x86, I need to look into it. The way the stubs and EH works is different over there and I am not that familiar with it.

[janvorli]

So here are the details for x86. Here, the exception personality routine of our interest is stored in FrameHandlerExRecord that's created by the thunk code, which in turn is generated by UMEntryThunk::CompileUMThunkWorker. You can see that it sets the exception handler to FastNExportExceptHandler. So for CoreCLR (FEATURE_CORECLR) , this handler body should be modified to call the DefaultCatchHandler and fail fast, like for the other case.
The same applies to UMThunkPrestubHandler.
Actually, now that I think about it, for the case of ARM, ARM64 and AMD64 that I've described before, we should also just modify the bodies of UMEntryPrestubUnwindFrameChainHandler and UMThunkStubUnwindFrameChainHandler for coreclr instead of creating a new personality routine.

[gkhanna79]

@janvorli @AndyAyersMS Are we planning to make this change for 2.0?

[AndyAyersMS]

It would probably be a good idea. I doubt we want anyone taking a dependence on the current behavior.

[janvorli]

@AndyAyersMS do you still think you'd find time to make the change or should we find someone else to do it?

[AndyAyersMS]

Not clear yet -- if you have somebody ready to take this on, please go ahead. Otherwise I'll try to get to it soon.

[mjsabby]

Will this cause a difference in behavior between an assembly running on .NET Desktop vs. .NET Core on Windows? If so, I don't think this is a good idea.

[AndyAyersMS]

@mjsabby It will cause differences, yes.

The proposal here is to make .Net Core behavior consistent across OS platforms, which means in this area .Net Core and .Net Desktop will diverge.

Under this proposal, for all .Net Core platforms including Windows, exceptions that cross a managed-native or native-managed boundary will cause the runtime to terminate with an unhandled exception.

cc @jkotas @richlander

[jkotas]

.Net Core behavior consistent across OS platforms

And also consistent with Mono runtime. There is a great write up on it here: http://www.mono-project.com/docs/advanced/pinvoke/#runtime-exception-propagation

[gkhanna79]

CC @rahku and @yizhang82