Add UTF8 (and maybe UTF16) to System.Runtime.InteropServices.CharSet + Marshal class

[migueldeicaza]

Currently marshaling in .NET has three modes: Ansi, Unicode (platform dependent) and "Auto" which picks a good default between those two. The meaning of Unicode is closely associated with Window's UTF16.

There is today no convenient and reliable way to do UTF8 encoding, and at best we have an ambiguous definition of what Unicode is.

There are enough bits on the metadata tables to add these two values.

People can resort to custom marshalers (slow, cumbersome, everyone has to do it), or manual marshaling, or hope that the platform does the right thing.

Anecdotally: this also happens to be oldest Mono bug that is still open.

The world has spoken, and UTF8 is the standard, we should have first-class support for it both for P/Invoke signatures as well as the various helper methods in Marshal.

[masonwheeler]

The world has spoken, and UTF8 is the standard

Hmm. Now, don't get me wrong, I like UTF8 as much as anyone. Its technical superiority over UTF-16 is obvious, and I'd definitely like to see it replace UTF-16. But there's more than a little exaggeration in that claim as stated!

The platform native string type in both Windows and OSX is UTF-16. The platform native string type in both the CLR and the JVM is UTF-16. Take those out... and there's precious little left of "the world"!

[josteink]

@masonwheeler: That may be, but I would argue your perspective is too narrow. You're looking at things from a framework-internal perspective.

When using P/Invoke you are typically (or at least often) interacting with libraries hosted outside the .NET framework, often written in plain C.

Most of those libraries will expect traditional 8-bit text ANSI/ASCII text and in those cases UTF8 is the only reliable way of getting Unicode-content across. And if we're going to start P/Invoking on other platforms than Windows, UTF8 is defacto the only way to represent unicode at system-level.

[masonwheeler]

@josteink Ah, you're right. That's a very good point.

[MikePopoloski]

+9000

I've worked with internal applications that pinvoked into custom C DLLs, and we had to maintain tons of custom marshaling code because using UTF-16 was out of the question due to the memory requirements. Automatic UTF-8 marshaling would be so nice to have.

[stephentoub]

While the name is misleading, on UNIX CoreCLR's ANSI marshaling is UTF8.

[migueldeicaza]

@stephentoub The name is not only misleading, it is incompatible with the behavior on Windows.

While this is what Mono has done, it causes problems for people that want to target both Unix and Windows, as evidenced by people that have to resort over and over to use the two approaches described above.

I do not need the extra work in Mono (just like I assume you guys do not want the extra work), but it makes the platform very unpleasant to work with for everyone that has to consider more than a platform. This is a request on behalf of the users of what is a half-baked and broken setup.

[jkotas]

cc @yizhang82 - maybe a good candidate for MCG (Marshaling Code Generator)

If this ever happen, would it mean we will not require to use MarshalString() between the native C++ and CLR strings, as the direct assignment would be possible? Which would also mean no more efforts to invent better (managed) variants of System::Interop like specified in @Cygon's blog: http://blog.nuclex-games.com/mono-dotnet/cxx-cli-string-marshaling/.
In that case 💯 👍

[yizhang82]

Supporting UTF-8 marshalling does sound like something we might want to support in the future, most likely in MCG @jkotas has mentioned earlier. MCG is the new interop technology we have in .NET Native, and is a great place for experimenting with newer features like this. Implementing marshalling support in the CLR itself is rather painful - you'll need to write code generation code that emits IL. While in MCG, you write code that spits out C#. Eventually we'd like to see all the CLR runtime (.NET native being one of them) use the same underlying interop technology so that we don't have to implement them twice and maintain two separate code bases.

@jasonwilliams200OK It's unlikely we'll be able to support marshalling directly to C++ std::strings. We don't necessarily want to tie to a particular layout of a C++ string implementation. I think C++/CLI libraries should provide a good way to marshal between managed string and C++ string.

[whoisj]

@yizhang82 C++ std:string is happy with c-string, as is C#. Why not just assume that native and C# will be using char[] or wchar_t[] for string import/export.

That said, the UTF-8 marshaller is critical. The Libgit2Sharp project has a fairly good implementation without too many contributors (one really), might be worth speaking with them about making their implementation part of the coreclr, or at least a good starting point.

/CC @nulltoken

[nulltoken]

👍 for me

@paulcbetts @phkelley @tclem @dahlbyk AFAIR you all contributed to it. Thoughts?

[dahlbyk]

First-class UTF8 seems reasonable to me. First-class Encoding support seems like it could also be useful, but I'm not exactly sure what that would look like.

Relevant LG2S classes:

• EncodingMarshaler base
• LaxUtf8Marshaler (throwOnInvalidBytes: false)
• StrictUtf8Marshaler (throwOnInvalidBytes: true)

[anaisbetts]

^^ this would be great, there's really no reason to assume text is ASCII when marshaling strings, the only two options in modern software today are UTF-16 (i.e. Windows / OS X native style) and UTF-8 (basically every OSS lib). +1 for this feature.

[yizhang82]

We have the API proposal out for review: https://github.com/dotnet/corefx/issues/7804

[myungjoo]

CC: @leemgs This seems to be related with the unexplained issue that depends on the LOCALE environment variables (not working if the locale is ko_KR.UTF8).