Compiler does not handle non-BMP characters in identifiers

[Serentty]

The compiler uses methods on Char extensively to check character properties, even in recent code such as #9199. This breaks down on non-BMP characters, and these properties should be checked on the full 32-bit code points instead. A similar bug exists in Roslyn, but it has been acknowledged as a bug (given that the C# standard allows non-BMP characters identifiers), and is in the process of being fixed. Therefore, for interoperability with C#, even putting aside the many minority languages around the world whose scripts are outside the BMP and the many CJK personal and place names outside of the BMP, these identifiers will need to be handled properly.

Example of the issue

let δ𐌰 = 5

Although the Greek letter δ is handled properly, the Gothic letter 𐌰 causes the compiler to explain of an unexpected character in the pattern.

[abelbraaksma]

This probably means that the parser should be made aware of surrogate pairs, as it uses UTF16 (technically, UCS-2, because it doesn't recognize surrogate pairs) for its parsing input.

There's now improved support for this in .NET through Rune, but I wonder how complex a fix would be.

[Serentty]

@abelbraaksma Does the compiler itself have to be able to run on .NET 4.8? If so, then Rune won't be available. In Roslyn, it seems the fix that they're working on uses int instead.

[abelbraaksma]

I actually don't know what the least common denominator is for the compiler wrt versions. Maybe @cartermp can answer that?

[Serentty]

I would be happy to work on a PR for this if anyone can tell me where in the compiler to look. Proper Unicode support in software is something that I'm very passionate about.

[cartermp]

The compiler must be able to run on downlevel .NET Frameworks. So we can't rely on using the Rune type to just handle things.

[Happypig375]

.NET Core executables can be self-contained. Why is .NET Framework still needed?

[abelbraaksma]

It isn't, but MS needs to support it until the support period is over, and anything in VS is framework based (hence that the FSI window is framework, not Core), which means FCS must be, too.

Note that Rune is not needed for this, it is just the simplest way. This can be done how it's always been done for higher plane characters: check if an UCS-2 character is a surrogate char, and if so, if it's a valid surrogate pair.

If both are true, test if the resulting codepoint belongs to accepted ranges (not sure which those are, I assume N).

But it's possible that this cannot be expressed easily in lex/yacc, which may not be aware of surrogates. While simple to do in F#, if it requires the lexer to be made Astral Plane aware, it's gonna be a little challenge (which is true even with Rune support).