Non-ASCII chars shouldn't compare equal to ASCII chars under OrdinalIgnoreCase comparison

[GrabYourPitchforks]

Under ICU, there are some non-ASCII code points that become ASCII code points after a simple case mapping transformation.

'K' (U+212A KELVIN SIGN) ~= 'k' (U+006B LATIN SMALL LETTER K) [simple lowercase mapping]
'ſ' (U+017F LATIN SMALL LETTER LONG S) ~= 'S' (U+0053 LATIN CAPITAL LETTER S) [simple uppercase mapping]

Since it's common for applications to use StringComparison.OrdinalIgnoreCase when comparing things like usernames, this could be a pit of failure for those applications, as it could lead to the following behavior at runtime.

string.Equals("administrator", "adminiſtrator", StringComparison.OrdinalIgnoreCase) // <-- FALSE on Windows, TRUE on Linux

A fairly straightfoward fix would be to prevent non-ASCII chars and ASCII characters from being equal under an OrdinalIgnoreCase comparison. It would mean that OrdinalIgnoreCase is no longer a direct wrapper around ICU's case mapping / case folding APIs, but it would bring the behavior more in line with what developers have come to expect over .NET's history.

With this proposal, ToUpperInvariant and ToLowerInvariant would be a direct wrapper around ICU's underlying simple case mapping APIs, and it wouldn't special-case any characters.

Related: #27540

The following APIs would be affected:

• string.Equals, string.Compare, string.GetHashCode, and any other APIs which might accept StringComparison.OrdinalIgnoreCase as a parameter
• StringComparer.OrdinalIgnoreCase.Equals and StringComparer.OrdinalIgnoreCase.GetHashCode
• TextInfo.Compare and similar APIs which might accept CompareOptions.OrdinalIgnoreCase

/cc @tarekgh

[tarekgh]

prevent non-ASCII chars from becoming ASCII chars

I would recommend doing that in the case of the ordinal casing but not for invariant. The reason here is Invariant casing still treated as a cultural casing and not ordinal and it makes sense to do the right casing according to Unicode standard as most of the people using it in scenarios that not specific to compare 2 strings securely. instead, the ordinal comparison should be used at that time.

[GrabYourPitchforks]

Just to clarify, you're suggesting:

string.Equals("administrator", "adminiſtrator", StringComparison.OrdinalIgnoreCase) // <-- with this change, will return FALSE on all platforms
"administrator".ToUpperInvariant() == "adminiſtrator".ToUpperInvariant() // <-- will continue to return FALSE on Windows, TRUE on Linux

This means that string.Equals(a, b, StringComparison.OrdinalIgnoreCase) will no longer be equivalent to string.Equals(a.ToUpperInvariant(), b.ToUpperInvariant(), StringComparison.Ordinal), which is a behavioral change from previous framework releases.

I'm ok with your suggestion here if you think we can swing the breaking change. :)

[tarekgh]

@GrabYourPitchforks yes you have articulated it accurately.

[GrabYourPitchforks]

@tarekgh I've updated the proposal text to match your suggestions. Thanks!

Do you suppose we'd need similar APIs char.EqualsOrdinalIgnoreCase or Rune.EqualsOrdinalIgnoreCase to accompany the string ordinal ignore case APIs? I can imagine some applications might be interested in doing the comparison on a character-by-character basis.

[tarekgh]

Char already has ToUpper/ToLower which should match the ordinal casing. Also, it has ToUpperInvariant/ToLowerInvariant for invariant case. I think the current APIs are enough for doing any needed operation.

[GrabYourPitchforks]

Char already has ToUpper/ToLower which should match the ordinal casing.

There's no such API, unfortunately. :( char.ToUpper implicitly uses the current culture. We don't have an "ordinal" version per se; we've just relied on ToUpperInvariant for this.

After this change, on Linux:

char.ToUpperInvariant('ſ') // <-- returns 'S'
"ſ".ToUpperInvariant() // <-- returns "S"
string.Equals("ſ", "S", StringComparison.OrdinalIgnoreCase) // <-- returns FALSE

[tarekgh]

right. but I expect:

string.Equals("ſ", "S", StringComparison.InvariantCultureIgnoreCase) // <-- returns true

Is that right?

[GrabYourPitchforks]

I guess my question is that in theory it should be possible to write the following code:

public static bool AreStringsEqualOrdinalCase(string a, string b)
{
    if (a.Length != b.Length) { return false; }
    for (int i = 0; i < a.Length; i++)
    {
        char charA = a[i];
        char charB = b[i];
        if (!CharsAreEqualOrdinalIgnoreCase(charA, charB)) { return false; }
    }
    return true;
}

And that it should behave identically to string.Equals(a, b, StringComparison.OrdinalIgnoreCase), handwaving away that we should be checking for surrogates, etc.

What is the CharsAreEqualOrdinalIgnoreCase method in the example above? Since we don't expose such a method, a developer won't be able to write the sample method above reliably.

[tarekgh]

CharsAreEqualOrdinalIgnoreCase should be equivalent to Char.Toupper(ch1) == Char.Toupper(ch2). We want to ensure char.ToUpper is behaving as ordinal casing we are going to have.

[GrabYourPitchforks]

char.ToUpper is defined to be culture-aware, though.

runtime/src/libraries/System.Private.CoreLib/src/System/Char.cs

Lines 350 to 353 in 0e0e852

	public static char ToUpper(char c)
	{
	return CultureInfo.CurrentCulture.TextInfo.ToUpper(c);
	}

We have no "ordinal" equivalent API. The closest we have is ToUpperInvariant.