Test failure System.Globalization.Tests.CultureInfoAll.TestAllCultures_Nls

[v-wenyuxu]

Failed in: runtime-coreclr crossgen2 20231223.1

Failed tests:

net9.0-windows-Release-x64-TestReadyToRun_Release-Windows.Amd64.Server2022.Open
    - System.Globalization.Tests.CultureInfoAll.TestAllCultures_Nls
net9.0-windows-Release-x64-TestReadyToRun_Release-Windows.11.Amd64.Client.Open
    - System.Globalization.Tests.CultureInfoAll.TestAllCultures_Nls

Error message:

Assert.All() Failure: 574 out of 575 items in the collection did not pass.
[1]:   Item:  aa-DJ
       Error: Assert.Equal() Failure: Strings differ
                         ↓ (pos 0)
              Expected: "Qafar (Yabuuti)"
              Actual:   "Afar (Djibouti)"
                         ↑ (pos 0)
[2]:   Item:  aa-ER
       Error: Assert.Equal() Failure: Strings differ
                         ↓ (pos 0)
              Expected: "Qafar (Eretria)"
              Actual:   "Afar (Eritrea)"
                         ↑ (pos 0)
[3]:   Item:  aa-ET
       Error: Assert.Equal() Failure: Strings differ
                         ↓ (pos 0)
              Expected: "Qafar (Otobbia)"
              Actual:   "Afar (Ethiopia)"
                         ↑ (pos 0)
[4]:   Item:  af-NA
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["So.", "Ma.", "Di.", "Wo.", "Do.", ···]
              Actual:   ["S", "M", "D", "W", "D", ···]
                         ↑ (pos 0)
[5]:   Item:  af-ZA
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["So.", "Ma.", "Di.", "Wo.", "Do.", ···]
              Actual:   ["S", "M", "D", "W", "D", ···]
                         ↑ (pos 0)
[6]:   Item:  agq-CM
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["nts", "kpa", "ghɔ", "tɔm", "ume", ···]
              Actual:   ["n", "k", "g", "t", "u", ···]
                         ↑ (pos 0)
[7]:   Item:  ak-GH
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["Kwe", "Dwo", "Ben", "Wuk", "Yaw", ···]
              Actual:   ["K", "D", "B", "W", "Y", ···]
                         ↑ (pos 0)
[8]:   Item:  am-ET
       Error: Assert.Equal() Failure: Values differ
              Expected: 2
              Actual:   3
[9]:   Item:  ar-001
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["أحد", "إثنين", "ثلاثاء", "أربعاء", "خميس", ···]
              Actual:   ["ح", "ن", "ث", "ر", "خ", ···]
                         ↑ (pos 0)
[10]:  Item:  ar-AE
       Error: Assert.Equal() Failure: Collections differ
                                  ↓ (pos 1)
              Expected: ["الأحد", "الإثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
              Actual:   ["الأحد", "الاثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
                                  ↑ (pos 1)
[11]:  Item:  ar-BH
       Error: Assert.Equal() Failure: Collections differ
                                  ↓ (pos 1)
              Expected: ["الأحد", "الإثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
              Actual:   ["الأحد", "الاثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
                                  ↑ (pos 1)
[12]:  Item:  ar-DJ
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["أحد", "إثنين", "ثلاثاء", "أربعاء", "خميس", ···]
              Actual:   ["ح", "ن", "ث", "ر", "خ", ···]
                         ↑ (pos 0)
[13]:  Item:  ar-DZ
       Error: Assert.Equal() Failure: Collections differ
                                  ↓ (pos 1)
              Expected: ["الأحد", "الإثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
              Actual:   ["الأحد", "الاثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
                                  ↑ (pos 1)
[14]:  Item:  ar-EG
       Error: Assert.Equal() Failure: Collections differ
                                  ↓ (pos 1)
              Expected: ["الأحد", "الإثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
              Actual:   ["الأحد", "الاثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
                                  ↑ (pos 1)
[15]:  Item:  ar-ER
       Error: Assert.Equal() Failure: Collections differ
     

Stack trace:

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Failed in: runtime-coreclr crossgen2 20231223.1

Failed tests:

net9.0-windows-Release-x64-TestReadyToRun_Release-Windows.Amd64.Server2022.Open
    - System.Globalization.Tests.CultureInfoAll.TestAllCultures_Nls
net9.0-windows-Release-x64-TestReadyToRun_Release-Windows.11.Amd64.Client.Open
    - System.Globalization.Tests.CultureInfoAll.TestAllCultures_Nls

Error message:

Assert.All() Failure: 574 out of 575 items in the collection did not pass.
[1]:   Item:  aa-DJ
       Error: Assert.Equal() Failure: Strings differ
                         ↓ (pos 0)
              Expected: "Qafar (Yabuuti)"
              Actual:   "Afar (Djibouti)"
                         ↑ (pos 0)
[2]:   Item:  aa-ER
       Error: Assert.Equal() Failure: Strings differ
                         ↓ (pos 0)
              Expected: "Qafar (Eretria)"
              Actual:   "Afar (Eritrea)"
                         ↑ (pos 0)
[3]:   Item:  aa-ET
       Error: Assert.Equal() Failure: Strings differ
                         ↓ (pos 0)
              Expected: "Qafar (Otobbia)"
              Actual:   "Afar (Ethiopia)"
                         ↑ (pos 0)
[4]:   Item:  af-NA
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["So.", "Ma.", "Di.", "Wo.", "Do.", ···]
              Actual:   ["S", "M", "D", "W", "D", ···]
                         ↑ (pos 0)
[5]:   Item:  af-ZA
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["So.", "Ma.", "Di.", "Wo.", "Do.", ···]
              Actual:   ["S", "M", "D", "W", "D", ···]
                         ↑ (pos 0)
[6]:   Item:  agq-CM
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["nts", "kpa", "ghɔ", "tɔm", "ume", ···]
              Actual:   ["n", "k", "g", "t", "u", ···]
                         ↑ (pos 0)
[7]:   Item:  ak-GH
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["Kwe", "Dwo", "Ben", "Wuk", "Yaw", ···]
              Actual:   ["K", "D", "B", "W", "Y", ···]
                         ↑ (pos 0)
[8]:   Item:  am-ET
       Error: Assert.Equal() Failure: Values differ
              Expected: 2
              Actual:   3
[9]:   Item:  ar-001
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["أحد", "إثنين", "ثلاثاء", "أربعاء", "خميس", ···]
              Actual:   ["ح", "ن", "ث", "ر", "خ", ···]
                         ↑ (pos 0)
[10]:  Item:  ar-AE
       Error: Assert.Equal() Failure: Collections differ
                                  ↓ (pos 1)
              Expected: ["الأحد", "الإثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
              Actual:   ["الأحد", "الاثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
                                  ↑ (pos 1)
[11]:  Item:  ar-BH
       Error: Assert.Equal() Failure: Collections differ
                                  ↓ (pos 1)
              Expected: ["الأحد", "الإثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
              Actual:   ["الأحد", "الاثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
                                  ↑ (pos 1)
[12]:  Item:  ar-DJ
       Error: Assert.Equal() Failure: Collections differ
                         ↓ (pos 0)
              Expected: ["أحد", "إثنين", "ثلاثاء", "أربعاء", "خميس", ···]
              Actual:   ["ح", "ن", "ث", "ر", "خ", ···]
                         ↑ (pos 0)
[13]:  Item:  ar-DZ
       Error: Assert.Equal() Failure: Collections differ
                                  ↓ (pos 1)
              Expected: ["الأحد", "الإثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
              Actual:   ["الأحد", "الاثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
                                  ↑ (pos 1)
[14]:  Item:  ar-EG
       Error: Assert.Equal() Failure: Collections differ
                                  ↓ (pos 1)
              Expected: ["الأحد", "الإثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
              Actual:   ["الأحد", "الاثنين", "الثلاثاء", "الأربعاء", "الخميس", ···]
                                  ↑ (pos 1)
[15]:  Item:  ar-ER
       Error: Assert.Equal() Failure: Collections differ
     

Stack trace:

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Author:	v-wenyuxu
Assignees:	-
Labels:	area-System.Globalization, os-windows, arch-x64, untriaged, blocking-clean-ci-optional, needs-area-label
Milestone:	-

[tarekgh]

@jkotas should this be fixed by your change #96288. The failing test case depends on PlatformDetection.IsNlsGlobalization to decide to run the test case while PlatformDetection.IsNlsGlobalization depends on if ICU is loaded or not.

runtime/src/libraries/System.Runtime/tests/System.Globalization.Tests/CultureInfo/CultureInfoAll.cs

Line 18 in 93caf8d

[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsNlsGlobalization))]

Also looking at the logs,

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-1c86dda4f46e4c39b4/System.Globalization.Tests/1/console.7f6b8543.log?helixlogtype=result

I am seeing

[FAIL] System.Globalization.Tests.IcuTests.IcuShouldBeLoaded
Assert.True() Failure
Expected: True
Actual:   False

[jkotas]

Yes, this should be fixed by #96288 .