[wasm][icu] ICU files sharding + custom icu data loading for WASM

docs/README.md

@@ -25,6 +25,7 @@ Design Docs
 =================
 
 - [.NET Globalization Invariant Mode](design/features/globalization-invariant-mode.md)
+- [WASM Globalization Icu](design/features/globalization-icu-wasm.md)
 - [Cross-Platform Cryptography](design/features/cross-platform-cryptography.md)
 - Many more under [design/features](design/features/)
 

docs/design/features/globalization-icu-wasm.md

@@ -0,0 +1,63 @@
+# WASM Globalization Icu
+
+In WASM applications when [globalization invariant mode](globalization-invariant-mode.md) is switched off, internalization data file is loaded. There are four basic types of these files:
+- `icudt.dat` - full data
+- `icudt_EFIGS.dat` - data for locales: "en-*", "fr-FR", "es-ES", "it-IT", and "de-DE".
+- `icudt_CJK.dat` - for locales: "en" "ja", "ko", and "zh".
+- `icudt_no_CJK.dat` - all locales from `icudt.dat`, excluding "ja", "ko", and "zh".
+
+## Wasm Console, Wasm Browser
+
+We can specify the file we want to load, e.g. `icudt_no_CJK.dat` by adding to .csproj:
+```
+<WasmIcuDataFileName>icudt_no_CJK.dat</WasmIcuDataFileName>
+```
+Only one value for `WasmIcuDataFileName` can be set. It can also be a custom file, created by the developer. To create a custom ICU file, see `Custom ICU` section below. If no `WasmIcuDataFileName` was specified, the application's culture will be checked and the corresponding file will be loaded, e.g. for `en-US` file `icudt_EFIGS.dat`, and for `zh-CN` - `icudt_CJK.dat`.
+
+## Custom ICU
+
+The easiest way to build ICU is to open https://github.com/dotnet/icu/ it in [Codespaces](docs\workflow\Codespaces.md). See files in https://github.com/dotnet/icu/tree/dotnet/main/icu-filters, and read https://unicode-org.github.io/icu/userguide/icu_data/buildtool.html#locale-slicing. Build your own filter or edit the existing file.
+We advise to edit the filters **only by adding/removing locales** from the `localeFilter/whitelist` to avoid removing important data. We recommend not to remove "en-US" locale from the localeFilter/whitelist because it is used as a fallback. Removing it for when
+- `<PredefinedCulturesOnly>true</PredefinedCulturesOnly>`: results in `Encountered infinite recursion while looking for resource in System.Private.Corelib.` exception
+- when predefined cultures only is not set: results in resolving data from ICU's `root.txt` files, e.g. `CultureInfo.DateTimeFormat.GetDayName(DateTime.Today.DayOfWeek)` will return an abbreviated form: `Mon` instead of `Monday`.
+Removing specific feature data might result in an exception that starts with `[CultureData.IcuGetLocaleInfo(LocaleStringData)] Failed`. It means you removed data necessary to extract basic information about the locale.
+
+ In the file `eng/icu.mk`, you can choose what filters to build. Choose the platform:
+
+### Building for Browser:
+* For prerequisites run `.devcontainer/postCreateCommand.sh` (it is run automatically on creation if using Codespaces)
+* Building:
+    ```
+    ./build.sh /p:TargetOS=Browser /p:TargetArchitecture=wasm /p:IcuTracing=true
+    ```
+  Output is located in `artifacts/bin/icu-browser-wasm`.
+
+### Building for Mobiles:
+* For prerequisites run:
+    ```bash
+    export ANDROID_NDK_ROOT=$PWD/artifacts/ndk/
+    mkdir  $ANDROID_NDK_ROOT
+    wget https://dl.google.com/android/repository/android-ndk-r25b-linux.zip
+    unzip android-ndk-r25b-linux.zip -d $ANDROID_NDK_ROOT
+    rm android-ndk-r25b-linux.zip
+    mv $ANDROID_NDK_ROOT/*/* $ANDROID_NDK_ROOT
+    rmdir $ANDROID_NDK_ROOT/android-ndk-r25b
+    ```
+* Building:
+ ```bash
+  ./build.sh /p:TargetOS=Android /p:TargetArchitecture=x64 /p:IcuTracing=true
+  ```
+
+Output from both builds will be located in subdirectories of `artifacts/bin`. Copy the generated `.dat` files to a suitable location and provide the full path to it in the `.csproj`, e.g.:
+```xml
+<WasmIcuDataFileName>C:\Users\wasmUser\icuSources\customIcu.dat</WasmIcuDataFileName>
+```
+
+## Blazor
+
+In Blazor we are loading the file based on the applications's culture.
+To force the full data to be loaded, add this to your `.csproj`:
+```xml
+<BlazorWebAssemblyLoadAllGlobalizationData>true</BlazorWebAssemblyLoadAllGlobalizationData>
+```
+Custom files loading for Blazor is not possible.

eng/testing/scenarios/BuildWasmAppsJobsList.txt

@@ -4,16 +4,18 @@ Wasm.Build.NativeRebuild.Tests.OptimizationFlagChangeTests
 Wasm.Build.NativeRebuild.Tests.ReferenceNewAssemblyRebuildTest
 Wasm.Build.NativeRebuild.Tests.SimpleSourceChangeRebuildTest
 Wasm.Build.Tests.Blazor.BuildPublishTests
-Wasm.Build.Tests.Blazor.MiscTests2
 Wasm.Build.Tests.Blazor.MiscTests
+Wasm.Build.Tests.Blazor.MiscTests2
 Wasm.Build.Tests.Blazor.NativeRefTests
 Wasm.Build.Tests.BuildPublishTests
 Wasm.Build.Tests.CleanTests
 Wasm.Build.Tests.ConfigSrcTests
+Wasm.Build.Tests.IcuShardingTests
 Wasm.Build.Tests.InvariantGlobalizationTests
 Wasm.Build.Tests.MainWithArgsTests
 Wasm.Build.Tests.NativeBuildTests
 Wasm.Build.Tests.NativeLibraryTests
+Wasm.Build.Tests.NonWasmTemplateBuildTests
 Wasm.Build.Tests.PInvokeTableGeneratorTests
 Wasm.Build.Tests.RebuildTests
 Wasm.Build.Tests.SatelliteAssembliesTests

eng/testing/tests.browser.targets

@@ -8,6 +8,9 @@
   <PropertyGroup>
     <IsBrowserWasmProject Condition="'$(IsBrowserWasmProject)' == ''">true</IsBrowserWasmProject>
 
+    <!-- some tests require full ICU data, force it for all the tests -->
+    <WasmIncludeFullIcuData>true</WasmIncludeFullIcuData>
+
     <!-- set this when provisioning emsdk on CI -->
     <EMSDK_PATH Condition="'$(EMSDK_PATH)' == '' and '$(ContinuousIntegrationBuild)' == 'true' and '$(MonoProjectRoot)' != ''">$([MSBuild]::NormalizeDirectory($(MonoProjectRoot), 'wasm', 'emsdk'))</EMSDK_PATH>
 
@@ -184,6 +187,8 @@
       <_WasmPropertyNames Include="DisableParallelEmccCompile" />
       <_WasmPropertyNames Include="EmccCompileOptimizationFlag" />
       <_WasmPropertyNames Include="EmccLinkOptimizationFlag" />
+      <_WasmPropertyNames Include="WasmIncludeFullIcuData" />
+      <_WasmPropertyNames Include="WasmIcuDataFileName" />
     </ItemGroup>
   </Target>
 

src/mono/wasi/build/WasiApp.targets

@@ -319,27 +319,29 @@
 
   <Target Name="_GetWasiGenerateAppBundleDependencies">
     <PropertyGroup>
-      <WasmIcuDataFileName Condition="'$(InvariantGlobalization)' != 'true'">icudt.dat</WasmIcuDataFileName>
-
       <_HasDotnetWasm Condition="'%(WasmNativeAsset.FileName)%(WasmNativeAsset.Extension)' == 'dotnet.wasm'">true</_HasDotnetWasm>
     </PropertyGroup>
 
     <ItemGroup>
       <!-- If dotnet.{wasm,js} weren't added already (eg. AOT can add them), then add the default ones -->
       <WasmNativeAsset Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)dotnet.wasm" Condition="'$(_HasDotnetWasm)' != 'true'" />
-
-      <WasmNativeAsset Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)$(WasmIcuDataFileName)" Condition="'$(InvariantGlobalization)' != 'true'" />
       <WasmNativeAsset Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)dotnet.timezones.blat" />
-
       <WasmFilesToIncludeInFileSystem Include="@(WasmNativeAsset)" Condition="'%(WasmNativeAsset.FileName)%(WasmNativeAsset.Extension)' == 'dotnet.js.symbols'" />
     </ItemGroup>
+
+    <ItemGroup Condition="'$(InvariantGlobalization)' != 'true'">
+      <WasmIcuDataFileNames Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)icudt.dat"/>
+      <WasmNativeAsset Include="@(WasmIcuDataFileNames)"/>
+    </ItemGroup>
   </Target>
 
   <Target Name="_WasiGenerateAppBundle"
-          Inputs="@(_WasmAssembliesInternal);$(WasmIcuDataFileName);@(WasmNativeAsset)"
+          Inputs="@(_WasmAssembliesInternal);@(WasmNativeAsset)"
           Outputs="$(WasmAppDir)\.stamp"
           Condition="'$(WasmGenerateAppBundle)' == 'true'"
-          DependsOnTargets="_WasmGenerateRuntimeConfig;_GetWasiGenerateAppBundleDependencies;_WasiDefaultGenerateAppBundle;_GenerateRunWasmtimeScript" />
+          DependsOnTargets="_WasmGenerateRuntimeConfig;_GetWasiGenerateAppBundleDependencies;_WasiDefaultGenerateAppBundle;_GenerateRunWasmtimeScript">
+  </Target>
+
 
   <Target Name="_WasiDefaultGenerateAppBundle">
 
@@ -353,7 +355,7 @@
       DefaultHostConfig="$(DefaultWasmHostConfig)"
       InvariantGlobalization="$(InvariantGlobalization)"
       SatelliteAssemblies="@(_WasmSatelliteAssemblies)"
-      IcuDataFileName="$(WasmIcuDataFileName)"
+      IcuDataFileNames="@(WasmIcuDataFileNames)"
       ExtraFilesToDeploy="@(WasmExtraFilesToDeploy)"
       NativeAssets="@(WasmNativeAsset)"
       DebugLevel="$(WasmDebugLevel)"

src/mono/wasm/Wasm.Build.Tests/BuildTestBase.cs

@@ -30,6 +30,7 @@ public abstract class BuildTestBase : IClassFixture<SharedBuildPerTestClassFixtu
     {
         public const string DefaultTargetFramework = "net8.0";
         public const string DefaultTargetFrameworkForBlazor = "net8.0";
+        private const string DefaultEnvironmentLocale = "en-US";
         protected static readonly bool s_skipProjectCleanup;
         protected static readonly string s_xharnessRunnerCommand;
         protected string? _projectDir;
@@ -39,8 +40,13 @@ public abstract class BuildTestBase : IClassFixture<SharedBuildPerTestClassFixtu
         protected SharedBuildPerTestClassFixture _buildContext;
         protected string _nugetPackagesDir = string.Empty;
 
+        private static bool s_isWindows = RuntimeInformation.IsOSPlatform(OSPlatform.Windows);
+        // changing Windows's language programistically is complicated and Node is using OS's language to determine
+        // what is client's preferred locale and then to load corresponding ICU => skip automatic icu testing with Node
+        // on Linux sharding does not work because we rely on LANG env var to check locale and emcc is overwriting it
+        protected static RunHost s_hostsForOSLocaleSensitiveTests = RunHost.Chrome;
         // FIXME: use an envvar to override this
-        protected static int s_defaultPerTestTimeoutMs = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ? 30*60*1000 : 15*60*1000;
+        protected static int s_defaultPerTestTimeoutMs = s_isWindows ? 30*60*1000 : 15*60*1000;
         protected static BuildEnvironment s_buildEnv;
         private const string s_runtimePackPathPattern = "\\*\\* MicrosoftNetCoreAppRuntimePackDir : '([^ ']*)'";
         private const string s_nugetInsertionTag = "<!-- TEST_RESTORE_SOURCES_INSERTION_LINE -->";
@@ -139,7 +145,8 @@ protected string RunAndTestWasmApp(BuildArgs buildArgs,
                                            string targetFramework = DefaultTargetFramework,
                                            string? extraXHarnessMonoArgs = null,
                                            string? extraXHarnessArgs = null,
-                                           string jsRelativePath = "test-main.js")
+                                           string jsRelativePath = "test-main.js",
+                                           string environmentLocale = DefaultEnvironmentLocale)
         {
             buildDir ??= _projectDir;
             envVars ??= new();
@@ -157,16 +164,15 @@ protected string RunAndTestWasmApp(BuildArgs buildArgs,
             }
 
             bundleDir ??= Path.Combine(GetBinDir(baseDir: buildDir, config: buildArgs.Config, targetFramework: targetFramework), "AppBundle");
-            if (host is RunHost.V8 && OperatingSystem.IsWindows())
+            IHostRunner hostRunner = GetHostRunnerFromRunHost(host);
+            if (!hostRunner.CanRunWBT())
                 throw new InvalidOperationException("Running tests with V8 on windows isn't supported");
 
             // Use wasm-console.log to get the xharness output for non-browser cases
-            (string testCommand, string xharnessArgs, bool useWasmConsoleOutput) = host switch
-            {
-                RunHost.V8     => ("wasm test", $"--js-file={jsRelativePath} --engine=V8 -v trace", true),
-                RunHost.NodeJS => ("wasm test", $"--js-file={jsRelativePath} --engine=NodeJS -v trace", true),
-                _              => ("wasm test-browser", $"-v trace -b {host} --web-server-use-cop", false)
-            };
+            string testCommand = hostRunner.GetTestCommand();
+            XHarnessArgsOptions options = new XHarnessArgsOptions(jsRelativePath, environmentLocale, host);
+            string xharnessArgs = s_isWindows ? hostRunner.GetXharnessArgsWindowsOS(options) : hostRunner.GetXharnessArgsOtherOS(options);
+            bool useWasmConsoleOutput = hostRunner.UseWasmConsoleOutput();
 
             extraXHarnessArgs += " " + xharnessArgs;
 
@@ -336,7 +342,7 @@ protected static BuildArgs ExpandBuildArgs(BuildArgs buildArgs, string extraProp
             if (buildArgs.AOT)
             {
                 extraProperties = $"{extraProperties}\n<RunAOTCompilation>true</RunAOTCompilation>";
-                extraProperties += $"\n<EmccVerbose>{RuntimeInformation.IsOSPlatform(OSPlatform.Windows)}</EmccVerbose>\n";
+                extraProperties += $"\n<EmccVerbose>{s_isWindows}</EmccVerbose>\n";
             }
 
             if (UseWebcil) {
@@ -429,7 +435,8 @@ protected static BuildArgs ExpandBuildArgs(BuildArgs buildArgs, string extraProp
                                          options.MainJS ?? "test-main.js",
                                          options.HasV8Script,
                                          options.TargetFramework ?? DefaultTargetFramework,
-                                         options.HasIcudt,
+                                         options.GlobalizationMode,
+                                         options.PredefinedIcudt ?? "",
                                          options.DotnetWasmFromRuntimePack ?? !buildArgs.AOT,
                                          UseWebcil);
                 }
@@ -633,7 +640,8 @@ protected static void AssertBasicAppBundle(string bundleDir,
                                                    string mainJS,
                                                    bool hasV8Script,
                                                    string targetFramework,
-                                                   bool hasIcudt = true,
+                                                   GlobalizationMode? globalizationMode,
+                                                   string predefinedIcudt = "",
                                                    bool dotnetWasmFromRuntimePack = true,
                                                    bool useWebcil = true)
         {
@@ -648,7 +656,7 @@ protected static void AssertBasicAppBundle(string bundleDir,
             });
 
             AssertFilesExist(bundleDir, new[] { "run-v8.sh" }, expectToExist: hasV8Script);
-            AssertFilesExist(bundleDir, new[] { "icudt.dat" }, expectToExist: hasIcudt);
+            AssertIcuAssets();
 
             string managedDir = Path.Combine(bundleDir, "managed");
             string bundledMainAppAssembly =
@@ -670,6 +678,53 @@ protected static void AssertBasicAppBundle(string bundleDir,
             }
 
             AssertDotNetWasmJs(bundleDir, fromRuntimePack: dotnetWasmFromRuntimePack, targetFramework);
+
+            void AssertIcuAssets()
+            {
+                bool expectEFIGS = false;
+                bool expectCJK = false;
+                bool expectNOCJK = false;
+                bool expectFULL = false;
+                switch (globalizationMode)
+                {
+                    case GlobalizationMode.Invariant:
+                        break;
+                    case GlobalizationMode.FullIcu:
+                        expectFULL = true;
+                        break;
+                    case GlobalizationMode.PredefinedIcu:
+                        if (string.IsNullOrEmpty(predefinedIcudt))
+                            throw new ArgumentException("WasmBuildTest is invalid, value for predefinedIcudt is required when GlobalizationMode=PredefinedIcu.");
+                        AssertFilesExist(bundleDir, new[] { predefinedIcudt }, expectToExist: true);
+                        // predefined ICU name can be identical with the icu files from runtime pack
+                        switch (predefinedIcudt)
+                        {
+                            case "icudt.dat":
+                                expectFULL = true;
+                                break;
+                            case "icudt_EFIGS.dat":
+                                expectEFIGS = true;
+                                break;
+                            case "icudt_CJK.dat":
+                                expectCJK = true;
+                                break;
+                            case "icudt_no_CJK.dat":
+                                expectNOCJK = true;
+                                break;
+                        }
+                        break;
+                    default:
+                        // icu shard chosen based on the locale
+                        expectCJK = true;
+                        expectEFIGS = true;
+                        expectNOCJK = true;
+                        break;
+                }
+                AssertFilesExist(bundleDir, new[] { "icudt.dat" }, expectToExist: expectFULL);
+                AssertFilesExist(bundleDir, new[] { "icudt_EFIGS.dat" }, expectToExist: expectEFIGS);
+                AssertFilesExist(bundleDir, new[] { "icudt_CJK.dat" }, expectToExist: expectCJK);
+                AssertFilesExist(bundleDir, new[] { "icudt_no_CJK.dat" }, expectToExist: expectNOCJK);
+            }
         }
 
         protected static void AssertDotNetWasmJs(string bundleDir, bool fromRuntimePack, string targetFramework)
@@ -1093,6 +1148,13 @@ public static int Main()
                     return 42;
                 }
             }";
+
+        private IHostRunner GetHostRunnerFromRunHost(RunHost host) => host switch
+        {
+            RunHost.V8 => new V8HostRunner(),
+            RunHost.NodeJS => new NodeJSHostRunner(),
+            _ => new BrowserHostRunner(),
+        };
     }
 
     public record BuildArgs(string ProjectName,
@@ -1106,20 +1168,21 @@ internal record BuildPaths(string ObjWasmDir, string ObjDir, string BinDir, stri
 
     public record BuildProjectOptions
     (
-        Action? InitProject               = null,
-        bool?   DotnetWasmFromRuntimePack = null,
-        bool    HasIcudt                  = true,
-        bool    UseCache                  = true,
-        bool    ExpectSuccess             = true,
-        bool    AssertAppBundle           = true,
-        bool    CreateProject             = true,
-        bool    Publish                   = true,
-        bool    BuildOnlyAfterPublish     = true,
-        bool HasV8Script = true,
-        string? Verbosity                 = null,
-        string? Label                     = null,
-        string? TargetFramework           = null,
-        string? MainJS                    = null,
+        Action?             InitProject               = null,
+        bool?               DotnetWasmFromRuntimePack = null,
+        GlobalizationMode?  GlobalizationMode         = null,
+        string?             PredefinedIcudt           = null,
+        bool                UseCache                  = true,
+        bool                ExpectSuccess             = true,
+        bool                AssertAppBundle           = true,
+        bool                CreateProject             = true,
+        bool                Publish                   = true,
+        bool                BuildOnlyAfterPublish     = true,
+        bool                HasV8Script               = true,
+        string?             Verbosity                 = null,
+        string?             Label                     = null,
+        string?             TargetFramework           = null,
+        string?             MainJS                    = null,
         IDictionary<string, string>? ExtraBuildEnvironmentVariables = null
     );
 
@@ -1132,5 +1195,12 @@ public record BlazorBuildOptions
         bool WarnAsError = true
     );
 
+    public enum GlobalizationMode
+    {
+        Invariant,       // no icu
+        FullIcu,         // full icu data: icudt.dat is loaded
+        PredefinedIcu   // user set WasmIcuDataFileName value and we are loading that file
+    };
+
     public enum NativeFilesType { FromRuntimePack, Relinked, AOT };
 }