Crossgen2 has been failing to load 'Microsoft.DiaSymReader.Native'

[BrennanConroy]

Description

Crossgen2 has started occasionally failing on aspnetcore CI builds. The earliest failure I found was on 5/5/22, but we've seen 10+ failures since then.

Reproduction Steps

Sadly, the most minimal steps I've been able to repro the issue with is run a CI build of the aspnetcore repo a couple times.

One odd thing I noticed is that when I capture a binlog while trying to reproduce the error I've never been able to repro the failure. Might imply some sort of race which doesn't happen when the machine is slower due to creating a binlog...

Expected behavior

crossgen2.exe to succeed 100% of the time

Actual behavior

Emitting R2R PE file: D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.App.Runtime\Release\net7.0\win-x64\Microsoft.AspNetCore.Mvc.ViewFeatures.dll
  Emitting PDB file: D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.App.Runtime\Release\net7.0\win-x64\Microsoft.AspNetCore.Mvc.ViewFeatures.ni.pdb
  Error: Unable to load native library 'Microsoft.DiaSymReader.Native' or one of its dependencies.
  System.DllNotFoundException: Unable to load native library 'Microsoft.DiaSymReader.Native' or one of its dependencies.
     at System.Runtime.InteropServices.NativeLibrary.LoadLibErrorTracker.Throw(String) + 0x69
     at Internal.Runtime.CompilerHelpers.InteropHelpers.FixupModuleCell(InteropHelpers.ModuleFixupCell*) + 0xdb
     at Internal.Runtime.CompilerHelpers.InteropHelpers.ResolvePInvokeSlow(InteropHelpers.MethodFixupCell*) + 0x2f
     at ILCompiler.Diagnostics.PdbWriter.CreateNGenPdbWriter(String, String, IntPtr&) + 0x52
     at ILCompiler.Diagnostics.PdbWriter.WritePDBDataHelper(String, IEnumerable`1) + 0x195
     at ILCompiler.Diagnostics.PdbWriter.WritePDBData(String, IEnumerable`1) + 0x1b
     at ILCompiler.DependencyAnalysis.ReadyToRunObjectWriter.EmitPortableExecutable() + 0xa30
     at ILCompiler.ReadyToRunCodegenCompilation.Compile(String) + 0x17a
     at ILCompiler.Program.RunSingleCompilation(Dictionary`2, InstructionSetSupport, String, Dictionary`2, HashSet`1, CompilerTypeSystemContext) + 0xff5
     at ILCompiler.Program.Run(String[]) + 0xd05
     at ILCompiler.Program.Main(String[]) + 0x32
D:\a\_work\1\s\src\Framework\App.Runtime\src\Microsoft.AspNetCore.App.Runtime.csproj(459,5): error MSB3073:
The command ""D:\a\_work\1\s\.packages\microsoft.netcore.app.crossgen2.win-x64\7.0.0-preview.5.22266.11\tools\crossgen2.exe" --targetarch:x64 --targetos:windows -O @"D:\a\_work\1\s\artifacts\obj\Microsoft.AspNetCore.App.Runtime\Release\net7.0\win-x64\crossgen\PlatformAssembliesPathsCrossgen2.rsp"
"-m:D:\a\_work\1\s\.packages\microsoft.netcore.app.runtime.win-x64\7.0.0-preview.5.22266.11\tools\StandardOptimizationData.mibc" --pdb --pdb-path:"D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.App.Runtime\Release\net7.0\win-x64\\"
-o:"D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.App.Runtime\Release\net7.0\win-x64\Microsoft.AspNetCore.Mvc.ViewFeatures.dll" "D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Mvc.ViewFeatures\Release\net7.0\Microsoft.AspNetCore.Mvc.ViewFeatures.dll"" exited with code 1.

Regression?

Yes, we haven't seen crossgen errors in a long time. And started seeing them ~5/5/22. Which is suspiciously close to when a change went in to switch crossgen to AOT: #67636

Known Workarounds

No response

Configuration

No response

Other information

No response

[MichalStrehovsky]

As a first course of action we need to fix this TODO in LoadLibErrorTracker to find out the exact problem with loading the DLL. It might be e.g. access denied, but we wouldn't currently know.

runtime/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/InteropServices/NativeLibrary.NativeAot.cs

Line 191 in b59c649

// TODO: copy the nice error logic from CoreCLR's LoadLibErrorTracker

I'll take a look at that today.

[MichalStrehovsky]

@agocke's comment in dotnet/aspnetcore#41891 (comment) pointed me in the right direction (there's no Microsoft.DiaSymReader.Native on disk) - I think once #69842 is merged, we would just see "module not found". Andy's comment saved us time waiting for that error message and digging into why Windows can't find the file.

The problem is this:

using System;
using System.Reflection;
using System.Runtime.InteropServices;

class Program
{
    [DllImport("none")]
    static extern void None();

    static void Register()
    {
        NativeLibrary.SetDllImportResolver(typeof(Program).Assembly, DllImportResolver);
    }

    private static IntPtr DllImportResolver(string libraryName, Assembly assembly, DllImportSearchPath? searchPath)
    {
        Console.WriteLine("Resolving...");
        return IntPtr.Zero;
    }

    static void Main()
    {
        Register();
        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();
        None();
    }
}

The above program never prints "Resolving" because we collect the System.Reflection.Assembly instance that was associated with the resolver. Assembly instances have a different lifetime on NativeAOT (they can be collected) vs on CoreCLR (we keep them forever, or until unload of the assembly load context, I guess).

We cannot keep these in a ConditionalWeakTable on NativeAOT or they'll be collected too early:

runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/NativeLibrary.cs

Lines 181 to 187 in cf36aae

	/// <summary>
	/// Map from assembly to native-library resolver.
	/// Interop specific fields and properties are generally not added to Assembly class.
	/// Therefore, this table uses weak assembly pointers to indirectly achieve
	/// similar behavior.
	/// </summary>
	private static ConditionalWeakTable<Assembly, DllImportResolver>? s_nativeDllResolveMap;

[jkotas]

Can we make runtime Assembly objects non-collectible, just like they are in CoreCLR? There is very likely other code out there that makes this same assumption about the lifetimes.