[Operational Issue] Crossgen crash

[ViktorHofer]

Build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2897935&view=logs&j=d20d6b9c-806c-57f3-6cb4-74f475b5613e&t=21cab82e-57e1-581d-a74d-6a14561af9f5&s=09ac2d0a-8250-5b89-9554-95ed439356a1

Job: osx-x64 (native)

  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error : Error: Unexpected CodeView data signature value. [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error : System.BadImageFormatException: Unexpected CodeView data signature value. [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at System.Reflection.PortableExecutable.PEReader.DecodeCodeViewDebugDirectoryData(AbstractMemoryBlock block) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at System.Reflection.PortableExecutable.PEReader.ReadCodeViewDebugDirectoryData(DebugDirectoryEntry entry) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.CompilerTypeSystemContext.OpenAssociatedSymbolFile(String peFilePath, PEReader peReader) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.CompilerTypeSystemContext.AddModule(String filePath, String expectedSimpleName, Boolean useForBinding, ModuleData oldModuleData, Boolean throwOnFailureToLoad) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.CompilerTypeSystemContext.GetModuleForSimpleName(String simpleName, Boolean throwIfNotFound) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaModule.ResolveAssemblyReference(AssemblyReferenceHandle handle) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaModule.EcmaObjectLookupHashtable.CreateValueFromKey(EntityHandle handle) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey key) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaModule.ResolveTypeReference(TypeReferenceHandle handle) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaModule.EcmaObjectLookupHashtable.CreateValueFromKey(EntityHandle handle) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey key) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaModule.GetObject(EntityHandle handle, NotFoundBehavior notFoundBehavior) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaSignatureParser.ResolveHandle(EntityHandle handle) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaSignatureParser.ParseTypeImpl(SignatureTypeCode typeCode) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaSignatureParser.ParseType() [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaSignatureParser.ParseFieldSignature(EmbeddedSignatureData[]& embeddedSigData) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.Ecma.EcmaField.InitializeFieldType() [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.MetadataFieldLayoutAlgorithm.ComputeInstanceLayout(DefType defType, InstanceLayoutKind layoutKind) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at Internal.TypeSystem.DefType.ComputeInstanceLayout(InstanceLayoutKind layoutKind) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(TypeDesc type) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.CompilerTypeSystemContext.EnsureLoadableType(TypeDesc type) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.CompilerTypeSystemContext.EnsureLoadableMethod(MethodDesc method) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.ReadyToRunLibraryRootProvider.CheckCanGenerateMethod(MethodDesc method) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.ReadyToRunLibraryRootProvider.RootMethods(MetadataType type, String reason, IRootingServiceProvider rootProvider) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.ReadyToRunLibraryRootProvider.AddCompilationRoots(IRootingServiceProvider rootProvider) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.Compilation..ctor(DependencyAnalyzerBase`1 dependencyGraph, NodeFactory nodeFactory, IEnumerable`1 compilationRoots, ILProvider ilProvider, DevirtualizationManager devirtualizationManager, IEnumerable`1 modulesBeingInstrumented, Logger logger, InstructionSetSupport instructionSetSupport) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.ReadyToRunCodegenCompilation..ctor(DependencyAnalyzerBase`1 dependencyGraph, NodeFactory nodeFactory, IEnumerable`1 roots, ILProvider ilProvider, Logger logger, DevirtualizationManager devirtualizationManager, IEnumerable`1 inputFiles, String compositeRootPath, InstructionSetSupport instructionSetSupport, Boolean resilient, Boolean generateMapFile, Boolean generateMapCsvFile, Boolean generatePdbFile, Func`2 printReproInstructions, String pdbPath, Boolean generatePerfMapFile, String perfMapPath, Int32 perfMapFormatVersion, Boolean generateProfileFile, Int32 parallelism, ProfileDataManager profileData, MethodLayoutAlgorithm methodLayoutAlgorithm, FileLayoutAlgorithm fileLayoutAlgorithm, Int32 customPESectionAlignment, Boolean verifyTypeAndFieldLayout, ReadyToRunContainerFormat format) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.ReadyToRunCodegenCompilationBuilder.ToCompilation() [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.Program.RunSingleCompilation(Dictionary`2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary`2 unrootedInputFilePaths, HashSet`1 versionBubbleModulesHash, ReadyToRunCompilerContext typeSystemContext, Logger logger) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.Program.Run() [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]
  /Users/runner/work/1/s/src/sdk/src/Layout/redist/targets/Crossgen.targets(171,5): error :    at ILCompiler.Crossgen2RootCommand.<>c__DisplayClass211_0.<.ctor>b__0(ParseResult result) [/Users/runner/work/1/s/src/sdk/src/Layout/redist/redist.csproj]

cc @dotnet/crossgen-contrib - Feel free to move the issue into runtime

I have seen that failure in a couple of builds over the last few days. In internal builds and usually real signed (which shouldn't matter as I assume that the crossgen used in dotnet/sdk is from the toolset SDK).

[steveisok]

@ViktorHofer have you seen any of these on the public side?

[ViktorHofer]

No, only on official builds which use macos-arm64

[ViktorHofer]

This one is showing up a lot in VMR builds. Would be great if someone could take a look.

[steveisok]

This one is showing up a lot in VMR builds. Would be great if someone could take a look.

I think we need a dump as a precondition. Have we got the VMR build to produce one yet?

[ViktorHofer]

Not yet.