Add perfmap debug directory entry to crossgen2 output as needed

docs/design/coreclr/botr/r2r-perfmap-format.md

@@ -0,0 +1,57 @@
+# Ready to run PerfMap format
+
+Traditionally in .NET symbols have been described using PDBs. These are used to map IL to source lines for code that the JIT will compile. The JIT usually emits the data that can then map from IL to a native address for symbolication purposes.
+
+Ready to run, however, avoids this IL to native code translation at runtime. For this reason, tools that emit R2R images often need to emit auxiliary artifacts to facilitate the mapping between source and native addresses. The Ready to Run PerfMap format describes such one map - where any method in the source code is associated with a region within the R2R image. That way any region from such image that gets executed can be linked back to a method at the source level. This facilitates tasks like stack symbolication for performance oriented investigations, although it is not appropriate to aid in tasks such as debugging at the source line level.
+
+## Version 1
+
+R2R PerfMaps of version 1 are usually found in files with the extension `.ni.r2rmap`. It's a plain text UTF-8 format where each entry is on a separate line. Each entry is composed of a triplet: an offset relative to the beginning of the image, a length, and a name. The file is laid out in the following as follows.
+
+### Header
+
+The header leads the file and is composed by special entries. Each entry contains a 4 byte integer token in place of an RVA signifying the type of information in the entry, a length that is always 0, and the entry data. The entries are emitted in the following order.
+
+| Token      | Description                                                           |
+|:-----------|-----------------------------------------------------------------------|
+| 0xFFFFFFFF | A 16 byte sequence representing a signature to correlate the perfmap with the r2r image. |
+| 0xFFFFFFFE | The version of the perfmap being emitted as a unsigned 4 byte integer. |
+| 0xFFFFFFFD | An unsigned 4 byte unsigned integer representing the OS the image targets. See [enumerables section](#enumerables-used-in-headers)  |
+| 0xFFFFFFFC | An unsigned 4 byte unsigned integer representing the architecture the image targets. See [enumerables section](#enumerables-used-in-headers) |
+| 0xFFFFFFFB | An unsigned 4 byte unsigned integer representing the ABI of the image. See [enumerables section](#enumerables-used-in-headers) |
+
+These entries contain information about the compilation that can be useful to tools and identifiers that can be used to correlate a perfmap with an image as described in ["Ready to Run format - debug directory entries"](./readytorun-format.md#additions-to-the-debug-directory).
+
+
+### Content
+
+Each entry is a triplet - the relative address of a method with respect to the image start as an unsigned 4 byte integer, the number of bytes used by the native code represented by an unsigned 2 byte integer, and the name of the method. There's one entry per line after the header, and a method can appear more than once since if may have gone through cold/hot path splitting.
+
+## Enumerables used in headers.
+
+```
+PerfMapArchitectureToken
+    Unknown = 0,
+    ARM     = 1,
+    ARM64   = 2,
+    X64     = 3,
+    X86     = 4,
+```
+
+```
+PerfMapOSToken
+    Unknown     = 0,
+    Windows     = 1,
+    Linux       = 2,
+    OSX         = 3,
+    FreeBSD     = 4,
+    NetBSD      = 5,
+    SunOS       = 6,
+```
+
+```
+PerfMapAbiToken
+    Unknown = 0,
+    Default = 1,
+    Armel = 2,
+```

docs/design/coreclr/botr/readytorun-format.md

@@ -44,6 +44,13 @@ without MSIL embedding are copied to the output folder next to the composite R2R
 and are rewritten by the compiler to include a formal ReadyToRun header with forwarding
 information pointing to the owner composite R2R executable (section `OwnerCompositeExecutable`).
 
+# Additions to the debug directory
+
+Currently shipping PE envelopes - both single-file and composite - can contain records for additional
+debug information in the debug directory. One such entry specific to R2R images is the one for R2R PerfMaps.
+The format of the auxiliary file is described [R2R perfmap format](./r2r-perfmap-format.md) and the corresponding
+debug directory entry is described in [PE COFF](../../../design/specs/PE-COFF.md#r2r-perfmap-debug-directory-entry-type-21).
+
 ## Future Improvements
 
 The limitations of the current format are:

docs/design/specs/PE-COFF.md

@@ -107,3 +107,15 @@ When validating that Windows PDB matches the debug directory record check that t
 
 > Note that when the debugger (or other tool) searches for the PDB only GUID and Age fields are used to match the PDB, but the timestamp of the CodeView debug directory entry does not need to match the timestamp stored in the PDB. Therefore, to verify byte-for-byte identity of the PDB, the timestamp field should also be checked.
 
+### R2R PerfMap Debug Directory Entry (type 21)
+
+Declares that the image has an associated PerfMap file containing a table mapping symbols to offsets for ready to run compilations.
+
+*Version Major=0x0001, Minor=0x0000* of the entry data format is following:
+
+| Offset | Size | Field             | Description                                                           |
+|:-------|:-----|:------------------|-----------------------------------------------------------------------|
+| 0      | 4    | Magic             | 0x52 0x32 0x52 0x4D (ASCII string: "R2RM")                            |
+| 4      | 16   | Signature         | Byte sequence uniquely identifying the associated PerfMap             |
+| 20     | 4    | Version           | Version number of the PerfMap. Currently only version 1 is supported. |
+| 24     |      | Path              | UTF-8 NUL-terminated path to the associated `.r2rmap` file.           |

src/coreclr/crossgen-corelib.proj

@@ -97,7 +97,6 @@
       <CrossGenDllCmd>$(CrossGenDllCmd) -o:$(CoreLibOutputPath)</CrossGenDllCmd>
       <CrossGenDllCmd>$(CrossGenDllCmd) -r:$([MSBuild]::NormalizePath('$(BinDir)', 'IL', '*.dll'))</CrossGenDllCmd>
       <CrossGenDllCmd>$(CrossGenDllCmd) --targetarch:$(TargetArchitecture)</CrossGenDllCmd>
-      <CrossGenDllCmd>$(CrossGenDllCmd) --perfmap-format-version:1</CrossGenDllCmd>
       <MibcArgs>@(OptimizationMibcFiles->'-m:$(MergedMibcPath)', ' ')</MibcArgs>
       <CrossGenDllCmd Condition="'$(UsingToolIbcOptimization)' != 'true' and '$(EnableNgenOptimization)' == 'true'">$(CrossGenDllCmd) $(MibcArgs) --embed-pgo-data</CrossGenDllCmd>
       <CrossGenDllCmd>$(CrossGenDllCmd) -O</CrossGenDllCmd>
@@ -110,6 +109,7 @@
     </PropertyGroup>
 
     <PropertyGroup Condition="$(BuildPerfMap)">
+      <CrossGenDllCmd>$(CrossGenDllCmd) --perfmap-format-version:1</CrossGenDllCmd>
       <CrossGenDllCmd>$(CrossGenDllCmd) --perfmap --perfmap-path:$(BinDir)</CrossGenDllCmd>
     </PropertyGroup>
 

src/coreclr/tools/aot/ILCompiler.Diagnostics/PerfMapWriter.cs

@@ -7,6 +7,7 @@
 using System.Linq;
 using System.Security.Cryptography;
 
+using Internal.ReadyToRunDiagnosticsConstants;
 using Internal.TypeSystem;
 
 namespace ILCompiler.Diagnostics
@@ -16,14 +17,8 @@ public class PerfMapWriter
         public const int LegacyCrossgen1FormatVersion = 0;
 
         public const int CurrentFormatVersion = 1;
-
-        public enum PseudoRVA : uint
-        {
-            OutputGuid         = 0xFFFFFFFF,
-            TargetOS           = 0xFFFFFFFE,
-            TargetArchitecture = 0xFFFFFFFD,
-            FormatVersion      = 0xFFFFFFFC,
-        }
+
+        const int HeaderEntriesPseudoLength = 0;
 
         private TextWriter _writer;
 
@@ -32,7 +27,7 @@ private PerfMapWriter(TextWriter writer)
             _writer = writer;
         }
 
-        public static void Write(string perfMapFileName, int perfMapFormatVersion, IEnumerable<MethodInfo> methods, IEnumerable<AssemblyInfo> inputAssemblies, TargetOS targetOS, TargetArchitecture targetArch)
+        public static void Write(string perfMapFileName, int perfMapFormatVersion, IEnumerable<MethodInfo> methods, IEnumerable<AssemblyInfo> inputAssemblies, TargetDetails details)
         {
             if (perfMapFormatVersion > CurrentFormatVersion)
             {
@@ -41,22 +36,10 @@ public static void Write(string perfMapFileName, int perfMapFormatVersion, IEnum
 
             using (TextWriter writer = new StreamWriter(perfMapFileName))
             {
-                IEnumerable<AssemblyInfo> orderedInputs = inputAssemblies.OrderBy(asm => asm.Name, StringComparer.OrdinalIgnoreCase);
 
                 PerfMapWriter perfMapWriter = new PerfMapWriter(writer);
-
-                List<byte> inputHash = new List<byte>();
-                foreach (AssemblyInfo inputAssembly in orderedInputs)
-                {
-                    inputHash.AddRange(inputAssembly.Mvid.ToByteArray());
-                }
-                inputHash.Add((byte)targetOS);
-                inputHash.Add((byte)targetArch);
-                Guid outputGuid = new Guid(MD5.HashData(inputHash.ToArray()));
-                perfMapWriter.WriteLine(outputGuid.ToString(), (uint)PseudoRVA.OutputGuid, 0);
-                perfMapWriter.WriteLine(targetOS.ToString(), (uint)PseudoRVA.TargetOS, 0);
-                perfMapWriter.WriteLine(targetArch.ToString(), (uint)PseudoRVA.TargetArchitecture, 0);
-                perfMapWriter.WriteLine(CurrentFormatVersion.ToString(), (uint)PseudoRVA.FormatVersion, 0);
+                byte[] signature = PerfMapV1SignatureHelper(inputAssemblies, details);
+                WritePerfMapV1Header(inputAssemblies, details, perfMapWriter);
 
                 foreach (MethodInfo methodInfo in methods)
                 {
@@ -72,6 +55,92 @@ public static void Write(string perfMapFileName, int perfMapFormatVersion, IEnum
             }
         }
 
+        private static void WritePerfMapV1Header(IEnumerable<AssemblyInfo> inputAssemblies, TargetDetails details, PerfMapWriter perfMapWriter)
+        {
+            byte[] signature = PerfMapV1SignatureHelper(inputAssemblies, details);
+
+            // Make sure these get emitted in this order, other tools in the ecosystem like the symbol uploader and PerfView rely on this.
+            // In particular, the order of it. Append only.
+            string signatureFormatted = Convert.ToHexString(signature);
+
+            PerfmapTokensForTarget targetTokens = TranslateTargetDetailsToPerfmapConstants(details);
+
+            perfMapWriter.WriteLine(signatureFormatted, (uint)PerfMapPseudoRVAToken.OutputSignature, HeaderEntriesPseudoLength);
+            perfMapWriter.WriteLine(CurrentFormatVersion.ToString(), (uint)PerfMapPseudoRVAToken.FormatVersion, HeaderEntriesPseudoLength);
+            perfMapWriter.WriteLine(((uint)targetTokens.OperatingSystem).ToString(), (uint)PerfMapPseudoRVAToken.TargetOS, HeaderEntriesPseudoLength);
+            perfMapWriter.WriteLine(((uint)targetTokens.Architecture).ToString(), (uint)PerfMapPseudoRVAToken.TargetArchitecture, HeaderEntriesPseudoLength);
+            perfMapWriter.WriteLine(((uint)targetTokens.Abi).ToString(), (uint)PerfMapPseudoRVAToken.TargetABI, HeaderEntriesPseudoLength);
+        }
+
+        public static byte[] PerfMapV1SignatureHelper(IEnumerable<AssemblyInfo> inputAssemblies, TargetDetails details)
+        {
+            IEnumerable<AssemblyInfo> orderedInputs = inputAssemblies.OrderBy(asm => asm.Name, StringComparer.OrdinalIgnoreCase);
+            List<byte> inputHash = new List<byte>();
+            foreach (AssemblyInfo inputAssembly in orderedInputs)
+            {
+                inputHash.AddRange(inputAssembly.Mvid.ToByteArray());
+            }
+
+            PerfmapTokensForTarget targetTokens = TranslateTargetDetailsToPerfmapConstants(details);
+
+            byte[] buffer = new byte[12];
+            if (!BitConverter.TryWriteBytes(buffer.AsSpan(0, sizeof(uint)), (uint)targetTokens.OperatingSystem)
+                || !BitConverter.TryWriteBytes(buffer.AsSpan(4, sizeof(uint)), (uint)targetTokens.Architecture)
+                || !BitConverter.TryWriteBytes(buffer.AsSpan(8, sizeof(uint)), (uint)targetTokens.Abi))
+            {
+                throw new InvalidOperationException();
+            }
+
+            if (!BitConverter.IsLittleEndian)
+            {
+                buffer.AsSpan(0, sizeof(uint)).Reverse();
+                buffer.AsSpan(4, sizeof(uint)).Reverse();
+                buffer.AsSpan(8, sizeof(uint)).Reverse();
+            }
+
+            inputHash.AddRange(buffer);
+            byte[] hash = MD5.HashData(inputHash.ToArray());
+
+            return hash;
+        }
+
+        internal record struct PerfmapTokensForTarget(PerfMapOSToken OperatingSystem, PerfMapArchitectureToken Architecture, PerfMapAbiToken Abi);
+
+        private static PerfmapTokensForTarget TranslateTargetDetailsToPerfmapConstants(TargetDetails details)
+        {
+            PerfMapOSToken osToken = details.OperatingSystem switch
+            {
+                TargetOS.Unknown => PerfMapOSToken.Unknown,
+                TargetOS.Windows => PerfMapOSToken.Windows,
+                TargetOS.Linux => PerfMapOSToken.Linux,
+                TargetOS.OSX => PerfMapOSToken.OSX,
+                TargetOS.FreeBSD => PerfMapOSToken.FreeBSD,
+                TargetOS.NetBSD => PerfMapOSToken.NetBSD,
+                TargetOS.SunOS => PerfMapOSToken.SunOS,
+                _ => throw new NotImplementedException(details.OperatingSystem.ToString())
+            };
+
+            PerfMapAbiToken abiToken = details.Abi switch
+            {
+                TargetAbi.Unknown => PerfMapAbiToken.Unknown,
+                TargetAbi.CoreRT => PerfMapAbiToken.Default,
+                TargetAbi.CoreRTArmel => PerfMapAbiToken.Armel,
+                _ => throw new NotImplementedException(details.Abi.ToString())
+            };
+
+            PerfMapArchitectureToken archToken = details.Architecture switch
+            {
+                TargetArchitecture.Unknown => PerfMapArchitectureToken.Unknown,
+                TargetArchitecture.ARM => PerfMapArchitectureToken.ARM,
+                TargetArchitecture.ARM64 => PerfMapArchitectureToken.ARM64,
+                TargetArchitecture.X64 => PerfMapArchitectureToken.X64,
+                TargetArchitecture.X86 => PerfMapArchitectureToken.X86,
+                _ => throw new NotImplementedException(details.Architecture.ToString())
+            };
+
+            return new PerfmapTokensForTarget(osToken, archToken, abiToken);
+        }
+
         private void WriteLine(string methodName, uint rva, uint length)
         {
             _writer.WriteLine($@"{rva:X8} {length:X2} {methodName}");

src/coreclr/tools/aot/ILCompiler.Diagnostics/ReadyToRunDiagnosticsConstants.cs

@@ -0,0 +1,40 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+namespace Internal.ReadyToRunDiagnosticsConstants;
+
+public enum PerfMapPseudoRVAToken : uint
+{
+    OutputSignature = 0xFFFFFFFF,
+    FormatVersion = 0xFFFFFFFE,
+    TargetOS = 0xFFFFFFFD,
+    TargetArchitecture = 0xFFFFFFFC,
+    TargetABI = 0xFFFFFFFB,
+}
+
+public enum PerfMapArchitectureToken : uint
+{
+    Unknown = 0,
+    ARM = 1,
+    ARM64 = 2,
+    X64 = 3,
+    X86 = 4,
+}
+
+public enum PerfMapOSToken : uint
+{
+    Unknown = 0,
+    Windows = 1,
+    Linux = 2,
+    OSX = 3,
+    FreeBSD = 4,
+    NetBSD = 5,
+    SunOS = 6,
+}
+
+public enum PerfMapAbiToken : uint
+{
+    Unknown = 0,
+    Default = 1,
+    Armel = 2,
+}