System.Private.CoreLib size regression when compiled with Crossgen2 compared to native Crossgen

[trylek]

As @am11 pointed out in the PR thread #44618, both Windows and Linux x64 SPC compiled with Crossgen2 is about 900K longer than Crossgen1. The statistics section of R2RDump --diff for the two libraries is as follows:

Left file:  D:\triage\SPC-size\CG1\System.Private.CoreLib.dll (9334784 B)
Right file: D:\triage\SPC-size\CG2\System.Private.CoreLib.dll (10193408 B)

 LEFT_SIZE RIGHT_SIZE       DIFF  PE sections (3 ELEMENTS)
----------------------------------------------------------
    143872     388608     244736  .data
   9163776    9769472     605696  .text
     26624      34816       8192  .reloc
   9334272   10192896     858624  

 LEFT_SIZE RIGHT_SIZE       DIFF  R2R sections (13 ELEMENTS)
------------------------------------------------------------
        23         28          5  CompilerIdentifier
       120        120             ImportSections
    368304     344532     -23772  RuntimeFunctions
    123504     220701      97197  MethodDefEntryPoints
      8848       8800        -48  ExceptionInfo
    807230     782522     -24708  DebugInfo
        33     386109     386076  DelayLoadMethodCallThunks
     11525      11525             AvailableTypes
    187472     240489      53017  InstanceMethodEntryPoints
     96140                -96140  InliningInfo
     16384      16384             AttributePresence
                  228        228  ManifestMetadata
               104880     104880  InliningInfo2
   1619583    2116318     496735  

LEFT COUNT | RIGHT COUNT | LEFT SIZE | RIGHT SIZE | R2R methods
---------------------------------------------------------------
      3173 |        --- |     221002 |        --- | LEFT METHODS
       --- |       1090 |        --- |      76176 | RIGHT METHODS
     26263 |      26263 |    2994580 |    2996609 | COMMON METHODS
     29436 |      27353 |    3215582 |    3072785 | ALL METHODS

(This includes some of my local improvements in R2RDump not yet merged in, I'll send out a separate PR for them.) According to the diff, slightly less than half of the size diff is due to the DelayLoadMethodThunks section that is vastly longer with CG2 compared to CG1. @nattress implemented the section in #39063, we need to find out why CG1 uses such a much more compact encoding - perhaps something like putting all thunks next to each other and representing them with a single interval. Also there's still more than 400K unaccounted for, I was originally thinking about methods but after improving their accounting in R2RDump it looks like the CG2 methods are actually slightly shorter than CG1.

[trylek]

/cc @dotnet/crossgen-contrib

[nattress]

The DelayLoadMethodCallThunks size discrepancy is due to Crossgen1 having an optimization to use VTable dispatch for virtual (non-interface) dispatch if the callee is in S.P.CoreLib and the caller is in the same version bubble as S.P.CoreLib.

Supporting this in Crossgen2 requires some scheme where we specify the VTable from Crossgen2 to the runtime (or otherwise guarantee they'll line up). This would be a benefit of implementing #45066.

[jkotas]

DelayLoadMethodCallThunks size discrepancy

Why are we not able to share these thunks accross all callsites?

[trylek]

I think that Crossgen2 currently adds call site identification to method call import cell identity so that we indeed create separate import cells / thunks per call site. I think I originally implemented it like this based on one of your older design docs where you mentioned it as a potential perf optimization (because in case of virtual dispatch the import cells for different call sites can converge to different callees); it's definitely a difference in comparison with CG1 and I guess we can either remove it altogether or at least put it under some aggressive perf optimization switch.

[jkotas]

Yes, we should match what CG1 does.

The cell duplication would only explain the difference in the .data section. Even when the cells are duplicated, the signatures and thunks should be shared between all cells with the same signature. We seem to have a problem with de-duplicating signatures and thunks.

[nattress]

Removing the thunk per-callsite duplicates yields a 160KB size reduction for S.P.C. There are still 300kb of thunks. I'll look into the signatures and verify we're properly sharing those.

 LEFT_SIZE RIGHT_SIZE       DIFF  PE sections by name (3 ELEMENTS)
------------------------------------------------------------------
    546816     492544     -54272  .data
     35328      26112      -9216  .reloc
  10028032    9932288     -95744  .text
  10610176   10450944    -159232  

 LEFT_SIZE RIGHT_SIZE       DIFF  R2R sections by delta (12 ELEMENTS)
---------------------------------------------------------------------
        28         28             CompilerIdentifier
       120        120             ImportSections
    358008     358008             RuntimeFunctions
    297781     297781             MethodDefEntryPoints
      8816       8816             ExceptionInfo
    791722     791722             DebugInfo
     11532      11532             AvailableTypes
    308595     308595             InstanceMethodEntryPoints
       228        228             ManifestMetadata
     16384      16384             AttributePresence
    105013     105013             InliningInfo2
    392301     301541     -90760  DelayLoadMethodCallThunks
   2290528    2199768     -90760  

 LEFT_SIZE RIGHT_SIZE       DIFF  R2R sections by name (12 ELEMENTS)
--------------------------------------------------------------------
     16384      16384             AttributePresence
     11532      11532             AvailableTypes
        28         28             CompilerIdentifier
    791722     791722             DebugInfo
    392301     301541     -90760  DelayLoadMethodCallThunks
      8816       8816             ExceptionInfo
       120        120             ImportSections
    105013     105013             InliningInfo2
    308595     308595             InstanceMethodEntryPoints
       228        228             ManifestMetadata
    297781     297781             MethodDefEntryPoints
    358008     358008             RuntimeFunctions
   2290528    2199768     -90760  

[jkotas]

There are still 300kb of thunks

This is still a lot. It should be <100 bytes, like with the CG1.

[am11]

Stats from current master suggest that we are getting closer (268288 bytes diff).

$ dotnet artifacts/bin//coreclr/OSX.x64.Release/R2RDump/R2RDump.dll --diff -i \
      /tmp/crossgen1-artifacts/System.Private.CoreLib.dll \
      /tmp/crossgen2-artifacts/System.Private.CoreLib.dll

Left file:  /tmp/crossgen1-artifacts/System.Private.CoreLib.dll (9152512 B)
Right file: /tmp/crossgen2-artifacts/System.Private.CoreLib.dll (9420800 B)

 LEFT_SIZE RIGHT_SIZE       DIFF  PE sections by delta (3 ELEMENTS)
-------------------------------------------------------------------
    141312    1977344    1836032  .data
     26112      25088      -1024  .reloc
   8984576    7417856   -1566720  .text
   9152000    9420288     268288  <TOTAL>

 LEFT_SIZE RIGHT_SIZE       DIFF  PE sections by name (3 ELEMENTS)
------------------------------------------------------------------
    141312    1977344    1836032  .data
     26112      25088      -1024  .reloc
   8984576    7417856   -1566720  .text
   9152000    9420288     268288  <TOTAL>

 LEFT_SIZE RIGHT_SIZE       DIFF  R2R sections by delta (13 ELEMENTS)
---------------------------------------------------------------------
               102850     102850  InliningInfo2
    121266     171158      49892  MethodDefEntryPoints
    194939     233939      39000  InstanceMethodEntryPoints
                  228        228  ManifestMetadata
        33        113         80  DelayLoadMethodCallThunks
        23         28          5  CompilerIdentifier
       120        120             ImportSections
     11117      11117             AvailableTypes
     16384      16384             AttributePresence
      8624       8560        -64  ExceptionInfo
    368352     342396     -25956  RuntimeFunctions
    791801     761881     -29920  DebugInfo
     94527                -94527  InliningInfo
   1607186    1648774      41588  <TOTAL>

 LEFT_SIZE RIGHT_SIZE       DIFF  R2R sections by name (13 ELEMENTS)
--------------------------------------------------------------------
     16384      16384             AttributePresence
     11117      11117             AvailableTypes
        23         28          5  CompilerIdentifier
    791801     761881     -29920  DebugInfo
        33        113         80  DelayLoadMethodCallThunks
      8624       8560        -64  ExceptionInfo
       120        120             ImportSections
     94527                -94527  InliningInfo
               102850     102850  InliningInfo2
    194939     233939      39000  InstanceMethodEntryPoints
                  228        228  ManifestMetadata
    121266     171158      49892  MethodDefEntryPoints
    368352     342396     -25956  RuntimeFunctions
   1607186    1648774      41588  <TOTAL>
# snip

[trylek]

I believe Simon made further changes so that CG2 SPC is actually smaller than the one compiled with Crossgen1. @am11, could you please confirm this so that we can close this issue as fixed?

[am11]

@trylek, I have tested with 366cd23. It looks like the difference has rather increased from 268288 to 287232 bytes since #45286 (comment).

Breakdown

$ ./dotnet.sh artifacts/bin/coreclr/OSX.x64.Release/R2RDump/R2RDump.dll --diff -i \
  /tmp/crossgen1-artifacts/System.Private.CoreLib.dll \
  /tmp/crossgen2-artifacts/System.Private.CoreLib.dll

Left file:  /tmp/crossgen1-artifacts/System.Private.CoreLib.dll (9139712 B)
Right file: /tmp/crossgen2-artifacts/System.Private.CoreLib.dll (9426944 B)

 LEFT_SIZE RIGHT_SIZE       DIFF  PE sections by delta (3 ELEMENTS)
-------------------------------------------------------------------
    140288    2315776    2175488  .data
     26624      26112       -512  .reloc
   8972288    7084544   -1887744  .text
   9139200    9426432     287232  <TOTAL>

 LEFT_SIZE RIGHT_SIZE       DIFF  PE sections by name (3 ELEMENTS)
------------------------------------------------------------------
    140288    2315776    2175488  .data
     26624      26112       -512  .reloc
   8972288    7084544   -1887744  .text
   9139200    9426432     287232  <TOTAL>

 LEFT_SIZE RIGHT_SIZE       DIFF  R2R sections by delta (15 ELEMENTS)
---------------------------------------------------------------------
               425070     425070  PgoInstrumentationData
               103076     103076  InliningInfo2
    788913     812626      23713  DebugInfo
      8648       8880        232  ExceptionInfo
                  228        228  ManifestMetadata
        33        113         80  DelayLoadMethodCallThunks
        23         28          5  CompilerIdentifier
       120        120             ImportSections
     11252      11252             AvailableTypes
     16384      16384             AttributePresence
                    0             ManifestAssemblyMvids
    122773     109492     -13281  MethodDefEntryPoints
    194058     167507     -26551  InstanceMethodEntryPoints
    370812     319476     -51336  RuntimeFunctions
     95739                -95739  InliningInfo
   1608755    1974252     365497  <TOTAL>

 LEFT_SIZE RIGHT_SIZE       DIFF  R2R sections by name (15 ELEMENTS)
--------------------------------------------------------------------
     16384      16384             AttributePresence
     11252      11252             AvailableTypes
        23         28          5  CompilerIdentifier
    788913     812626      23713  DebugInfo
        33        113         80  DelayLoadMethodCallThunks
      8648       8880        232  ExceptionInfo
       120        120             ImportSections
     95739                -95739  InliningInfo
               103076     103076  InliningInfo2
    194058     167507     -26551  InstanceMethodEntryPoints
                    0             ManifestAssemblyMvids
                  228        228  ManifestMetadata
    122773     109492     -13281  MethodDefEntryPoints
               425070     425070  PgoInstrumentationData
    370812     319476     -51336  RuntimeFunctions
   1608755    1974252     365497  <TOTAL>
# snip