JIT and one-time source generated methods performance cost

[thomhurst]

Description

When leveraging source generation, it is sometimes required to register an object or some data at application startup. To do this, we generally have to emit a static method with a [ModuleInitializer] if we want this to happen automatically without the user having to know or do anything.

I do this in TUnit, and was emitting some metadata about the test and registering on application startup. I was triggerring the source generation by looking for a [Test] attribute, and so there would generally be 1 new emitted method for 1 test.

However for users with large test suites, this would mean for example 10k tests means 10k new emitted methods. This means it of course scales when test suites grow.

I (and many others I'm sure) was under the impression that source generated code will always be faster than reflection, and while I think that's true once an application is warmed up, I've found that this approach is actually slower due to the JIT compilations that need to happen on each of these new methods. And since these will only ever be called once, could we avoid the overhead?

I'm not even sure if there's a solution here - But I thought I'd raise this for awareness and maybe start a discussion.

I don't know much about compilers and how the runtime itself works so I don't know any solutions to propose or suggest.

My first thought was could we add some sort of flag to the [MethodImpl] attribute to tell the compiler that this is a one-time called method and could the compiler do some pre-optimisations to it to avoid the JIT costs during app start up?

Data

Approach	Median	Range
Source Generated (10K JIT)	~103ms	97-140ms
Reflection	~36ms	35-40ms

Here's a repo for the trivial benchmarking apps: https://github.com/thomhurst/JitOverheadBenchmark/tree/main

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[EgorBo]

I am not sure I understand what exactly you're suggesting, the fact that JIT was able to compile & execute 10000 methods in 103ms sounds very quick to me (maybe it could do it quicker speculatively by compiling in parallel but this scenario sounds very synthetic to me). It's obvious that Reflection would be quicker for this scenario.

My first thought was could we add some sort of flag to the [MethodImpl] attribute to tell the compiler that this is a one-time called method

There is no need, it's already handled by TieredCompilation. if you run your repro with DOTNET_JitDisasmSummary=1 you will see something like:

...
2191: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2188() [Tier0, IL size=1, code size=6]
2192: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2189() [Tier0, IL size=1, code size=6]
2193: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2190() [Tier0, IL size=1, code size=6]
2194: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2191() [Tier0, IL size=1, code size=6]
2195: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2192() [Tier0, IL size=1, code size=6]
2196: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2193() [Tier0, IL size=1, code size=6]
2197: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2194() [Tier0, IL size=1, code size=6]
2198: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2195() [Tier0, IL size=1, code size=6]
2199: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2196() [Tier0, IL size=1, code size=6]
2200: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2197() [Tier0, IL size=1, code size=6]
2201: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2198() [Tier0, IL size=1, code size=6]
2202: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2199() [Tier0, IL size=1, code size=6]
2203: JIT compiled SlowJitExample.Generated.TestRegistrations:Register_2200() [Tier0, IL size=1, code size=6]
...

It is possible it can be quicker if we ever integrate the interpreter as Tier0, but I bet reflection still will be faster in this case.

[EgorBo]

Btw, if you need a faster startup time, you should use PublishReadyToRun=true - it makes SG scenario 3-4x faster for me (so effectively faster than Reflection).

[jkotas]

source generated code will always be faster than reflection

That's true for a code that executes many times (e.g. invokes a method). If you have a code that executes once, it depends on what it does exactly.

this would mean for example 10k tests means 10k new emitted methods.

Executing 10k methods is always going to be relatively slow. Can you emit one method per assembly to register all 10k tests?

Depending on what types of queries you typically run against the registration database, you can also consider layered registration. For example, register all types (one source generated method per assembly) and register the methods in given type lazily only once they are needed (one source generated method per type).

[thomhurst]

Executing 10k methods is always going to be relatively slow. Can you emit one method per assembly to register all 10k tests?

This seems to be a trade-off with source generators. To do this, I'd have to Collect all usages, which kinda breaks incremental generation 😖

[jkotas]

The layered registration that I have suggested may be a good tradeoff between the incremental source generator performance and the runtime performance.

[teo-tsirpanis]

I'm starting to think that aiming to avoid the use of reflection under any circumstances is not without its disadvantages, and apart from the upfront initialization cost mentioned here, it can lead to effectively reimplementing what reflection has already been doing. One example I found for TUnit is this consolidated listing of all attributes of the assembly and the test class (IIUC), even though typeof(T).GetCustomAttributes() and typeof(T).Assembly.GetCustomAttributes() are supported in Native AOT¹:

private static global::System.Attribute[] __Attributes(int groupIndex)
{
    switch (groupIndex)
    {
        case 0:
        {
            return
            [
                new global::TUnit.Core.TestAttribute(),
                new global::System.Runtime.Versioning.TargetFrameworkAttribute(".NETCoreApp,Version=v10.0")
            {FrameworkDisplayName = ".NET 10.0",},
                new global::System.Reflection.AssemblyCompanyAttribute("SlowJitExample.TestProject"),
                new global::System.Reflection.AssemblyConfigurationAttribute("Debug"),
                new global::System.Reflection.AssemblyFileVersionAttribute("1.0.0.0"),
                new global::System.Reflection.AssemblyInformationalVersionAttribute("1.0.0+75d6fcc775e957fe3fb4c1a8533e91babd902173"),
                new global::System.Reflection.AssemblyProductAttribute("SlowJitExample.TestProject"),
                new global::System.Reflection.AssemblyTitleAttribute("SlowJitExample.TestProject"),
                new global::System.Reflection.AssemblyVersionAttribute("1.0.0.0"),
                new global::System.Reflection.AssemblyMetadataAttribute("Microsoft.Testing.Platform.Application", "true")
            ];
        }
        default:
            throw new global::System.ArgumentOutOfRangeException(nameof(groupIndex));
    }
}

The above snippet is generated once per test class. In a project with multiple test classes, it's not certain whether the increased code size would outweigh the performance winnings of avoiding reflection.

It has been suggested above that R2R or Native AOT can increase performance, but I'm not sure if it's a useful workaround for a testing framework. Even if users ran dotnet publish to build their tests, it's not certain that the total time of building and running tests during a development inner loop would be smaller.

@thomhurst I would recommend you to explore whether using more runtime reflection in TUnit would improve performance. You can use both runtime reflection and extensive source-generation if you find out that they are better for JIT and AOT environments respectively; one way would be with a feature switch that is on by default when $(PublishAot) == true.

---

This seems to be a trade-off with source generators. To do this, I'd have to Collect all usages, which kinda breaks incremental generation 😖

For each class with tests, you can emit something like [assembly:TestClass(typeof(MyClass))], where TestClassAttribute is defined like this:

[AttributeUsage(AttributeTargets.Assembly, AllowMultiple = true)]
public sealed class TestClassAttribute([DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicMethods | DynamicallyAccessedMemberTypes.PublicConstructors)] Type type) : Attribute
{
    [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicMethods | DynamicallyAccessedMemberTypes.PublicConstructors)]
    public Type TestClassType => type;
}

Then you can get all test classes of an assembly by calling assembly.GetAttributes<TestClassAttribute>().Select(x => x.TestClassType). If you want to avoid reflecting on MyClass, you can generate proxy classes that implement a factory interface that returns TestMetadata. This way you limit the amount of reflection to just constructing a class, and in both cases you avoid the use of module initializers.

Footnotes

1. Native AOT used to have a reflection-free mode, but has never been officially supported or documented, and was deleted in Delete reflection-free mode #109857. ↩