Performance regression in .NET Core 3.1 when awaiting Task that completes synchronously?

[CodeBlanch]

Consider this simple code:

using System.Threading.Tasks;

namespace Benchmarks
{
    internal static class Program
    {
        public static void Main(string[] args)
        {
            TestClassBenchmark().GetAwaiter().GetResult();
        }

        public static async Task TestClassBenchmark()
        {
            for (int i = 0; i < 5000; i++)
            {
                await WorkerNoParameter();
            }
        }

        private static async Task WorkerNoParameter()
        {
            await NothingWorker();
        }

        private static Task NothingWorker()
        {
            return Task.CompletedTask;
        }
    }
}

Basically a bunch of awaits but everything is really completing synchronously.

That program when run on .NET Core 3.1 allocates the async state machine 5000 times:

Same test run on net462 and netcoreapp2.1 do not allocate at all, which was the expected result.

Hopefully I'm mistaken on this and someone can point me in the right direction, but I thought it was worth running up the flag pole just in case.

[benaadams]

/cc @stephentoub

[stephentoub]

Are you building debug rather than release? The state machine generated by Roslyn is a class rather than struct in debug builds.

[stephentoub]

If you are in fact building release, you might also try disabling tiered compilation (set COMPlus_TieredCompilation=0). It's possible you're measuring code without the JIT's boxing optimizations applied, with the whole test running before the runtime gets a chance to recompile at a higher tier.

[CodeBlanch]

@stephentoub Yes I am using release builds. Adding...

<TieredCompilation>false</TieredCompilation>

...into my csproj got it working as expected. Thanks!

[stephentoub]

FYI @kouvel, @karpinsn

[CodeBlanch]

Some musings on this...

VS shows me a warning if I try to launch a diagnostic session in Debug:

It would have saved ma a lot of time if I also got a warning when I used Release w/ Tiered Compilation turned ON. Or if it was smart enough to do a full compile for these automatically. A dropdown to select target runtime if multiple targets are used would be great too.

Where would I even put a ticket for the tooling folks? "Suggest a feature" in VS?

[kouvel]

Tiered compilation is currently not geared towards certain kinds of benchmarks, especially those that measure allocation bytes or GC performance. I'm curious to understand whether this issue is seen only in a benchmark or in a more realistic scenario. For benchmarks that measure allocation amounts or counts and/or GC performance, tiered compilation can have adverse and unnatural effects on the results. For such benchmarks there may be a configuration change that would be appropriate for such benchmarks (things like this have been discussed in the past). I'm interested in knowing whether a realistic scenario led to this finding or if this is only seen in benchmarks.

[kouvel]

CC @AndyAyersMS

[CodeBlanch]

@kouvel Define "benchmark" for me :)

How I got here...

I wrote a bunch of microbenchmarks to run using BenchmarkDotNet so I could gauge allocations in some code I was interested in. The output I got was showing a lot of allocations, not anywhere near where I needed it to be. So the natural question was, what's being allocated and by whom? To answer that I hacked the code so I could run it in VS using the object allocation tracking tool. The hack is more or less:

        public static void Main(string[] args)
        {
            // BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

            var b = new MyBenchmarkClass();

            b.NumberOfIterations = 1000;

            b.MemoryBenchmark();
        }

I set the benchmark project as startup, ALT+F2, launch a diagnostic session tracking object allocations, look at the results. So they started as "benchmarks" but when I run in VS, they aren't really.

What I saw in the allocation output was the usual stuff. Lots of buffers. But it also showed me a ton of state machine allocations for things I knew were executing synchronously. That's where the tiered compilation led me into the weeds. For hours I was wracking my brain trying to figure it out. Decompiled things, searched the web, read @stephentoub's Async Performance: Understanding the Costs of Async and Await 100 times, went on a vision quest, etc. I learned a lot, so no big deal, but the tooling could have been slightly more helpful. When it finally occurred to me to try my tests using net48, that's when I started to piece it together and ended up here.

[kouvel]

@kouvel Define "benchmark" for me :)

I won't attempt that here :) but what I mean is that it is a measurement that is not something that would represent something that would happen in practice. If a measurement is not representative, the finding may lead investigation of the issue in an irrelevant direction. I'm only trying to narrow it down. I can tell you for sure that if you're writing microbenchmarks that measure allocations or otherwise GC-related metrics, then in .NET Core 3.x the appropriate thing to do would be to disable tiered compilation. I'm aware that that is not an ideal long-term suggestion (and I have thoughts on that), but I just want to learn more about this scenario so that I can suggest something better now or later.

[AndyAyersMS]

We tried to enable most of the box elimination opts in Tier0 as they came as unwelcome surprises like the one you got here -- especially boxing from the async machinery as it's not easy for the end user to remove -- but didn't get them all.

This is the same issue as #1713; there is a fix in 5.0 for it at #1741.

In 3.1 the system will eventually rejit at Tier1 and the allocations will stop. But as #1713 shows you can create quite a lot of boxes bit before this happens.