Error in ASP.NET Core benchmarks after updating to use net5.0 tfm

[JamesNK]

This is the error written to the console when trying to run benchmarks:

// Benchmark: GetHeaderSplitBenchmark.SplitDoubleHeader: Job-IHVIBP(Server=True, Toolchain=.NET Core 5.0, RunStrategy=Throughput)
// *** Execute ***
// Launch: 1 / 1
// Execute: dotnet "47fa91ed-99c2-4976-8c16-a2a9a36949c5.exe" --benchmarkName "Microsoft.AspNetCore.Http.Abstractions.Microbenchmarks.GetHeaderSplitBenchmark.SplitDoubleHeader" --job "Server=True, Toolchain=.NET Core 5.0, RunStrategy=Throughput" --benchmarkId 2 in C:\Development\Source\AspNetCore\artifacts\bin\Microsoft.AspNetCore.Http.Microbenchmarks\Release\net5.0\47fa91ed-99c2-4976-8c16-a2a9a36949c5\bin\Release\net5.0
Error:
  An assembly specified in the application dependencies manifest (47fa91ed-99c2-4976-8c16-a2a9a36949c5.deps.json) has already been found but with a different file extension:
    package: '47fa91ed-99c2-4976-8c16-a2a9a36949c5', version: '5.0.0-dev'
    path: '47fa91ed-99c2-4976-8c16-a2a9a36949c5.dll'
    previously found assembly: 'C:\Development\Source\AspNetCore\artifacts\bin\Microsoft.AspNetCore.Http.Microbenchmarks\Release\net5.0\47fa91ed-99c2-4976-8c16-a2a9a36949c5\bin\Release\net5.0\47fa91ed-99c2-4976-8c16-a2a9a36949c5.exe'
ExitCode != 0
// Benchmark Process 83440 has exited with code -2147450740
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!

We believe this issue began when dotnet/aspnetcore was updated to the net5.0 target framework.

App is using BenchmarkDotNet 0.12.1.

How do we fix this error?

[adamsitnik]

I am able to repro. But to be honest, I see this weird error for the first time ;) Let me try a few things

[adamsitnik]

I was able to find the root cause:

dotnet build is now producing both a .dll and an .exe file:

BDN tries to run:

dotnet job.exe and it fails:

I was able to reproduce it:

PS C:\Projects\aspnetcore\artifacts\bin\Microsoft.AspNetCore.Http.Performance\Release\net5.0\Job-JPNGZL\bin\Release\net5.0> C:\Projects\aspnetcore\.dotnet\dotnet.exe "Job-JPNGZL.exe" --benchmarkName "Microsoft.Net.Http.Headers.HeaderUtilitiesBenchmark.EscapeAsQuotedString" --job "Server=True, Toolchain=.NET 5.0, RunStrategy=Throughput" --benchmarkId 1
Error:
  An assembly specified in the application dependencies manifest (Job-JPNGZL.deps.json) has already been found but with a different file extension:
    package: 'Job-JPNGZL', version: '5.0.0-dev'
    path: 'Job-JPNGZL.dll'
    previously found assembly: 'C:\Projects\aspnetcore\artifacts\bin\Microsoft.AspNetCore.Http.Performance\Release\net5.0\Job-JPNGZL\bin\Release\net5.0\Job-JPNGZL.exe'

The fix is to pass .dll, not the .exe to dotnet. I need to wrap my head about how we could do this properly

PS C:\Projects\aspnetcore\artifacts\bin\Microsoft.AspNetCore.Http.Performance\Release\net5.0\Job-JPNGZL\bin\Release\net5.0> C:\Projects\aspnetcore\.dotnet\dotnet.exe "Job-JPNGZL.dll" --benchmarkName "Microsoft.Net.Http.Headers.HeaderUtilitiesBenchmark.EscapeAsQuotedString" --job "Server=True, Toolchain=.NET 5.0, RunStrategy=Throughput" --benchmarkId 1
// BeforeAnythingElse

[adamsitnik]

I've found the source of the problem:

the new moniker (net5.0) does not contain core in the name ;)

BenchmarkDotNet/src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliGenerator.cs

Line 29 in 3223c94

protected override string GetExecutableExtension() => TargetFrameworkMoniker.Contains("core") ? ".dll" : ".exe";

I love this rebranding. It brought so much value to my products