Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion azure-pipelines.yml
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@ jobs:
channels:
- master
- release/3.1.2xx
- 3.0
- 2.1

# Windows x64 net461 micro benchmarks
- template: /eng/performance/benchmark_jobs.yml
Expand Down Expand Up @@ -167,7 +169,6 @@ jobs:
- master
- release/3.1.2xx
- 3.0
- 2.2
- 2.1

# Ubuntu 1804 x64 ML.NET benchmarks
Expand Down
22 changes: 11 additions & 11 deletions docs/benchmarkdotnet.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ In order to build or run the benchmarks you will need the **.NET Core command-li

### Using .NET Cli

To build the benchmarks you need to have the right `dotnet cli`. This repository allows to benchmark .NET Core 2.0, 2.1, 2.2, 3.0 and 5.0 so you need to install all of them.
To build the benchmarks you need to have the right `dotnet cli`. This repository allows to benchmark .NET Core 2.1, 3.0, 3.1 and 5.0 so you need to install all of them.

All you need to do is run the following command:

Expand All @@ -70,8 +70,8 @@ dotnet build -c Release
If you don't want to install all of them and just run the benchmarks for selected runtime(s), you need to manually edit the [common.props](../build/common.props) file.

```diff
-<TargetFrameworks>netcoreapp2.1;netcoreapp2.2;netcoreapp3.0</TargetFrameworks>
+<TargetFrameworks>netcoreapp3.0</TargetFrameworks>
-<TargetFrameworks>netcoreapp2.1;netcoreapp3.0;netcoreapp3.1;netcoreapp5.0</TargetFrameworks>
+<TargetFrameworks>netcoreapp5.0</TargetFrameworks>
```

### Using Python script
Expand Down Expand Up @@ -283,30 +283,30 @@ M00_L00:

The `--runtimes` or just `-r` allows you to run the benchmarks for **multiple Runtimes**.

Available options are: Mono, CoreRT, net461, net462, net47, net471, net472, netcoreapp2.1, netcoreapp2.2, netcoreapp3.0.
Available options are: Mono, CoreRT, net461, net462, net47, net471, net472, netcoreapp2.1, netcoreapp3.0, netcoreapp3.1 and netcoreapp5.0.

Example: run the benchmarks for .NET Core 2.2 and 3.0:
Example: run the benchmarks for .NET Core 3.1 and 5.0:

```cmd
dotnet run -c Release -f netcoreapp2.2 --runtimes netcoreapp2.1 netcoreapp3.0
dotnet run -c Release -f netcoreapp3.1 --runtimes netcoreapp3.1 netcoreapp5.0
```

**Important: The host process needs to be the lowest common API denominator of the runtimes you want to compare!** In this case, it was`netcoreapp2.2`.
**Important: The host process needs to be the lowest common API denominator of the runtimes you want to compare!** In this case, it was`netcoreapp3.1`.

## Regressions

To perform a Mann–Whitney U Test and display the results in a dedicated column you need to provide the Threshold for Statistical Test via `--statisticalTest` argument. The value can be relative (5%) or absolute (10ms, 100ns, 1s)

Example: run Mann–Whitney U test with relative ratio of 5% for `BinaryTrees_2` for .NET Core 2.1 (base) vs .NET Core 2.2 (diff). .NET Core 2.1 will be baseline because it was first.
Example: run Mann–Whitney U test with relative ratio of 5% for `BinaryTrees_2` for .NET Core 3.1 (base) vs .NET Core 5.0 (diff). .NET Core 3.1 will be baseline because it was first.

```cmd
dotnet run -c Release -f netcoreapp2.1 --filter *BinaryTrees_2* --runtimes netcoreapp2.1 netcoreapp2.2 --statisticalTest 5%
dotnet run -c Release -f netcoreapp3.1 --filter *BinaryTrees_2* --runtimes netcoreapp3.1 netcoreapp5.0 --statisticalTest 5%
```

| Method | Toolchain | Mean | MannWhitney(5%) |
|-------------- |-------------- |---------:|---------------- |
| BinaryTrees_2 | netcoreapp2.1 | 124.4 ms | Base |
| BinaryTrees_2 | netcoreapp2.2 | 153.7 ms | Slower |
| BinaryTrees_2 | netcoreapp3.1 | 124.4 ms | Base |
| BinaryTrees_2 | netcoreapp5.0 | 153.7 ms | Slower |

**Note:** to compare the historical results you need to use [Results Comparer](../src/tools/ResultsComparer/README.md)

Expand Down
2 changes: 1 addition & 1 deletion docs/benchmarking-workflow-dotnet-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ If profiling using the `--profiler ETW` is not enough, you should use a differen

BenchmarkDotNet has some extra features that might be useful when doing performance investigation:

- You can run the benchmarks against [multiple Runtimes](./benchmarkdotnet.md#Multiple-Runtimes). It can be very useful when the regression has been introduced between .NET Core releases, for example: between netcoreapp2.2 and netcoreapp3.0.
- You can run the benchmarks against [multiple Runtimes](./benchmarkdotnet.md#Multiple-Runtimes). It can be very useful when the regression has been introduced between .NET Core releases, for example: between netcoreapp3.1 and netcoreapp5.0.
- You can run the benchmarks using provided [dotnet cli](./benchmarkdotnet.md#dotnet-cli). You can download few dotnet SDKs, unzip them and just run the benchmarks to spot the version that has introduced the regression to narrow down your investigation.
- You can run the benchmarks using few [CoreRuns](./benchmarkdotnet.md#CoreRun). You can build the latest [dotnet/runtime](https://github.com/dotnet/runtime) in Release, create a copy of the folder with CoreRun and use git to checkout an older commit. Then rebuild [dotnet/runtime](https://github.com/dotnet/runtime) and run the benchmarks against the old and new builds. This can narrow down your investigation to the commit that has introduced the bug.

Expand Down
2 changes: 1 addition & 1 deletion scripts/benchmarks_ci.py
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ def __main(args: list) -> int:
)

# WORKAROUND
# The MicroBenchmarks.csproj targets .NET Core 2.0, 2.1, 2.2 and 3.0
# The MicroBenchmarks.csproj targets .NET Core 2.1, 3.0, 3.1 and 5.0
# to avoid a build failure when using older frameworks (error NETSDK1045:
# The current .NET SDK does not support targeting .NET Core $XYZ)
# we set the TFM to what the user has provided.
Expand Down
4 changes: 0 additions & 4 deletions scripts/channel_map.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,6 @@ class ChannelMap():
'tfm': 'netcoreapp3.0',
'branch': 'release/3.0'
},
'2.2': {
'tfm': 'netcoreapp2.2',
'branch': 'release/2.2'
},
'release/2.1.6xx': {
'tfm': 'netcoreapp2.1',
'branch': 'release/2.1.6xx'
Expand Down
1 change: 0 additions & 1 deletion scripts/dotnet.py
Original file line number Diff line number Diff line change
Expand Up @@ -813,7 +813,6 @@ def __process_arguments(args: list):
# TODO: Could pull this information from repository.
SUPPORTED_CHANNELS = [
'master', # Default channel
'2.2',
'2.1',
'LTS',
]
Expand Down
4 changes: 2 additions & 2 deletions src/benchmarks/micro/MicroBenchmarks.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
<!-- Used by Python script to narrow down the specified target frameworks to test, and avoid downloading all supported SDKs -->
<TargetFrameworks>$(PERFLAB_TARGET_FRAMEWORKS)</TargetFrameworks>
<!-- Supported target frameworks -->
<TargetFrameworks Condition="'$(TargetFrameworks)' == '' AND '$(OS)' == 'Windows_NT'">net461;netcoreapp2.1;netcoreapp2.2;netcoreapp3.0;netcoreapp5.0</TargetFrameworks>
<TargetFrameworks Condition="'$(TargetFrameworks)' == ''">netcoreapp2.1;netcoreapp2.2;netcoreapp3.0;netcoreapp5.0</TargetFrameworks>
<TargetFrameworks Condition="'$(TargetFrameworks)' == '' AND '$(OS)' == 'Windows_NT'">net461;netcoreapp2.1;netcoreapp3.0;netcoreapp3.1;netcoreapp5.0</TargetFrameworks>
<TargetFrameworks Condition="'$(TargetFrameworks)' == ''">netcoreapp2.1;netcoreapp3.0;netcoreapp3.1;netcoreapp5.0</TargetFrameworks>

<OutputType>Exe</OutputType>
<PlatformTarget>AnyCPU</PlatformTarget>
Expand Down
12 changes: 6 additions & 6 deletions src/benchmarks/micro/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,38 +12,38 @@ To learn more about designing benchmarks, please read [Microbenchmark Design Gui

## Quick Start

The first thing that you need to choose is the Target Framework. Available options are: `netcoreapp2.1|netcoreapp2.2|netcoreapp3.0|net461`. You can specify the target framework using `-f|--framework` argument. For the sake of simplicity, all examples below use `netcoreapp3.0` as the target framework.
The first thing that you need to choose is the Target Framework. Available options are: `netcoreapp2.1|netcoreapp3.0|netcoreapp3.1|netcoreapp5.0|net461`. You can specify the target framework using `-f|--framework` argument. For the sake of simplicity, all examples below use `netcoreapp5.0` as the target framework.

The following commands are run from the `src/benchmarks/micro` directory.

To run the benchmarks in Interactive Mode, where you will be asked which benchmark(s) to run:

```cmd
dotnet run -c Release -f netcoreapp3.0
dotnet run -c Release -f netcoreapp5.0
```

To list all available benchmarks ([read more](../../../docs/benchmarkdotnet.md#Listing-the-Benchmarks)):

```cmd
dotnet run -c Release -f netcoreapp3.0 --list flat|tree
dotnet run -c Release -f netcoreapp5.0 --list flat|tree
```

To filter the benchmarks using a glob pattern applied to namespace.typeName.methodName ([read more](../../../docs/benchmarkdotnet.md#Filtering-the-Benchmarks)):

```cmd
dotnet run -c Release -f netcoreapp3.0 --filter *Span*
dotnet run -c Release -f netcoreapp5.0 --filter *Span*
```

To profile the benchmarked code and produce an ETW Trace file ([read more](../../../docs/benchmarkdotnet.md#Profiling)):

```cmd
dotnet run -c Release -f netcoreapp3.0 --filter $YourFilter --profiler ETW
dotnet run -c Release -f netcoreapp5.0 --filter $YourFilter --profiler ETW
```

To run the benchmarks for multiple runtimes ([read more](../../../docs/benchmarkdotnet.md#Multiple-Runtimes)):

```cmd
dotnet run -c Release -f netcoreapp2.1 --filter * --runtimes netcoreapp2.1 netcoreapp3.0 corert
dotnet run -c Release -f netcoreapp3.1 --filter * --runtimes netcoreapp3.1 netcoreapp5.0 corert
```

## Private Runtime Builds
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ public class ReadOnlySequenceTests<T>
[Benchmark(OperationsPerInvoke = 16)]
public int FirstArray() => First(new ReadOnlySequence<T>(_array));

#if !NETFRAMEWORK && !NETCOREAPP2_1 && !NETCOREAPP2_2
#if !NETFRAMEWORK && !NETCOREAPP2_1
[Benchmark(OperationsPerInvoke = 16)]
public int FirstSpanArray() => FirstSpan(new ReadOnlySequence<T>(_array));

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ public bool Array()
return result;
}

#if !NETFRAMEWORK && !NETCOREAPP2_1 && !NETCOREAPP2_2
#if !NETFRAMEWORK && !NETCOREAPP2_1
[BenchmarkCategory(Categories.Span)]
[Benchmark]
public bool Span()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ public bool Array()
return result;
}

#if !NETFRAMEWORK && !NETCOREAPP2_1 && !NETCOREAPP2_2
#if !NETFRAMEWORK && !NETCOREAPP2_1
[BenchmarkCategory(Categories.Span)]
[Benchmark]
public bool Span()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -402,7 +402,7 @@ public IEnumerable<object> ElementAtArguments()
[Benchmark]
public void EmptyTakeSelectToArray() => Enumerable.Empty<int>().Take(10).Select(i => i).ToArray();

#if !NETFRAMEWORK && !NETCOREAPP2_1 && !NETCOREAPP2_2 // API Available in .NET Core 3.0+
#if !NETFRAMEWORK && !NETCOREAPP2_1 // API Available in .NET Core 3.0+
// Append() has two execution paths: AppendPrependIterator (a result of another Append or Prepend) and IEnumerable, this benchmark tests both
// https://github.com/dotnet/corefx/blob/dcf1c8f51bcdbd79e08cc672e327d50612690a25/src/System.Linq/src/System/Linq/AppendPrepend.cs
[Benchmark]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
using System.Numerics;

// HW Intrinsic APIs are available only in .NET Core 3.0+
#if !NETCOREAPP2_1 && !NETCOREAPP2_2 && !NETFRAMEWORK
#if !NETCOREAPP2_1 && !NETFRAMEWORK
using System.Runtime.Intrinsics;
using System.Runtime.Intrinsics.X86;
#endif
Expand Down Expand Up @@ -193,7 +193,7 @@ private double[] BilinearInterpol_Vector(
return z;
}

#if !NETCOREAPP2_1 && !NETCOREAPP2_2 && !NETFRAMEWORK
#if !NETCOREAPP2_1 && !NETFRAMEWORK
private static unsafe double[] BilinearInterpol_AVX(
double[] x,
double[] A,
Expand Down Expand Up @@ -273,7 +273,7 @@ private static unsafe double[] BilinearInterpol_AVX(
}
return z;
}
#endif // !NETCOREAPP2_1 && !NETCOREAPP2_2 && !NETFRAMEWORK
#endif // !NETCOREAPP2_1 && !NETFRAMEWORK

// This method is currently unused. It is useful, when generating a new vectorized version
// of the benchmark (e.g. with a different set of HW intrinsics), to ensure that it is
Expand Down Expand Up @@ -309,7 +309,7 @@ public double[] Interpol_Vector()
return vectorOutput;
}

#if !NETCOREAPP2_1 && !NETCOREAPP2_2 && !NETFRAMEWORK
#if !NETCOREAPP2_1 && !NETFRAMEWORK
[BenchmarkCategory(Categories.Runtime)]
[Benchmark]
public double[] Interpol_AVX()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ public class Activator<T>
[Benchmark]
public object CreateInstanceType() => System.Activator.CreateInstance(typeof(T));

#if !NETCOREAPP2_1 && !NETCOREAPP2_2 // API available in Full .NET Framework and .NET Core 3.0+
#if !NETCOREAPP2_1 // API available in Full .NET Framework and .NET Core 3.0+
[Benchmark]
public object CreateInstanceNames() => System.Activator.CreateInstance(_assemblyName, _typeName);
#endif
Expand Down
2 changes: 1 addition & 1 deletion src/tools/ResultsComparer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ It can be used to compare:
* historical results (eg. before and after my changes)
* results for different OSes (eg. Windows vs Ubuntu)
* results for different CPU architectures (eg. x64 vs ARM64)
* results for different target frameworks (eg. .NET Core 2.1 vs 2.2)
* results for different target frameworks (eg. .NET Core 3.1 vs 5.0)

All you need to provide is:
* `--base` - path to folder/file with baseline results
Expand Down