diff --git a/DEVGUIDE.md b/DEVGUIDE.md index e605c711dba..e5480e39ae0 100644 --- a/DEVGUIDE.md +++ b/DEVGUIDE.md @@ -275,9 +275,25 @@ Where `` corresponds to the latest Visual Studio version on your machin Use the `Debug` configuration to test your changes locally. It is the default. Do not use the `Release` configuration! Local development and testing of Visual Studio tooling is not designed for the `Release` configuration. -### Writing and running benchmarks +### Benchmarking -Existing compiler benchmarks can be found in `tests\benchmarks\`. +Existing compiler benchmarks can be found in `tests\benchmarks\`. The folder contains READMEs describing specific benchmark projects as well as guidelines for creating new benchmarks. There is also `FSharp.Benchmarks.sln` solution containing all the benchmark project and their dependencies. + +To exercise the benchmarking infrastructure locally, run: + +(Windows) +```cmd +build.cmd -configuration Release -testBenchmarks +``` + +(Linux/Mac) +```shell +./build.sh --configuration Release --testBenchmarks +``` + +This is executed in CI as well. It does the following: +- builds all the benchmarking projects +- does smoke testing for fast benchmarks (executes them once to check they don't fail in the runtime) ### Benchmarking and profiling the compiler @@ -286,151 +302,6 @@ Existing compiler benchmarks can be found in `tests\benchmarks\`. * Always build both versions of compiler/FCS from source and not use pre-built binaries from SDK (SDK binaries are crossgen'd, which can affect performance). * To run `Release` build of compiler/FCS. -### Example benchmark setup using [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet) - -1. Perform a clean build of the compiler and FCS from source (as described in this document, build can be done with `-noVisualStudio` in case if FCS/FSharp.Core is being benchmarked/profiled). - -2. Create a benchmark project (in this example, the project will be created in `tests\benchmarks\FCSBenchmarks`). - - ```shell - cd tests\benchmarks\FCSBenchmarks - dotnet new console -o FcsBench --name FcsBench -lang F# - ``` - -3. Add needed packages and project references. - - ```shell - cd FcsBench - dotnet add package BenchmarkDotNet - dotnet add reference ..\..\..\src\Compiler\FSharp.Compiler.Service.fsproj - ``` - -4. Additionally, if you want to test changes to the FSharp.Core (note that the relative path can be different) - - ```shell - dotnet add reference ..\..\..\src\FSharp.Core\FSharp.Core.fsproj - ``` - - > as well as the following property have to be added to `FcsBench.fsproj`: - - ```xml - - true - - ``` - -5. Add a new benchmark for FCS/FSharp.Core by editing `Program.fs`. - - ```fsharp - open System.IO - open FSharp.Compiler.CodeAnalysis - open FSharp.Compiler.Diagnostics - open FSharp.Compiler.Text - open BenchmarkDotNet.Attributes - open BenchmarkDotNet.Running - - [] - type CompilerService() = - let mutable checkerOpt = None - let mutable sourceOpt = None - - let parsingOptions = - { - SourceFiles = [|"CheckExpressions.fs"|] - ConditionalDefines = [] - DiagnosticOptions = FSharpDiagnosticOptions.Default - LangVersionText = "default" - IsInteractive = false - LightSyntax = None - CompilingFsLib = false - IsExe = false - } - - [] - member _.Setup() = - match checkerOpt with - | None -> - checkerOpt <- Some(FSharpChecker.Create(projectCacheSize = 200)) - | _ -> () - - match sourceOpt with - | None -> - sourceOpt <- Some <| SourceText.ofString(File.ReadAllText("""C:\Users\vlza\code\fsharp\src\Compiler\Checking\CheckExpressions.fs""")) - | _ -> () - - - [] - member _.ParsingTypeCheckerFs() = - match checkerOpt, sourceOpt with - | None, _ -> failwith "no checker" - | _, None -> failwith "no source" - | Some(checker), Some(source) -> - let results = checker.ParseFile("CheckExpressions.fs", source, parsingOptions) |> Async.RunSynchronously - if results.ParseHadErrors then failwithf "parse had errors: %A" results.Diagnostics - - [] - member _.ParsingTypeCheckerFsSetup() = - match checkerOpt with - | None -> failwith "no checker" - | Some(checker) -> - checker.InvalidateAll() - checker.ClearLanguageServiceRootCachesAndCollectAndFinalizeAllTransients() - checker.ParseFile("dummy.fs", SourceText.ofString "dummy", parsingOptions) |> Async.RunSynchronously |> ignore - - [] - let main _ = - BenchmarkRunner.Run() |> ignore - 0 - ``` - - > For more detailed information about available BenchmarkDotNet options, please refer to [BenchmarkDotNet Documentation](https://benchmarkdotnet.org/articles/overview.html). - -6. Build and run the benchmark. - - ```shell - dotnet build -c Release - dotnet run -c Release - ``` - -7. You can find results in `.\BenchmarkDotNet.Artifacts\results\` in the current benchmark project directory. - - ```shell - > ls .\BenchmarkDotNet.Artifacts\results\ - - Directory: C:\Users\vlza\code\fsharp\tests\benchmarks\FCSBenchmarks\FcsBench\BenchmarkDotNet.Artifacts\results - - Mode LastWriteTime Length Name - ---- ------------- ------ ---- - -a--- 4/25/2022 1:42 PM 638 Program.CompilerService-report-github.md - -a--- 4/25/2022 1:42 PM 1050 Program.CompilerService-report.csv - -a--- 4/25/2022 1:42 PM 1169 Program.CompilerService-report.html - ``` - - > *-report-github.md can be used to post benchmark results to GitHub issue/PR/discussion or RFC. - > - >*-report.csv can be used for comparison purposes. - - **Example output:** - - ``` ini - - BenchmarkDotNet=v0.13.1, OS=Windows 10.0.25102 - Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores - .NET SDK=6.0.200 - [Host] : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT DEBUG - Job-GDIBXX : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT - - InvocationCount=1 UnrollFactor=1 - - ``` - - | Method | Mean | Error | StdDev | Median | Gen 0 | Gen 1 | Allocated | - |--------------------- |---------:|--------:|--------:|---------:|----------:|----------:|----------:| - | ParsingTypeCheckerFs | 199.4 ms | 3.84 ms | 9.78 ms | 195.5 ms | 4000.0000 | 1000.0000 | 28 MB | - -8. Repeat for any number of changes you would like to test. -9. **Optionally:** benchmark code and results can be included as part of the PR for future reference. - ## Additional resources The primary technical guide to the core compiler code is [The F# Compiler Technical Guide](https://github.com/dotnet/fsharp/blob/main/docs/index.md). Please read and contribute to that guide. diff --git a/eng/build-utils.ps1 b/eng/build-utils.ps1 index d5375c29732..a0c5299cb95 100644 --- a/eng/build-utils.ps1 +++ b/eng/build-utils.ps1 @@ -179,7 +179,7 @@ function Get-PackageDir([string]$name, [string]$version = "") { } function Run-MSBuild([string]$projectFilePath, [string]$buildArgs = "", [string]$logFileName = "", [switch]$parallel = $true, [switch]$summary = $true, [switch]$warnAsError = $true, [string]$configuration = $script:configuration, [string]$verbosity = $script:verbosity) { - # Because we override the C#/VB toolset to build against our LKG package, it is important + # Because we override the C#/VB toolset to build against our LKG (Last Known Good) package, it is important # that we do not reuse MSBuild nodes from other jobs/builds on the machine. Otherwise, # we'll run into issues such as https://github.com/dotnet/roslyn/issues/6211. # MSBuildAdditionalCommandLineArgs= diff --git a/eng/build.sh b/eng/build.sh index 3b992d6bfab..b8915397d25 100755 --- a/eng/build.sh +++ b/eng/build.sh @@ -25,6 +25,7 @@ usage() echo "Test actions:" echo " --testcoreclr Run unit tests on .NET Core (short: --test, -t)" echo " --testCompilerComponentTests Run FSharp.Compiler.ComponentTests on .NET Core" + echo " --testBenchmarks Build and Run Benchmark suite" echo "" echo "Advanced settings:" echo " --ci Building in CI" @@ -56,6 +57,7 @@ pack=false publish=false test_core_clr=false test_compilercomponent_tests=false +test_benchmarks=false configuration="Debug" verbosity='minimal' binary_log=false @@ -126,6 +128,9 @@ while [[ $# > 0 ]]; do --testcompilercomponenttests) test_compilercomponent_tests=true ;; + --testbenchmarks) + test_benchmarks=true + ;; --ci) ci=true ;; @@ -330,4 +335,10 @@ if [[ "$test_compilercomponent_tests" == true ]]; then TestUsingNUnit --testproject "$repo_root/tests/FSharp.Compiler.ComponentTests/FSharp.Compiler.ComponentTests.fsproj" --targetframework $coreclrtestframework --notestfilter fi +if [[ "$test_benchmarks" == true ]]; then + pushd "$repo_root/tests/benchmarks" + ./SmokeTestBenchmarks.sh + popd +fi + ExitWithExitCode 0 diff --git a/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md b/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md index d3bd6c823e6..72a6acc3f1e 100644 --- a/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md +++ b/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md @@ -12,7 +12,7 @@ To run a benchmark for a local FCS in the current codebase you can run the bench ```dotnet run --project HistoricalBenchmark.fsproj -c Release --filter *``` -To run a comparison use the `runner.ipynb` .NET notebook +To run a comparison use the `runner.ipynb` .NET notebook. ## How it works @@ -35,9 +35,4 @@ As of now the minimum supported version of FCS is 13.0.0 ## Sample results -Below is a sample result of running the notebook locally with a selection of versions: -![a](./sample_result.png?raw=true) - -## Other - -You can find this document under 'tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md'. \ No newline at end of file +See sample results in the dedicated [sample_results](./sample_results/) folder. \ No newline at end of file diff --git a/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/sample_results/README.md b/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/sample_results/README.md index 098d4f8a0c4..803e47ed266 100644 --- a/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/sample_results/README.md +++ b/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/sample_results/README.md @@ -1,13 +1,16 @@ # Sample results -This folder contains a selection of results obtained by running the notebook located in `../runner.ipynb` + +This folder contains a selection of results obtained by running the notebook located in `../runner.ipynb`. ## Timings are not accurate + The results were gathered on a busy machine without much care taken to provide a reliable performance environment. While this means the timing metrics are not very useful, the results can still be useful for two reasons: * allocation data is quite accurate as it doesn't tend to depend much on the environment * they work as examples that can make using the benchmarks easier ## Structure + Each directory contains 3 files output by `HistoricalBenchmark.Runner.runAll` function for a given selection of versions. The three different version sets are: @@ -16,10 +19,12 @@ The three different version sets are: - `10_latest_nuget_versions` - 10 FCS NuGet versions between `v41.0.2` and ``v41.0.5-preview.22327.2` ## Observations -One thing that can be observed by looking at the results in `between_2_nuget_versions` is the noticable increase of allocations in https://github.com/dotnet/fsharp/pull/11517 + +One thing that can be observed by looking at the results in `between_2_nuget_versions` is the noticeable increase of allocations in https://github.com/dotnet/fsharp/pull/11517. While this isn't necessarily something worth addressing, partly because later revisions show reduced allocations, it shows how running a historical benchmark can be potentially useful. ## Notes + - The metrics gathered here are very limited - much more data can be gathered from each benchmark. - Such historical benchmarks run locally might be mostly deprecated once CI setup exists for performance tests that will provide the necessary historical information \ No newline at end of file diff --git a/tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/README.md b/tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/README.md index 90e90b0fd6f..1b574ec1b56 100644 --- a/tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/README.md +++ b/tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/README.md @@ -2,8 +2,8 @@ ## What is it -* A selection of BDN benchmarks analysing FCS performance -* Uses BDN's commandline API +* A selection of BDN benchmarks analyzing FCS performance +* Uses BDN's command line API ## How to run it @@ -11,12 +11,15 @@ Running all benchmarks: ```dotnet run -c Release --filter *``` Running a specific benchmark: -```dotnet run -c Release --filter *ParsingTypeCheckerFs*``` +```dotnet run -c Release --filter *ParsingCheckExpressionsFs*``` ## Sample results -*TODO* - -## Other - -You can find this document under 'tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md'. \ No newline at end of file +| Method | Job | UnrollFactor | Mean | Error | StdDev | Median | Gen 0 | Gen 1 | Gen 2 | Allocated | +|--------------------------------------- |----------- |------------- |---------------:|-------------:|--------------:|---------------:|------------:|-----------:|----------:|-----------:| +| SimplifyNames | DefaultJob | 16 | 17,221.4 us | 378.14 us | 1,097.04 us | 17,164.1 us | 1875.0000 | 31.2500 | - | 11,654 KB | +| UnusedOpens | DefaultJob | 16 | 852.7 us | 16.96 us | 36.87 us | 852.0 us | 120.1172 | 37.1094 | - | 736 KB | +| UnusedDeclarations | DefaultJob | 16 | 208.2 us | 6.65 us | 19.09 us | 202.7 us | 71.5332 | 3.6621 | - | 438 KB | +| ParsingCheckExpressionsFs | Job-CXFNSP | 1 | 255,107.0 us | 39,778.24 us | 117,287.03 us | 186,340.7 us | 4000.0000 | 1000.0000 | - | 30,082 KB | +| ILReading | Job-CXFNSP | 1 | 1,256,653.6 us | 24,802.85 us | 48,958.41 us | 1,249,170.3 us | 102000.0000 | 31000.0000 | 2000.0000 | 671,507 KB | +| TypeCheckFileWith100ReferencedProjects | Job-CXFNSP | 1 | 6,541.1 us | 242.62 us | 700.00 us | 6,614.2 us | - | - | - | 3,547 KB | diff --git a/tests/benchmarks/FCSBenchmarks/FCSSourceFiles/README.md b/tests/benchmarks/FCSBenchmarks/FCSSourceFiles/README.md index 534e2762961..0cd8807aa16 100644 --- a/tests/benchmarks/FCSBenchmarks/FCSSourceFiles/README.md +++ b/tests/benchmarks/FCSBenchmarks/FCSSourceFiles/README.md @@ -9,12 +9,18 @@ ## How to run it 1. Run `dotnet run -c release` -2. Output available on the commandline and in `BenchmarkDotNet.Artifacts/` +2. Output available on the command line and in `BenchmarkDotNet.Artifacts/` ## Sample results -*TODO* - -## Other - -You can find this document under 'tests/benchmarks/FCSBenchmarks/FCSSourceFiles/README.md'. \ No newline at end of file +``` +BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22621 +11th Gen Intel Core i7-1185G7 3.00GHz, 1 CPU, 8 logical and 4 physical cores +.NET SDK=6.0.320 + [Host] : .NET 6.0.25 (6.0.2523.51912), X64 RyuJIT DEBUG + DefaultJob : .NET 6.0.25 (6.0.2523.51912), X64 RyuJIT +``` + +| Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated | +|--------------------------- |--------:|--------:|--------:|-------------:|------------:|----------:|----------:| +| ParseAndCheckFileInProject | 22.14 s | 0.543 s | 1.522 s | 1645000.0000 | 307000.0000 | 6000.0000 | 10 GB | diff --git a/tests/benchmarks/FCSBenchmarks/README.md b/tests/benchmarks/FCSBenchmarks/README.md index 6d4a3a674f0..aedaf8c38aa 100644 --- a/tests/benchmarks/FCSBenchmarks/README.md +++ b/tests/benchmarks/FCSBenchmarks/README.md @@ -2,26 +2,19 @@ ## What can be found here -Benchmarks that exercise performance of `FSharp.Compiler.Service` +Benchmarks that exercise performance of `FSharp.Compiler.Service`. + +## Testing performance of FSharp.Compiler.Service -## Testing performance of FSharp.Compiler.Service Performance of the compiler service is crucial for having good developer experience. This includes compilation, type checking and any other parts of the API used by IDEs. When making changes to the FCS source code, consider running some of these to assess impact of the changes on performance. ## Benchmark list + * `BenchmarkComparison/` - a Notebook-based benchmark that analyses performance of `FSharpChecker.ParseAndCheckFileInProject` on a single-file F# project. Supports comparing different revisions of the FCS codebase and fetching the code automatically. -* `CompilerServiceBenchmarks/` - +* `CompilerServiceBenchmarks/` - a selection of BDN benchmarks analyzing FCS performance * `FCSSourceFiles/` - analyses performance of `FSharpChecker.ParseAndCheckFileInProject` for the `FSharp.Core` project. Uses locally available source code. All the above benchmarks use BenchmarkDotNet. - -## Quickly validating that the benchmarks work -`SmokeTestAllBenchmarks.ps1` allows to run all BDN benchmarks in this directory with a minimum number of iterations, as a way to verify that the benchmarks still work. - -This doesn't validate the notebook-based meta-benchmarks. - -## Other - -You can find this document under 'tests/benchmarks/FCSBenchmarks/README.md'. \ No newline at end of file diff --git a/tests/benchmarks/README.md b/tests/benchmarks/README.md index c3500eb8bb5..dc304fed2ce 100644 --- a/tests/benchmarks/README.md +++ b/tests/benchmarks/README.md @@ -12,10 +12,6 @@ Each of them assesses a slightly different use case and is run in a different wa Since there is currently no dedicated hardware setup for running benchmarks in a highly accurate fashion, the results obtained by running them locally have to be treated carefully. Specifically results obtained on different hardware or in different environments should be treated differently. -Note that there are plans to update the performance testing infrastructure. More information can be found at the following links: -* https://github.com/dotnet/fsharp/discussions/12526 -* https://github.com/dotnet/performance/issues/2457 - ### Types of performance tests Performance tests in this codebase can be broadly put into two groups: @@ -26,9 +22,10 @@ Group 1. affects end users of programs, while group 2. affects developer experie ### Directory structure -Tests are structured as follows +The code is structured as follows * `CompiledCodeBenchmarks/` - benchmarks that test compiled code performance. * `FCSBenchmarks/` - benchmarks of the compiler service itself. +* `FSharp.Benchmarks.Common/` - the library with common code. ### Jupyter notebooks @@ -43,7 +40,11 @@ It helps avoid common benchmarking pitfalls and provide highly-accurate, repeata A BDN benchmark is an executable. To run it, simply run `dotnet run %BenchmarkProject.fsproj%` in the benchmark's directory. -### Writing a new benchmark +## Quickly validating that the benchmarks work + +`SmokeTestBenchmarks.ps1` allows to run faster BDN benchmarks with a minimum number of iterations, as a way to verify that the benchmarks still work. This doesn't validate the notebook-based meta-benchmarks. + +## Authoring benchmarks When adding a benchmark, consider: * choosing an appropriate subdirectory @@ -52,8 +53,147 @@ When adding a benchmark, consider: * * how to run the benchmark, including any environment requirements * * an example of the results it produces -For instructions on how to write a BDN benchmark see [DEVGUIDE](https://github.com/dotnet/fsharp/blob/main/DEVGUIDE.md). - -## Other +Here are the steps for creating benchmarks: + +1. Perform a clean build of the compiler and FCS from source (as described in this document, build can be done with `-noVisualStudio` in case if FCS/FSharp.Core is being benchmarked/profiled). + +2. Create a benchmark project (in this example, the project will be created in `tests\benchmarks\FCSBenchmarks`). + + ```shell + cd tests\benchmarks\FCSBenchmarks + dotnet new console -o FcsBench --name FcsBench -lang F# + ``` + +3. Add needed packages and project references. + + ```shell + cd FcsBench + dotnet add package BenchmarkDotNet + dotnet add reference ..\..\..\src\Compiler\FSharp.Compiler.Service.fsproj + ``` + +4. Additionally, if you want to test changes to the FSharp.Core (note that the relative path can be different) + + ```shell + dotnet add reference ..\..\..\src\FSharp.Core\FSharp.Core.fsproj + ``` + + And the following property has to be added to `FcsBench.fsproj`: + + ```xml + + true + + ``` + +5. Add a new benchmark for FCS/FSharp.Core by editing `Program.fs`. + + ```fsharp + open System.IO + open FSharp.Compiler.CodeAnalysis + open FSharp.Compiler.Diagnostics + open FSharp.Compiler.Text + open BenchmarkDotNet.Attributes + open BenchmarkDotNet.Running + + [] + type CompilerService() = + let mutable checkerOpt = None + let mutable sourceOpt = None + + let parsingOptions = + { + SourceFiles = [|"CheckExpressions.fs"|] + ConditionalDefines = [] + DiagnosticOptions = FSharpDiagnosticOptions.Default + LangVersionText = "default" + IsInteractive = false + LightSyntax = None + CompilingFsLib = false + IsExe = false + } + + [] + member _.Setup() = + match checkerOpt with + | None -> + checkerOpt <- Some(FSharpChecker.Create(projectCacheSize = 200)) + | _ -> () + + match sourceOpt with + | None -> + sourceOpt <- Some <| SourceText.ofString(File.ReadAllText("""C:\Users\vlza\code\fsharp\src\Compiler\Checking\CheckExpressions.fs""")) + | _ -> () + + [] + member _.ParsingCheckExpressionsFs() = + match checkerOpt, sourceOpt with + | None, _ -> failwith "no checker" + | _, None -> failwith "no source" + | Some(checker), Some(source) -> + let results = checker.ParseFile("CheckExpressions.fs", source, parsingOptions) |> Async.RunSynchronously + if results.ParseHadErrors then failwithf "parse had errors: %A" results.Diagnostics + + [] + member _.ParsingCheckExpressionsFsSetup() = + match checkerOpt with + | None -> failwith "no checker" + | Some(checker) -> + checker.InvalidateAll() + checker.ClearLanguageServiceRootCachesAndCollectAndFinalizeAllTransients() + checker.ParseFile("dummy.fs", SourceText.ofString "dummy", parsingOptions) |> Async.RunSynchronously |> ignore + + [] + let main _ = + BenchmarkRunner.Run() |> ignore + 0 + ``` + + For more detailed information about available BenchmarkDotNet options, please refer to [BenchmarkDotNet Documentation](https://benchmarkdotnet.org/articles/overview.html). + +6. Build and run the benchmark. + + ```shell + dotnet build -c Release + dotnet run -c Release + ``` + +7. You can find results in `.\BenchmarkDotNet.Artifacts\results\` in the current benchmark project directory. + + ```shell + > ls .\BenchmarkDotNet.Artifacts\results\ + + Directory: C:\Users\vlza\code\fsharp\tests\benchmarks\FCSBenchmarks\FcsBench\BenchmarkDotNet.Artifacts\results + + Mode LastWriteTime Length Name + ---- ------------- ------ ---- + -a--- 4/25/2022 1:42 PM 638 Program.CompilerService-report-github.md + -a--- 4/25/2022 1:42 PM 1050 Program.CompilerService-report.csv + -a--- 4/25/2022 1:42 PM 1169 Program.CompilerService-report.html + ``` + + *-report-github.md can be used to post benchmark results to GitHub issue/PR/discussion or RFC. + + *-report.csv can be used for comparison purposes. + + **Example output:** + + ``` ini + + BenchmarkDotNet=v0.13.1, OS=Windows 10.0.25102 + Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores + .NET SDK=6.0.200 + [Host] : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT DEBUG + Job-GDIBXX : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT + + InvocationCount=1 UnrollFactor=1 + + ``` + + | Method | Mean | Error | StdDev | Median | Gen 0 | Gen 1 | Allocated | + |-------------------------- |---------:|--------:|--------:|---------:|----------:|----------:|----------:| + | ParsingCheckExpressionsFs | 199.4 ms | 3.84 ms | 9.78 ms | 195.5 ms | 4000.0000 | 1000.0000 | 28 MB | + +8. Repeat for any number of changes you would like to test. +9. **Optionally:** benchmark code and results can be included as part of the PR for future reference. -You can find this document under 'tests/benchmarks/README.md'. diff --git a/tests/benchmarks/SmokeTestBenchmarks.sh b/tests/benchmarks/SmokeTestBenchmarks.sh new file mode 100644 index 00000000000..0da2c8f2213 --- /dev/null +++ b/tests/benchmarks/SmokeTestBenchmarks.sh @@ -0,0 +1,18 @@ +# Smoke test for checking that some (faster) benchmarks work. +# The test is successful if all the benchmarks run and produce results. +# The actual numbers produced aren't accurate. + +run() { + local path=$1 + dotnet run \ + --project $path \ + -c release \ + --no-build \ + --job Dry \ + --allCategories short \ + --stopOnFirstError +} + +run "FCSBenchmarks/BenchmarkComparison/HistoricalBenchmark.fsproj" +run "FCSBenchmarks/CompilerServiceBenchmarks/FSharp.Compiler.Benchmarks.fsproj" +run "FCSBenchmarks/FCSSourceFiles/FCSSourceFiles.fsproj"