Skip to content

TESTGUIDE.md

Latest commit

 

History

History
285 lines (184 loc) · 14.4 KB

TESTGUIDE.md

File metadata and controls

285 lines (184 loc) · 14.4 KB

F# Compiler, Core Library and Visual F# Tools Tests

Where this guide mentions the command build it means either build.cmd in the root folder for Windows, or build.sh for Linux/macOS.

In this guide

Quick start: Running Tests

To run the tests in Release mode:

build -testCompiler -c Release
build -testCompilerService -c Release
build -testCompilerComponentTests -c Release
build -testCambridge -c Release -ci -nobl
build -testFSharpCore -c Release
build -testScripting -c Release
build -testVs -c Release
build -testAOT -c Release
build -testAll -c Release

Tests grouping summary

Group name OS Description
testDesktop Windows Runs all net472 tests in 32 bit processes, this includes tests from other groups
testCoreClr Linux/Mac/Windows Runs all .NetStandard and .NETCore tests in 64 bit processes, this includes tests from other groups
testFSharpCore Windows Runs all test for FSharp.Core.dll
testCambridge Windows Runs the Cambridge suite tests
testVS Windows + VS Runs all VS integration tests
testAOT Windows Run AOT/Trimming tests
testCompiler Windows Runs a few quick compiler tests
testScripting Windows Runs scripting fsx and fsi commandline tests
test Windows Same as testDesktop
testAll Windows Runs all above tests
testAllButIntegration Windows Runs all minus integration tests

Some test groups can only be run in CI configuration, for that, you need to pass the -ci -bl or -ci -nobl arguments. Some test groups can only be run in Release mode, this is indicated below. Some tests can only be run on Windows.

To run tests, from a command prompt, use variations such as the following, depending on which test suite and build configuration you want.

Tests that can be run on Linux and macOS

If you're using Linux or macOS to develop, the group of tests that are known to succeed are all in -testCoreClr. Any other -testXXX argument will currently fail. An effort is underway to make testing and running tests easier on all systems.

Tests that can only be run in Release mode

The following tests must be run in Release mode with -c Release:

build -testAll -c Release
build -test -c Release
build -testDesktop -c Release
build -testCoreClr -c Release

Tests that open other windows

The following testsets open other windows and may interfere with you using your workstation, or change focus while you're doing something else:

  • Cambridge

Running tests online in CI

You can also submit pull requests to https://github.com/dotnet/fsharp and run the tests via continuous integration. Most people do wholesale testing that way. A few notes:

  • Online, sometimes unrelated tests or builds may fail, a rerun may solve this
  • A CI build can be restarted by closing/reopening the PR
  • A new CI build will be started on each pushed commit
  • CI builds that are not finished will be canceled on new commits. If you need to complete such runs, you can do so in the Checks tab of the PR by selecting the actual commit from the dropdown.

Analyzing CI results

Finding the logs in the online CI results can be tricky, a small video can be found below under "Test gotchas".

Prerequisites

The prerequisites are the same as for building the FSharp.sln, plus:

  • Run git clean -xdf -e .vs before running tests when:
    • Making changes to the lexer or parser
    • Between switching git branches
    • When merging with latest main upstream branch.

Test suites

The F# tests are split as follows:

FSharp Suite

This is compiled using tests\fsharp\FSharpSuite.Tests.fsproj as a test executable. Each individual test is an xUnit test case, and so you can run it like any other xUnit test.

Tests are grouped in folders per area. Each test compiles and executes a test.fsx|fs file in its folder using some combination of compiler or FSI flags specified in the FSharpSuite test project.
If the compilation and execution encounter no errors, the test is considered to have passed.

There are also negative tests checking code expected to fail compilation. See note about baseline under "Other Tips" below for tests checking expectations against "baseline" (.bsl) files.

Logs and output

All test execution logs and result files will be dropped into the tests\TestResults folder, and have file names matching

    net40-fsharp-suite-*.*
    net40-compilerunit-suite-*.*
    net40-coreunit-suite-*.*
    vs-ideunit-suite-*.*

Working with baseline tests

FSharp Test Suite works with a couple of .bsl (or .bslpp) files describing "expected test results" and are called the Baseline Tests. Those are matched against the actual output that resides under .err or .vserr files of the same name during test execution. When doing so keep in mind to carefully review the diff before committing updated baseline files.

The .bslpp (for: baseline pre-process) files are specially designed to enable substitution of certain tokens to generate the .bsl file. You can look further about the pre-processing logic under tests/fsharp/TypeProviderTests.fs, this is used only for type provider tests for now.

To update baselines use this:

fsi tests\scripts\update-baselines.fsx

Use -n to dry-run:

fsi tests\scripts\update-baselines.fsx -n

Other Tips and gotchas

This section contains general tips, for solving errors see next section.

Close any open VisualFSharp.sln

If you have the VisualFSharp.sln open, or if you recently debugged it through VisualFSharpFull as start-up project, certain tests may fail because files will be in use. It's best to close Visual Studio and any debugging sessions during a test run. It is fine to have VS open on a different solution, or to have it open from a different F# repo folder.

Finding the logs on CI

Finding the proper logs in the CI system can be daunting. The console output contains enough info for most tests and can be found by clicking Raw output in the CI window, or clicking download logs:

download logs

Increase command screen line buffer on Windows

You can increase the window buffer so that more lines of the console output can be scrolled back to, as opposed to them disappearing off the top. The default size on Windows is very small:

  • Click top-left icon of the command window
  • Go to Properties then Layout
  • Select a higher Screen buffer size than the window size (this will add a scroll bar)
  • You may want to increase the width and height as well
  • Click OK.

Run as Administrator

Running tests should now be possible without admin privileges. If you find tests that don't run unless you are an admin, please create an issue.

Running tests on other (feature) branches

When you switch branches, certain temporary files, as well as the .NET version (downloaded to .dotnet folder) are likely to not be in sync anymore and can lead to curious build errors. Fix this by running git clean like this (this will leave your VS settings intact):

git clean -xdf -e .vs

If you get "file in use" errors during cleaning, make sure to close Visual Studio and any running dotnet.exe and VBCSCompiler.exe, esp those that show up at the bottom of Process Explorer without parent process.

Running tests on release/dev16.6 etc branches

Some tests are known to fail on these older branches when run using one of the testXXXX commandline arguments. However, -test, -testAll, -testCoreClr and -testDesktop are known to work on at least the dev16.6 and dev16.7 branches.

Making the tests run faster

  • Adding the -norestore flag to the commandline speeds up the build part a little bit.
  • When using the -ci flag (mandatory for some testsets), adding the -nobl flag prevents creating the binary log files.

Solving common errors

The following are common errors that users have encountered while running tests on their system.

Error that a file cannot be accessed

The build often leaves dangling processes like VBCSCompiler.exe or MSBuild.exe. In Process Explorer you can see these processes having no parent process anymore. You can also use this to kill such processes.

StackOverflow exception

This usually happens when you try to run tests without specifying -c Release, or as -c Debug (which is the default). Run the same set with -c Release instead and the SOE should disappear.

Approximate running times

Some tests can run for several minutes, this doesn't mean that your system froze:

image

To get an idea of how long it may take, or how much coffee you'll need while waiting, here are some rough indications from an older workstation run, using arguments -c Release -nobl -norestore:

Testset Approx running time Ngen'ed running time
sln build time 1 min* n/a
-testDesktop 5 min ?
-testCoreClr 36 min ?
-testCambridge 72 min 35 min
-testCompiler 30 seconds n/a
-testFSharpCore 2 min ?
-testScripting 2 min 1.5 min
-testVS 13 min ?
  • This is the build time when a previous build with the same configuration succeeded, and without -ci present, which always rebuilds the solution. With -norestore the build part can go down to about 10-20 seconds, before tests are being run

Test Infrastructure

Testing Framework

The F# repository uses xUnit v3 (3.2.2) for unit testing, running on the Microsoft Testing Platform (MTP) instead of the legacy VSTest runner. Key components:

  • xunit.v3.mtp-v2: 3.2.2 (test framework + MTP integration)
  • xunit.v3.runner.console: 3.2.2 (console runner)
  • FsCheck: 2.16.6 (property-based testing)
  • Microsoft.Testing.Extensions.HangDump: 2.0.2 (hang detection, replaces VSTest blame-hang-timeout)

All test projects are <OutputType>Exe</OutputType> executables (an xUnit v3 requirement). Package references are centrally managed in tests/Directory.Build.props.

Test Execution

Tests run via dotnet test using the MTP runner (enabled via <UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>). The build script function TestUsingMSBuild in eng/Build.ps1 handles test invocation with xUnit-flavored TRX reporting (--report-xunit-trx) and hang dump timeouts.

To run a specific test project directly:

dotnet test --project tests\FSharp.Compiler.ComponentTests\FSharp.Compiler.ComponentTests.fsproj -c Release -f net10.0

To filter tests by method name:

dotnet test --project <project> -c Release -- --filter-method "*YourTestName*"

Custom Test Utilities

The repository includes custom xUnit v3 extensions in tests/FSharp.Test.Utilities/:

Active:

  • Console output capture (TestConsole): Each test case captures and reports its console output via TestConsole.ExecutionCapture. Note: xUnit v3's built-in [<assembly: CaptureTrace>] is intentionally disabled because it intercepts console output before TestConsole's redirectors, breaking FSI test capture.
  • Custom data attributes: DirectoryAttribute and FileInlineDataAttribute for file-based test discovery, StressAttribute for repeated parallel stress testing.
  • Assembly initialization (XunitSetup): Installs TestConsole and (on .NET Framework) the AssemblyResolver once per assembly via a lazy module initializer, replacing the old FSharpXunitFramework approach.

Disabled (pending xUnit v3 API adaptation):

  • FSharpXunitFramework (XunitHelpers.fs): The custom XunitTestFramework subclass is commented out. It previously handled one-time setup, OpenTelemetry tracing/metrics export for test runs, and post-run temp directory cleanup. xUnit v3 changed the XunitTestFramework/XunitTestFrameworkExecutor APIs, making the override incompatible.
  • XUNIT_EXTRAS — internal parallelization and batch traits (XunitHelpers.fs, gated by the XUNIT_EXTRAS define which is commented out in FSharp.Test.Utilities.fsproj): This included CustomTestCase/CustomTheoryTestCase types and a custom discoverer that assigned each test case a unique test collection for fine-grained parallelization, and injected batch=1..4 traits for CI multi-agent filtering (--filter-trait batch=N). xUnit v3 changed the XunitTestCase, XunitTheoryTestCase, and discovery APIs. The test suite now relies on xUnit v3's default collection-level parallelization.

Test Configuration

Test execution behavior is controlled by testconfig.json files (xUnit v3 format) in test projects that need non-default settings. Example from tests/fsharp/testconfig.json:

{
  "xUnit": {
    "parallelizeTestCollections": false,
    "maxParallelThreads": 1
  }
}

.NET Framework (net472) test projects are forced to x64 via <PlatformTarget>x64</PlatformTarget> in tests/Directory.Build.props to avoid OOM issues.