From 96ad31bf7bc85032a86956cc3c52452bc6fab7e1 Mon Sep 17 00:00:00 2001 From: Gauthier Segay Date: Wed, 6 Jun 2018 05:15:21 -0700 Subject: [PATCH 1/2] * helper script to update baseline files under tests/fsharp * rework "Other Tips" section to expand on the system of baselines files used in FSharp Test Suite --- TESTGUIDE.md | 29 +++++++++++++++---- .../fsharp/update.base.line.with.actuals.fsx | 27 +++++++++++++++++ 2 files changed, 50 insertions(+), 6 deletions(-) create mode 100644 tests/fsharp/update.base.line.with.actuals.fsx diff --git a/TESTGUIDE.md b/TESTGUIDE.md index 408e49d987..9cb6377f0a 100644 --- a/TESTGUIDE.md +++ b/TESTGUIDE.md @@ -39,7 +39,11 @@ The F# tests are split as follows: This is compiled using [tests\fsharp\FSharp.Tests.FSharpSuite.fsproj](tests/fsharp/FSharp.Tests.FSharpSuite.fsproj) to a unit test DLL which acts as a driver script. Each individual test is an NUnit test case, and so you can run it like any other NUnit test. Tests are grouped in folders per area. Each test compiles and executes a `test.fsx|fs` file in its folder using some combination of compiler or FSI flags specified in the FSharpSuite test project. -If the compilation and execution encounter no errors, the test is considered to have passed. +If the compilation and execution encounter no errors, the test is considered to have passed. + +There are also negative tests checking code expected to fail compilation. + +See note about baseline under "Other Tips" bellow for tests checking expectations against "baseline" files. ### FSharpQA Suite @@ -81,9 +85,22 @@ All test execution logs and result files will be dropped into the `tests\TestRes ### Other Tips -* Run as Administrator, or a handful of tests will fail +#### Run as Administrator + +Do this, or a handful of tests will fail. + +#### Making the tests run faster + +* NGen-ing the F# bits (fsc, fsi, FSharp.Core, etc) will result in tests executing much faster. Make sure you run `src\update.cmd` with the `-ngen` flag before running tests. +* The FSharp and FSharpQA suites will run test cases in parallel by default. You can comment out the relevant line (look for `PARALLEL_ARG`) to disable this. +* By default, tests from the FSharpQA suite are run using a persistent, hosted version of the compiler. This speeds up test execution, as there is no need for the `fsc.exe` process to spin up repeatedly. To disable this, uncomment the relevant line (look for `HOSTED_COMPILER`). + +#### Test outcome against baseline + +FSharp Test Suite works with couples of .bsl (or .bslpp) files considered "expected" and called baseline, those are matched against the actual output which resides under .err or .vserr files of same name at the during test execution. + +When working on changes generating conflicts with the baseline, you can use the helper script [tests/fsharp/update.base.line.with.actuals.fsx](tests/fsharp/update.base.line.with.actuals.fsx) to update all .bsl based on the matching .err file. + +When doing so keep in mind to carefully review the diff before comitting updated baseline files. -* Making the tests run faster - * NGen-ing the F# bits (fsc, fsi, FSharp.Core, etc) will result in tests executing much faster. Make sure you run `src\update.cmd` with the `-ngen` flag before running tests. - * The FSharp and FSharpQA suites will run test cases in parallel by default. You can comment out the relevant line (look for `PARALLEL_ARG`) to disable this. - * By default, tests from the FSharpQA suite are run using a persistent, hosted version of the compiler. This speeds up test execution, as there is no need for the `fsc.exe` process to spin up repeatedly. To disable this, uncomment the relevant line (look for `HOSTED_COMPILER`). +.bslpp (baseline pre-process) files are specially designed to enable substitution of certain tokens to generate the .bsl file. You can look further about the pre-processing logic under [tests/fsharp/TypeProviderTests.fs](tests/fsharp/TypeProviderTests.fs), this is used only for type provider tests for now. \ No newline at end of file diff --git a/tests/fsharp/update.base.line.with.actuals.fsx b/tests/fsharp/update.base.line.with.actuals.fsx new file mode 100644 index 0000000000..8b391a2f45 --- /dev/null +++ b/tests/fsharp/update.base.line.with.actuals.fsx @@ -0,0 +1,27 @@ +open System.IO + +// this script is usefull for tests using a .bsl file (baseline) containing expected compiler output +// which is matched against .vserr or .err file aside once the test has run +// the script replaces all the .bsl/.bslpp with either .err or .vserr + +let directories = + [ + "typecheck/sigs" + "typeProviders/negTests" + ] + |> List.map (fun d -> Path.Combine(__SOURCE_DIRECTORY__, d) |> DirectoryInfo) + +let extensionPatterns = ["*.err"; "*.vserr"] +for d in directories do + for p in extensionPatterns do + for errFile in d.GetFiles p do + let baseLineFile = FileInfo(Path.ChangeExtension(errFile.FullName, "bsl")) + let baseLineFilePreProcess = FileInfo(Path.ChangeExtension(errFile.FullName, "bslpp")) + + if File.ReadAllText(errFile.FullName) <> File.ReadAllText(baseLineFile.FullName) then + let expectedFile = + if baseLineFilePreProcess.Exists then baseLineFilePreProcess + else baseLineFile + + printfn "%s not matching, replacing with %s" expectedFile.FullName errFile.FullName + errFile.CopyTo(expectedFile.FullName, true) |> ignore From 3c126a61598fad63b00a9d7d2e427c4c236557d5 Mon Sep 17 00:00:00 2001 From: Gauthier Segay Date: Wed, 6 Jun 2018 05:29:19 -0700 Subject: [PATCH 2/2] add the script to the project where it is most likely to be useful --- tests/fsharp/FSharp.Tests.FSharpSuite.fsproj | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/fsharp/FSharp.Tests.FSharpSuite.fsproj b/tests/fsharp/FSharp.Tests.FSharpSuite.fsproj index 20052813a6..be325d2219 100644 --- a/tests/fsharp/FSharp.Tests.FSharpSuite.fsproj +++ b/tests/fsharp/FSharp.Tests.FSharpSuite.fsproj @@ -39,6 +39,7 @@ +