From f5cfba11686fc4f9a2574855a91b87847d07c803 Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 07:47:31 -0700 Subject: [PATCH 01/19] Remove hard dependency on llvm-project --- CMakeLists.txt | 7 ++++--- azure-devops/run-build.yml | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 97adae96212..7c7dee67a91 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -88,6 +88,7 @@ set(TOOLSET_LIB "${TOOLSET_ROOT_DIR}/lib/${VCLIBS_X86_OR_X64}") add_subdirectory(stl) -# Since we don't configure any of the tests with CMake, there is no downside to enabling testing unconditionally. -enable_testing() -add_subdirectory(tests) +if(ENABLE_TESTS) + enable_testing() + add_subdirectory(tests) +endif() diff --git a/azure-devops/run-build.yml b/azure-devops/run-build.yml index afe6304e470..7b4f131d436 100644 --- a/azure-devops/run-build.yml +++ b/azure-devops/run-build.yml @@ -76,7 +76,8 @@ jobs: cmakeListsTxtPath: '$(Build.SourcesDirectory)/CMakeLists.txt' buildDirectory: $(Build.ArtifactStagingDirectory)/${{ parameters.targetPlatform }} useVcpkgToolchainFile: true - cmakeAppendedArgs: '-G Ninja -DENABLE_XUNIT_OUTPUT=TRUE -DADDITIONAL_LIT_FLAGS=-j$(testParallelism)' + cmakeAppendedArgs: | + -G Ninja -DENABLE_TESTS=True -DENABLE_XUNIT_OUTPUT=TRUE -DADDITIONAL_LIT_FLAGS=-j$(testParallelism) - task: PowerShell@2 displayName: 'Run Tests' timeoutInMinutes: 120 From 0ffb6f1d4333eebbf53a61943c2496cc6e46ecd0 Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Fri, 27 Mar 2020 13:29:21 -0700 Subject: [PATCH 02/19] First pass at adding testing instructions to the README --- README.md | 45 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 43 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index baaca174e16..8e22afdb2c1 100644 --- a/README.md +++ b/README.md @@ -31,8 +31,9 @@ flavor of the STL (native desktop). We need to extend this to build all of the f because they need to be updated whenever source files are added/renamed/deleted. We'll delete the legacy machinery as soon as possible.) -* Tests: **Coming soon.** We rely on three test suites: std, tr1, and [libcxx][]. We need to replace our current test -harness, which extensively uses Microsoft-internal machinery. +* Tests: **In progress.** We rely on three test suites: std, tr1, and [libcxx][]. We've partially ported std and fully +ported libcxx to run under [lit][] using the various configuration/compilers we test internally. tr1 and portions of +std are still a work in progress. * Continuous Integration: **In progress.** We've set up Azure Pipelines to validate changes to the repository. Currently, it builds the STL (native desktop for x86, x64, ARM, and ARM64). Also, it strictly verifies that all of our @@ -220,6 +221,45 @@ C:\Users\bion\Desktop>dumpbin /IMPORTS .\example.exe | findstr msvcp msvcp140d_oss.dll ``` +# How To Run The Tests + +In order to run any of the tests you must be in a correctly configured environment for the architecture you are +targeting, have python 3.8 or newer and have `clang-cl` on the PATH. If you are testing the x64 flavor of the STL then +running from the "x64 Native Tools Command Prompt" is sufficient for setting up the environment. + +## Running All The Tests + +After configuring the project, running `ctest` from the root of the build output directory will run all the tests. +CTest will only display the standard error output of tests which failed. In order to get more details from CTest's +`lit` invocations, run the tests with `ctest -V`. + +## Running A Subset Of The Tests + +In order to run a subset of one of the testsuites you must invoke `lit` yourself and point it to the desired +subdirectory of tests. + +The CMake configuration step produces a Python script, `${PROJECT_BINARY_DIR}\tests\llvm-lit\llvm-lit.py`. +`llvm-lit.py` simply calls into the standard `lit` found in `llvm-project\llvm\utils\lit` with certain options +pre-configured. Namely a mapping of `lit.cfg` files to `lit.site.cfg` files is provided, which informs lit which +site-specific config file to load whenever it discovers a general config file. + +While one can directly use `lit` to run our tests by hand, we only support using the generated `llvm-lit`. + +### Examples + +``` +C:\Dev\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake .. + +:: This command will run all of the std testsuite. +C:\Dev\STL\build>python tests\llvm-lit\llvm-lit.py tests\std + +:: However if you want to run a subset of std you need to point it to the right place in the sources. +C:\Dev\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std\tests\VSO_0000000_any_calling_conventions + +:: The same applies for libcxx. The following will run all the libcxx containers tests. +C:\Dev\STL\build>python tests\llvm-lit\llvm-lit.py ..\llvm-project\libcxx\test\std\containers +``` + # Block Diagram The STL is built atop other compiler support libraries that ship with Windows and Visual Studio, like the UCRT, @@ -270,6 +310,7 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception [enhancement tag]: https://github.com/microsoft/STL/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement [hub]: https://support.microsoft.com/en-us/help/4021566/windows-10-send-feedback-to-microsoft-with-feedback-hub-app [libcxx]: https://libcxx.llvm.org +[lit]: http://llvm.org/docs/CommandGuide/lit.html [opencode@microsoft.com]: mailto:opencode@microsoft.com [redistributables]: https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads [vcpkg]: https://github.com/microsoft/vcpkg From 6ef27ba5a826174af02c7b193c439351aab96142 Mon Sep 17 00:00:00 2001 From: Curtis J Bezault Date: Fri, 27 Mar 2020 18:07:48 -0700 Subject: [PATCH 03/19] Apply suggestions from code review Co-Authored-By: Stephan T. Lavavej Co-Authored-By: Casey Carter --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 8e22afdb2c1..28634250d70 100644 --- a/README.md +++ b/README.md @@ -223,14 +223,14 @@ C:\Users\bion\Desktop>dumpbin /IMPORTS .\example.exe | findstr msvcp # How To Run The Tests -In order to run any of the tests you must be in a correctly configured environment for the architecture you are -targeting, have python 3.8 or newer and have `clang-cl` on the PATH. If you are testing the x64 flavor of the STL then +In order to run any of the tests, you must be in a correctly configured environment for the architecture you are +targeting, have Python 3.8 or newer, and have `clang-cl` on the PATH. If you are testing the x64 flavor of the STL then running from the "x64 Native Tools Command Prompt" is sufficient for setting up the environment. ## Running All The Tests -After configuring the project, running `ctest` from the root of the build output directory will run all the tests. -CTest will only display the standard error output of tests which failed. In order to get more details from CTest's +After configuring the project, running `ctest` from the build output directory will run all the tests. +CTest will only display the standard error output of tests that failed. In order to get more details from CTest's `lit` invocations, run the tests with `ctest -V`. ## Running A Subset Of The Tests @@ -245,7 +245,7 @@ site-specific config file to load whenever it discovers a general config file. While one can directly use `lit` to run our tests by hand, we only support using the generated `llvm-lit`. -### Examples +## Examples ``` C:\Dev\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake .. @@ -253,7 +253,7 @@ C:\Dev\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=.. :: This command will run all of the std testsuite. C:\Dev\STL\build>python tests\llvm-lit\llvm-lit.py tests\std -:: However if you want to run a subset of std you need to point it to the right place in the sources. +:: However, if you want to run a subset of std, you need to point it to the right place in the sources. C:\Dev\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std\tests\VSO_0000000_any_calling_conventions :: The same applies for libcxx. The following will run all the libcxx containers tests. @@ -310,7 +310,7 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception [enhancement tag]: https://github.com/microsoft/STL/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement [hub]: https://support.microsoft.com/en-us/help/4021566/windows-10-send-feedback-to-microsoft-with-feedback-hub-app [libcxx]: https://libcxx.llvm.org -[lit]: http://llvm.org/docs/CommandGuide/lit.html +[lit]: https://llvm.org/docs/CommandGuide/lit.html [opencode@microsoft.com]: mailto:opencode@microsoft.com [redistributables]: https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads [vcpkg]: https://github.com/microsoft/vcpkg From 323e0d679ee627aa9048b98b81a16cc7edeb23b1 Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Fri, 27 Mar 2020 19:43:27 -0700 Subject: [PATCH 04/19] Incorporate Casey's and STL's feedback --- README.md | 46 ++++++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 28634250d70..7ebad16bc08 100644 --- a/README.md +++ b/README.md @@ -221,43 +221,44 @@ C:\Users\bion\Desktop>dumpbin /IMPORTS .\example.exe | findstr msvcp msvcp140d_oss.dll ``` -# How To Run The Tests +# How To Run The Tests From The Developer Command Prompt For VS -In order to run any of the tests, you must be in a correctly configured environment for the architecture you are -targeting, have Python 3.8 or newer, and have `clang-cl` on the PATH. If you are testing the x64 flavor of the STL then -running from the "x64 Native Tools Command Prompt" is sufficient for setting up the environment. +Before running the tests you must configure and build the project as described above. You must also have [Python][] 3.8 +or newer, and have LLVM's `bin` directory on the PATH. ## Running All The Tests -After configuring the project, running `ctest` from the build output directory will run all the tests. -CTest will only display the standard error output of tests that failed. In order to get more details from CTest's +After configuring and building the project, running `ctest` from the build output directory will run all the tests. +CTest will only display the standard error output of tests which failed. In order to get more details from CTest's `lit` invocations, run the tests with `ctest -V`. ## Running A Subset Of The Tests -In order to run a subset of one of the testsuites you must invoke `lit` yourself and point it to the desired -subdirectory of tests. - -The CMake configuration step produces a Python script, `${PROJECT_BINARY_DIR}\tests\llvm-lit\llvm-lit.py`. -`llvm-lit.py` simply calls into the standard `lit` found in `llvm-project\llvm\utils\lit` with certain options -pre-configured. Namely a mapping of `lit.cfg` files to `lit.site.cfg` files is provided, which informs lit which -site-specific config file to load whenever it discovers a general config file. - -While one can directly use `lit` to run our tests by hand, we only support using the generated `llvm-lit`. +`${PROJECT_BINARY_DIR}\tests\llvm-lit\llvm-lit.py` can be invoked on a subdirectory of a testsuite and will execute all +the tests under that subdirectory. This can mean executing the entirety of a single testsuite, running all tests under +a category in libcxx, or running a single test in `std` and `tr1`. ## Examples ``` -C:\Dev\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake .. +:: These two commands configure and build the stl. They are unecessary if you have already followed the build steps +:: as described above. +C:\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake .. +C:\STL\build>ninja + +:: This command will run all of the testsuites with verbose output. +C:\STL\build>ctest -V -:: This command will run all of the std testsuite. -C:\Dev\STL\build>python tests\llvm-lit\llvm-lit.py tests\std +:: This command will run all of the std testuite. +C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std -:: However, if you want to run a subset of std, you need to point it to the right place in the sources. -C:\Dev\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std\tests\VSO_0000000_any_calling_conventions +:: If you want to run a subset of std you need to point it to the right place in the sources. The following will run +:: the single test found under VSO_0000000_any_calling_conventions. +C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std\tests\VSO_0000000_any_calling_conventions -:: The same applies for libcxx. The following will run all the libcxx containers tests. -C:\Dev\STL\build>python tests\llvm-lit\llvm-lit.py ..\llvm-project\libcxx\test\std\containers +:: You can invoke llvm-lit with any arbitrary subdirectory of a testuite. In libcxx this allows you to have a finer +:: control over what category of tests you would like to run. The following will run all the libcxx map tests. +C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\llvm-project\libcxx\test\std\containers\associative\map ``` # Block Diagram @@ -303,6 +304,7 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception [NOTICE.txt]: NOTICE.txt [Ninja]: https://ninja-build.org [Pipelines]: https://dev.azure.com/vclibs/STL/_build/latest?definitionId=2&branchName=master +[Python]: https://www.python.org/downloads/windows/ [Roadmap]: https://github.com/microsoft/STL/wiki/Roadmap [Wandbox]: https://wandbox.org [bug tag]: https://github.com/microsoft/STL/issues?q=is%3Aopen+is%3Aissue+label%3Abug From 38b701b15f66dc132ba9154010dff7dff5566bf9 Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 08:38:39 -0700 Subject: [PATCH 05/19] Make instruction to run better --- README.md | 43 ++++++++++++++++++++++++++------------ azure-devops/run-build.yml | 2 +- 2 files changed, 31 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 7ebad16bc08..769b0e22dfc 100644 --- a/README.md +++ b/README.md @@ -31,9 +31,8 @@ flavor of the STL (native desktop). We need to extend this to build all of the f because they need to be updated whenever source files are added/renamed/deleted. We'll delete the legacy machinery as soon as possible.) -* Tests: **In progress.** We rely on three test suites: std, tr1, and [libcxx][]. We've partially ported std and fully -ported libcxx to run under [lit][] using the various configuration/compilers we test internally. tr1 and portions of -std are still a work in progress. +* Tests: **In progress.** We rely on three test suites: std, tr1, and [libcxx][]. We've partially ported std and tr1, +and fully ported libcxx to run under [lit][] using the various configuration/compilers we test internally. * Continuous Integration: **In progress.** We've set up Azure Pipelines to validate changes to the repository. Currently, it builds the STL (native desktop for x86, x64, ARM, and ARM64). Also, it strictly verifies that all of our @@ -223,13 +222,24 @@ C:\Users\bion\Desktop>dumpbin /IMPORTS .\example.exe | findstr msvcp # How To Run The Tests From The Developer Command Prompt For VS -Before running the tests you must configure and build the project as described above. You must also have [Python][] 3.8 -or newer, and have LLVM's `bin` directory on the PATH. +1. Follow steps 1-9 of +[How To Build With A Native Tools Command Prompt](#how-to-build-with-a-native-tools-command-prompt). +2. Invoke `git submodule init llvm-project` +3. Invoke `git submodule update` +4. Invoke `cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE={where your vcpkg clone is located}\scripts\buildsystems\vcpkg.cmake +-DENABLE_TESTS=TRUE -S . -B {wherever you want binaries}`. This differs from above only in `-DENABLE_TESTS=TRUE`. +5. If you have already followed the steps from +[How To Build With A Native Tools Command Prompt](#how-to-build-with-a-native-tools-command-prompt), and have not +changed the value of {wherever you want binaries} in step four, then there is no need to rebuild to run the tests. +Otherwise, invoke `ninja -C {wherever you want binaries}` to build the project. + +In addition to following the above steps you must also have [Python][] 3.8 or newer, and have LLVM's `bin` directory on +the PATH. ## Running All The Tests After configuring and building the project, running `ctest` from the build output directory will run all the tests. -CTest will only display the standard error output of tests which failed. In order to get more details from CTest's +CTest will only display the standard error output of tests that failed. In order to get more details from CTest's `lit` invocations, run the tests with `ctest -V`. ## Running A Subset Of The Tests @@ -241,19 +251,26 @@ a category in libcxx, or running a single test in `std` and `tr1`. ## Examples ``` -:: These two commands configure and build the stl. They are unecessary if you have already followed the build steps -:: as described above. -C:\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake .. +:: This command configures the project with tests enabled. It assumes you are using the vcpkg submodule and have +:: already installed boost. It also assumes you have inited and updated the llvm-project submodule. +C:\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake -DENABLE_TESTS=TRUE .. + +:: As stated above this step is only strictly necessary if you have yet to build the STL or if you have changed the +:: output directory of the binaries. Any changes or additions in any of the existing testsuites do not require +:: recompilation or reconfiguration to take effect when running the tests. C:\STL\build>ninja :: This command will run all of the testsuites with verbose output. C:\STL\build>ctest -V -:: This command will run all of the std testuite. -C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std +:: This command will run all of the tr1 testuite +C:\STL\build>ctest -R tr1 + +:: This command will also run all of the tr1 testuite. +C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\tr1 -:: If you want to run a subset of std you need to point it to the right place in the sources. The following will run -:: the single test found under VSO_0000000_any_calling_conventions. +:: If you want to run a subset of a testsuite you need to point it to the right place in the sources. The following +:: will run the single test found under VSO_0000000_any_calling_conventions. C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std\tests\VSO_0000000_any_calling_conventions :: You can invoke llvm-lit with any arbitrary subdirectory of a testuite. In libcxx this allows you to have a finer diff --git a/azure-devops/run-build.yml b/azure-devops/run-build.yml index 7b4f131d436..e19fc9027e8 100644 --- a/azure-devops/run-build.yml +++ b/azure-devops/run-build.yml @@ -77,7 +77,7 @@ jobs: buildDirectory: $(Build.ArtifactStagingDirectory)/${{ parameters.targetPlatform }} useVcpkgToolchainFile: true cmakeAppendedArgs: | - -G Ninja -DENABLE_TESTS=True -DENABLE_XUNIT_OUTPUT=TRUE -DADDITIONAL_LIT_FLAGS=-j$(testParallelism) + -G Ninja -DENABLE_TESTS=TRUE -DENABLE_XUNIT_OUTPUT=TRUE -DADDITIONAL_LIT_FLAGS=-j$(testParallelism) - task: PowerShell@2 displayName: 'Run Tests' timeoutInMinutes: 120 From eb63d3e276ef124b988397f84921b9baa5609f8a Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 08:55:07 -0700 Subject: [PATCH 06/19] Fix trailing whitespace --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 769b0e22dfc..50b40f14f54 100644 --- a/README.md +++ b/README.md @@ -228,7 +228,7 @@ C:\Users\bion\Desktop>dumpbin /IMPORTS .\example.exe | findstr msvcp 3. Invoke `git submodule update` 4. Invoke `cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE={where your vcpkg clone is located}\scripts\buildsystems\vcpkg.cmake -DENABLE_TESTS=TRUE -S . -B {wherever you want binaries}`. This differs from above only in `-DENABLE_TESTS=TRUE`. -5. If you have already followed the steps from +5. If you have already followed the steps from [How To Build With A Native Tools Command Prompt](#how-to-build-with-a-native-tools-command-prompt), and have not changed the value of {wherever you want binaries} in step four, then there is no need to rebuild to run the tests. Otherwise, invoke `ninja -C {wherever you want binaries}` to build the project. From 25541e5309912d277ff1d669cfd17b5bb4956595 Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 11:09:06 -0700 Subject: [PATCH 07/19] Skip another flaky test --- tests/libcxx/expected_results.txt | 1 + tests/libcxx/skipped_tests.txt | 1 + 2 files changed, 2 insertions(+) diff --git a/tests/libcxx/expected_results.txt b/tests/libcxx/expected_results.txt index 39a88e49139..3d49a8274da 100644 --- a/tests/libcxx/expected_results.txt +++ b/tests/libcxx/expected_results.txt @@ -733,6 +733,7 @@ std/thread/futures/futures.async/async.pass.cpp SKIP std/thread/futures/futures.shared_future/get.pass.cpp SKIP std/thread/futures/futures.shared_future/wait_for.pass.cpp SKIP std/thread/futures/futures.shared_future/wait.pass.cpp SKIP +std/thread/futures/futures.unique_future/get.pass.cpp SKIP std/thread/futures/futures.unique_future/wait_for.pass.cpp SKIP std/thread/futures/futures.unique_future/wait.pass.cpp SKIP std/thread/thread.condition/thread.condition.condvar/notify_all.pass.cpp SKIP diff --git a/tests/libcxx/skipped_tests.txt b/tests/libcxx/skipped_tests.txt index 27551a81790..efb8e47769e 100644 --- a/tests/libcxx/skipped_tests.txt +++ b/tests/libcxx/skipped_tests.txt @@ -733,6 +733,7 @@ thread\futures\futures.async\async.pass.cpp thread\futures\futures.shared_future\get.pass.cpp thread\futures\futures.shared_future\wait_for.pass.cpp thread\futures\futures.shared_future\wait.pass.cpp +thread\futures\futures.unique_future\get.pass.cpp thread\futures\futures.unique_future\wait_for.pass.cpp thread\futures\futures.unique_future\wait.pass.cpp thread\thread.condition\thread.condition.condvar\notify_all.pass.cpp From c36f88b80faa39ade745c77d4d58981489d17c84 Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 12:48:55 -0700 Subject: [PATCH 08/19] Add interpreting results --- README.md | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 84 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 50b40f14f54..f6cf26cbafd 100644 --- a/README.md +++ b/README.md @@ -224,11 +224,10 @@ C:\Users\bion\Desktop>dumpbin /IMPORTS .\example.exe | findstr msvcp 1. Follow steps 1-9 of [How To Build With A Native Tools Command Prompt](#how-to-build-with-a-native-tools-command-prompt). -2. Invoke `git submodule init llvm-project` -3. Invoke `git submodule update` -4. Invoke `cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE={where your vcpkg clone is located}\scripts\buildsystems\vcpkg.cmake +2. Invoke `git submodule update --init llvm-project` +3. Invoke `cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE={where your vcpkg clone is located}\scripts\buildsystems\vcpkg.cmake -DENABLE_TESTS=TRUE -S . -B {wherever you want binaries}`. This differs from above only in `-DENABLE_TESTS=TRUE`. -5. If you have already followed the steps from +4. If you have already followed the steps from [How To Build With A Native Tools Command Prompt](#how-to-build-with-a-native-tools-command-prompt), and have not changed the value of {wherever you want binaries} in step four, then there is no need to rebuild to run the tests. Otherwise, invoke `ninja -C {wherever you want binaries}` to build the project. @@ -278,6 +277,86 @@ C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std\tests\VSO_0000000_an C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\llvm-project\libcxx\test\std\containers\associative\map ``` +## Interpreting The Results Of Tests + +## CTest + +When running the tests via CTest each of the testsuites is considered to be a single test. If any single test in a +testsuite fails, CTest will report the test which represents that testsuite as failed. + +Example: +``` +67% tests passed, 1 tests failed out of 3 + +Total Test time (real) = 2441.55 sec + +The following tests FAILED: + 1 - libcxx (Failed) +``` + +CTest will output everything that was sent to stderr for each of the failed testsuites, which can be used to identify +which individual test within the testsuite failed. It can sometimes be helpful to run CTest with the `-V` option in +order to see the stdout of the tests. + +## llvm-lit + +When running the tests directly via the generated `llvm-lit.py` script the result of each test will be printed. The +format of each result is +`<[Result Code](#result-code-values)>: :: :`. + +Example: +``` +-- Testing: 28 tests, 12 workers -- +PASS: tr1 :: tests/cwchar1:01 (1 of 28) +PASS: tr1 :: tests/cwchar1:11 (2 of 28) +PASS: tr1 :: tests/cwchar1:02 (3 of 28) +PASS: tr1 :: tests/cwchar1:03 (4 of 28) +PASS: tr1 :: tests/cwchar1:00 (5 of 28) +PASS: tr1 :: tests/cwchar1:04 (6 of 28) +PASS: tr1 :: tests/cwchar1:05 (7 of 28) +PASS: tr1 :: tests/cwchar1:09 (8 of 28) +PASS: tr1 :: tests/cwchar1:06 (9 of 28) +UNSUPPORTED: tr1 :: tests/cwchar1:20 (10 of 28) +UNSUPPORTED: tr1 :: tests/cwchar1:21 (11 of 28) +UNSUPPORTED: tr1 :: tests/cwchar1:22 (12 of 28) +UNSUPPORTED: tr1 :: tests/cwchar1:23 (13 of 28) +UNSUPPORTED: tr1 :: tests/cwchar1:24 (14 of 28) +PASS: tr1 :: tests/cwchar1:07 (15 of 28) +PASS: tr1 :: tests/cwchar1:08 (16 of 28) +PASS: tr1 :: tests/cwchar1:10 (17 of 28) +PASS: tr1 :: tests/cwchar1:16 (18 of 28) +PASS: tr1 :: tests/cwchar1:17 (19 of 28) +PASS: tr1 :: tests/cwchar1:14 (20 of 28) +PASS: tr1 :: tests/cwchar1:12 (21 of 28) +PASS: tr1 :: tests/cwchar1:13 (22 of 28) +PASS: tr1 :: tests/cwchar1:19 (23 of 28) +PASS: tr1 :: tests/cwchar1:18 (24 of 28) +PASS: tr1 :: tests/cwchar1:15 (25 of 28) +PASS: tr1 :: tests/cwchar1:25 (26 of 28) +PASS: tr1 :: tests/cwchar1:26 (27 of 28) +PASS: tr1 :: tests/cwchar1:27 (28 of 28) + +Testing Time: 3.96s + Expected Passes : 23 + Unsupported Tests : 5 +``` + +In the above example we see that twenty-three tests succeeded and five were unsupported. + +### Result Code Values + +Our tests use the standard [lit result codes][] and the addition of a non-standard result code `SKIP`. For our tests +one need mostly only concern themselves with the `PASS`, `XFAIL`, `XPASS`, `FAIL`, and `UNSUPPORTED` result codes. + +The `PASS` and `FAIL` result codes are self explanatory. We want our tests to `PASS` and not `FAIL`. +The `XPASS` and `XFAIL` are less obvious. `XFAIL` is actually a successfull result and indicates that we expected the +test to fail and it did. `XPASS` is a failure result and indicates that we expected a test to fail but it passed. +Typically this means that the `expected_results.txt` file for the testsuite needs to be modified and a `FAIL` entry +needs to be removed. + +The `UNSUPPORTED` result code means that the requirements for a test are not met and so it will not be run. Currently +all tests which use the `/BE` or `clr:pure` flags are unsupported. + # Block Diagram The STL is built atop other compiler support libraries that ship with Windows and Visual Studio, like the UCRT, @@ -330,6 +409,7 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception [hub]: https://support.microsoft.com/en-us/help/4021566/windows-10-send-feedback-to-microsoft-with-feedback-hub-app [libcxx]: https://libcxx.llvm.org [lit]: https://llvm.org/docs/CommandGuide/lit.html +[lit result codes]: https://llvm.org/docs/CommandGuide/lit.html#test-status-results [opencode@microsoft.com]: mailto:opencode@microsoft.com [redistributables]: https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads [vcpkg]: https://github.com/microsoft/vcpkg From fbb5e4bdacc809399333a5410fc0eca1314e671f Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 12:56:47 -0700 Subject: [PATCH 09/19] No links in code block --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index f6cf26cbafd..b4dd61ce14d 100644 --- a/README.md +++ b/README.md @@ -301,8 +301,7 @@ order to see the stdout of the tests. ## llvm-lit When running the tests directly via the generated `llvm-lit.py` script the result of each test will be printed. The -format of each result is -`<[Result Code](#result-code-values)>: :: :`. +format of each result is <[Result Code](#result-code-values)>: :: :. Example: ``` From a5223544ebecbca381e10361ea179b8d3096ba90 Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 13:26:37 -0700 Subject: [PATCH 10/19] Just always use the cwd as TMP --- tests/tr1/tests/cwchar1/__init__.py | 2 -- tests/tr1/tests/cwchar1/custom_format.py | 17 ----------------- tests/tr1/tests/cwchar1/lit.local.cfg | 10 ---------- tests/utils/stl/test/format.py | 5 ++++- 4 files changed, 4 insertions(+), 30 deletions(-) delete mode 100644 tests/tr1/tests/cwchar1/__init__.py delete mode 100644 tests/tr1/tests/cwchar1/custom_format.py delete mode 100644 tests/tr1/tests/cwchar1/lit.local.cfg diff --git a/tests/tr1/tests/cwchar1/__init__.py b/tests/tr1/tests/cwchar1/__init__.py deleted file mode 100644 index 2ac2a854cb0..00000000000 --- a/tests/tr1/tests/cwchar1/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception diff --git a/tests/tr1/tests/cwchar1/custom_format.py b/tests/tr1/tests/cwchar1/custom_format.py deleted file mode 100644 index 456c91bb9bf..00000000000 --- a/tests/tr1/tests/cwchar1/custom_format.py +++ /dev/null @@ -1,17 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception - -from stl.test.format import STLTestFormat, TestStep - - -class CustomTestFormat(STLTestFormat): - def getTestSteps(self, test, lit_config, shared): - if shared.exec_file is None: - for step in super().getTestSteps(test, lit_config, shared): - yield step - else: - exec_env = test.cxx.compile_env - exec_env['TMP'] = str(shared.exec_dir) - - yield TestStep([str(shared.exec_file)], shared.exec_dir, - [shared.exec_file], exec_env) diff --git a/tests/tr1/tests/cwchar1/lit.local.cfg b/tests/tr1/tests/cwchar1/lit.local.cfg deleted file mode 100644 index c1dd34543ab..00000000000 --- a/tests/tr1/tests/cwchar1/lit.local.cfg +++ /dev/null @@ -1,10 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception - -import cwchar1.custom_format - -config.test_format = \ - cwchar1.custom_format.CustomTestFormat(config.test_format.cxx, - config.test_format.execute_external, - config.test_format.build_executor, - config.test_format.test_executor) diff --git a/tests/utils/stl/test/format.py b/tests/utils/stl/test/format.py index 717fb926c9c..db3b707d63e 100644 --- a/tests/utils/stl/test/format.py +++ b/tests/utils/stl/test/format.py @@ -254,8 +254,11 @@ def getBuildSteps(self, test, lit_config, shared): def getTestSteps(self, test, lit_config, shared): if shared.exec_file is not None: + exec_env = test.cxx.compile_env + exec_env['TMP'] = str(shared.exec_dir) + yield TestStep([str(shared.exec_file)], shared.exec_dir, - [shared.exec_file], test.cxx.compile_env) + [shared.exec_file], exec_env) elif test.path_in_suite[-1].endswith('.fail.cpp'): exec_dir = test.getExecDir() source_path = Path(test.getSourcePath()) From c21ac439608f1fcf6054a7b4a7325cbe031d84fd Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 13:37:10 -0700 Subject: [PATCH 11/19] some fixups --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b4dd61ce14d..0b0b3b7f0ad 100644 --- a/README.md +++ b/README.md @@ -252,7 +252,8 @@ a category in libcxx, or running a single test in `std` and `tr1`. ``` :: This command configures the project with tests enabled. It assumes you are using the vcpkg submodule and have :: already installed boost. It also assumes you have inited and updated the llvm-project submodule. -C:\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake -DENABLE_TESTS=TRUE .. +C:\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake^ + -DENABLE_TESTS=TRUE .. :: As stated above this step is only strictly necessary if you have yet to build the STL or if you have changed the :: output directory of the binaries. Any changes or additions in any of the existing testsuites do not require @@ -301,7 +302,7 @@ order to see the stdout of the tests. ## llvm-lit When running the tests directly via the generated `llvm-lit.py` script the result of each test will be printed. The -format of each result is <[Result Code](#result-code-values)>: :: :. +format of each result is {[Result Code](#result-code-values)}: {Testsuite Name} :: {Test Name}:{Configuration Number}. Example: ``` From 6bde406f00762695f8a1aa00f9ffef2a5dfc7d4c Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 14:41:52 -0700 Subject: [PATCH 12/19] Remove whitespace --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0b0b3b7f0ad..4aba87e5dee 100644 --- a/README.md +++ b/README.md @@ -341,7 +341,7 @@ Testing Time: 3.96s Unsupported Tests : 5 ``` -In the above example we see that twenty-three tests succeeded and five were unsupported. +In the above example we see that twenty-three tests succeeded and five were unsupported. ### Result Code Values From 00c1bfd46ee7463497804af721301d8b67b34d3d Mon Sep 17 00:00:00 2001 From: Curtis J Bezault Date: Mon, 30 Mar 2020 15:29:01 -0700 Subject: [PATCH 13/19] Update README.md Co-Authored-By: Casey Carter --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4aba87e5dee..c6979a765e1 100644 --- a/README.md +++ b/README.md @@ -349,7 +349,8 @@ Our tests use the standard [lit result codes][] and the addition of a non-standa one need mostly only concern themselves with the `PASS`, `XFAIL`, `XPASS`, `FAIL`, and `UNSUPPORTED` result codes. The `PASS` and `FAIL` result codes are self explanatory. We want our tests to `PASS` and not `FAIL`. -The `XPASS` and `XFAIL` are less obvious. `XFAIL` is actually a successfull result and indicates that we expected the +The `XPASS` and `XFAIL` are less obvious. `XFAIL` is actually a successful result and indicates that we expected the + test to fail and it did. `XPASS` is a failure result and indicates that we expected a test to fail but it passed. Typically this means that the `expected_results.txt` file for the testsuite needs to be modified and a `FAIL` entry needs to be removed. From dfe01143730b5df4070be61b0ce1bc48f1213496 Mon Sep 17 00:00:00 2001 From: Curtis J Bezault Date: Mon, 30 Mar 2020 20:10:45 -0700 Subject: [PATCH 14/19] Apply suggestions from code review Co-Authored-By: Stephan T. Lavavej --- README.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index c6979a765e1..2c43e4b4a51 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ because they need to be updated whenever source files are added/renamed/deleted. soon as possible.) * Tests: **In progress.** We rely on three test suites: std, tr1, and [libcxx][]. We've partially ported std and tr1, -and fully ported libcxx to run under [lit][] using the various configuration/compilers we test internally. +and fully ported libcxx to run under [lit][] using the various configurations/compilers we test internally. * Continuous Integration: **In progress.** We've set up Azure Pipelines to validate changes to the repository. Currently, it builds the STL (native desktop for x86, x64, ARM, and ARM64). Also, it strictly verifies that all of our @@ -229,7 +229,7 @@ C:\Users\bion\Desktop>dumpbin /IMPORTS .\example.exe | findstr msvcp -DENABLE_TESTS=TRUE -S . -B {wherever you want binaries}`. This differs from above only in `-DENABLE_TESTS=TRUE`. 4. If you have already followed the steps from [How To Build With A Native Tools Command Prompt](#how-to-build-with-a-native-tools-command-prompt), and have not -changed the value of {wherever you want binaries} in step four, then there is no need to rebuild to run the tests. +changed the value of `{wherever you want binaries}` in step 4, then there is no need to rebuild to run the tests. Otherwise, invoke `ninja -C {wherever you want binaries}` to build the project. In addition to following the above steps you must also have [Python][] 3.8 or newer, and have LLVM's `bin` directory on @@ -252,10 +252,10 @@ a category in libcxx, or running a single test in `std` and `tr1`. ``` :: This command configures the project with tests enabled. It assumes you are using the vcpkg submodule and have :: already installed boost. It also assumes you have inited and updated the llvm-project submodule. -C:\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake^ - -DENABLE_TESTS=TRUE .. +C:\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake ^ +-DENABLE_TESTS=TRUE .. -:: As stated above this step is only strictly necessary if you have yet to build the STL or if you have changed the +:: As stated above, this step is only strictly necessary if you have yet to build the STL or if you have changed the :: output directory of the binaries. Any changes or additions in any of the existing testsuites do not require :: recompilation or reconfiguration to take effect when running the tests. C:\STL\build>ninja @@ -263,17 +263,17 @@ C:\STL\build>ninja :: This command will run all of the testsuites with verbose output. C:\STL\build>ctest -V -:: This command will run all of the tr1 testuite +:: This command will run all of the tr1 testsuite. C:\STL\build>ctest -R tr1 -:: This command will also run all of the tr1 testuite. +:: This command will also run all of the tr1 testsuite. C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\tr1 :: If you want to run a subset of a testsuite you need to point it to the right place in the sources. The following :: will run the single test found under VSO_0000000_any_calling_conventions. C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std\tests\VSO_0000000_any_calling_conventions -:: You can invoke llvm-lit with any arbitrary subdirectory of a testuite. In libcxx this allows you to have a finer +:: You can invoke llvm-lit with any arbitrary subdirectory of a testsuite. In libcxx this allows you to have finer :: control over what category of tests you would like to run. The following will run all the libcxx map tests. C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\llvm-project\libcxx\test\std\containers\associative\map ``` @@ -341,22 +341,22 @@ Testing Time: 3.96s Unsupported Tests : 5 ``` -In the above example we see that twenty-three tests succeeded and five were unsupported. +In the above example we see that 23 tests succeeded and 5 were unsupported. ### Result Code Values Our tests use the standard [lit result codes][] and the addition of a non-standard result code `SKIP`. For our tests one need mostly only concern themselves with the `PASS`, `XFAIL`, `XPASS`, `FAIL`, and `UNSUPPORTED` result codes. -The `PASS` and `FAIL` result codes are self explanatory. We want our tests to `PASS` and not `FAIL`. -The `XPASS` and `XFAIL` are less obvious. `XFAIL` is actually a successful result and indicates that we expected the +The `PASS` and `FAIL` result codes are self-explanatory. We want our tests to `PASS` and not `FAIL`. +The `XPASS` and `XFAIL` result codes are less obvious. `XFAIL` is actually a successful result and indicates that we expected the test to fail and it did. `XPASS` is a failure result and indicates that we expected a test to fail but it passed. Typically this means that the `expected_results.txt` file for the testsuite needs to be modified and a `FAIL` entry needs to be removed. The `UNSUPPORTED` result code means that the requirements for a test are not met and so it will not be run. Currently -all tests which use the `/BE` or `clr:pure` flags are unsupported. +all tests which use the `/BE` or `/clr:pure` options are unsupported. # Block Diagram From ab16ecc8c685dd8b9981947928b8ea21319afac1 Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 20:36:39 -0700 Subject: [PATCH 15/19] Fix some stuff STL commented on --- README.md | 39 +++++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 2c43e4b4a51..6fd90c36ac1 100644 --- a/README.md +++ b/README.md @@ -222,13 +222,11 @@ C:\Users\bion\Desktop>dumpbin /IMPORTS .\example.exe | findstr msvcp # How To Run The Tests From The Developer Command Prompt For VS -1. Follow steps 1-9 of -[How To Build With A Native Tools Command Prompt](#how-to-build-with-a-native-tools-command-prompt). +1. Follow steps 1-9 of [How To Build With A Native Tools Command Prompt][]. 2. Invoke `git submodule update --init llvm-project` 3. Invoke `cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE={where your vcpkg clone is located}\scripts\buildsystems\vcpkg.cmake -DENABLE_TESTS=TRUE -S . -B {wherever you want binaries}`. This differs from above only in `-DENABLE_TESTS=TRUE`. -4. If you have already followed the steps from -[How To Build With A Native Tools Command Prompt](#how-to-build-with-a-native-tools-command-prompt), and have not +4. If you have already followed the steps from [How To Build With A Native Tools Command Prompt][], and have not changed the value of `{wherever you want binaries}` in step 4, then there is no need to rebuild to run the tests. Otherwise, invoke `ninja -C {wherever you want binaries}` to build the project. @@ -263,11 +261,11 @@ C:\STL\build>ninja :: This command will run all of the testsuites with verbose output. C:\STL\build>ctest -V -:: This command will run all of the tr1 testsuite. -C:\STL\build>ctest -R tr1 +:: This command will run all of the std testsuite. +C:\STL\build>ctest -R std -:: This command will also run all of the tr1 testsuite. -C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\tr1 +:: This command will also run all of the std testsuite. +C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std :: If you want to run a subset of a testsuite you need to point it to the right place in the sources. The following :: will run the single test found under VSO_0000000_any_calling_conventions. @@ -280,7 +278,7 @@ C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\llvm-project\libcxx\test\std\c ## Interpreting The Results Of Tests -## CTest +### CTest When running the tests via CTest each of the testsuites is considered to be a single test. If any single test in a testsuite fails, CTest will report the test which represents that testsuite as failed. @@ -299,7 +297,7 @@ CTest will output everything that was sent to stderr for each of the failed test which individual test within the testsuite failed. It can sometimes be helpful to run CTest with the `-V` option in order to see the stdout of the tests. -## llvm-lit +### llvm-lit When running the tests directly via the generated `llvm-lit.py` script the result of each test will be printed. The format of each result is {[Result Code](#result-code-values)}: {Testsuite Name} :: {Test Name}:{Configuration Number}. @@ -345,19 +343,27 @@ In the above example we see that 23 tests succeeded and 5 were unsupported. ### Result Code Values -Our tests use the standard [lit result codes][] and the addition of a non-standard result code `SKIP`. For our tests -one need mostly only concern themselves with the `PASS`, `XFAIL`, `XPASS`, `FAIL`, and `UNSUPPORTED` result codes. +Our tests use the standard [lit result codes][], and a non-standard result code: `SKIP`. For our tests, only the +`PASS`, `XFAIL`, `XPASS`, `FAIL`, and `UNSUPPORTED` standard result codes are relevant. The `PASS` and `FAIL` result codes are self-explanatory. We want our tests to `PASS` and not `FAIL`. -The `XPASS` and `XFAIL` result codes are less obvious. `XFAIL` is actually a successful result and indicates that we expected the -test to fail and it did. `XPASS` is a failure result and indicates that we expected a test to fail but it passed. -Typically this means that the `expected_results.txt` file for the testsuite needs to be modified and a `FAIL` entry -needs to be removed. +The `XPASS` and `XFAIL` result codes are less obvious. `XPASS` is actually a failure result and indicates that we +expected a test to fail but it passed. `XFAIL` is a successful result and indicates that we expected the test to fail +and it did. Typically an `XPASS` result means that the `expected_results.txt` file for the testsuite needs to be +modified. If the `XPASS` result is a test legitimately passing the usual course of action would be to remove a `FAIL` +entry from the `expected_results.txt`. However, some tests from `libcxx` mark themselves as `XFAIL` (meaning they +expect to fail) for features they have added tests for but have yet to implement in `libcxx`. If the STL implements +those features first the tests will begin passing unexpectedly for us and return `XPASS` results. In order to resolve +this it is necessary to add a `PASS` entry to the `expected_results.txt` of the testsuite in question. The `UNSUPPORTED` result code means that the requirements for a test are not met and so it will not be run. Currently all tests which use the `/BE` or `/clr:pure` options are unsupported. +The `SKIP` result code indicates that a given test was explicitly skipped by adding a `SKIP` entry to the +`expected_results.txt`. A test may be skipped for a number or reasons, which include, but are not limited to: it being +an incorrect test; it taking a very long time to run; it failing or passing for the incorrect reason. + # Block Diagram The STL is built atop other compiler support libraries that ship with Windows and Visual Studio, like the UCRT, @@ -393,6 +399,7 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception [Code of Conduct FAQ]: https://opensource.microsoft.com/codeofconduct/faq/ [Compiler Explorer]: https://godbolt.org [Developer Community]: https://developercommunity.visualstudio.com/spaces/62/index.html +[How To Build With A Native Tools Command Prompt]: #how-to-build-with-a-native-tools-command-prompt [LICENSE.txt]: LICENSE.txt [LWG issues]: https://cplusplus.github.io/LWG/lwg-toc.html [LWG tag]: https://github.com/microsoft/STL/issues?q=is%3Aopen+is%3Aissue+label%3ALWG From 9bf48dc343b0461f2034c0e58e87b7c27197f9ca Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Mon, 30 Mar 2020 20:40:18 -0700 Subject: [PATCH 16/19] Drop link from format string --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6fd90c36ac1..ea6fa3b059f 100644 --- a/README.md +++ b/README.md @@ -300,7 +300,7 @@ order to see the stdout of the tests. ### llvm-lit When running the tests directly via the generated `llvm-lit.py` script the result of each test will be printed. The -format of each result is {[Result Code](#result-code-values)}: {Testsuite Name} :: {Test Name}:{Configuration Number}. +format of each result is `{Result Code}: {Testsuite Name} :: {Test Name}:{Configuration Number}`. Example: ``` From 9a4df0391a0045d44d9b2fd85e29c32d9953608d Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Tue, 31 Mar 2020 08:09:34 -0700 Subject: [PATCH 17/19] Add newlines --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index ea6fa3b059f..3c0b7e8d0bd 100644 --- a/README.md +++ b/README.md @@ -250,29 +250,36 @@ a category in libcxx, or running a single test in `std` and `tr1`. ``` :: This command configures the project with tests enabled. It assumes you are using the vcpkg submodule and have :: already installed boost. It also assumes you have inited and updated the llvm-project submodule. + C:\STL\build>cmake -GNinja -DCMAKE_CXX_COMPILER=cl -DCMAKE_TOOLCHAIN_FILE=..\vcpkg\scripts\buildsystems\vcpkg.cmake ^ -DENABLE_TESTS=TRUE .. :: As stated above, this step is only strictly necessary if you have yet to build the STL or if you have changed the :: output directory of the binaries. Any changes or additions in any of the existing testsuites do not require :: recompilation or reconfiguration to take effect when running the tests. + C:\STL\build>ninja :: This command will run all of the testsuites with verbose output. + C:\STL\build>ctest -V :: This command will run all of the std testsuite. + C:\STL\build>ctest -R std :: This command will also run all of the std testsuite. + C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std :: If you want to run a subset of a testsuite you need to point it to the right place in the sources. The following :: will run the single test found under VSO_0000000_any_calling_conventions. + C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\tests\std\tests\VSO_0000000_any_calling_conventions :: You can invoke llvm-lit with any arbitrary subdirectory of a testsuite. In libcxx this allows you to have finer :: control over what category of tests you would like to run. The following will run all the libcxx map tests. + C:\STL\build>python tests\llvm-lit\llvm-lit.py ..\llvm-project\libcxx\test\std\containers\associative\map ``` From 90ddb7d8bce0a07d1ee14150beb04e13778a8ac9 Mon Sep 17 00:00:00 2001 From: "Curtis.Bezault" Date: Tue, 31 Mar 2020 08:17:05 -0700 Subject: [PATCH 18/19] Add a comma --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3c0b7e8d0bd..44d0a22cf1c 100644 --- a/README.md +++ b/README.md @@ -358,7 +358,7 @@ The `PASS` and `FAIL` result codes are self-explanatory. We want our tests to `P The `XPASS` and `XFAIL` result codes are less obvious. `XPASS` is actually a failure result and indicates that we expected a test to fail but it passed. `XFAIL` is a successful result and indicates that we expected the test to fail and it did. Typically an `XPASS` result means that the `expected_results.txt` file for the testsuite needs to be -modified. If the `XPASS` result is a test legitimately passing the usual course of action would be to remove a `FAIL` +modified. If the `XPASS` result is a test legitimately passing, the usual course of action would be to remove a `FAIL` entry from the `expected_results.txt`. However, some tests from `libcxx` mark themselves as `XFAIL` (meaning they expect to fail) for features they have added tests for but have yet to implement in `libcxx`. If the STL implements those features first the tests will begin passing unexpectedly for us and return `XPASS` results. In order to resolve From 754a8794142e00c912895ab87288ab84ff7e0fee Mon Sep 17 00:00:00 2001 From: Curtis J Bezault Date: Tue, 31 Mar 2020 13:06:52 -0700 Subject: [PATCH 19/19] Update README.md Co-Authored-By: Stephan T. Lavavej --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 44d0a22cf1c..48a53a4feaa 100644 --- a/README.md +++ b/README.md @@ -368,8 +368,10 @@ The `UNSUPPORTED` result code means that the requirements for a test are not met all tests which use the `/BE` or `/clr:pure` options are unsupported. The `SKIP` result code indicates that a given test was explicitly skipped by adding a `SKIP` entry to the -`expected_results.txt`. A test may be skipped for a number or reasons, which include, but are not limited to: it being -an incorrect test; it taking a very long time to run; it failing or passing for the incorrect reason. +`expected_results.txt`. A test may be skipped for a number of reasons, which include, but are not limited to: +* being an incorrect test +* taking a very long time to run +* failing or passing for the incorrect reason # Block Diagram