Currently, if the user selects tests for execution, we parse the tests's executables and extract all tests within those executables. The reason for this is two-fold:
- In the past, our impression was that the TestCase objects provided via
ITestExecutor.RunTests(IEnumerable<TestCase>, IRunContext, IFrameworkHandle) do not contain all information we put in there. However, we apparently were doing something wrong, since that information is indeed kept (at least with VS2017).
- Our
CommandLineGenerator needs information about all test cases to decide whether the test cases to be executed can be adressed by executing
- the whole executable,
- a complete test suite,
- single test cases or
- a mixture of the above
The CommandLineGenerator can already work without being provided all test cases (in addition to the test cases to be run). Not parsing the complete executable when executing a single test case speeds up execution enormously (>3s vs. <1s for a test of our LoadTests project). However, if executing all tests of the LoadTests project, execution time rises again, since we due to restrictions on the length of command line arguments, we have to run the executable several times.
We should evaluate adding additional information to the TestCases via TestProperties which enables the CommandLineGenerator to still use suite-based test filters (or no filter at all in case all tests of an executable are run). This could e.g. be achieved by storing the number of testcases per test suite and test executable within the properties of each test. This information could be used to decide whether we are given all tests of a suite/executable (and thus provide an according filter).
Currently, if the user selects tests for execution, we parse the tests's executables and extract all tests within those executables. The reason for this is two-fold:
ITestExecutor.RunTests(IEnumerable<TestCase>, IRunContext, IFrameworkHandle)do not contain all information we put in there. However, we apparently were doing something wrong, since that information is indeed kept (at least with VS2017).CommandLineGeneratorneeds information about all test cases to decide whether the test cases to be executed can be adressed by executingThe
CommandLineGeneratorcan already work without being provided all test cases (in addition to the test cases to be run). Not parsing the complete executable when executing a single test case speeds up execution enormously (>3s vs. <1s for a test of ourLoadTestsproject). However, if executing all tests of theLoadTestsproject, execution time rises again, since we due to restrictions on the length of command line arguments, we have to run the executable several times.We should evaluate adding additional information to the TestCases via
TestPropertieswhich enables theCommandLineGeneratorto still use suite-based test filters (or no filter at all in case all tests of an executable are run). This could e.g. be achieved by storing the number of testcases per test suite and test executable within the properties of each test. This information could be used to decide whether we are given all tests of a suite/executable (and thus provide an according filter).