From 6ba0aba61515b7aeed0d0d834b788e34f0640b6c Mon Sep 17 00:00:00 2001 From: Jeremy Walker Date: Tue, 5 Jan 2021 16:36:20 +0000 Subject: [PATCH 1/3] Specify that tests must be run in order In the Research Site, the responsibly of ordering the tests lay with the website. It used the `config.json` file as its basis for ordering. Now that file has been deprecated, the responsibility for ordering moves to the individual test-runners. --- anatomy/track-tooling/test-runners/interface.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/anatomy/track-tooling/test-runners/interface.md b/anatomy/track-tooling/test-runners/interface.md index aa5c69fd..febf6990 100644 --- a/anatomy/track-tooling/test-runners/interface.md +++ b/anatomy/track-tooling/test-runners/interface.md @@ -53,6 +53,15 @@ Where the status is `error` (the tests fail to execute cleanly), the top level ` When the status is not `error`, either set the value to `null` or omit the key entirely. +#### Tests + +> key: `tests` + +This is an array of the test results, specified in the "Per-test" section below. + +The tests **MUST** be returned in the order they are specified in the tests file. +For languages that execute tests in a random order, this may mean re-ordering the results in line with the order specified in the tests file. + ### Per-test #### Name From 57dd0e65f1a33a13958e5061d5ca9665f90a1d86 Mon Sep 17 00:00:00 2001 From: Erik Schierboom Date: Wed, 6 Jan 2021 09:36:20 +0100 Subject: [PATCH 2/3] Update anatomy/track-tooling/test-runners/interface.md Co-authored-by: Jeremy Walker --- anatomy/track-tooling/test-runners/interface.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/anatomy/track-tooling/test-runners/interface.md b/anatomy/track-tooling/test-runners/interface.md index febf6990..f11cb668 100644 --- a/anatomy/track-tooling/test-runners/interface.md +++ b/anatomy/track-tooling/test-runners/interface.md @@ -62,6 +62,8 @@ This is an array of the test results, specified in the "Per-test" section below. The tests **MUST** be returned in the order they are specified in the tests file. For languages that execute tests in a random order, this may mean re-ordering the results in line with the order specified in the tests file. +The rationale for this is that only the first failure is show to students and therefore it is important that the correct failure is shown. Because tests are generally ordered in that file in a TDD way, and because for Practice Exercises the students see the tests file in the editor, aligning the results with the file is critical. + ### Per-test #### Name From e604ea7b97caa199b35be945dd893a8fe780ca5c Mon Sep 17 00:00:00 2001 From: Jeremy Walker Date: Wed, 6 Jan 2021 12:53:09 +0000 Subject: [PATCH 3/3] Update anatomy/track-tooling/test-runners/interface.md Co-authored-by: Erik Schierboom --- anatomy/track-tooling/test-runners/interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/anatomy/track-tooling/test-runners/interface.md b/anatomy/track-tooling/test-runners/interface.md index f11cb668..dfceaf55 100644 --- a/anatomy/track-tooling/test-runners/interface.md +++ b/anatomy/track-tooling/test-runners/interface.md @@ -62,7 +62,7 @@ This is an array of the test results, specified in the "Per-test" section below. The tests **MUST** be returned in the order they are specified in the tests file. For languages that execute tests in a random order, this may mean re-ordering the results in line with the order specified in the tests file. -The rationale for this is that only the first failure is show to students and therefore it is important that the correct failure is shown. Because tests are generally ordered in that file in a TDD way, and because for Practice Exercises the students see the tests file in the editor, aligning the results with the file is critical. +The rationale for this is that only the first failure is shown to students and therefore it is important that the correct failure is shown. Because tests are generally ordered in the tests file in a TDD way, and because for Practice Exercises the students see the tests file in the editor, aligning the results with the tests file is critical. ### Per-test