diff --git a/docs/get_started.md b/docs/get_started.md index c9f508043..a7b893d13 100644 --- a/docs/get_started.md +++ b/docs/get_started.md @@ -109,5 +109,8 @@ If you encounter any issues or need assistance, consider the following: - Check the configuration settings. - Refer to the Testrun documentation or ask for assistance in the issues page: https://github.com/google/testrun/issues +# Reviewing +Once you have completed a test attempt, you may want to review the test report provided by Testrun. For more information about what Testrun looks for when testing, and what the output means, take a look at the testing documentation: [Testing](/docs/test/index.md). + # Uninstall -To uninstall Testrun, use the built-in dpkg uninstall command to remove Testrun correctly. For Testrun, this would be: ```sudo apt-get remove testrun``` +To uninstall Testrun, use the built-in dpkg uninstall command to remove Testrun correctly. For Testrun, this would be: ```sudo apt-get remove testrun```. Note that this diff --git a/docs/test/index.md b/docs/test/index.md new file mode 100644 index 000000000..b440a86f1 --- /dev/null +++ b/docs/test/index.md @@ -0,0 +1,5 @@ +# Testing + +The test requirements that are investigated by Testrun can be found in the [test modules documentation](/docs/test/modules.md). + +To understand the testing results, various definitions of test results and requirements are specified in the [statuses documentation](/docs/test/statuses.md). \ No newline at end of file diff --git a/docs/test/statuses.md b/docs/test/statuses.md new file mode 100644 index 000000000..5a2ba626d --- /dev/null +++ b/docs/test/statuses.md @@ -0,0 +1,32 @@ +# Test Statuses +Testrun will output the result and description of each automated test. The test results will be one of the following: + +| Name | Description | What next? | +|---|---|---| +| Compliant | The device implements the required feature correctly | Nothing | +| Non-Compliant | The device does not support the specified requirements for the test | Modify or implement the required functionality on the device | +| Feature Not Present | The device does not implement a feature covered by the test | You may implement the functionality (not required) | +| Error | An error occured whilst running the test | Create a bug report requesting additional support to diagnose the issue | +| Skipped | The test has not been executed because a linked test did not produce a compliant result | You may implement the functionality (not required) | + +## Test requirement +Testrun also determines whether each test is required for the device to receive an overall compliant result. These rules are: + +| Name | Description | +|---|---| +| Required | The device must implement the feature | +| Recommended | The device should implement the feature, but will not receive an overall Non-Compliant if not implemented | +| Roadmap | The device should implement this feature in the future, but is not required at the moment | +| Required If Applicable | If the device implements this feature, it must be implemented correctly (as per the test requirements) | + +## Testrun Statuses +Once testing is completed, an overall result for the test attempt will be produced. This is calculated by comparing the result of all tests, and whether they are required or not required. + +### Compliant +All required tests are implemented correctly, and all required if applicable tests are implemented correctly (where the feature has been implemented). + +### Non-Compliant +One or more of the required tests (or required if applicable tests) have produced a non-compliant result. + +### Error +One of more of the required tests (or required if applicable tests) have not executed correctly. This does not necessarily indicate that the device is compliant or non-compliant. \ No newline at end of file