Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/get_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,5 +109,8 @@ If you encounter any issues or need assistance, consider the following:
- Check the configuration settings.
- Refer to the Testrun documentation or ask for assistance in the issues page: https://github.com/google/testrun/issues

# Reviewing
Once you have completed a test attempt, you may want to review the test report provided by Testrun. For more information about what Testrun looks for when testing, and what the output means, take a look at the testing documentation: [Testing](/docs/test/index.md).

# Uninstall
To uninstall Testrun, use the built-in dpkg uninstall command to remove Testrun correctly. For Testrun, this would be: ```sudo apt-get remove testrun```
To uninstall Testrun, use the built-in dpkg uninstall command to remove Testrun correctly. For Testrun, this would be: ```sudo apt-get remove testrun```. Note that this
5 changes: 5 additions & 0 deletions docs/test/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Testing

The test requirements that are investigated by Testrun can be found in the [test modules documentation](/docs/test/modules.md).

To understand the testing results, various definitions of test results and requirements are specified in the [statuses documentation](/docs/test/statuses.md).
32 changes: 32 additions & 0 deletions docs/test/statuses.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Test Statuses
Testrun will output the result and description of each automated test. The test results will be one of the following:

| Name | Description | What next? |
|---|---|---|
| Compliant | The device implements the required feature correctly | Nothing |
| Non-Compliant | The device does not support the specified requirements for the test | Modify or implement the required functionality on the device |
| Feature Not Present | The device does not implement a feature covered by the test | You may implement the functionality (not required) |
| Error | An error occured whilst running the test | Create a bug report requesting additional support to diagnose the issue |
| Skipped | The test has not been executed because a linked test did not produce a compliant result | You may implement the functionality (not required) |

## Test requirement
Testrun also determines whether each test is required for the device to receive an overall compliant result. These rules are:

| Name | Description |
|---|---|
| Required | The device must implement the feature |
| Recommended | The device should implement the feature, but will not receive an overall Non-Compliant if not implemented |
| Roadmap | The device should implement this feature in the future, but is not required at the moment |
| Required If Applicable | If the device implements this feature, it must be implemented correctly (as per the test requirements) |

## Testrun Statuses
Once testing is completed, an overall result for the test attempt will be produced. This is calculated by comparing the result of all tests, and whether they are required or not required.

### Compliant
All required tests are implemented correctly, and all required if applicable tests are implemented correctly (where the feature has been implemented).

### Non-Compliant
One or more of the required tests (or required if applicable tests) have produced a non-compliant result.

### Error
One of more of the required tests (or required if applicable tests) have not executed correctly. This does not necessarily indicate that the device is compliant or non-compliant.