From 44eec63863374840513a0dd765796ab6087b2f4d Mon Sep 17 00:00:00 2001 From: Jacob Boddey Date: Sat, 9 Mar 2024 13:27:40 +0000 Subject: [PATCH 1/2] Update documentation --- README.md | 11 ++++---- modules/test/dns/README.md | 18 +++++++++++++ modules/test/nmap/README.md | 27 +++++++++++++++++++ modules/test/nmap/conf/module_config.json | 14 +++++----- modules/test/ntp/README.md | 18 +++++++++++++ modules/test/protocol/README.md | 19 +++++++++++++ modules/test/protocol/conf/module_config.json | 6 ++--- modules/test/tls/README.md | 18 +++++++++++++ 8 files changed, 116 insertions(+), 15 deletions(-) create mode 100644 modules/test/dns/README.md create mode 100644 modules/test/nmap/README.md create mode 100644 modules/test/ntp/README.md create mode 100644 modules/test/protocol/README.md create mode 100644 modules/test/tls/README.md diff --git a/README.md b/README.md index 0fbf71271..273d78391 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,6 @@ [![CodeQL](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql/badge.svg?branch=main)](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql) [![Testrun test suite](https://github.com/google/testrun/actions/workflows/testing.yml/badge.svg?branch=main&event=push)](https://github.com/google/testrun/actions/workflows/testing.yml) -Disclaimer: Testrun uses Google Analytics to learn about how our users use Testrun. By installing and running Testrun, you understand and accept the Terms of Service found [here](https://policies.google.com/technologies/partner-sites). - ## Introduction :wave: Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows manufacturers of IP capable devices to test their devices for the purposes of Device Qualification within the BOS program. @@ -42,7 +40,10 @@ When manual testing or configuration changes are required, Testrun will provide Once you have met the hardware and software requirements, you can get started with Testrun by following the [Get started guide](docs/get_started.md). ## Roadmap :chart_with_upwards_trend: -Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards. +Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards. For further information on upcoming features, check out the [Roadmap](docs/roadmap.pdf). + +## Accessibility :busts_in_silhouette: +We are proud to support accessibility throughout the Testrun product and constantly strive to provide an enjoyable experience for all of our users. You can read more about [Google and Accessibility here](https://www.google.co.uk/accessibility). ## Issue reporting :triangular_flag_on_post: If the application has come across a problem at any point during setup or use, please raise an issue under the [issues tab](https://github.com/auto-iot/test-run/issues). Issue templates exist for both bug reports and feature requests. If neither of these are appropriate for your issue, raise a blank issue instead. @@ -70,10 +71,10 @@ The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md) - IPv6 SLAAC - DNS - NTPv4 - + 4) Can I run Testrun on a virtual machine? - Testrun can be virtualized if the 2x ethernet adapters are passed through to a Virtual Box VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested. + Testrun can be virtualized if the 2x ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested. 5) Can I connect multiple devices to Testrun? diff --git a/modules/test/dns/README.md b/modules/test/dns/README.md new file mode 100644 index 000000000..13f0df5fd --- /dev/null +++ b/modules/test/dns/README.md @@ -0,0 +1,18 @@ +# DNS Test Module + +The DNS test module inspects the device's behavior when attempting to resolve hostnames. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| dns.network.hostname_resolution | Verifies that the device resolves hostnames | The device sends DNS requests | Required | +| dns.network.from_dhcp | Verifies that the device allows for a DNS server to be provided by the DHCP server | The device sends DNS requests to the DNS server provided by the DHCP server | Roadmap | \ No newline at end of file diff --git a/modules/test/nmap/README.md b/modules/test/nmap/README.md new file mode 100644 index 000000000..eae8a0bd0 --- /dev/null +++ b/modules/test/nmap/README.md @@ -0,0 +1,27 @@ +# Services Test Module + +The services test module checks for all running servers that are available to other devices on the network. Unsecure network services should be disabled. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| security.services.ftp | Check FTP port 20/21 is disabled and FTP is not running on any port | There is no FTP service running on any port | Required | +| security.ssh.version | If the device is running an SSH server ensure it is SSHv2 | SSH server is not running or service is SSHv2 | Required | +| security.services.telnet | Check TELNET port 23 is disabled and TELNET is not running on any port | There is no Telnet service running on any port | Required | +| security.services.smtp | Check SMTP ports 25, 465 and 587 are not enabled and SMTP is not running on any port | There is no SMTP service running on any port | Required | +| security.services.http | Check that there is no HTTP server running on any port | Device is unreachable on port 80 (or any other port) and only responds to HTTPS requests if required | Required | +| security.services.pop | Check POP ports 109 and 110 are disabled and POP is not running on any port | There is no POP service running on any port | Required | +| security.services.imap | Check IMAP port 143 is disabled and IMAP is not running on any port | There is no IMAP service running on any port | Required | +| security.services.snmpv3 | Check SNMP port 161/162 is disabled. If SNMP is an essential service, it should be v3 | Device is unreachable on port 161/162 unless SNMP is essential in which case it is SNMPv3 that is used | Required | +| security.services.vnc | Check VNS is disabled on any port | Device cannot be accessed via VNC on any port | Required | +| security.services.tftp | Check TFTP port 69 is disabled (UDP) | There is no TFTP service running on any port | Required | +| ntp.network.ntp_server | Check NTP port 123 is disabled and the device is not acting as an NTP server | The devices does not respond to NTP requests | Required | \ No newline at end of file diff --git a/modules/test/nmap/conf/module_config.json b/modules/test/nmap/conf/module_config.json index 8fc50f0f5..1c1115afe 100644 --- a/modules/test/nmap/conf/module_config.json +++ b/modules/test/nmap/conf/module_config.json @@ -68,7 +68,7 @@ { "name": "security.services.telnet", "test_description": "Check TELNET port 23 is disabled and TELNET is not running on any port", - "expected_behavior": "There is no FTP service running on any port", + "expected_behavior": "There is no Telnet service running on any port", "required_result": "Required", "config": { "services": [ @@ -93,7 +93,7 @@ { "name": "security.services.smtp", "test_description": "Check SMTP ports 25, 465 and 587 are not enabled and SMTP is not running on any port.", - "expected_behavior": "There is no smtp service running on any port", + "expected_behavior": "There is no SMTP service running on any port", "required_result": "Required", "config": { "services": [ @@ -156,7 +156,7 @@ { "name": "security.services.pop", "test_description": "Check POP ports 109 and 110 are disabled and POP is not running on any port", - "expected_behavior": "There is no pop service running on any port", + "expected_behavior": "There is no POP service running on any port", "required_result": "Required", "config": { "services": [ @@ -198,7 +198,7 @@ { "name": "security.services.imap", "test_description": "Check IMAP port 143 is disabled and IMAP is not running on any port", - "expected_behavior": "There is no imap service running on any port", + "expected_behavior": "There is no IMAP service running on any port", "required_result": "Required", "config": { "services": [ @@ -272,7 +272,7 @@ { "name": "security.services.vnc", "test_description": "Check VNC is disabled on any port", - "expected_behavior": "Device cannot be accessed /connected to via VNC on any port", + "expected_behavior": "Device cannot be accessed / connected to via VNC on any port", "required_result": "Required", "config": { "services": [ @@ -327,7 +327,7 @@ { "name": "security.services.tftp", "test_description": "Check TFTP port 69 is disabled (UDP)", - "expected_behavior": "There is no tftp service running on any port", + "expected_behavior": "There is no TFTP service running on any port", "required_result": "Required", "config": { "services": [ @@ -360,7 +360,7 @@ { "name": "ntp.network.ntp_server", "test_description": "Check NTP port 123 is disabled and the device is not operating as an NTP server", - "expected_behavior": "The device dos not respond to NTP requests when it's IP is set as the NTP server on another device", + "expected_behavior": "The device does not respond to NTP requests when it's IP is set as the NTP server on another device", "required_result": "Required", "config": { "services": [ diff --git a/modules/test/ntp/README.md b/modules/test/ntp/README.md new file mode 100644 index 000000000..36e9981a6 --- /dev/null +++ b/modules/test/ntp/README.md @@ -0,0 +1,18 @@ +# NTP Test Module + +The NTP test module verifies the device behavior when syncing time with an NTP server. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| ntp.network.ntp_support | Does the device request network time using NTPv4 | The device sends an NTPv4 request to the configured NTP server | Required | +| ntp.network.ntp_dhcp | Checks the device can accept an NTP server address from the DHCP server | Device can accept NTP server address and sends an NTP request to that server | Roadmap | \ No newline at end of file diff --git a/modules/test/protocol/README.md b/modules/test/protocol/README.md new file mode 100644 index 000000000..765fbf758 --- /dev/null +++ b/modules/test/protocol/README.md @@ -0,0 +1,19 @@ +# Protocol Test Module + +The protocol test module verifies whether the device communicates using BMS protocols. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| protocol.valid_bacnet | Can valid BACnet traffic be seen | BACnet traffic can be seen on the network and packets are valid | Required if Applicable | +| protocol.bacnet.version | Obtain the version of BACnet client used | The BACnet client implements an up to date version of BACnet | Recommended | +| protocol.valid_modbus | Can valid Modbus traffic be seen | Any Modbus functionality works as expected and valid Modbus traffic can be observed | Recommended | \ No newline at end of file diff --git a/modules/test/protocol/conf/module_config.json b/modules/test/protocol/conf/module_config.json index 486751fb1..365bd346b 100644 --- a/modules/test/protocol/conf/module_config.json +++ b/modules/test/protocol/conf/module_config.json @@ -21,14 +21,14 @@ }, { "name": "protocol.bacnet.version", - "test_description": "Can valid BACnet traffic be seen", - "expected_behavior": "BACnet traffic can be seen on the network and packets are valid and not malformed", + "test_description": "Obtain the version of BACnet client used", + "expected_behavior": "The BACnet client implements an up to date version of BACnet", "required_result": "Recommended" }, { "name": "protocol.valid_modbus", "test_description": "Can valid Modbus traffic be seen", - "expected_behavior": "Any Modbus functionality works as expected and valid modbus traffic can be observed", + "expected_behavior": "Any Modbus functionality works as expected and valid Modbus traffic can be observed", "required_result": "Recommended", "config":{ "port": 502, diff --git a/modules/test/tls/README.md b/modules/test/tls/README.md new file mode 100644 index 000000000..ba1c6b1db --- /dev/null +++ b/modules/test/tls/README.md @@ -0,0 +1,18 @@ +# TLS Test Module + +The TLS test module verifies that any peer or cloud connections are secure. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| security.tls.v1_2_server | Check the device web server is TLSv1.2 minimum and the certificate is valid | TLS 1.2 certificate is issues to the client when accessed | Required | +| security.tls.v1_2_client | Device uses TLS with connections to external services on any port | The packet indicates a TLS connection with at least TLS v1.2 and support for ECDH and ECDSA ciphers | Required | \ No newline at end of file From 91007114a68a8a6b4b14873a6740c8fcdc7f7ce4 Mon Sep 17 00:00:00 2001 From: Jacob Boddey Date: Sat, 11 May 2024 15:33:08 +0100 Subject: [PATCH 2/2] Update docs --- docs/get_started.md | 2 +- docs/test/statuses.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/get_started.md b/docs/get_started.md index a7b893d13..0bf3b9f2c 100644 --- a/docs/get_started.md +++ b/docs/get_started.md @@ -113,4 +113,4 @@ If you encounter any issues or need assistance, consider the following: Once you have completed a test attempt, you may want to review the test report provided by Testrun. For more information about what Testrun looks for when testing, and what the output means, take a look at the testing documentation: [Testing](/docs/test/index.md). # Uninstall -To uninstall Testrun, use the built-in dpkg uninstall command to remove Testrun correctly. For Testrun, this would be: ```sudo apt-get remove testrun```. Note that this +To uninstall Testrun, use the built-in dpkg uninstall command to remove Testrun correctly. For Testrun, this would be: ```sudo apt-get remove testrun```. diff --git a/docs/test/statuses.md b/docs/test/statuses.md index 5a2ba626d..c83259fb3 100644 --- a/docs/test/statuses.md +++ b/docs/test/statuses.md @@ -9,7 +9,7 @@ Testrun will output the result and description of each automated test. The test | Error | An error occured whilst running the test | Create a bug report requesting additional support to diagnose the issue | | Skipped | The test has not been executed because a linked test did not produce a compliant result | You may implement the functionality (not required) | -## Test requirement +## Test Requirement Testrun also determines whether each test is required for the device to receive an overall compliant result. These rules are: | Name | Description |