diff --git a/README.md b/README.md index 0fbf71271..273d78391 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,6 @@ [![CodeQL](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql/badge.svg?branch=main)](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql) [![Testrun test suite](https://github.com/google/testrun/actions/workflows/testing.yml/badge.svg?branch=main&event=push)](https://github.com/google/testrun/actions/workflows/testing.yml) -Disclaimer: Testrun uses Google Analytics to learn about how our users use Testrun. By installing and running Testrun, you understand and accept the Terms of Service found [here](https://policies.google.com/technologies/partner-sites). - ## Introduction :wave: Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows manufacturers of IP capable devices to test their devices for the purposes of Device Qualification within the BOS program. @@ -42,7 +40,10 @@ When manual testing or configuration changes are required, Testrun will provide Once you have met the hardware and software requirements, you can get started with Testrun by following the [Get started guide](docs/get_started.md). ## Roadmap :chart_with_upwards_trend: -Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards. +Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards. For further information on upcoming features, check out the [Roadmap](docs/roadmap.pdf). + +## Accessibility :busts_in_silhouette: +We are proud to support accessibility throughout the Testrun product and constantly strive to provide an enjoyable experience for all of our users. You can read more about [Google and Accessibility here](https://www.google.co.uk/accessibility). ## Issue reporting :triangular_flag_on_post: If the application has come across a problem at any point during setup or use, please raise an issue under the [issues tab](https://github.com/auto-iot/test-run/issues). Issue templates exist for both bug reports and feature requests. If neither of these are appropriate for your issue, raise a blank issue instead. @@ -70,10 +71,10 @@ The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md) - IPv6 SLAAC - DNS - NTPv4 - + 4) Can I run Testrun on a virtual machine? - Testrun can be virtualized if the 2x ethernet adapters are passed through to a Virtual Box VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested. + Testrun can be virtualized if the 2x ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested. 5) Can I connect multiple devices to Testrun? diff --git a/docs/test/modules.md b/docs/test/modules.md index a3016e17f..40c66f29e 100644 --- a/docs/test/modules.md +++ b/docs/test/modules.md @@ -10,4 +10,5 @@ Testrun provides some pre-built test modules for you to use when testing your ow | DNS | Verify DNS functionality | [DNS module](/modules/test/dns/README.md) | | NMAP | Ensure unsecure services are disabled | [NMAP module](/modules/test/nmap/README.md) | | NTP | Verify NTP functionality | [NTP module](/modules/test/ntp/README.md) | -| TLS | Determine TLS client and server behavior | [TLS module](/modules/test/tls/README.md) | \ No newline at end of file +| Protocol | Inspect BMS protocol implementation | [Protocol Module](/modules/test/protocol/README.md) | +| TLS | Determine TLS client and server behavior | [TLS module](/modules/test/tls/README.md) | diff --git a/modules/test/dns/README.md b/modules/test/dns/README.md new file mode 100644 index 000000000..13f0df5fd --- /dev/null +++ b/modules/test/dns/README.md @@ -0,0 +1,18 @@ +# DNS Test Module + +The DNS test module inspects the device's behavior when attempting to resolve hostnames. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| dns.network.hostname_resolution | Verifies that the device resolves hostnames | The device sends DNS requests | Required | +| dns.network.from_dhcp | Verifies that the device allows for a DNS server to be provided by the DHCP server | The device sends DNS requests to the DNS server provided by the DHCP server | Roadmap | \ No newline at end of file diff --git a/modules/test/nmap/README.md b/modules/test/nmap/README.md new file mode 100644 index 000000000..eae8a0bd0 --- /dev/null +++ b/modules/test/nmap/README.md @@ -0,0 +1,27 @@ +# Services Test Module + +The services test module checks for all running servers that are available to other devices on the network. Unsecure network services should be disabled. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| security.services.ftp | Check FTP port 20/21 is disabled and FTP is not running on any port | There is no FTP service running on any port | Required | +| security.ssh.version | If the device is running an SSH server ensure it is SSHv2 | SSH server is not running or service is SSHv2 | Required | +| security.services.telnet | Check TELNET port 23 is disabled and TELNET is not running on any port | There is no Telnet service running on any port | Required | +| security.services.smtp | Check SMTP ports 25, 465 and 587 are not enabled and SMTP is not running on any port | There is no SMTP service running on any port | Required | +| security.services.http | Check that there is no HTTP server running on any port | Device is unreachable on port 80 (or any other port) and only responds to HTTPS requests if required | Required | +| security.services.pop | Check POP ports 109 and 110 are disabled and POP is not running on any port | There is no POP service running on any port | Required | +| security.services.imap | Check IMAP port 143 is disabled and IMAP is not running on any port | There is no IMAP service running on any port | Required | +| security.services.snmpv3 | Check SNMP port 161/162 is disabled. If SNMP is an essential service, it should be v3 | Device is unreachable on port 161/162 unless SNMP is essential in which case it is SNMPv3 that is used | Required | +| security.services.vnc | Check VNS is disabled on any port | Device cannot be accessed via VNC on any port | Required | +| security.services.tftp | Check TFTP port 69 is disabled (UDP) | There is no TFTP service running on any port | Required | +| ntp.network.ntp_server | Check NTP port 123 is disabled and the device is not acting as an NTP server | The devices does not respond to NTP requests | Required | \ No newline at end of file diff --git a/modules/test/nmap/conf/module_config.json b/modules/test/nmap/conf/module_config.json index 8fc50f0f5..1c1115afe 100644 --- a/modules/test/nmap/conf/module_config.json +++ b/modules/test/nmap/conf/module_config.json @@ -68,7 +68,7 @@ { "name": "security.services.telnet", "test_description": "Check TELNET port 23 is disabled and TELNET is not running on any port", - "expected_behavior": "There is no FTP service running on any port", + "expected_behavior": "There is no Telnet service running on any port", "required_result": "Required", "config": { "services": [ @@ -93,7 +93,7 @@ { "name": "security.services.smtp", "test_description": "Check SMTP ports 25, 465 and 587 are not enabled and SMTP is not running on any port.", - "expected_behavior": "There is no smtp service running on any port", + "expected_behavior": "There is no SMTP service running on any port", "required_result": "Required", "config": { "services": [ @@ -156,7 +156,7 @@ { "name": "security.services.pop", "test_description": "Check POP ports 109 and 110 are disabled and POP is not running on any port", - "expected_behavior": "There is no pop service running on any port", + "expected_behavior": "There is no POP service running on any port", "required_result": "Required", "config": { "services": [ @@ -198,7 +198,7 @@ { "name": "security.services.imap", "test_description": "Check IMAP port 143 is disabled and IMAP is not running on any port", - "expected_behavior": "There is no imap service running on any port", + "expected_behavior": "There is no IMAP service running on any port", "required_result": "Required", "config": { "services": [ @@ -272,7 +272,7 @@ { "name": "security.services.vnc", "test_description": "Check VNC is disabled on any port", - "expected_behavior": "Device cannot be accessed /connected to via VNC on any port", + "expected_behavior": "Device cannot be accessed / connected to via VNC on any port", "required_result": "Required", "config": { "services": [ @@ -327,7 +327,7 @@ { "name": "security.services.tftp", "test_description": "Check TFTP port 69 is disabled (UDP)", - "expected_behavior": "There is no tftp service running on any port", + "expected_behavior": "There is no TFTP service running on any port", "required_result": "Required", "config": { "services": [ @@ -360,7 +360,7 @@ { "name": "ntp.network.ntp_server", "test_description": "Check NTP port 123 is disabled and the device is not operating as an NTP server", - "expected_behavior": "The device dos not respond to NTP requests when it's IP is set as the NTP server on another device", + "expected_behavior": "The device does not respond to NTP requests when it's IP is set as the NTP server on another device", "required_result": "Required", "config": { "services": [ diff --git a/modules/test/ntp/README.md b/modules/test/ntp/README.md new file mode 100644 index 000000000..36e9981a6 --- /dev/null +++ b/modules/test/ntp/README.md @@ -0,0 +1,18 @@ +# NTP Test Module + +The NTP test module verifies the device behavior when syncing time with an NTP server. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| ntp.network.ntp_support | Does the device request network time using NTPv4 | The device sends an NTPv4 request to the configured NTP server | Required | +| ntp.network.ntp_dhcp | Checks the device can accept an NTP server address from the DHCP server | Device can accept NTP server address and sends an NTP request to that server | Roadmap | \ No newline at end of file diff --git a/modules/test/protocol/README.md b/modules/test/protocol/README.md new file mode 100644 index 000000000..765fbf758 --- /dev/null +++ b/modules/test/protocol/README.md @@ -0,0 +1,19 @@ +# Protocol Test Module + +The protocol test module verifies whether the device communicates using BMS protocols. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| protocol.valid_bacnet | Can valid BACnet traffic be seen | BACnet traffic can be seen on the network and packets are valid | Required if Applicable | +| protocol.bacnet.version | Obtain the version of BACnet client used | The BACnet client implements an up to date version of BACnet | Recommended | +| protocol.valid_modbus | Can valid Modbus traffic be seen | Any Modbus functionality works as expected and valid Modbus traffic can be observed | Recommended | \ No newline at end of file diff --git a/modules/test/protocol/conf/module_config.json b/modules/test/protocol/conf/module_config.json index 486751fb1..365bd346b 100644 --- a/modules/test/protocol/conf/module_config.json +++ b/modules/test/protocol/conf/module_config.json @@ -21,14 +21,14 @@ }, { "name": "protocol.bacnet.version", - "test_description": "Can valid BACnet traffic be seen", - "expected_behavior": "BACnet traffic can be seen on the network and packets are valid and not malformed", + "test_description": "Obtain the version of BACnet client used", + "expected_behavior": "The BACnet client implements an up to date version of BACnet", "required_result": "Recommended" }, { "name": "protocol.valid_modbus", "test_description": "Can valid Modbus traffic be seen", - "expected_behavior": "Any Modbus functionality works as expected and valid modbus traffic can be observed", + "expected_behavior": "Any Modbus functionality works as expected and valid Modbus traffic can be observed", "required_result": "Recommended", "config":{ "port": 502, diff --git a/modules/test/tls/README.md b/modules/test/tls/README.md new file mode 100644 index 000000000..ba1c6b1db --- /dev/null +++ b/modules/test/tls/README.md @@ -0,0 +1,18 @@ +# TLS Test Module + +The TLS test module verifies that any peer or cloud connections are secure. + +## What's inside? + +The ```bin``` folder contains the startup script for the module. + +The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out. + +Within the ```python/src``` directory, the below tests are executed. + +## Tests covered + +| ID | Description | Expected behavior | Required result +|---|---|---|---| +| security.tls.v1_2_server | Check the device web server is TLSv1.2 minimum and the certificate is valid | TLS 1.2 certificate is issues to the client when accessed | Required | +| security.tls.v1_2_client | Device uses TLS with connections to external services on any port | The packet indicates a TLS connection with at least TLS v1.2 and support for ECDH and ECDSA ciphers | Required | \ No newline at end of file