Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@
[![CodeQL](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql/badge.svg?branch=main)](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql)
[![Testrun test suite](https://github.com/google/testrun/actions/workflows/testing.yml/badge.svg?branch=main&event=push)](https://github.com/google/testrun/actions/workflows/testing.yml)

<strong>Disclaimer</strong>: Testrun uses Google Analytics to learn about how our users use Testrun. By installing and running Testrun, you understand and accept the Terms of Service found [here](https://policies.google.com/technologies/partner-sites).

## Introduction :wave:
Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows manufacturers of IP capable devices to test their devices for the purposes of Device Qualification within the BOS program.

Expand Down Expand Up @@ -42,7 +40,10 @@ When manual testing or configuration changes are required, Testrun will provide
Once you have met the hardware and software requirements, you can get started with Testrun by following the [Get started guide](docs/get_started.md).

## Roadmap :chart_with_upwards_trend:
Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards.
Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards. For further information on upcoming features, check out the [Roadmap](docs/roadmap.pdf).

## Accessibility :busts_in_silhouette:
We are proud to support accessibility throughout the Testrun product and constantly strive to provide an enjoyable experience for all of our users. You can read more about [Google and Accessibility here](https://www.google.co.uk/accessibility).

## Issue reporting :triangular_flag_on_post:
If the application has come across a problem at any point during setup or use, please raise an issue under the [issues tab](https://github.com/auto-iot/test-run/issues). Issue templates exist for both bug reports and feature requests. If neither of these are appropriate for your issue, raise a blank issue instead.
Expand Down Expand Up @@ -70,10 +71,10 @@ The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md)
- IPv6 SLAAC
- DNS
- NTPv4

4) Can I run Testrun on a virtual machine?

Testrun can be virtualized if the 2x ethernet adapters are passed through to a Virtual Box VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested.
Testrun can be virtualized if the 2x ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested.

5) Can I connect multiple devices to Testrun?

Expand Down
3 changes: 2 additions & 1 deletion docs/test/modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ Testrun provides some pre-built test modules for you to use when testing your ow
| DNS | Verify DNS functionality | [DNS module](/modules/test/dns/README.md) |
| NMAP | Ensure unsecure services are disabled | [NMAP module](/modules/test/nmap/README.md) |
| NTP | Verify NTP functionality | [NTP module](/modules/test/ntp/README.md) |
| TLS | Determine TLS client and server behavior | [TLS module](/modules/test/tls/README.md) |
| Protocol | Inspect BMS protocol implementation | [Protocol Module](/modules/test/protocol/README.md) |
| TLS | Determine TLS client and server behavior | [TLS module](/modules/test/tls/README.md) |
18 changes: 18 additions & 0 deletions modules/test/dns/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# DNS Test Module

The DNS test module inspects the device's behavior when attempting to resolve hostnames.

## What's inside?

The ```bin``` folder contains the startup script for the module.

The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out.

Within the ```python/src``` directory, the below tests are executed.

## Tests covered

| ID | Description | Expected behavior | Required result
|---|---|---|---|
| dns.network.hostname_resolution | Verifies that the device resolves hostnames | The device sends DNS requests | Required |
| dns.network.from_dhcp | Verifies that the device allows for a DNS server to be provided by the DHCP server | The device sends DNS requests to the DNS server provided by the DHCP server | Roadmap |
27 changes: 27 additions & 0 deletions modules/test/nmap/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Services Test Module

The services test module checks for all running servers that are available to other devices on the network. Unsecure network services should be disabled.

## What's inside?

The ```bin``` folder contains the startup script for the module.

The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out.

Within the ```python/src``` directory, the below tests are executed.

## Tests covered

| ID | Description | Expected behavior | Required result
|---|---|---|---|
| security.services.ftp | Check FTP port 20/21 is disabled and FTP is not running on any port | There is no FTP service running on any port | Required |
| security.ssh.version | If the device is running an SSH server ensure it is SSHv2 | SSH server is not running or service is SSHv2 | Required |
| security.services.telnet | Check TELNET port 23 is disabled and TELNET is not running on any port | There is no Telnet service running on any port | Required |
| security.services.smtp | Check SMTP ports 25, 465 and 587 are not enabled and SMTP is not running on any port | There is no SMTP service running on any port | Required |
| security.services.http | Check that there is no HTTP server running on any port | Device is unreachable on port 80 (or any other port) and only responds to HTTPS requests if required | Required |
| security.services.pop | Check POP ports 109 and 110 are disabled and POP is not running on any port | There is no POP service running on any port | Required |
| security.services.imap | Check IMAP port 143 is disabled and IMAP is not running on any port | There is no IMAP service running on any port | Required |
| security.services.snmpv3 | Check SNMP port 161/162 is disabled. If SNMP is an essential service, it should be v3 | Device is unreachable on port 161/162 unless SNMP is essential in which case it is SNMPv3 that is used | Required |
| security.services.vnc | Check VNS is disabled on any port | Device cannot be accessed via VNC on any port | Required |
| security.services.tftp | Check TFTP port 69 is disabled (UDP) | There is no TFTP service running on any port | Required |
| ntp.network.ntp_server | Check NTP port 123 is disabled and the device is not acting as an NTP server | The devices does not respond to NTP requests | Required |
14 changes: 7 additions & 7 deletions modules/test/nmap/conf/module_config.json
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@
{
"name": "security.services.telnet",
"test_description": "Check TELNET port 23 is disabled and TELNET is not running on any port",
"expected_behavior": "There is no FTP service running on any port",
"expected_behavior": "There is no Telnet service running on any port",
"required_result": "Required",
"config": {
"services": [
Expand All @@ -93,7 +93,7 @@
{
"name": "security.services.smtp",
"test_description": "Check SMTP ports 25, 465 and 587 are not enabled and SMTP is not running on any port.",
"expected_behavior": "There is no smtp service running on any port",
"expected_behavior": "There is no SMTP service running on any port",
"required_result": "Required",
"config": {
"services": [
Expand Down Expand Up @@ -156,7 +156,7 @@
{
"name": "security.services.pop",
"test_description": "Check POP ports 109 and 110 are disabled and POP is not running on any port",
"expected_behavior": "There is no pop service running on any port",
"expected_behavior": "There is no POP service running on any port",
"required_result": "Required",
"config": {
"services": [
Expand Down Expand Up @@ -198,7 +198,7 @@
{
"name": "security.services.imap",
"test_description": "Check IMAP port 143 is disabled and IMAP is not running on any port",
"expected_behavior": "There is no imap service running on any port",
"expected_behavior": "There is no IMAP service running on any port",
"required_result": "Required",
"config": {
"services": [
Expand Down Expand Up @@ -272,7 +272,7 @@
{
"name": "security.services.vnc",
"test_description": "Check VNC is disabled on any port",
"expected_behavior": "Device cannot be accessed /connected to via VNC on any port",
"expected_behavior": "Device cannot be accessed / connected to via VNC on any port",
"required_result": "Required",
"config": {
"services": [
Expand Down Expand Up @@ -327,7 +327,7 @@
{
"name": "security.services.tftp",
"test_description": "Check TFTP port 69 is disabled (UDP)",
"expected_behavior": "There is no tftp service running on any port",
"expected_behavior": "There is no TFTP service running on any port",
"required_result": "Required",
"config": {
"services": [
Expand Down Expand Up @@ -360,7 +360,7 @@
{
"name": "ntp.network.ntp_server",
"test_description": "Check NTP port 123 is disabled and the device is not operating as an NTP server",
"expected_behavior": "The device dos not respond to NTP requests when it's IP is set as the NTP server on another device",
"expected_behavior": "The device does not respond to NTP requests when it's IP is set as the NTP server on another device",
"required_result": "Required",
"config": {
"services": [
Expand Down
18 changes: 18 additions & 0 deletions modules/test/ntp/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# NTP Test Module

The NTP test module verifies the device behavior when syncing time with an NTP server.

## What's inside?

The ```bin``` folder contains the startup script for the module.

The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out.

Within the ```python/src``` directory, the below tests are executed.

## Tests covered

| ID | Description | Expected behavior | Required result
|---|---|---|---|
| ntp.network.ntp_support | Does the device request network time using NTPv4 | The device sends an NTPv4 request to the configured NTP server | Required |
| ntp.network.ntp_dhcp | Checks the device can accept an NTP server address from the DHCP server | Device can accept NTP server address and sends an NTP request to that server | Roadmap |
19 changes: 19 additions & 0 deletions modules/test/protocol/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Protocol Test Module

The protocol test module verifies whether the device communicates using BMS protocols.

## What's inside?

The ```bin``` folder contains the startup script for the module.

The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out.

Within the ```python/src``` directory, the below tests are executed.

## Tests covered

| ID | Description | Expected behavior | Required result
|---|---|---|---|
| protocol.valid_bacnet | Can valid BACnet traffic be seen | BACnet traffic can be seen on the network and packets are valid | Required if Applicable |
| protocol.bacnet.version | Obtain the version of BACnet client used | The BACnet client implements an up to date version of BACnet | Recommended |
| protocol.valid_modbus | Can valid Modbus traffic be seen | Any Modbus functionality works as expected and valid Modbus traffic can be observed | Recommended |
6 changes: 3 additions & 3 deletions modules/test/protocol/conf/module_config.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,14 @@
},
{
"name": "protocol.bacnet.version",
"test_description": "Can valid BACnet traffic be seen",
"expected_behavior": "BACnet traffic can be seen on the network and packets are valid and not malformed",
"test_description": "Obtain the version of BACnet client used",
"expected_behavior": "The BACnet client implements an up to date version of BACnet",
"required_result": "Recommended"
},
{
"name": "protocol.valid_modbus",
"test_description": "Can valid Modbus traffic be seen",
"expected_behavior": "Any Modbus functionality works as expected and valid modbus traffic can be observed",
"expected_behavior": "Any Modbus functionality works as expected and valid Modbus traffic can be observed",
"required_result": "Recommended",
"config":{
"port": 502,
Expand Down
18 changes: 18 additions & 0 deletions modules/test/tls/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# TLS Test Module

The TLS test module verifies that any peer or cloud connections are secure.

## What's inside?

The ```bin``` folder contains the startup script for the module.

The ```config/module_config.json``` provides the name and description of the module, and specifies which tests will be caried out.

Within the ```python/src``` directory, the below tests are executed.

## Tests covered

| ID | Description | Expected behavior | Required result
|---|---|---|---|
| security.tls.v1_2_server | Check the device web server is TLSv1.2 minimum and the certificate is valid | TLS 1.2 certificate is issues to the client when accessed | Required |
| security.tls.v1_2_client | Device uses TLS with connections to external services on any port | The packet indicates a TLS connection with at least TLS v1.2 and support for ECDH and ECDSA ciphers | Required |