Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 9 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,37 +5,35 @@
[![Testrun test suite](https://github.com/google/testrun/actions/workflows/testing.yml/badge.svg?branch=main&event=push)](https://github.com/google/testrun/actions/workflows/testing.yml)

## Introduction :wave:
Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows external manufacturers to test their devices for the purposes of Device Qualification within the BOS program.
Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows manufacturers of IP capable devices to test their devices for the purposes of Device Qualification within the BOS program.

## Motivation :bulb:
Without tools like Testrun, testing labs may be maintaining a large and complex network using equipment such as: A managed layer 3 switch, an enterprise-grade network router, virtualized or physical servers to provide DNS, NTP, 802.1x etc. With this amount of moving parts, all with dynamic configuration files and constant software updates, more time is likely to be spent on preparation and clean up of functinality or penetration testing - not forgetting the number of software tools required to perform the testing. The major issues which can and should be solved:
Without tools like Testrun, test labs and engineers may need to maintain a large and complex network coupled with dynamic configuration files and constant software updates. The major issues which can and should be solved are:
1) The complexity of managing a testing network
2) The time required to perform testing of network functionality
3) The accuracy and consistency of testing network functionality

## How it works :triangular_ruler:
Testrun creates an isolated and controlled network environment to fully simulate enterprise network deployments in your device testing lab.
This removes the necessity for complex hardware, advanced knowledge and networking experience whilst enabling semi-technical engineers to validate device
behaviour against industry cyber standards.
Testrun creates an isolated and controlled network environment on a linux machine. This removes the necessity for complex hardware, advanced knowledge and networking experience whilst enabling test engineers to validate device behaviour against Google’s Building Operating System requirements.

Two runtime modes are supported by Testrun:
Two modes are supported by Testrun:

1) <strong>Automated testing</strong>

Once the device has become operational (steady state), automated testing of the device under test will begin. Containerized test modules will then execute against the device (one module at a time). Once all test modules have completed execution, a final test report will be produced - presenting the results and further description of findings.
Once the device has become operational (steady state), automated testing of the DUT (device under test) will begin. Containerized test modules will then execute against the device, one module at a time. Once all test modules have been executed, a report will be produced - presenting the results.

2) <strong>Lab network</strong>

Testrun cannot automate everything, and so additional manual testing may be required (or configuration changes may be required on the device). Rather than having to maintain a separate but idential lab network, Testrun will provide the network and some tools to assist an engineer performing the additional testing. At the same time, packet captures of the device behaviour will be recorded, alongside logs for each network service, for further debugging.
When manual testing or configuration changes are required, Testrun will provide the network and some tools to assist an engineer performing the additional testing. This reduces the need to maintain a separate but identical lab network. Testrun will take care of packet captures and logs for each network service for further debugging.

## Minimum requirements :computer:
### Hardware
- PC running Ubuntu LTS (laptop or desktop)
- PC running Ubuntu LTS 20.04 or 22.04 (laptop or desktop)
- 2x USB ethernet adapter (One may be built in ethernet)
- Internet connection
### Software
- Docker - installation guide: [https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository)
### Device
### Device under test (DUT)
- DHCP client - The device must be able to obtain an IP address via DHCP

## Get started ▶️
Expand Down Expand Up @@ -68,8 +66,7 @@ The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md)

3) Can I run Testrun on a virtual machine?

Probably. Provided that the required 2x USB ethernet adapters are passed to the virtual machine as USB devices rather than network adapters, Testrun should
still work. We will look to test and approve the use of virtualisation to run Testrun in the future.
Testrun can be virtualized if the 2x ethernet adapters are passed through to a Virtual Box VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested.

4) Can I connect multiple devices to Testrun?

Expand Down
6 changes: 0 additions & 6 deletions modules/test/dns/conf/module_config.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,6 @@
"test_description": "Verify the device allows for a DNS server to be entered automatically",
"expected_behavior": "The device sends DNS requests to the DNS server provided by the DHCP server",
"required_result": "Roadmap"
},
{
"name": "dns.mdns",
"test_description": "If the device has MDNS (or any kind of IP multicast), can it be disabled",
"expected_behavior": "Device may send MDNS requests",
"required_result": "Recommended"
}
]
}
Expand Down
12 changes: 0 additions & 12 deletions modules/test/tls/conf/module_config.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,11 @@
"expected_behavior": "TLS 1.2 certificate is issued to the web browser client when accessed",
"required_result": "Required"
},
{
"name": "security.tls.v1_3_server",
"test_description": "Check the device web server TLS 1.3 & certificate is valid",
"expected_behavior": "TLS 1.3 certificate is issued to the web browser client when accessed",
"required_result": "Recommended"
},
{
"name": "security.tls.v1_2_client",
"test_description": "Device uses TLS with connection to an external service on port 443 (or any other port which could be running the webserver-HTTPS)",
"expected_behavior": "The packet indicates a TLS connection with at least TLS 1.2 and support for ECDH and ECDSA ciphers",
"required_result": "Required"
},
{
"name": "security.tls.v1_3_client",
"test_description": "Device uses TLS with connection to an external service on port 443 (or any other port which could be running the webserver-HTTPS)",
"expected_behavior": "The packet indicates a TLS connection with at least TLS 1.3",
"required_result": "Recommended"
}
]
}
Expand Down
4 changes: 2 additions & 2 deletions modules/test/tls/python/src/tls_util.py
Original file line number Diff line number Diff line change
Expand Up @@ -380,8 +380,8 @@ def validate_tls_client(self, client_ip, tls_version, capture_file):
tls_client_details += 'Completed handshake detected from server: '
tls_client_details += result + '\n'
else:
LOGGER.info('No client hello packets detected. Skipping')
tls_client_details = 'No client hello packets detected. Skipping'
LOGGER.info('No client hello packets detected')
tls_client_details = 'No client hello packets detected'
return tls_client_valid, tls_client_details

def is_ecdh_and_ecdsa(self, ciphers):
Expand Down