diff --git a/README.md b/README.md index fa49f5ed4..404de4915 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ Testrun logo ## Introduction :wave: -Testrun is a tool to automate the validation of network-based functionality of IoT devices. Any device which is capable of receiving an IP address via DHCP is considered an IoT device by Testrun and can be tested. +Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows external manufacturers to test their devices for the purposes of Device Qualification within the BOS program. ## Motivation :bulb: Without tools like Testrun, testing labs may be maintaining a large and complex network using equipment such as: A managed layer 3 switch, an enterprise-grade network router, virtualized or physical servers to provide DNS, NTP, 802.1x etc. With this amount of moving parts, all with dynamic configuration files and constant software updates, more time is likely to be spent on preparation and clean up of functinality or penetration testing - not forgetting the number of software tools required to perform the testing. The major issues which can and should be solved: diff --git a/docs/get_started.md b/docs/get_started.md index 31f48181c..f201ee8b4 100644 --- a/docs/get_started.md +++ b/docs/get_started.md @@ -1,5 +1,7 @@ # Getting Started +It is recommended that you run Testrun on a standalone machine running a fresh-install of Ubuntu 22.04.3 LTS. + ## Prerequisites ### Hardware @@ -24,29 +26,66 @@ Ensure the following software is installed on your Ubuntu LTS PC: ## Installation -1. Download the latest version of Testrun from the [releases page](https://github.com/google/test-run/releases) +1. Download the latest version of the Testrun installer from the [releases page](https://github.com/google/test-run/releases) -2. Install the package using ``sudo apt install ./testrun*.deb`` +2. Open a terminal and navigate to location of the Testrun installer (most likely your downloads folder) -## Test Your Device +3. Install the package using ``sudo apt install ./testrun*.deb`` -1. Attach network interfaces: + - Testrun will be installed under the /usr/local/testrun directory. + - Testing data will be available in the ``local/devices/{device}/reports`` folders + - Additional configuration options are available in the ``local/system.json`` file - - Connect one USB Ethernet adapter to the internet source (e.g., router or switch) using an Ethernet cable. - - Connect the other USB Ethernet adapter directly to the IoT device you want to test using an Ethernet cable. +## Start Testrun + +1. Attach network interfaces: + - Connect one USB Ethernet adapter to the internet source (e.g., router or switch) using an ethernet cable. + - Connect the other USB Ethernet adapter directly to the IoT device you want to test using an ethernet cable. **NOTE: Both adapters should be disabled in the host system (IPv4, IPv6 and general). You can do this by going to Settings > Network** 2. Start Testrun. -Start Testrun with the command `sudo testrun --no-validate` +Start Testrun with the command `sudo testrun` - To run Testrun in network-only mode (without running any tests), use the `--net-only` option. - - To skip network validation before use and not launch the faux device on startup, use the `--no-validate` option. - - To run Testrun with just one interface (connected to the device), use the ``--single-intf`` option. +## Test Your Device + +1. Once Testrun has started, open your browser to http://localhost:8080. + +2. Configure your network interfaces under the settings menu - located in the top right corner of the application. Settings can be changed at any time. + + ![](/docs/ui/settings_icon.png) + +3. Navigate to the device repository icon to add a new device for testing. + + ![](/docs/ui/device_icon.png) + +4. Click the button 'Add Device'. + +5. Enter the MAC address, manufacturer name and model number. + +6. Select the test modules you wish to enable for this device (Hint: All are required for qualification purposes) and click save. + +7. Navigate to the Testrun progress icon and click the button 'Start New Testrun'. + + ![](/docs/ui/progress_icon.png) + +8. Select the device you would like to test. + +9. Enter the version number of the firmware running on the device. + +10. Click 'Start Testrun' + + - During testing, if you would like to stop Testrun, click 'Stop' next to the test name. + +11. On completion of the test sequence, a report will appear under the history icon. + + ![](/docs/ui/history_icon.png) + # Troubleshooting If you encounter any issues or need assistance, consider the following: diff --git a/docs/ui/device_icon.png b/docs/ui/device_icon.png new file mode 100644 index 000000000..2472f7da2 Binary files /dev/null and b/docs/ui/device_icon.png differ diff --git a/docs/ui/history_icon.png b/docs/ui/history_icon.png new file mode 100644 index 000000000..eb95a8663 Binary files /dev/null and b/docs/ui/history_icon.png differ diff --git a/docs/ui/progress_icon.png b/docs/ui/progress_icon.png new file mode 100644 index 000000000..c326d185e Binary files /dev/null and b/docs/ui/progress_icon.png differ diff --git a/docs/ui/settings_icon.png b/docs/ui/settings_icon.png new file mode 100644 index 000000000..8fc83b9bb Binary files /dev/null and b/docs/ui/settings_icon.png differ diff --git a/docs/ui/settings_menu.png b/docs/ui/settings_menu.png new file mode 100644 index 000000000..046526b25 Binary files /dev/null and b/docs/ui/settings_menu.png differ diff --git a/docs/ui/test_name.png b/docs/ui/test_name.png new file mode 100644 index 000000000..3d18df19d Binary files /dev/null and b/docs/ui/test_name.png differ diff --git a/framework/python/src/core/test_runner.py b/framework/python/src/core/test_runner.py index 9962c3995..f3db52eec 100644 --- a/framework/python/src/core/test_runner.py +++ b/framework/python/src/core/test_runner.py @@ -34,7 +34,7 @@ class TestRunner: def __init__(self, config_file=None, - validate=True, + validate=False, net_only=False, single_intf=False, no_ui=False): @@ -76,9 +76,10 @@ def parse_args(): help="Define the configuration file for Test Run and Network Orchestrator" ) parser.add_argument( - "--no-validate", + "--validate", + default=False, action="store_true", - help="Turn off the validation of the network after network boot") + help="Turn on the validation of the network after network boot") parser.add_argument("-net", "--net-only", action="store_true", @@ -97,7 +98,7 @@ def parse_args(): if __name__ == "__main__": args = parse_args() runner = TestRunner(config_file=args.config_file, - validate=not args.no_validate, + validate=args.validate, net_only=args.net_only, single_intf=args.single_intf, no_ui=args.no_ui) diff --git a/framework/python/src/core/testrun.py b/framework/python/src/core/testrun.py index e1b3f9aab..d66f599e3 100644 --- a/framework/python/src/core/testrun.py +++ b/framework/python/src/core/testrun.py @@ -66,7 +66,7 @@ class TestRun: # pylint: disable=too-few-public-methods def __init__(self, config_file, - validate=True, + validate=False, net_only=False, single_intf=False, no_ui=False): @@ -91,8 +91,8 @@ def __init__(self, self._session.add_runtime_param('single_intf') if net_only: self._session.add_runtime_param('net_only') - if not validate: - self._session.add_runtime_param('no-validate') + if validate: + self._session.add_runtime_param('validate') self.load_all_devices() @@ -290,6 +290,7 @@ def start(self): self.get_net_orc().start_listener() self._set_status('Waiting for Device') LOGGER.info('Waiting for devices on the network...') + time.sleep(self.get_session().get_runtime()) if not (self._test_orc.test_in_progress() or diff --git a/framework/python/src/net_orc/network_orchestrator.py b/framework/python/src/net_orc/network_orchestrator.py index 852748953..05733dfe0 100644 --- a/framework/python/src/net_orc/network_orchestrator.py +++ b/framework/python/src/net_orc/network_orchestrator.py @@ -119,7 +119,7 @@ def start_network(self): self.create_net() self.start_network_services() - if 'no-validate' not in self._session.get_runtime_params(): + if 'validate' in self._session.get_runtime_params(): # Start the validator after network is ready self.validator.start() diff --git a/testing/baseline/test_baseline b/testing/baseline/test_baseline index 4bc6d9e9f..7ecc6ba19 100755 --- a/testing/baseline/test_baseline +++ b/testing/baseline/test_baseline @@ -43,7 +43,7 @@ sudo cp testing/baseline/system.json $TESTRUN_DIR/local/system.json # Copy device configs to testrun sudo cp -r testing/device_configs/* $TESTRUN_DIR/local/devices -sudo testrun --single-intf --no-ui > $TESTRUN_OUT 2>&1 & +sudo testrun --single-intf --no-ui --validate > $TESTRUN_OUT 2>&1 & TPID=$! # Time to wait for testrun to be ready diff --git a/testing/tests/test_tests b/testing/tests/test_tests index da24d0d64..7277d3e7f 100755 --- a/testing/tests/test_tests +++ b/testing/tests/test_tests @@ -62,7 +62,7 @@ for tester in $TESTERS; do args=$(jq -r .$tester.args $MATRIX) touch $testrun_log - sudo timeout 900 testrun --single-intf --no-ui --no-validate > $testrun_log 2>&1 & + sudo timeout 900 testrun --single-intf --no-ui > $testrun_log 2>&1 & TPID=$! # Time to wait for testrun to be ready