Add docs, disable network validation by default

README.md

@@ -1,7 +1,7 @@
   <img width="200" alt="Testrun logo" src="https://user-images.githubusercontent.com/7399056/221927867-4190a4e8-a571-4e40-9c2b-65780ad9264c.png" alt="Testrun">
 
 ## Introduction :wave:
-Testrun is a tool to automate the validation of network-based functionality of IoT devices. Any device which is capable of receiving an IP address via DHCP is considered an IoT device by Testrun and can be tested.
+Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows external manufacturers to test their devices for the purposes of Device Qualification within the BOS program.
 
 ## Motivation :bulb:
 Without tools like Testrun, testing labs may be maintaining a large and complex network using equipment such as: A managed layer 3 switch, an enterprise-grade network router, virtualized or physical servers to provide DNS, NTP, 802.1x etc. With this amount of moving parts, all with dynamic configuration files and constant software updates, more time is likely to be spent on preparation and clean up of functinality or penetration testing - not forgetting the number of software tools required to perform the testing. The major issues which can and should be solved:

docs/get_started.md

@@ -1,5 +1,7 @@
 # Getting Started
 
+It is recommended that you run Testrun on a standalone machine running a fresh-install of Ubuntu 22.04.3 LTS.
+
 ## Prerequisites
 
 ### Hardware
@@ -24,29 +26,66 @@ Ensure the following software is installed on your Ubuntu LTS PC:
 
 ## Installation
 
-1. Download the latest version of Testrun from the [releases page](https://github.com/google/test-run/releases)
+1. Download the latest version of the Testrun installer from the [releases page](https://github.com/google/test-run/releases)
 
-2. Install the package using ``sudo apt install ./testrun*.deb``
+2. Open a terminal and navigate to location of the Testrun installer (most likely your downloads folder)
 
-## Test Your Device
+3. Install the package using ``sudo apt install ./testrun*.deb``
 
-1. Attach network interfaces:
+ - Testrun will be installed under the /usr/local/testrun directory.
+   - Testing data will be available in the ``local/devices/{device}/reports`` folders
+   - Additional configuration options are available in the ``local/system.json`` file
 
-   - Connect one USB Ethernet adapter to the internet source (e.g., router or switch) using an Ethernet cable.
-   - Connect the other USB Ethernet adapter directly to the IoT device you want to test using an Ethernet cable.
+## Start Testrun
+
+1. Attach network interfaces:
+   - Connect one USB Ethernet adapter to the internet source (e.g., router or switch) using an ethernet cable.
+   - Connect the other USB Ethernet adapter directly to the IoT device you want to test using an ethernet cable.
 
    **NOTE: Both adapters should be disabled in the host system (IPv4, IPv6 and general). You can do this by going to Settings > Network** 
 
 2. Start Testrun.
 
-Start Testrun with the command `sudo testrun --no-validate`
+Start Testrun with the command `sudo testrun`
 
    - To run Testrun in network-only mode (without running any tests), use the `--net-only` option.
 
-   - To skip network validation before use and not launch the faux device on startup, use the `--no-validate` option.
-
    - To run Testrun with just one interface (connected to the device), use the ``--single-intf`` option.
 
+## Test Your Device
+
+1. Once Testrun has started, open your browser to http://localhost:8080.
+
+2.  Configure your network interfaces under the settings menu - located in the top right corner of the application. Settings can be changed at any time.
+
+    ![](/docs/ui/settings_icon.png)
+
+3. Navigate to the device repository icon to add a new device for testing.
+
+    ![](/docs/ui/device_icon.png)
+
+4. Click the button 'Add Device'.
+
+5. Enter the MAC address, manufacturer name and model number.
+
+6. Select the test modules you wish to enable for this device (Hint: All are required for qualification purposes) and click save.
+
+7. Navigate to the Testrun progress icon and click the button 'Start New Testrun'.
+
+    ![](/docs/ui/progress_icon.png)
+
+8. Select the device you would like to test.
+
+9. Enter the version number of the firmware running on the device.
+
+10. Click 'Start Testrun'
+
+ - During testing, if you would like to stop Testrun, click 'Stop' next to the test name.
+
+11. On completion of the test sequence, a report will appear under the history icon. 
+
+    ![](/docs/ui/history_icon.png)
+
 # Troubleshooting
 
 If you encounter any issues or need assistance, consider the following:

framework/python/src/core/test_runner.py

@@ -34,7 +34,7 @@ class TestRunner:
 
   def __init__(self,
                config_file=None,
-               validate=True,
+               validate=False,
                net_only=False,
                single_intf=False,
                no_ui=False):
@@ -76,9 +76,10 @@ def parse_args():
       help="Define the configuration file for Test Run and Network Orchestrator"
   )
   parser.add_argument(
-      "--no-validate",
+      "--validate",
+      default=False,
       action="store_true",
-      help="Turn off the validation of the network after network boot")
+      help="Turn on the validation of the network after network boot")
   parser.add_argument("-net",
                       "--net-only",
                       action="store_true",
@@ -97,7 +98,7 @@ def parse_args():
 if __name__ == "__main__":
   args = parse_args()
   runner = TestRunner(config_file=args.config_file,
-                      validate=not args.no_validate,
+                      validate=args.validate,
                       net_only=args.net_only,
                       single_intf=args.single_intf,
                       no_ui=args.no_ui)

framework/python/src/core/testrun.py

@@ -66,7 +66,7 @@ class TestRun:  # pylint: disable=too-few-public-methods
 
   def __init__(self,
                config_file,
-               validate=True,
+               validate=False,
                net_only=False,
                single_intf=False,
                no_ui=False):
@@ -91,8 +91,8 @@ def __init__(self,
       self._session.add_runtime_param('single_intf')
     if net_only:
       self._session.add_runtime_param('net_only')
-    if not validate:
-      self._session.add_runtime_param('no-validate')
+    if validate:
+      self._session.add_runtime_param('validate')
 
     self.load_all_devices()
 
@@ -290,6 +290,7 @@ def start(self):
       self.get_net_orc().start_listener()
       self._set_status('Waiting for Device')
       LOGGER.info('Waiting for devices on the network...')
+
       time.sleep(self.get_session().get_runtime())
 
       if not (self._test_orc.test_in_progress() or

framework/python/src/net_orc/network_orchestrator.py

@@ -119,7 +119,7 @@ def start_network(self):
     self.create_net()
     self.start_network_services()
 
-    if 'no-validate' not in self._session.get_runtime_params():
+    if 'validate' in self._session.get_runtime_params():
       # Start the validator after network is ready
       self.validator.start()
 

testing/baseline/test_baseline

@@ -43,7 +43,7 @@ sudo cp testing/baseline/system.json $TESTRUN_DIR/local/system.json
 # Copy device configs to testrun
 sudo cp -r testing/device_configs/* $TESTRUN_DIR/local/devices
 
-sudo testrun --single-intf --no-ui > $TESTRUN_OUT 2>&1 & 
+sudo testrun --single-intf --no-ui --validate > $TESTRUN_OUT 2>&1 & 
 TPID=$!
 
 # Time to wait for testrun to be ready

testing/tests/test_tests

@@ -62,7 +62,7 @@ for tester in $TESTERS; do
   args=$(jq -r .$tester.args $MATRIX)
 
   touch $testrun_log
-  sudo timeout 900 testrun --single-intf --no-ui --no-validate > $testrun_log 2>&1 &
+  sudo timeout 900 testrun --single-intf --no-ui > $testrun_log 2>&1 &
   TPID=$!
 
   # Time to wait for testrun to be ready