Merge dev into main (Release 1.1-alpha)

.github/workflows/testing.yml

@@ -31,7 +31,7 @@ jobs:
     name:  Tests
     runs-on: ubuntu-20.04
     needs: testrun_baseline
-    timeout-minutes: 40
+    timeout-minutes: 45
     steps:
       - name: Checkout source
         uses: actions/checkout@v2.3.4
@@ -44,13 +44,12 @@ jobs:
       - name: Install Testrun
         shell: bash {0}
         run: sudo dpkg -i testrun*.deb
-        timeout-minutes: 10
       - name: Run tests
         shell: bash {0}
         run: testing/tests/test_tests
       - name: Archive runtime results
         if: ${{ always() }}
-        run: sudo tar --exclude-vcs -czf runtime.tgz /usr/local/testrun/runtime/ /usr/local/testrun/local/
+        run: sudo tar --exclude-vcs -czf runtime.tgz /usr/local/testrun/runtime/
       - name: Upload runtime results
         uses: actions/upload-artifact@v3
         if: ${{ always() }}
@@ -69,16 +68,26 @@ jobs:
       - name: Install dependencies
         shell: bash {0}
         run: cmd/prepare
-      - name: Package Testrun
+      - name: Package Testrun  
         shell: bash {0}
         run: cmd/package
       - name: Install Testrun
         shell: bash {0}
         run: sudo dpkg -i testrun*.deb
-        timeout-minutes: 10
+        timeout-minutes: 30
       - name: Run tests
         shell: bash {0}
         run: testing/api/test_api
+      - name: Archive runtime results
+        if: ${{ always() }}
+        run: sudo tar --exclude-vcs -czf runtime.tgz /usr/local/testrun/runtime/ /usr/local/testrun/local/
+      - name: Upload runtime results
+        uses: actions/upload-artifact@v3
+        if: ${{ always() }}
+        with:
+          if-no-files-found: error
+          name: runtime_${{ github.workflow }}_${{ github.run_id }}
+          path: runtime.tgz
 
   pylint:
     name:  Pylint
@@ -105,4 +114,26 @@ jobs:
         uses: actions/upload-artifact@v3
         with:
           name: Testrun Installer
-          path: testrun*.deb
+          path: testrun*.deb
+
+  testrun_ui:
+    name: UI
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Install Node
+      uses: actions/setup-node@v3
+      with:
+        node-version: 18.10.0
+    - name: Install Chromium Browser
+      run: sudo apt install chromium-browser
+    - name: Install dependencies
+      run: npm install && npm ci
+      working-directory: ./modules/ui
+    - name: Run tests
+      run: |
+        export CHROME_BIN=/usr/bin/chromium-browser
+        CI=true npm run test-headless
+      env:
+        CI: true
+      working-directory: ./modules/ui

README.md

@@ -5,37 +5,35 @@
 [![Testrun test suite](https://github.com/google/testrun/actions/workflows/testing.yml/badge.svg?branch=main&event=push)](https://github.com/google/testrun/actions/workflows/testing.yml)
 
 ## Introduction :wave:
-Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows external manufacturers to test their devices for the purposes of Device Qualification within the BOS program.
+Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows manufacturers of IP capable devices to test their devices for the purposes of Device Qualification within the BOS program.
 
 ## Motivation :bulb:
-Without tools like Testrun, testing labs may be maintaining a large and complex network using equipment such as: A managed layer 3 switch, an enterprise-grade network router, virtualized or physical servers to provide DNS, NTP, 802.1x etc. With this amount of moving parts, all with dynamic configuration files and constant software updates, more time is likely to be spent on preparation and clean up of functinality or penetration testing - not forgetting the number of software tools required to perform the testing. The major issues which can and should be solved:
+Without tools like Testrun, test labs and engineers may need to  maintain a large and complex network coupled with dynamic configuration files and constant software updates. The major issues which can and should be solved are:
  1) The complexity of managing a testing network
  2) The time required to perform testing of network functionality
  3) The accuracy and consistency of testing network functionality
 
 ## How it works :triangular_ruler:
-Testrun creates an isolated and controlled network environment to fully simulate enterprise network deployments in your device testing lab. 
-This removes the necessity for complex hardware, advanced knowledge and networking experience whilst enabling semi-technical engineers to validate device 
-behaviour against industry cyber standards. 
+Testrun creates an isolated and controlled network environment on a linux machine. This removes the necessity for complex hardware, advanced knowledge and networking experience whilst enabling test engineers to validate device behaviour against Google’s Building Operating System requirements.
 
-Two runtime modes are supported by Testrun:
+Two modes are supported by Testrun:
 
 1) <strong>Automated testing</strong>
 
-Once the device has become operational (steady state), automated testing of the device under test will begin. Containerized test modules will then execute against the device (one module at a time). Once all test modules have completed execution, a final test report will be produced - presenting the results and further description of findings.
+Once the device has become operational (steady state), automated testing of the DUT (device under test) will begin. Containerized test modules will then execute against the device, one module at a time. Once all test modules have been executed, a report will be produced - presenting the results.
 
 2) <strong>Lab network</strong>
 
-Testrun cannot automate everything, and so additional manual testing may be required (or configuration changes may be required on the device). Rather than having to maintain a separate but idential lab network, Testrun will provide the network and some tools to assist an engineer performing the additional testing. At the same time, packet captures of the device behaviour will be recorded, alongside logs for each network service, for further debugging.
+When manual testing or configuration changes are required, Testrun will provide the network and some tools to assist an engineer performing the additional testing. This reduces the need to maintain a separate but identical lab network. Testrun will take care of packet captures and logs for each network service for further debugging.
 
 ## Minimum requirements :computer:
 ### Hardware
- - PC running Ubuntu LTS (laptop or desktop)
+ - PC running Ubuntu LTS 20.04 or 22.04 (laptop or desktop)
  - 2x USB ethernet adapter (One may be built in ethernet)
  - Internet connection
 ### Software
 - Docker - installation guide: [https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository)
-### Device
+### Device under test (DUT)
  - DHCP client - The device must be able to obtain an IP address via DHCP
 
 ## Get started ▶️
@@ -68,8 +66,7 @@ The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md)
 
 3) Can I run Testrun on a virtual machine?
 
-  Probably. Provided that the required 2x USB ethernet adapters are passed to the virtual machine as USB devices rather than network adapters, Testrun should
-  still work. We will look to test and approve the use of virtualisation to run Testrun in the future.
+  Testrun can be virtualized if the 2x ethernet adapters are passed through to a Virtual Box VM as a USB device rather than managed network adapters. A full guide will be provided once virtualization of Testrun has been fully tested.
 
  4) Can I connect multiple devices to Testrun?
 

cmd/build

@@ -14,6 +14,25 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+# Check if Docker is installed
+if [[ $(which docker) && $(docker --version) ]]; then
+    # Docker is installed
+    echo Using $(docker --version)
+  else
+    # Docker is not installed
+    echo Docker is not installed. Please follow the Get Started guide at https://github.com/google/testrun
+    exit 1
+fi
+
+# Stop any running containers
+echo Stopping any running containers
+running_containers=$(docker container ls -q --filter name=tr-*)
+if [ -n "$running_containers" ]; then
+  docker container kill $running_containers
+else
+  echo No containers were found running
+fi
+
 # Builds all docker images
 echo Building docker images
 

cmd/package

@@ -44,7 +44,7 @@ mkdir -p $MAKE_SRC_DIR/usr/local/testrun/local/devices
 
 # Copy root_certs folder
 mkdir -p local/root_certs
-cp -r local/root_certs $MAKE_SRC_DIR/usr/local/testrun/local/root_certs
+cp -r local/root_certs $MAKE_SRC_DIR/usr/local/testrun/local/
 
 # Copy framework and modules into testrun folder
 cp -r {framework,modules} $MAKE_SRC_DIR/usr/local/testrun
@@ -53,4 +53,4 @@ cp -r {framework,modules} $MAKE_SRC_DIR/usr/local/testrun
 dpkg-deb --build --root-owner-group make
 
 # Rename the .deb file
-mv make.deb testrun_1-0-3_amd64.deb
+mv make.deb testrun_1-1_amd64.deb

cmd/prepare

@@ -19,6 +19,6 @@
 
 echo Installing system dependencies
 
-sudo apt-get install openvswitch-common openvswitch-switch python3 libpangocairo-1.0-0
+sudo apt-get update && sudo apt-get install openvswitch-common openvswitch-switch python3 libpangocairo-1.0-0
 
 echo Finished installing system dependencies

docs/configure_device.md

@@ -0,0 +1,29 @@
+# Device Configuration
+
+The device configuration file allows you to customize the testing behavior for a specific device. This file is located at `local/devices/{Device Name}/device_config.json`. Below is an overview of how to configure the device tests.
+
+## Device Information
+
+The device information section includes the manufacturer, model, and MAC address of the device. These details help identify the specific device being tested.
+
+## Test Modules
+
+Test modules are groups of tests that can be enabled or disabled as needed. You can choose which test modules to run on your device.
+
+### Enabling and Disabling Test Modules
+
+To enable or disable a test module, modify the `enabled` field within the respective module. Setting it to `true` enables the module, while setting it to `false` disables the module.
+
+## Customizing the Device Configuration
+
+To customize the device configuration for your specific device, follow these steps:
+
+1. Copy the default configuration file provided in the `resources/devices/template` folder.
+   - Create a new folder for your device under `local/devices` directory.
+   - Copy the `device_config.json` file from `resources/devices/template` to the newly created device folder.
+
+This ensures that you have a copy of the default configuration file, which you can then modify for your specific device.
+
+> Note: Ensure that the device configuration file is properly formatted, and the changes made align with the intended test behavior. Incorrect settings or syntax may lead to unexpected results during testing.
+
+If you encounter any issues or need assistance with the device configuration, refer to the Testrun documentation or ask a question on the Issues page.

docs/get_started.md

@@ -41,6 +41,8 @@ However, to achieve a compliant test outcome, your device must be configured cor
    - Testing data will be available in the ``local/devices/{device}/reports`` folders
    - Additional configuration options are available in the ``local/system.json`` file
 
+    **NOTE: Place your local CA certificate in local/root_certs (any name with a .crt extension) to perform TLS server tests**
+
 ## Start Testrun
 
 1. Attach network interfaces:
@@ -105,4 +107,4 @@ If you encounter any issues or need assistance, consider the following:
 - Refer to the Testrun documentation or ask for assistance in the issues page: https://github.com/google/testrun/issues
 
 # Uninstall
-To uninstall Testrun, use the built-in dpkg uninstall command to remove Testrun correctly. For Testrun, this would be:  ```sudo apt-get remove testrun```
+To uninstall Testrun, use the built-in dpkg uninstall command to remove Testrun correctly. For Testrun, this would be:  ```sudo apt-get remove testrun```