Skip to content

Update docs #839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jboddey merged 6 commits into from
Oct 3, 2024
Merged

Update docs #839

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6792fdb
Update docs
jboddey Oct 1, 2024
b3fa91d
Update links and icons
jboddey Oct 1, 2024
9254ab3
Add header images
jboddey Oct 1, 2024
cfb2300
Update docs
jboddey Oct 2, 2024
dea6a5c
Update links
jboddey Oct 2, 2024
c8f37e0
Merge branch 'dev' into update_docs
jboddey Oct 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 67 additions & 58 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,84 +4,93 @@
[![CodeQL](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql/badge.svg?branch=main)](https://github.com/google/testrun/actions/workflows/github-code-scanning/codeql)
[![Testrun test suite](https://github.com/google/testrun/actions/workflows/testing.yml/badge.svg?branch=main&event=push)](https://github.com/google/testrun/actions/workflows/testing.yml)

## Introduction :wave:
Testrun automates specific test cases to verify network and security functionality in IoT devices. It is an open source tool which allows manufacturers of IP capable devices to test their devices for the purposes of Device Qualification within the BOS program.
# Introduction :wave:

## Motivation :bulb:
Without tools like Testrun, test labs and engineers may need to maintain a large and complex network coupled with dynamic configuration files and constant software updates. The major issues which can and should be solved are:
1) The complexity of managing a testing network
2) The time required to perform testing of network functionality
3) The accuracy and consistency of testing network functionality
Testrun automates specific test cases to verify network and security functionality in IoT devices. It's an open-source tool that manufacturers use to test their IP-capable devices for the purpose of device qualification within Google's Building Operating System (BOS) program.

## How it works :triangular_ruler:
Testrun creates an isolated and controlled network environment on a linux machine. This removes the necessity for complex hardware, advanced knowledge and networking experience whilst enabling test engineers to validate device behaviour against Google’s Building Operating System requirements.
# Motivation :bulb:

Two modes are supported by Testrun:
Test labs and engineers often need to maintain a large and complex network coupled with dynamic configuration files and constant software updates. Testrun helps address major issues like:

<details>
<summary>
<strong>Automated testing</strong>
</summary>
- The complexity of managing a testing network
- The time required to perform testing of network functionality
- The accuracy and consistency of testing network functionality

Once the device has become operational (steady state), automated testing of the DUT (device under test) will begin. Containerized test modules will then execute against the device, one module at a time. Once all test modules have been executed, a report will be produced - presenting the results.
</details>
# How it works :triangular_ruler:

<details>
Testrun creates an isolated and controlled network environment on a Linux machine. This removes the necessity for complex hardware, advanced knowledge, and networking experience while enabling test engineers to validate device behavior against Google's BOS requirements.

<summary>
<strong>Lab network</strong>
</summary>
Testrun supports two modes: automated testing and lab network.

When manual testing or configuration changes are required, Testrun will provide the network and some tools to assist an engineer performing the additional testing. This reduces the need to maintain a separate but identical lab network. Testrun will take care of packet captures and logs for each network service for further debugging.
## Automated testing

</details>
Automated testing of the device under test (DUT) begins once the device is operational (steady state). Containerized test modules execute against the device one module at a time. Testrun produces a report with the results after all modules are executed.

## Minimum requirements :computer:
### Hardware
- PC running Ubuntu LTS 20.04, 22.04 or 24.04 (laptop or desktop)
- 2x USB ethernet adapter (One may be built in ethernet)
- Internet connection
### Software
- Docker - installation guide: [https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository)
### Device under test (DUT)
- DHCP client - The device must be able to obtain an IP address via DHCP
## Lab network

## Get started ▶️
Once you have met the hardware and software requirements, you can get started with Testrun by following the [Get started guide](docs/get_started.md). Further docs are available in the [docs directory](docs)
Testrun provides the network and assistive tools for engineers when manual testing or configuration changes are required, reducing the need to maintain a separate but identical lab network. Testrun handles packet captures and logs for each network service for further debugging.

## Roadmap :chart_with_upwards_trend:
Testrun will constantly evolve to further support end-users by automating device network behaviour against industry standards. For further information on upcoming features, check out the [Roadmap](docs/roadmap.pdf).
# Minimum requirements :computer:

## Accessibility :busts_in_silhouette:
We are proud of our tool and strive to provide an enjoyable experience for all of our users. Testrun goes through rigorous accessibility testing at each release. You can read more about [Google and Accessibility here](https://www.google.co.uk/accessibility). You are welcome to submit a new issue and provide feedback on our implementations. To find out how Testrun implements accessibility features, you can view a [short video here](docs/ui/accessibility.mp4).
## Hardware

## Issue reporting :triangular_flag_on_post:
If the application has come across a problem at any point during setup or use, please raise an issue under the [issues tab](https://github.com/google/testrun/issues). Issue templates exist for both bug reports and feature requests. If neither of these are appropriate for your issue, raise a blank issue instead.
- PC running Ubuntu LTS 20.04, 22.04, or 24.04 (laptop or desktop)
- 2x USB Ethernet adapter (one may be built-in Ethernet)
- Internet connection

## Contributing :keyboard:
The contributing requirements can be found in [CONTRIBUTING.md](CONTRIBUTING.md). In short, checkout the [Google CLA](https://cla.developers.google.com/) site to get started. After that, check out our [developer documentation](docs/dev/README.md).
## Software

## FAQ :raising_hand:
1) I have an issue whilst installing/upgrading Testrun, what do I do?
Testrun requires Docker. Refer to the [installation guide](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) for more information.

Sometimes, issues may arise when installing or upgrading Testrun - this may happen due to one of many reasons due to the nature of the application. However, most of the time, it can be resolved by following a full Testrun re-install by using these commands:
- ```sudo docker system prune -a```
- ```sudo apt install ./testrun-*.deb```
## Device under test (DUT)

2) What device networking functionality is validated by Testrun?
The DUT must be able to obtain an IP address via DHCP.

Best practices and requirements for IoT devices are constantly changing due to technological advances and discovery of vulnerabilities.
The current expectations for IoT devices on Google deployments can be found in the [Application Security Requirements for IoT Devices](https://partner-security.withgoogle.com/docs/iot_requirements).
Testrun aims to automate as much of the Application Security Requirements as possible.
# Get started :arrow_forward:

3) What services are provided on the virtual network?
Once you meet the hardware and software requirements, follow the Testrun [Get started guide](/docs/get_started.md). Additional guidance is available in the [docs directory](/docs).

The following are network services that are containerized and accessible to the device under test though are likely to change over time:
- DHCP in failover configuration with internet connectivity
- IPv6 SLAAC
- DNS
- NTPv4
# Roadmap :chart_with_upwards_trend:

4) Can I run Testrun on a virtual machine?
Testrun continually evolves to further support end users by automating device network behavior against industry standards. For information on upcoming features, check out the [Roadmap](/docs/roadmap.pdf).

Testrun can be virtualized if the 2x ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. You can view the guide to working on a [virtual machine here](docs/virtual_machine.md).
# Accessibility :busts_in_silhouette:

We're proud of our tool and strive to provide an enjoyable experience for everyone. Testrun goes through rigorous accessibility testing at each release. Download the [Testrun: Accessible features](https://github.com/google/testrun/raw/refs/heads/main/docs/ui/accessibility.mp4) video to learn more.You're welcome to [submit a new issue](https://github.com/google/testrun/issues) and provide feedback on our implementations. To learn more about Google's [Belonging initiative](https://www.google.co.uk/accessibility) and their approach to accessibility, visit their site.

# Issue reporting :triangular_flag_on_post:

If you encounter a problem during setup or use, raise an issue under the [Issues tab](https://github.com/google/testrun/issues). Issue templates exist for both bug reports and feature requests. If neither of these apply, raise a blank issue instead.

# Contributing :keyboard:

We strongly encourage contributions from the community. Review the requirements on the ["How to Contribute" page](CONTRIBUTING.md), then follow the [developer guidelines](/docs/dev/README.md).

# FAQ :raising_hand:

#### 1. What should I do if I have an issue while installing or upgrading Testrun?

You can resolve most issues by reinstalling Testrun using these commands:
- `sudo docker system prune -a`
- `sudo apt install ./testrun-*.deb`

If this doesn't resolve the problem, [raise an issue](https://github.com/google/testrun/issues).

#### 2. What device networking functionality does Testrun validate?

Best practices and requirements for IoT devices change often due to technological advances and discovery of vulnerabilities. You can find the current expectations for IoT devices on Google deployments in the [Application Security Requirements for IoT Devices](https://partner-security.withgoogle.com/docs/iot_requirements). Testrun aims to automate as much of the Application Security Requirements as possible.

#### 3. What services are provided on the virtual network?

The following network services are containerized and accessible to the DUT:

- DHCP in failover configuration with internet connectivity
- IPv6 SLAAC
- DNS
- NTPv4

Note that this list is likely to change over time.

#### 4. Can I run Testrun on a virtual machine?

Testrun can be virtualized if the 2x Ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. Visit the [virtual machine guide](/docs/virtual_machine.md) for additional details.
30 changes: 14 additions & 16 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,19 @@
<img width="200" alt="Testrun logo" src="https://user-images.githubusercontent.com/7399056/221927867-4190a4e8-a571-4e40-9c2b-65780ad9264c.png" alt="Testrun">

# Contents

## Contents
- [Get started](/docs/get_started.md)
- [Run on a virtual machine](/docs/virtual_machine.md)
- [Network](/docs/network/README.md)
- [Network addresses](/docs/network/addresses.md)
- [Add a new network service](/docs/network/add_new_service.md)
- [Testing](/docs/test/README.md)
- [Test modules](/docs/test/modules.md)
- [Test results](/docs/test/statuses.md)
- [Developer guidelines](/docs/dev/README.md)
- [Accessibility](/docs/ui/accessibility.md)
- [Roadmap](/docs/roadmap.pdf)

- [Get Started](get_started.md)
- [Network](network/README.md)
- [Network Overview](network/README.md)
- [How to identify network interfaces](network/identify_interfaces.md)
- [Addresses](network/addresses.md)
- [Add a new network service](network/add_new_service.md)
- [Testing](test/README.md)
- [Test modules](test/modules.md)
- [Test statuses](test/statuses.md)
- [Development](dev/README.md)
- [Running on a virtual machine](virtual_machine.md)
- [Accessibility](ui/accessibility.mp4)
- [Roadmap](roadmap.pdf)
# Something missing?

## Something missing?
If you feel there is some documentation that you would find useful, or have found an issue with existing documentation, please raise an issue on GitHub by navigating [here](https://github.com/google/testrun/issues/new/choose)
To request additional documentation or report an issue with existing resources, visit [the Issues tab](https://github.com/google/testrun/issues/new/choose).
31 changes: 0 additions & 31 deletions docs/configure_device.md
View file
Open in desktop

This file was deleted.

38 changes: 24 additions & 14 deletions docs/dev/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,35 @@
<img width="200" alt="Testrun logo" src="https://user-images.githubusercontent.com/7399056/221927867-4190a4e8-a571-4e40-9c2b-65780ad9264c.png" alt="Testrun">

## Developer docs
# Developer guidelines

## Table of Contents
1) General guidelines (this page)
2) [Code quality](code_quality.md)
## How to contribute

## General guidelines
As an open source project, we absolutely encourage contributions from the community to help Testrun remain an expanding but stable product. However, before contributing there are a number of things to take into consideration.
As an open source project, we encourage contributions from the community to help Testrun remain an expanding but stable product. To contribute, follow the steps below:

1) [Sign the Google CLA](https://cla.developers.google.com/): Whether you are an individual or contributing on behalf of your organisation, you must be covered by a Google CLA.
1. Sign the [Google Contributor License Agreement (CLA)](https://cla.developers.google.com/).
- Whether you're an individual or contributing on behalf of your organization, you must be covered by a Google CLA.

2) Determine the scope of your contribution
1. Determine the scope of your contribution.
- Keep it simple. Your contribution is more likely to be accepted if you change fewer files.
- Ensure your pull request addresses one thing, such as a bug fix, dependency issue, or new framework capability.

- Your contribution is more likely to be accepted if fewer files are changed (keep it simple)
- Are you going to be fixing a bug, dependency issue or a new framework capability? Whatever it is, ensure your pull request fixes or changes just one thing.
1. Reach out to the core maintainers at [testrun-team@googlegroups.com](mailto:testrun-team@googlegroups.com).
- They can provide confirmation that your proposed changes meet our objectives and align with Testrun principles, making them more likely to be accepted.

3) Get in touch to discuss whether your proposed changes are likely to be accepted
1. Fork Testrun and get developing.
- We aim to provide thorough and clear developer documentation to help you contribute successfully.

- It is best to get the opinion from the core maintainers whether your proposed changes meet our objectives and align with Testrun principles.
## Code quality

4) Fork Testrun and get developing
To ensure code quality, use the appropriate style guide when developing code for Testrun:

- We aim to provide thorough and easy to ready developer documentation to help you contribute successfully.
- [Python](https://google.github.io/styleguide/pyguide.html)
- [Angular](https://google.github.io/styleguide/angularjs-google-style.html)
- [Shell](https://google.github.io/styleguide/shellguide.html)
- [HTML/CSS](https://google.github.io/styleguide/htmlcssguide.html)
- [JSON](https://google.github.io/styleguide/jsoncstyleguide.xml)
- [Markdown](https://google.github.io/styleguide/docguide/style.html)

## Automated actions

The current code base has zero code lint issues. To maintain this, all lint checks are enforced on pull requests to dev and main. You should ensure these lint checks pass before marking your pull requests as Ready for review.
16 changes: 0 additions & 16 deletions docs/dev/code_quality.md
View file
Open in desktop

This file was deleted.

Loading