Test Vectors Runner

This project is a reference implementation of a Test Vector runner which executes Test Vectors based tests for black-box testing of Stratum enabled switches.

Build status (master):

testvectors-runner works with various switch targets that expose P4Runtime and gNMI, including Stratum switches. To get started, you'll need Switch Under Test (SUT) and set of corresponding Test Vectors.

Start a Stratum Switch

To start Stratum's behavioral model software switch (stratum-bmv2) in a Docker container for testing, run:

make bmv2

Note: The bmv2 container runs on the host network and creates two veth pairs on host machine which are used for testing data plane scenarios.

To start Stratum on hardware switches, including devices with Barefoot Tofino and Broadcom Tomahawk, visit the Stratum Project repo for details of how to get Stratum running on supported devices.

Get Test Vectors

Download Test Vector files matching your SUT (tofino/bcm/bmv2) from Test Vectors repo or create your own Test Vectors.

Another options is to download Test Vector templates which work with any switch platform together with a template configuration file which is used for rendering the templates and producing Test Vector files that match your SUT. See more details in the Run with Test Vector Templates section below.

In addition to Test Vector files, a target.pb.txt file and a portmap.pb.txt file are mandatory for starting testvectors-runner. target.pb.txt stores the IP and port that your SUT is using, and portmap.pb.txt stores information related to specific switch ports used in the Test Vectors. Check examples in Test Vectors repo as well as the readme for more details.

Test with testvectors-runner

For running with hardware switches, testvectors-runner could be deployed on a server which has both gRPC and data plane connections to the SUT. For running with stratum-bmv2, testvectors-runner needs to be deployed on the same network where the bmv2 container is deployed.

When loopback mode is enabled on hardware switches, it's also supported to deploy testvectors-runner directly on the switch. See the loopback section below for more details.

Use existing testvectors-runner binary docker image

./tvrunner.sh --target <TARGET_FILE> --portmap <PORT_MAP_FILE> --tv-dir <TESTVECTORS_DIR>

Above command uses tvrunner binary docker image, executes testvectors from tv-dir on switch running on target. In addition to --tv-dir argument, you can also use --tv-name <TEST_NAME_REGEX> to run tests matching provided regular expression from tv-dir.

For example, assuming bmv2 container is deployed by make bmv2 command and Test Vectors repo is downloaded to ~/testvectors, first push a pipeline configuration to the bmv2 switch before running any tests:

./tvrunner.sh --target ~/testvectors/bmv2/target.pb.txt --portmap ~/testvectors/bmv2/portmap.pb.txt --tv-dir ~/testvectors/bmv2 --tv-name PipelineConfig

Above command finds and executes Test Vector with name PipelineConfig.pb.txt under ~/testvectors/bmv2. Then run p4runtime test suite by:

./tvrunner.sh --target ~/testvectors/bmv2/target.pb.txt --portmap ~/testvectors/bmv2/portmap.pb.txt --tv-dir ~/testvectors/bmv2/p4runtime

Build and use local testvectors-runner binary docker image

Build testvectors-runner binary image locally using below command:

docker build -t <IMAGE_NAME> -f build/test/Dockerfile .

Run tests with below command:

./tvrunner.sh --target <TARGET_FILE> --portmap <PORT_MAP_FILE> --tv-dir <TESTVECTORS_DIR> --image <IMAGE_NAME>

In both cases, tvrunner.sh runs docker container in host network. To run docker container in another container's network, use below command:

./tvrunner.sh --target <TARGET_FILE> --portmap <PORT_MAP_FILE> --tv-dir <TESTVECTORS_DIR> --network <NETWORK>

Note: For more optional arguments, run ./tvrunner.sh -h

Use go run command to run tests

go run cmd/main/testvectors-runner.go --target <TARGET_FILE> --portmap <PORT_MAP_FILE> --tv-dir <TESTVECTORS_DIR>

Build go binary, run tests

Build testvectors-runner go binary using below command:

make build

Use the executed binary to run tests

./tvrunner --target <TARGET_FILE> --portmap <PORT_MAP_FILE> --tv-dir <TESTVECTORS_DIR>

Note: For more optional arguments, run go run cmd/main/testvectors-runner.go -h or ./tvrunner -h

Loopback mode

To run tests in loopback mode just add --dp-mode loopback to the commands. It applies to all the options above. Take a Tofino switch as an example. First push a pipeline configuration by:

./tvrunner.sh --target ~/testvectors/tofino/target.pb.txt --portmap ~/testvectors/tofino/portmap.pb.txt --tv-dir ~/testvectors/tofino --tv-name PipelineConfig --dp-mode loopback

As part of loopback mode setup, extra Insert* Test Vectors need to be executed before running any tests (see more details here).

./tvrunner.sh --target ~/testvectors/tofino/target.pb.txt --portmap ~/testvectors/tofino/portmap.pb.txt --tv-dir ~/testvectors/tofino --tv-name Insert.* --dp-mode loopback

Then run p4runtime test suite by:

./tvrunner.sh --target ~/testvectors/tofino/target.pb.txt --portmap ~/testvectors/tofino/portmap.pb.txt --tv-dir ~/testvectors/tofino/p4runtime --dp-mode loopback

After all tests are done, run the Delete* Test Vectors to clean up.

./tvrunner.sh --target ~/testvectors/tofino/target.pb.txt --portmap ~/testvectors/tofino/portmap.pb.txt --tv-dir ~/testvectors/tofino --tv-name Delete.* --dp-mode loopback

Run with Test Vector Templates

Test Vector templates are tokenized Test Vector files and were created with the goal of maintaining a single set of tests that works across multiple switch platforms. As an alternative way of running Test Vectors, now it is also supported to run Test Vector templates together with a template configuration file (get more details in Test Vectors repo) by pointing --tv-dir and --tv-name to the template file and using --template-config argument to specify the template configuration file, and all the other options above still apply:

./tvrunner.sh --target ~/testvectors/tofino/target.pb.txt --portmap ~/testvectors/tofino/portmap.pb.txt --tv-dir ~/testvectors/templates/p4runtime --tv-name L3ForwardTest --template-config ~/testvectors/tofino/template_config.json

Additional Documents

• Test Vectors Runner Architecture