Yaml Tests

Yaml Tests is a simple composer plugin tthat make it as simple as possible to define and run a set of tests.

The plugin provides a composer command that simply reads a YML file and runs the lines as a process.

The output is rendered in a way to be easier to read, and proper exit code returns if a single process fails.

It can be installed "locally" (included in your composer.json file) or "globally" (installed into the global "composer" command.

Installation

Keeping yaml-tests in your composer.json (local install) is the most stable way to operate, since the version is pinned.

Local Install

1. cd my-composer-project
2. composer require provision-ops/yaml-tests

Global Install

composer global require provision-ops/yaml-tests

To confirm the command is installed, ask for help:

composer yaml-tests --help

GitHub Integration

If you pass yaml-tests a GitHub Token, it will send the test results as "commit status" indicators.

There are 3 ways to pass the GitHub Token to YamlTests:

1. Use the --github-token command line option. Don't use this in CI, or you might expose your GitHub token in logs.

2. Set a GITHUB_TOKEN environment variable. This is pretty simple in Docker, but can be a challenge if your tests get run in different environments.

3. Recommended: Create a .env file in your repo, or in your user's home directory:

   GITHUB_TOKEN=abcdefg
   

   There is a .env.example file in this directory you can use as an example.

Writing Tests

Create tests.yml file

By default the composer yaml-tests command looks for a tests.yml file in the project root. You can also pass a path using the --tests-file option.

The tests.yml file is read as a simple collection of commands. The key can be any string, as long as it is unique in the test suite.

test/dir: pwd
test/environment: env

You can also include commands in a list:

lint:
  - find src -name '*.php' -print0 | xargs -0 -n1 php -l
  - find web/modules/custom -name '*.php' -print0 | xargs -0 -n1 php -l
  - find tests/src -name '*.php' -print0 | xargs -0 -n1 php -l

You can include a description for each test like:

debug: 
  command: env
  description: Current Environment

Commands in tests.yml

Yaml Tests work like Composer Scripts: If your project has the config.bin-dir set in composer.json, Composer will automatically add that directory to the PATH when scripts or other commands are run.

For example, you can include PHPUnit and call it without specifying the full path in composer scripts or tests.yml

composer.json:

{
    "config": {
        "bin-dir": "bin/"
    },
    "require": {
        "provision-ops/yaml-tests": "^1.1",
        "phpunit/phpunit": "^8.1"
    },
    "scripts": {
        "test": [
            "which phpunit",
            "phpunit --version"
        ]
    }
}

Having the scripts.test section in composer.json creates a composer command called composer test.

tests.yml:

test/debug: 
  - which phpunit
  - phpunit --version

If you want to only maintain one set of scripts, you can reference composer scripts in tests.yml:

tests.yml:

test/debug: composer test 

Running tests

Once the tests.yml file is in place, and the composer yaml-tests command is available, you can trigger test runs.

Dry Runs vs Normal

This plugin was also designed to pass these tests as "Commit Statuses" on GitHub. This allows us to tag the results to the specific commit, pass or fail.

If the environment variable GITHUB_TOKEN or the command line option --github-token is NOT set, the --dry-run option will be forced.

Use the --dry-run option if you have a token set but do not want to post test results to GitHub.

Run composer yaml-tests or, just like all composer commands, you can use shortcuts like compose y.

composer yaml-tests

The output will look something like this:

And you will get a nice summary at the end like this:

Yaml-Tests executable

There is now a "bin" for yaml-tests, allowing the command to be run by itself.

If you require provision-ops/yaml-tests, you will see a link to yaml-tests in your bin-dir.