Proper, full-stack test suite

[mpasternacki]

[JayBusch]

I'm happy to help. I'm trying to deploy jetpack on AWS/EC2 right now and have forked the project with the intent of contributing to the documentation at least; I'd be happy to write tests. This would be my first GoLang effort but I welcome that.

[mpasternacki]

Thank you! Help with documentation and tests is definitely needed. There's a bit of undocumented work in progress regarding full-stack test (testing the utility as a whole rather than individual libraries and functions). I'll describe it a bit better over the weekend, but here's a short overview:

Test cases are in the t/ directory. Tests are written in Perl (after some research I found Perl's testing framework to be most straightforward for this use case). Tests are started using prove command from lang/perl5.20, and need a couple Perl modules. They need to have jetpack command in $PATH, full run needs uninitialized host and starts by initializing, and appc/spec's validator ACIs need to be built (there's a target in Makefile that does it).

The biggest missing piece is automating the whole flow (installing Jetpack on a fresh host and running the test suite there; making it possible to test Jetpack on both 10.2-STABLE and 11-CURRENT host systems). Two approaches I'm considering now are: use bhyve (which would require host to run FreeBSD), or automate that part somehow with chef's test-kitchen framework. If you have any input for this part of the test workflow, I'd appreciate your help.

[mpasternacki]

The Perl modules needed for running the test suite are in packages p5-File-Slurp, p5-Test-Most, and p5-Test-Command

[JayBusch]

I think the approach using bhyve makes the most sense; it should be designed in such a way that qemu could also be used in the future if non FreeBSD hosts need to be supported. I don't find requiring FreeBSD to be an undue burden since it is my understanding that jetpack is intended to be a FreeBSD specific implementation of the App Container spec; and therefore it is reasonable to assume that anyone wanting to use jetpack has reasonable access to a FreeBSD host that can run bhyve.

Regarding chef; I'm not sure that adding ruby and other dependencies is a good idea. You are already using ansible; so it seems that could provide the configuration of the test host; or frankly even a shell script might suffice. I've also used the fabric library for Python to some success; again it would introduce another dependency but I would personally prefer that over a Ruby implementation (experience and some prejudice on my behalf driving that preference).

I'll see if I can put together a shell script that will create and run a bhyve host, install the jetpack source and necessary dependencies, and run the current test suite.

[mattbostock]

@mpasternacki If I were to open a pull request that converted the Perl tests to Go, would it be accepted?

I don't know Perl and I think the project might see more contributions from the Go community if the tests were in Go. I think the number of developers that know Go is likely greater than the number of developers that know both Go and Perl.

(PS I'm here because I was motivated by your talk at FOSDEM 2016 :-)

[JayBusch]

@mattbostock I second this and am actually sitting down now to further that goal.

In addition to standard Go unit tests I intend to add BDD tests using the Ginkgo framework.

[mikepea]

I'm interested in joining this effort too .

On Mon, Feb 8, 2016 at 1:29 AM, JayBusch notifications@github.com wrote:

@mattbostock https://github.com/mattbostock I second this and am
actually sitting down now to further that goal.

In addition to standard Go unit tests I intend to add BDD tests using the
Ginkgo framework.

—
Reply to this email directly or view it on GitHub
#8 (comment).

[mpasternacki]

Thanks for the feedback! The Perl tests are not perfect, but I've chosen these as best fitting the needs I had for this project. Any PR that uses something nicer (Go tests, Cucumber, whatever) will be most likely accepted.

For unit tests, standard Go testing framework is and should be used. There are some of these. Not enough. Any additional Go unit tests are most welcome.

The problem is full stack / acceptance test stack. What we need here is a system that:

1. Tests "from the outside" (black-box tests), running jetpack command and checking the effects on the real system;
2. Can run on its own, preferably without whole development/build environment (i.e. I want to be able to install Jetpack to a server, configure it, and then run the acceptance test suite right there). This is to test Jetpack's behaviour in real environments, for different setups. It might also help with debugging.
3. Maybe it could be worked around, but right now I need tests to be ordered. First test file needs an uninitialized system, because they check jetpack init. Following files require an initialized system.

The third item is most tricky, and right now I realize it might be problem with my approach rather than a requirement – if jetpack init on an initialized system is okay to run, then the scenarios other than first might just start with jetpack init and not check that system is not initialized. Teardown can include zfs destroy -r (but it should be easy to disable – I want to run individual test scenarios on my own workstation too). This way the tests can provide better isolation. The main issue here would be the metadata service, but this can also be supported with a smart setup/teardown logic (mds could be started anew for each scenario, again – better isolation).

I'm not defending Perl. I went with it by timeboxing research: this was the best option I could find at the time to get some working tests, which are better than none. I'm quite hesitant to use a sophisticated Go testing framework, these seem overcomplicated and overengineered to me. On the other hand, I find Cucumber's Gherkin scenario syntax (given/when/then) and its separating scenarios from step implementation to be a nice approach. Maybe we could look at either Cucumber (there's a port for it, so installation is not a problem), or a Go implementation that would use its scenario language? There is an official parser (https://github.com/cucumber/gherkin-go) and there seems to be a Cucumber-like tool natively in Go (http://gnuu.org/2015/03/30/gucumber-is-cucumber-for-go/).

I'd lean towards real Cucumber, because I'm pretty familiar with it and only one port would be required on the tested machine, but I also see the benefit in sticking to a single language. If gucumber could compile to one binary that would include all the step definitions (and maybe also scenarios themselves), this would be perfect.

[mpasternacki]

For the record: https://data-dog.github.io/godog/ looks nice too, and it uses official Gherkin parser library.

[JayBusch]

I'm looking at each of these and I can say initially that the Cucumber style BDD does seem better designed. In addition Cucumber seems to be a project that is very interested in cross language support.

I think the stability of the project we rely on is important, so I'm inclined towards a tool chain that is produced by the Cucumber project itself if there are no significant reasons as to why not. I will try and determine if these other projects exist for any special reason (killer feature, compatibility with FreeBSD, doesn't require some other language interpreter etc.) that we may care about.

[JayBusch]

BTW, per our previous conversation some months ago I wrote this shell script to ease the creation of bhyve test hosts. It is most definitely incomplete and not ready for a pull request. I am now actively learning Go for reasons beyond this project so I wonder if I should re-write it in Go and if that would be a good start to the "installing Jetpack on a fresh host and running the test suite there; making it possible to test Jetpack on both 10.2-STABLE and 11-CURRENT host systems" problem you had described at that time?

https://github.com/JayBusch/jetpack/blob/master/t/util/automation/build_initial_test_vm.sh

[mpasternacki]

Please take a look at this branch: https://github.com/3ofcoins/jetpack/tree/godog-tests

I have added a prototype of tests based on the godog tool. Instead of using their CLI, I added a cmd/test (built by default), which can then be redistributed with jetpack. To run it, make, and then run ./bin/test in repo's root.

I'd feel slightly more comfortable using Cucumber with its native Ruby, as I'm more used to that language for high-level scripting, but I feel rather good about godog. Possibility of building a binary to run acceptance tests seems very nice, and it uses official Cucumber's Go parser for the *.feature files. What do you think?

I'll take a look at the VM script tomorrow, thanks for linking it here!