Testing with multiple suites has them run in parallel

[seantalts]

If use go test on all of your packages at once and there are multiple test suites, the suites will end up running in parallel with other suites.

[matryer]

Interesting - is this a bug or feature? :)

On May 2, 2014, at 2:15 PM, seantalts notifications@github.com wrote:

If use go test on all of your packages at once and there are multiple test suites, the suites will end up running in parallel with other suites.

—
Reply to this email directly or view it on GitHub.

[seantalts]

I think it's a bug, as normally go test will only execute tests in parallel if they mark themselves as parallel by calling *testing.T.Parallel().

[nelsam]

I've been running in to this lately, and have been trying to find a fix. Work on it is slow, though - I'm swamped.

[tylerstillwater]

Thanks for looking in to this, @nelsam :)

[MysticHLE]

I believe our team is running into this too today.

Basically I created a new test file in a particular package with some barebone test structure - no actual tests...just an empty struct type that embeds suite.Suite, and a function that takes in a *testing.T object and calls suite.Run() on said struct. This immediately caused all our other tests to start failing indeterministically. The nature of the failures were associated with database unique key integrity violations on inserts and deletes into a single Postgres DB. This is leading me to believe that the tests were being run concurrently without calling our setup methods to prepare the environment properly between tests. Needless to say, the moment I move this test file to another package, everything magically works!

We are not choosing to run any of the tests concurrently.

[matryer]

Would an option to run synchronously be the solution?

[nelsam]

The issue really boils down to the fact that I don't know what's causing suites to run in parallel. I've done some research (again, I'm swamped), but haven't really gotten much worthwhile about what's causing it.

Technically, tests that can be run in parallel should call t.Parallel(), but all suite methods are being run in parallel without having that called.

I'll see what I can find out today, but no promises. This is on my priority list, but it's below several other things I'm working on.

[matryer]

@tylerb how about two eyes on it?

On Oct 29, 2014, at 16:51, nelsam notifications@github.com wrote:

The issue really boils down to the fact that I don't know what's causing suites to run in parallel. I've done some research (again, I'm swamped), but haven't really gotten much worthwhile about what's causing it.

Technically, tests that can be run in parallel should call t.Parallel(), but all suite methods are being run in parallel without having that called.

I'll see what I can find out today, but no promises. This is on my priority list, but it's below several other things I'm working on.

—
Reply to this email directly or view it on GitHub.

[seantalts]

They're running in parallel because of the implementation of golang's
testing.Run that testify's suite.Run calls. It spawns a new goroutine and
communicates with it over a channel.

HTH, on a bus sorry.
On Oct 29, 2014 5:53 PM, "Mat Ryer" notifications@github.com wrote:

@tylerb how about two eyes on it?

On Oct 29, 2014, at 16:51, nelsam notifications@github.com wrote:

The issue really boils down to the fact that I don't know what's causing
suites to run in parallel. I've done some research (again, I'm swamped),
but haven't really gotten much worthwhile about what's causing it.

Technically, tests that can be run in parallel should call t.Parallel(),
but all suite methods are being run in parallel without having that called.

I'll see what I can find out today, but no promises. This is on my
priority list, but it's below several other things I'm working on.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#52 (comment).

[nelsam]

Found it.

It basically looks like testing.RunTests and testing.InternalTest are not intended for use externally. I'll see what I can do. I think our original version just walked through methods calling them individually (instead of adding them to a slice and calling testing.RunTests), and I think that would be a better choice than what we currently have; however, I'd love it if I could get individual test methods to respect both testing.T.Parallel() and the -parallel flag.

For the time being, I'll probably just strip out the testing.InternalTest and testing.RunTests usage, because (as I mentioned previously) I'm swamped and don't have much time to dedicate to this. However, I'll keep an item on my todo list to figure out a way to make use of the parallel test stuff. I'm pretty sure GoCheck works with those flags, but their solution is pretty complicated, and I think I'd rather have a simple solution than a complete solution. Anyone who really needs that functionality can probably just use GoCheck.

EDIT: Actually, I think I have a better idea. Sorta. Maybe. We'll see how much it breaks things.

[nelsam]

... Argh.

Okay, so, I was pretty sure I had found the problem, but I was (to put it simply) wrong. I added channels to suite.Run() to ensure that only one instance of suite.Run() was calling testing methods at a time, and then I just removed everything so that suite.Run() called every method in sequence (without using testing.InternalTest or testing.RunTests). Same issue, both times.

My boss found this, which suggests that the issue isn't exclusive to our suites: http://stackoverflow.com/questions/15721238/go-serial-execution-of-package-tests

It looks like relying on serial execution when using go test ./... just isn't supported without tweaking. I'm going to try to find a way to swap things around so that I can use transactions, because I still want the queries to actually run against my database engine (for checking query syntax); that might require a pretty big change to my application code, but we'll see.

[MysticHLE]

As that StackOverflow thread pointed out, it's possible to use the -p=1 flag to force the maximum number of packages to be ran in parallel (with 1 effectively serializing it all). This doesn't seem to be a problem with testify - but rather, just how go test works.

Another thread for this issue is here:

http://stackoverflow.com/questions/23715302/go-how-to-run-tests-for-multiple-packages

[hectorj]

Can confirm @MysticHLE 's solution works. Go documentation matches: https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies

-p n
the number of programs, such as build commands or
test binaries, that can be run in parallel.
The default is the number of CPUs available, except
on darwin/arm which defaults to 1.

Ticket should be closed I think.