Conversation
Here, experiment with parallelizing the main package's example tests by putting each in their own subdirectory. Example tests within any given package must run sequentially unfortunately, and because of that they're now about 50% of the main package's test suite's total runtime, which is pretty bad. The downside to subdirectories are that: * We trade parallelization for some additional compile time. (Although even so, we still see something like a 40% reduction in test time. Locally, about 8 seconds versus 13 seconds.) * The examples don't show up in Godoc anymore, so we'd have to link them. This might not be that bad because currently, they're so many examples now that they kind of look bad in the docs anyway (just the list of them takes up almost a whole page by itself). An alternative would be to see if we could speed up examples as much as possible, although this is somewhat difficult because there's going to be some fixed time in getting a client up and running, then working a job and stopped. It would be possible though, some ideas would be to use a non-batch completer (the batch completer has a ~250 ms overhead where it waits for jobs to come in), and to rewrite examples to make sure there's no sleeps. We also eliminate use of the `waitForNJobs` helper in favor of using `riversharedtest.WaitOrTimeoutN` combined with `testutil.PanicTB` which was introduced in #848. This was a nice thing anyway, but required here so we don't have to replicate the helper in every single example.
brandur
force-pushed
the
brandur-parallelized-examples-2
branch
from
b2f8459 to
ce670e3
Compare
|
Damn, this definitely has a nice local speedup effect, but actually makes CI 5-10 seconds slower, I guess because there isn't as much power available for compilation. |
|
I think this one is a pretty big negative due to examples not showing on the main doc page in the usual way. I think I'm opposed to it unless it turns out to be totally necessary, like if the perf gets so bad that there is no other way to speed up CI. |
|
Yeah, fair. If we intend to keep all the example tests in one package though, unless we can get them sped up in a general sense, we may have to be more judicious about which new ones we want to add in. Or try to farm more logic out to subpackages (e.g. like |
2 participants
Here, experiment with parallelizing the main package's example tests by
putting each in their own subdirectory. Example tests within any given
package must run sequentially unfortunately, and because of that they're
now about 50% of the main package's test suite's total runtime, which is
pretty bad.
The downside to subdirectories are that:
We trade parallelization for some additional compile time. (Although
even so, we still see something like a 40% reduction in test time.
Locally, about 8 seconds versus 13 seconds.)
The examples don't show up in Godoc anymore, so we'd have to link
them. This might not be that bad because currently, they're so many
examples now that they kind of look bad in the docs anyway (just the
list of them takes up almost a whole page by itself).
An alternative would be to see if we could speed up examples as much as
possible, although this is somewhat difficult because there's going to
be some fixed time in getting a client up and running, then working a
job and stopped. It would be possible though, some ideas would be to use
a non-batch completer (the batch completer has a ~250 ms overhead where
it waits for jobs to come in), and to rewrite examples to make sure
there's no sleeps.
We also eliminate use of the
waitForNJobshelper in favor of usingriversharedtest.WaitOrTimeoutNcombined withtestutil.PanicTBwhichwas introduced in #848. This was a nice thing anyway, but required here
so we don't have to replicate the helper in every single example.