Should we actually run fixtures concurrently?

[njsmith]

Upsides:

• Can make test setup/teardown faster
• Since we have to run fixtures in independent tasks anyway (see It's very annoying that fixtures can't yield inside a nursery/cancel scope #55), running them concurrently was easier than running them sequentially.

Counterarguments:

• Might increase the chance of flaky tests, race conditions, etc.
  • In particular, race conditions involving ContextVars (see Rework fixtures #50 (comment)) – currently fixtures all share a contextvars.Context, so if multiple fixtures are mutating the same ContextVar, the results will be random. (Of course, doing this is inherently nonsensical. But nonsensical-and-consistent is still better than nonsensical-and-random.)
• Might make failing tests harder to debug
• Is there really that much concurrency to extract from fixture setup? How many people use multiple independent fixtures that need to block for significant amounts of time?
• Running them sequentially isn't actually that hard, if we let pytest pick the order for us (see Rework fixtures #50 (comment))

Leaving this issue open for now for discussion, and to collect any positive/negative experiences people want to report.

[njsmith]

#120 is a genuine bug that is partly due to the complexity of managing concurrent fixture startup -- specifically, if one fixture setup crashes, we stop creating new ones, but we don't do anything to cancel any concurrent fixtures. And fixing it is made more difficult by the complexity of our fixture setup code. If we switched to serial setup, it might make this more maintainable.

[gc-ss]

If we switched to serial setup, it might make this more maintainable.

I don't think we had to take that route. The fix was ensuring the cancel scopes got passed through properly instead?

[njsmith]

Not so much passing them through properly, but keeping track of which fixtures were currently being set up, and cancelling all of them instead of just the test.

[FPtje]

Hi, I see a call on your website for people who would benefit from pytest fixtures that are being set up concurrently.

At my company we've been using pytest to run integration tests for some of our programs. These integration tests generally look as follows:

• Start an empty local database
• Create the necessary tables on the database
• Start services X, Y, Z
• Wait for services X, Y, Z to be online (poll until some HTTP endpoint returns a value)
• Run tests
• Shut down X, Y, Z
• Shut down local database

Quite a few of these setup tasks are slow. Starting the empty database takes a couple of seconds, creating the tables, and finally polling the services takes a couple of seconds each depending on the service.

Naively implemented, these startup times all add up. Where each individual thing is just a couple of seconds, it adds up to minutes. Besides, not every service needs the database, and services generally don't need other services.

Right now we have a workaround that kind of looks like this:

@pytest.fixture(scope="session", autouse=True)
def start_whole_environment():
    with start_db() as db:
        with start_x() as x:
            with start_y() as y:
                with start_z() as z:
                    x.poll()
                    y.poll()
                    z.poll()
                    yield (x, y, z)
                    x.terminate()
                    y.terminate()
                    z.terminate()

The benefit of that workaround is that all services start at about the same time, and polling is about as slow as the slowest one. The downside is that it now always starts the entire environment, even if you just want to run a single test that only needs a single service.

Using a solution where there is a simple fixture per service would be much nicer. So far this GitHub repo is the only thing I've found after searching specifically for running fixtures concurrently.