Intermittent test failures in CI

[tschneidereit]

On the last merge to main we had an intermittent test failure. The test passed on a manually triggered re-run.

@vados-cosmonic it seems like this is due to getRandomPort not waiting for the temporary server it creates to be closed before reporting the port.

However, I think there's a fundamental race condition here: either we have the race seen in this test failure, or we have a potential race between this and another test, where that other test might steal the port between the temporary server shutting down and our test server starting up.

That second one is probably less likely, but I'm not sure. A way to avoid either would be to not create a temporary server at all: why not create the actual test server in the global script instead of the test function, and return its port once it got created?

[vados-cosmonic]

Yeah the temporary server is really just to get a random port -- it has to be closed before the port is reported because we want to actually use the port relatively soon after. I agree that there's a race condition here.

That second one is probably less likely, but I'm not sure. A way to avoid either would be to not create a temporary server at all: why not create the actual test server in the global script instead of the test function, and return its port once it got created?

Yeah I think that would work -- basically the server itself should just be created outside, w/ a randomized port it holds on to, that sounds good to me!