"unacceptable anti-replay nonce" integration test flake on main

[aarongable]

We've been living with this one for a while now, and should buckle down and fix it:

acme.messages.Error: urn:ietf:params:acme:error:badNonce :: The client sent an unacceptable anti-replay nonce :: Unable to validate JWS :: JWS has an invalid anti-replay nonce: "GKXSL_3j2RxCob8usrXV29mjC7FCrOELZH4Ta45Jd8JwNaqXMh8"

Example run: https://github.com/letsencrypt/boulder/actions/runs/17559123440/job/49870768932

[aarongable]

My working theory is that the flaky failure is as a result of a race that happens immediately after startservers starts the nonce servers:

1. Consul tries to notice that the nonce servers are healthy so it can populate their SRV records into DNS
2. Startservers launches the WFEs, which in turn look up the SRV records for their nonce backends

When a WFE needs to hand out or redeem a nonce, if it has all the nonce server backends, it obviously succeeds. So if consul fully wins the race, we're happy.

When a WFE needs to hand out or redeem a nonce, if it has no nonce server backends, it re-resolves. So if the WFEs fully win the race, we're happy.

However, if a WFE comes up at a point in time when consul has noticed some but not all of the nonce servers, the WFE will have backends (and therefore not attempt re-resolution) but it won't have all the backends and therefore may report "no backends match the nonce prefix", depending on which nonce servers it is missing.

One fix would be for the WFE to try to re-resolve its nonce backends any time it receives a nonce whose prefix it doesn't recognize. This would work, but would also cause tons of re-resolution whenever a nonce server goes down and we receive a (temporary) flood of no-longer-valid nonces.

Another fix would be for startservers to wait for consul to recognize the service as up, rather than merely waiting for each service to report itself as healthy. This would likely slow down startservers significantly, but would guarantee that all WFEs can see all the nonce servers by the time the first request comes in.

[jsha]

Health checking for our dev Consul was set up in #6908 and revised in #6928. Note that it's only set up for the SA; all other services have a straightforward config that always resolves them.

So the WFE in dev should always be able to look up the SRV record for a nonce server. Also, there is only one WFE so there's no issue of "this WFE sees ABC; this other WFE sees XYZ".

Also, for the WFE to get "no backends match the nonce prefix", it must receive a nonce with that prefix, which means it must have previously given out a nonce with that prefix, which means it must have talked to a nonce server matching that prefix.

[aarongable]

Also, there is only one WFE so there's no issue of "this WFE sees ABC; this other WFE sees XYZ".

Oh geez, I'd totally missed that we only spin up one WFE in startservers.py, despite spinning up multiples of many other services. Yep, that throws my whole theory out the window.