With high concurrency (75-100 concurrency), users hit Nighthawk Scheduled Start Time Exception

[dubious90]

Nighthawk Exception:

terminating with uncaught exception of type Nighthawk::NighthawkException: Scheduled starting time needs to be in the future

We should find a way to make this work. Easiest solution would probably be to allow the worker delay to be configurable. Current code at

nighthawk/source/client/process_impl.cc

Line 110 in 24a11ff

const double inter_worker_delay_usec =

However, what is the purpose of the actual failure at

nighthawk/source/common/rate_limiter_impl.cc

Line 60 in 1b69aaf

throw NighthawkException("Scheduled starting time needs to be in the future");

? Maybe there's another way to accomplish this, that doesn't require our consumers to know about the internals.

[oschaaf]

Some background on the actual purpose:

The code involved tries to schedule the start of each workers so that each worker will have non-overlapping request-release timings. With that, --rps 2 --concurrency 1 should be the equivalent of --rps 1 --concurrency 2 from the perspective of request-release timings.

However, in the case of a worker not readying up before the time at which we had intended it to start, the above would no longer hold. Timings would be skewed, at least initially, probably resulting in distorted measurements.

In the happy flow, the workers compare the date against their designated start time in a spin loop. When it's time to start their actual work.

Some thoughts:
Maybe we should use synchronisation primitives for this in the case where --concurrency outnumbers the number of physical cpus; alternatively we could also greatly lower the risk by making this a two step process: the main thread waits for the workers to signal that they are ready, and then it computes the timing offsets and hands those to the workers.

In retrospect, I wished I had put a warning or error in there, instead of throwing an exception.

-- editted a typo --

[oschaaf]

Above I wrote --rps 2 --concurrency 1 two times. This should have been --rps 1 --concurrency 2 vs --rps 2 --concurrency 1. (Edited that in the original comment as well for clarity).

[dubious90]

Here is an example of the run command that failed within google (redacted url target)

./nighthawk_client --rps 200 --duration 10000 --open-loop --concurrency 75 --max-requests-per-connection 200 -v info --failure-predicate benchmark.http_4xx:999999999 --failure-predicate benchmark.http_5xx:999999999 --failure-predicate benchmark.pool_connection_failure:999999999 {{redacted_url}}

[dubious90]

We talked within our team, and agree with your assessment that this should probably be a LOG statement rather than an exception/error. Would you be okay with us submitting such a change?

In the longterm, we can also discuss better ways to configure this worker delay.

[oschaaf]

@dubious90 a change to make this a LOG line sounds good to me.

[oschaaf]

Note: #53 has a request to optionize the worker start delay, so that is still being tracked.