Wrk scenario test cases #641

diegolovison · 2025-12-22T19:42:24Z

The goal of this test is to demonstrate two scenarios that are failing:

During the test we have the following log

16:34:51,699 INFO  (main) [i.h.s.WrkScenarioTest] Test duration: 49s
16:34:51,700 INFO  (main) [i.h.s.WrkScenarioTest] --- phase: calibration ---
16:34:51,703 INFO  (main) [i.h.s.WrkScenarioTest] StatusHistory{previousStatus=NOT_STARTED, currentStatus=RUNNING, when=2025-12-22T16:34:14.995-03:00[America/Sao_Paulo], threadId=1}
16:34:51,704 INFO  (main) [i.h.s.WrkScenarioTest] 2025-12-22T16:34:20.995-03:00[America/Sao_Paulo] elapsed: 6s -> StatusHistory{previousStatus=RUNNING, currentStatus=FINISHED, when=2025-12-22T16:34:20.995-03:00[America/Sao_Paulo], threadId=1}
16:34:51,704 INFO  (main) [i.h.s.WrkScenarioTest] 2025-12-22T16:34:22.995-03:00[America/Sao_Paulo] elapsed: 2s -> StatusHistory{previousStatus=FINISHED, currentStatus=TERMINATING, when=2025-12-22T16:34:22.995-03:00[America/Sao_Paulo], threadId=1}
16:34:51,704 INFO  (main) [i.h.s.WrkScenarioTest] 2025-12-22T16:34:33.909-03:00[America/Sao_Paulo] elapsed: 10s -> StatusHistory{previousStatus=TERMINATING, currentStatus=TERMINATED, when=2025-12-22T16:34:33.909-03:00[America/Sao_Paulo], threadId=56}
16:34:51,704 INFO  (main) [i.h.s.WrkScenarioTest] 2025-12-22T16:34:33.917-03:00[America/Sao_Paulo] elapsed: 0s -> StatusHistory{previousStatus=TERMINATED, currentStatus=STATS_COMPLETE, when=2025-12-22T16:34:33.917-03:00[America/Sao_Paulo], threadId=56}
16:34:51,704 INFO  (main) [i.h.s.WrkScenarioTest] --- phase: test ---
16:34:51,705 INFO  (main) [i.h.s.WrkScenarioTest] StatusHistory{previousStatus=NOT_STARTED, currentStatus=RUNNING, when=2025-12-22T16:34:33.917-03:00[America/Sao_Paulo], threadId=1}
16:34:51,705 INFO  (main) [i.h.s.WrkScenarioTest] 2025-12-22T16:34:43.917-03:00[America/Sao_Paulo] elapsed: 10s -> StatusHistory{previousStatus=RUNNING, currentStatus=FINISHED, when=2025-12-22T16:34:43.917-03:00[America/Sao_Paulo], threadId=1}
16:34:51,705 INFO  (main) [i.h.s.WrkScenarioTest] 2025-12-22T16:34:43.917-03:00[America/Sao_Paulo] elapsed: 0s -> StatusHistory{previousStatus=FINISHED, currentStatus=TERMINATING, when=2025-12-22T16:34:43.917-03:00[America/Sao_Paulo], threadId=1}
16:34:51,705 INFO  (main) [i.h.s.WrkScenarioTest] 2025-12-22T16:34:51.450-03:00[America/Sao_Paulo] elapsed: 7s -> StatusHistory{previousStatus=TERMINATING, currentStatus=TERMINATED, when=2025-12-22T16:34:51.450-03:00[America/Sao_Paulo], threadId=55}
16:34:51,705 INFO  (main) [i.h.s.WrkScenarioTest] 2025-12-22T16:34:51.450-03:00[America/Sao_Paulo] elapsed: 0s -> StatusHistory{previousStatus=TERMINATED, currentStatus=STATS_COMPLETE, when=2025-12-22T16:34:51.450-03:00[America/Sao_Paulo], threadId=55}

Between FINISHED and TERMINATED, What should we do with in-flight requests?

franz1981 · 2025-12-22T19:58:27Z

Just focus on one issue a time bud 🙏

core/src/main/java/io/hyperfoil/core/impl/PhaseInstanceImpl.java

diegolovison · 2025-12-22T20:53:12Z

Just focus on one issue a time bud

Yes. The focus is WrkScenarioTest::wrk2Test.

Any idea about what we should do if in-flight requests?

Issue: Requests started in the calibration phase are still running when we switch to the test phase.

Impact: The test phase cannot process anything. Its 10-second time limit runs out while waiting for the old calibration work to finish.

Question: Should we make the system wait for all requests to finish before moving to the next phase?

franz1981 · 2025-12-22T21:19:46Z

Why it is waiting?
It is not using its own connections?

franz1981 · 2025-12-22T21:26:46Z

Try to understand why we have a calibration phase and what it is for; if is not needed or the reason why was there no longer valid, we should get rid of it.
Said that; usually phases should wait till the previous operations are completed: it is not like that?

diegolovison · 2025-12-23T00:43:38Z

Try to understand why we have a calibration phase and what it is for; if is not needed or the reason why was there no longer valid, we should get rid of it.

The calibration, imho, is valid to warm up the loader.

usually phases should wait till the previous operations are completed: it is not like that?

Ok. So you are expecting that.

The idea of this PR is not to fix any issue. Just create a reproducer for it.

franz1981 · 2025-12-29T13:07:21Z

The calibration, imho, is valid to warm up the loader.

Yes and no. To be fair you don't know when is warmed up (it depends by both client and server hotspot or anything which the server is running..) so we should have an option to decide to not make it happen too.
It could be surprising to users which expect all requests to be within the steady state only (the layden people would noticed that..)

franz1981 · 2025-12-29T13:07:22Z

The calibration, imho, is valid to warm up the loader.

Yes and no. To be fair you don't know when is warmed up so we should have an option to decide to not make it happen too (or decide the calibration duration, which is better IMHO).
It could be surprising to users which expect all requests to be within the steady state only (the layden people would noticed that..)

franz1981 · 2026-01-08T12:58:02Z

I'm not able to see the tests failing on my machine: check what's going on @diegolovison 🙏

franz1981 · 2026-01-08T12:36:50Z

cli/src/main/java/io/hyperfoil/cli/commands/WrkScenario.java

+      test
+   }
+
+   public BenchmarkBuilder getBenchmarkBuilder(String name, String url, boolean enableHttp2, int connections,


Suggested change

public BenchmarkBuilder getBenchmarkBuilder(String name, String url, boolean enableHttp2, int connections,

public BenchmarkBuilder createBenchmarkBuilder(String name, String url, boolean enableHttp2, int connections,

franz1981 · 2026-01-08T12:37:46Z

core/src/main/java/io/hyperfoil/core/impl/PhaseInstanceImpl.java

      synchronized (this) {
         assert status == Status.NOT_STARTED : "Status is " + status;
-         status = Status.RUNNING;
+         this.changeStatus(Status.RUNNING);


no need to use this. for the these...

Fixed by e77b7c0

I removed those unrelated changes from this PR

franz1981 · 2026-01-08T12:39:24Z

core/src/main/java/io/hyperfoil/core/impl/SimulationRunner.java

      }
   }
+
+   public void setEnableWatchdog(boolean enableWatchdog) {


this seems unrelated to this issue: what it is?

Fixed by e77b7c0

I removed those unrelated changes from this PR

franz1981 · 2026-01-08T12:43:31Z

test-suite/src/test/java/io/hyperfoil/benchmark/BaseBenchmarkTest.java

      return benchmark.getAbsolutePath();
   }
+
+   public class TestStatistics implements StatisticsCollector.StatisticsConsumer {


TestStatistics is already on BaseScenarioTest - no need of a new class which does the same (if it's the same)

TestStatistics from BaseScenarioTest is storing only the metric.
The new TestStatistics are separated per phase, which is where we can show the issue in the assert.

cannot we unify them and have just one type?

Wrk scenario

00387b7

franz1981 reviewed Dec 22, 2025

View reviewed changes

core/src/main/java/io/hyperfoil/core/impl/PhaseInstanceImpl.java Outdated Show resolved Hide resolved

Remove StatusHistory

5bcf083

Fix stats metric

b0b35d9

diegolovison mentioned this pull request Dec 29, 2025

Pulse Continues During PhaseInstance.Status.TERMINATING #642

Open

diegolovison mentioned this pull request Dec 29, 2025

Connection Leak During PhaseInstance.Status.TERMINATING #643

Open

diegolovison force-pushed the wrk_scenario branch from f5fa9fc to 4a710b0 Compare January 7, 2026 16:24

Reduce verbosity for tests

cd01e7f

franz1981 reviewed Jan 8, 2026

View reviewed changes

diegolovison added 2 commits January 8, 2026 10:25

Revert not related changes

e77b7c0

Reduce a chance of the test being flaky

76d76ef

diegolovison force-pushed the wrk_scenario branch from 4a710b0 to 76d76ef Compare January 8, 2026 14:44

	public BenchmarkBuilder getBenchmarkBuilder(String name, String url, boolean enableHttp2, int connections,
	public BenchmarkBuilder createBenchmarkBuilder(String name, String url, boolean enableHttp2, int connections,

Wrk scenario test cases #641

Are you sure you want to change the base?

Wrk scenario test cases #641

Uh oh!

Conversation

diegolovison commented Dec 22, 2025

Uh oh!

franz1981 commented Dec 22, 2025

Uh oh!

Uh oh!

diegolovison commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

franz1981 commented Dec 22, 2025

Uh oh!

franz1981 commented Dec 22, 2025

Uh oh!

diegolovison commented Dec 23, 2025

Uh oh!

franz1981 commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

franz1981 commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

franz1981 commented Jan 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

diegolovison commented Dec 22, 2025 •

edited

Loading

franz1981 commented Dec 29, 2025 •

edited

Loading

franz1981 commented Dec 29, 2025 •

edited

Loading