Skip to content

Attempt fetch immediately when a queue is unpaused#854

Merged
brandur merged 1 commit intomasterfrom
brandur-queue-unpause-fetch
Apr 28, 2025
Merged

Attempt fetch immediately when a queue is unpaused#854
brandur merged 1 commit intomasterfrom
brandur-queue-unpause-fetch

Conversation

@brandur
Copy link
Contributor

@brandur brandur commented Apr 26, 2025

Here, try to speed up the test suite by triggering a job fetch in the
producer when receiving a notice that a queue's been unpaused.

Currently, the "queue paused" example is by far the slowest test in the
entire project, clocking in at an impressive 1+ second:

$ go test . -count 1 -run Example_queuePause -test.v
=== RUN   Example_queuePause
Initial: 42ns
After start: 102.926042ms
Before pause: 109.6845ms
After pause: 109.879458ms
After insert: 112.385292ms
After reliable receive: 208.227042ms
After unreliable receive: 1.106444792s
After stop: 1.111110333s
--- PASS: Example_queuePause (1.11s)
PASS
ok      github.com/riverqueue/river     1.318s

(Extra timing information's been added to the test case above, but
notice how waiting for the receive from the unreliable queue after
unpausing it takes 800 ms+.)

The test's written in a way that it's hard to make it really fast, but
this change does succeed in more than doubling its speed:

$ go test . -count 1 -run Example_queuePause -test.v
=== RUN   Example_queuePause
Initial: 41ns
After start: 74.874791ms
Before pause: 77.679708ms
After pause: 83.262416ms
After insert: 88.872ms
After reliable receive: 177.465125ms
After unreliable receive: 276.994541ms
After stop: 284.25225ms
--- PASS: Example_queuePause (0.28s)
PASS
ok      github.com/riverqueue/river     0.516s

Because example tests have to run sequentially, that means we knock
about 800 ms off every run of the top level package's tests.

Here, try to speed up the test suite by triggering a job fetch in the
producer when receiving a notice that a queue's been unpaused.

Currently, the "queue paused" example is by far the slowest test in the
entire project, clocking in at an impressive 1+ second:

    $ go test . -count 1 -run Example_queuePause -test.v
    === RUN   Example_queuePause
    Initial: 42ns
    After start: 102.926042ms
    Before pause: 109.6845ms
    After pause: 109.879458ms
    After insert: 112.385292ms
    After reliable receive: 208.227042ms
    After unreliable receive: 1.106444792s
    After stop: 1.111110333s
    --- PASS: Example_queuePause (1.11s)
    PASS
    ok      github.com/riverqueue/river     1.318s

(Extra timing information's been added to the test case above, but
notice how waiting for the receive from the unreliable queue after
unpausing it takes 800 ms+.)

The test's written in a way that it's hard to make it really fast, but
this change does succeed in more than doubling its speed:

    $ go test . -count 1 -run Example_queuePause -test.v
    === RUN   Example_queuePause
    Initial: 41ns
    After start: 74.874791ms
    Before pause: 77.679708ms
    After pause: 83.262416ms
    After insert: 88.872ms
    After reliable receive: 177.465125ms
    After unreliable receive: 276.994541ms
    After stop: 284.25225ms
    --- PASS: Example_queuePause (0.28s)
    PASS
    ok      github.com/riverqueue/river     0.516s

Because example tests have to run sequentially, that means we knock
about 800 ms off every run of the top level package's tests.
@brandur brandur force-pushed the brandur-queue-unpause-fetch branch from 3d222ea to 0b68fcc Compare April 26, 2025 05:05
@brandur brandur requested a review from bgentry April 26, 2025 05:08
Copy link
Contributor

@bgentry bgentry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a thought of how we could add coverage to this but it seems like it'd be pretty difficult and probably not worth it 🤔 This should work though!

@brandur
Copy link
Contributor Author

brandur commented Apr 28, 2025

I had a thought of how we could add coverage to this but it seems like it'd be pretty difficult and probably not worth it 🤔 This should work though!

Yeah, had the same thought. It'd be kind of nice, but hard to test properly and might bring in more intermittency. Hopefully this is reasonably unlikely to regress.

@brandur brandur merged commit 6e09e4b into master Apr 28, 2025
10 checks passed
@brandur brandur deleted the brandur-queue-unpause-fetch branch April 28, 2025 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants