Skip to content

Option to reset tries in ping()#1

Merged
alecgibson merged 1 commit intomainfrom
reset-tries
Jun 25, 2021
Merged

Option to reset tries in ping()#1
alecgibson merged 1 commit intomainfrom
reset-tries

Conversation

@alecgibson
Copy link
Collaborator

The ping() method can be useful for setting up recurring jobs if we
deliberately avoid acking the job. For example:

  1. Submit job
  2. Pull job from queue
  3. Process job
  4. ping()
  5. Go to Step 2

A real-world example of this might be notifying for recurring
appointments, or setting up long-running, cross-process, periodic jobs.

The main advantage this has over using ack() and add() is that it
effectively requeues a job in a single, atomic commit. If we tried the
above with ack() and add():

  1. Submit job
  2. Pull job from queue
  3. Process job
  4. ack()
  5. add()
  6. Go to Step 2

In this version, the process could crash or quit between Steps 4 & 5,
and our recurring job would be lost.

We could also try inverting Steps 4 & 5, but then we get the opposite
issue: if the process crashes or quits, then we might accidentally
duplicate our recurring job. It also prevents us from setting up any
unique indexes on our payload.

Using ping() perfectly solves this problem: there's only ever one
version of the job, and it's never dropped (because it's never acked).
If the process crashes before we ping(), we'll retry it, as with any
other normal job.

The one issue with this approach is that tries will steadily increase,
and - if you have maxRetries set up - the job will eventually be moved
to the dead queue, which isn't what we want.

This change adds an option to the ping() method: resetTries, which
will reset tries to zero, so that the job is treated like a "new" job
when it's pinged, and is only moved to the dead queue if it's genuinely
retried.

The `ping()` method can be useful for setting up recurring jobs if we
deliberately avoid acking the job. For example:

 1. Submit job
 2. Pull job from queue
 3. Process job
 4. `ping()`
 5. Go to Step 2

A real-world example of this might be notifying for recurring
appointments, or setting up long-running, cross-process, periodic jobs.

The main advantage this has over using `ack()` and `add()` is that it
effectively requeues a job in a single, atomic commit. If we tried the
above with `ack()` and `add()`:

 1. Submit job
 2. Pull job from queue
 3. Process job
 4. `ack()`
 5. `add()`
 6. Go to Step 2

In this version, the process could crash or quit between Steps 4 & 5,
and our recurring job would be lost.

We could also try inverting Steps 4 & 5, but then we get the opposite
issue: if the process crashes or quits, then we might accidentally
duplicate our recurring job. It also prevents us from setting up any
unique indexes on our `payload`.

Using `ping()` perfectly solves this problem: there's only ever one
version of the job, and it's never dropped (because it's never acked).
If the process crashes before we `ping()`, we'll retry it, as with any
other normal job.

The one issue with this approach is that `tries` will steadily increase,
and - if you have `maxRetries` set up - the job will eventually be moved
to the dead queue, which isn't what we want.

This change adds an option to the `ping()` method: `resetTries`, which
will reset `tries` to zero, so that the job is treated like a "new" job
when it's pinged, and is only moved to the dead queue if it's genuinely
retried.
Copy link

@ricardoferrolho ricardoferrolho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@alecgibson alecgibson merged commit a32799a into main Jun 25, 2021
@alecgibson alecgibson deleted the reset-tries branch June 25, 2021 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants