When spawning jobs, is possible that the same job may be requested multiple times. An API could check if it already has the results and, if not, spawn a job. This approach could result in the job being spawned multiple times if it's already in the queue but not completed when it's requested again. We currently deal with this by having each worker check that the job it receives hasn't already been completed, and returning immediately if it has been.
Perhaps a neater approach is to add the job to the queue only if it isn't already in the queue or being processed.
Users can use a hash of the job to generate an ID for it, therefore, to achieve this, we need a method to add an item only if there isn't an item with the same ID already in the queue.
When spawning jobs, is possible that the same job may be requested multiple times. An API could check if it already has the results and, if not, spawn a job. This approach could result in the job being spawned multiple times if it's already in the queue but not completed when it's requested again. We currently deal with this by having each worker check that the job it receives hasn't already been completed, and returning immediately if it has been.
Perhaps a neater approach is to add the job to the queue only if it isn't already in the queue or being processed.
Users can use a hash of the job to generate an ID for it, therefore, to achieve this, we need a method to add an item only if there isn't an item with the same ID already in the queue.