[14][queue_job] Vacuum done jobs in batch #313
Conversation
queue_job/models/queue_job.py
Outdated
| ) | ||
| if jobs: | ||
| jobs.unlink() | ||
| with api.Environment.manage(): |
There was a problem hiding this comment.
@florian-dacosta Why do you need to use a new cursor?
|
On the principle yes, on the implementation, as @lmignon noted, you shouldn't need a new cursor, and as the db snapshot can change after a commit, it may be better to do a search after every batch (with a limit). |
Huge amount of jobs to delete may take a lot of time and the cron may crash, because of timeout for instance This issue would make the cron permanently run and permanently failing... Deleting batch allow to avoid this issue if the cron fails, it still will have delete some of the job history
florian-dacosta
force-pushed
the
14-queue-vacuum-batch
branch
from
4c38400 to
085d5ba
Compare
|
I thought it was always better to do explicit commit in new cursor.
Nice addition, I had not see it, but I've also done it in this v8 project, mainly for the gain at deletion indeed. |
The cron are already executed in a second cursor |
|
Thanks to you :) |
lmignon
left a comment
lmignon
left a comment
There was a problem hiding this comment.
Thank you @florian-dacosta. LGTM (code review)
|
This PR has the |
|
/ocabot merge patch |
|
On my way to merge this fine PR! |
|
Congratulations, your PR was merged at 470d6b2. Thanks a lot for contributing to OCA. ❤️ |
6 participants
I ran into an issue several times (on version 8 but no reason it will be different on version 14) where the AutoVacuum Job Queue cron fails due to timeout.
As a result it starts again, fail again, and have to be unblocked manually (by deleting some jobs by script for instance)
After some digging, it seems some people had this kind of issue also in past version :
#115
and was even merged in v7 : OCA/connector#158
So I'd like to introduce the same in v14, commit every 1000 deleted jobs. (the 1000 is arbitrary, we could change it if needed, but I don't really see why it should be configurable.
I think it is important because depending on the project, the jobs may take more or less time to delete, for instance, if some mail.messages are added, it takes more time, if attachments are added (base_async_import), it takes more time, etc...
And depending how intensively jobs are used of course.
What do you think @guewen @rousseldenis @lmignon ?