Skip to content

Conversation

@rousseldenis
Copy link
Contributor

@rousseldenis rousseldenis commented Dec 14, 2018

@guewen @lmignon

Did this as sometimes, the amount of jobs is huge and there is no need fetching fields.

@rousseldenis
Copy link
Contributor Author

@guewen I bumped version as it was not done for long. But in separate commit if you disagree.

"""
deadline = datetime.now() - timedelta(days=self._removal_interval)
jobs = self.search(
jobs = self.with_context(prefetch_fields=False).search(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was sure that the prefetch only happen the first time you access a field (other than id)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try to objective that with a test script (not in the module)

@rousseldenis rousseldenis force-pushed the 10.0-imp-queue_no_prefetch-dro branch from 2dc130c to e44038d Compare December 18, 2018 13:46
@rousseldenis
Copy link
Contributor Author

@guewen I think you're right.
The problem is the timeout when your backlog is huge. I was inspired by odoo orderpoint runner (https://github.com/odoo/odoo/blob/10.0/addons/stock/models/procurement.py#L308) to make sets of jobs to unlink and commit.
I'm not sure of all the parts I've done, so if you have suggestions, I will implement them !

job_obj = self.env['queue.job']
deadline = datetime.now() - timedelta(days=self._removal_interval)
jobs = self.search(
jobs_ids = self.with_context(prefetch_fields=True).search(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rousseldenis IMO prefetch_field is useless

cr.commit()
except OperationalError:
if use_new_cursor:
cr.rollback()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rousseldenis IMO all the logic should be enclosed into a try except finally block

    if use_new_cursor:
        cr = registry(self._cr.dbname).cursor()
    try:
        self = self.with_env(self.env(cr=cr))
        jobs = job_obj.browse(jobs_ids[:jobs_set])
        jobs_ids = jobs_ids[jobs_set:]
        jobs.unlink()
        if use_new_cursor:
            cr.commit()
    except OperationalError:
         if use_new_cursor:
            cr.rollback()
            raise
    finally:
        if use_new_cursor:
            cr.close()

@rousseldenis rousseldenis force-pushed the 10.0-imp-queue_no_prefetch-dro branch from e44038d to b5d8160 Compare December 18, 2018 14:28
@rousseldenis rousseldenis force-pushed the 10.0-imp-queue_no_prefetch-dro branch from b5d8160 to c4e8caf Compare December 18, 2018 14:42
@rousseldenis
Copy link
Contributor Author

@lmignon @guewen

Copy link
Contributor

@lmignon lmignon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rousseldenis
Copy link
Contributor Author

@guewen As suggested by @lmignon, should I fill in changelog as it should includes all commits descriptions since the previous version?

P.S.: The modification has been tested on a real database.

@sbidoul
Copy link
Member

sbidoul commented Jan 14, 2019

@rousseldenis are you sure the benefit is worth the additional code complexity?
I understood you faced the issue in a real pathological situation (which could be resolved manually by a sql delete). When the vacuum cron runs regularly nobody faces the issue, even on the largest queue job delployments AFAIK.

@rousseldenis
Copy link
Contributor Author

@sbidoul You have to handle also mail messages. So, in fact there should be multiple sql deletes.

@github-actions
Copy link

There hasn't been any activity on this pull request in the past 4 months, so it has been marked as stale and it will be closed automatically if no further activity occurs in the next 30 days.
If you want this PR to never become stale, please ask a PSC member to apply the "no stale" label.

@github-actions github-actions bot added the stale PR/Issue without recent activity, it'll be soon closed automatically. label Nov 14, 2021
@github-actions github-actions bot closed this Dec 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale PR/Issue without recent activity, it'll be soon closed automatically.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants