Investigate ways to resolve running jobs queue not reflecting reality

[rafrombrc]

We know that the running jobs queue is often not right bc worker crash before updating the status to remove the current job from the jobs queue. Celery doesn't seem to provide the current list of running jobs (but maybe it does?), but somehow we'd like to try to keep this queue synchronized with what's actually happening.

[rafrombrc]

Related: https://bugzilla.mozilla.org/show_bug.cgi?id=1423553

[arikfr]

Trying to get a reliable status from Celery is something I've been digging into several times. The best I came up with is:

from redash.worker import celery
active = celery.control.inspect().active()

And then I parse it with:

def parse_active(active):
    rows = []

    for task in itertools.chain(*active.values()):
        data_source_id = None
        query_id = None

        if task['name'] == 'redash.tasks.execute_query':
            args = eval(task['args'])
            data_source_id = args[1]
            query_id = args[2].get('Query ID')
            if query_id == 'adhoc':
                query_id = None

        rows.append({
            'task_name': task['name'],
            'worker': task['hostname'],
            'queue': task['delivery_info']['routing_key'],
            'task_id': task['id'],
            'data_source_id': data_source_id,
            'query_id': query_id
        })

    return {
        'rows': rows,
        'columns': [column('task_name'), column('worker'), column('queue'),
                    column('task_id'), column('data_source_id', 'integer'),
                    column('query_id', 'integer')]
    }

As you can see I return a Redash results format here, and then I query it with the URL query runner, to load it into Redash and create dashboards :)

It's mostly reliable, but it's not 100% reliable -- if the worker doesn't respond for some reason, you get partial data... And usually workers don't respond when shit hits the fan, i.e. when you need this info 🤔

Having said that, it's better than what we have now, so I'm thinking of adding this into the project and replacing our current queue info with it while adding some warnings about reliability.

A longer term solution is to look into RQ as an alternative to Celery, which feels simpler so might be easier to poke around and get reliable info from.

[rafrombrc]

We're working around this issue in part by deploying flower (see #396 & #397) and in part by improving the per-data-source status reporting on status.json (see #379 & #388). Closing this since the work is being tracked in those other issues.