Proposal: Async (Background) Queue

[ostark]

First off, using the new yii2-queue is a great improvement!

But the HTTP/ajax based queue is still the default, right? What's wrong with it? In PHP FPM environments, when all PHP FPM processes are blocked by ajax requests, the site can become unresponsive. Fortunately, we can turn off runQueueAutomatically and run the processes in the background.

However, when we trigger craft queue/run by a cron job, the queue handling is delayed a bit. And for craft queue/listen we may use a process monitor to ensure that the queue listener does not stop running.

Proposal to fix this

If we could run a daemonized php process in the background to handle the jobs right after the job was created - without blocking the rendering of the site. For laravel there is a package that does exactly this.

Since there is no package for Yii so far, I wrote a plugin for Craft 3 - just as a proof of concept:
https://github.com/ostark/craft-async-queue

Please let me know what you think. Is this worth a PR here?

---

This is how the process execution looks like:

Running processes

Empty queue (only php-fpm master is running)

xxx:~$ ps auxf | grep php
root      2953  0.0  0.0 399552 13520 ?        Ss   12:27   0:00 php-fpm: master process (/etc/php/fpm.conf)

New jobs pushed (php-fpm master + child + bin/php daemon started)

xxx:~$ ps auxf | grep php
root      2953  0.0  0.0 399552 13520 ?        Ss   12:27   0:00 php-fpm: master process (/etc/php/fpm.conf)
app       3031  2.2  0.2 718520 45992 ?        S    12:31   0:00  \_ php-fpm: pool www
app       3033  1.2  0.2 280936 32808 ?        S    12:31   0:00 /usr/bin/php craft queue/run -v
app       3034  0.0  0.0   4460   784 ?        S    12:31   0:00  \_ sh -c /opt/php/7.1/bin/php craft queue/exec "297" "0" "1" --verbose=1 --color=
app       3035  1.2  0.2 280928 32280 ?        S    12:31   0:00      \_ /opt/php/7.1/bin/php craft queue/exec 297 0 1 --verbose=1 --color=

No requests (only php-fpm master + bin/php daemon still running until queue is empty)

xxx:~$ ps auxf | grep php
root      2953  0.0  0.0 399552 13520 ?        Ss   12:27   0:00 php-fpm: master process (/etc/php/fpm.conf)
app       3033  0.3  0.2 280936 32844 ?        S    12:31   0:00 /usr/bin/php craft queue/run -v
app       3044  0.0  0.0   4460   688 ?        S    12:31   0:00  \_ sh -c /opt/php/7.1/bin/php craft queue/exec "301" "0" "1" --verbose=1 --color=
app       3045  1.1  0.2 280928 32396 ?        S    12:31   0:00      \_ /opt/php/7.1/bin/php craft queue/exec 301 0 1 --verbose=1 --color=

[angrybrad]

Sorry if I'm being dense, @ostark, but can you achieve the same thing by daemonizing craft queue/listen and passing in 0 for the delay parameter?

[ostark]

Sorry I closed this by accident.

queue/listen would work the same way. However listening to potential jobs does not make much sense, as the jobs get processed when they arrive.

And why do you prefer a delay of 0? This means a sleep(0) in the while loop and can be very expensive for the DB and on the 'php worker' as well.

[ostark]

@takobell do you see the benefits of this approach?

[angrybrad]

@ostark Ahh... I misunderstood. The difference is that queue/listen is constantly polling at a set interval and could be wasted CPU cycles in the case of low queue activity. Your change doesn't poll, but will trigger a queue/run asynchronously/immediately when a new job is added to the queue. Correct?

[ostark]

Right

[angrybrad]

I like it... thoughts @brandonkelly?

[brandonkelly]

Like the concept too. I don’t think we should assume that PHP will be capable of spawning a child process though, so we’d need to do one of these things:

1. Add a new runQueueAsNewProcess config setting. Only checked if runQueueAutomatically is set to true (is by default), and determines whether auto-run queues should happen via Ajax (stays the default behavior) or as a separate PHP process, similar to @ostark’s proof of concept.
2. Replace runQueueAutomatically with a new config setting – maybe called queueMode, with possible values:
   • "ajax": same as current behavior with runQueueAutomatically enabled (default value)
   • "process": same as proof of concept behavior
   • "manual" / null / false: same as current behavior with runQueueAutomatically disabled

[ostark]

Glad you like it @takobell + @brandonkelly 👍

I prefer the second config option queueMode (queueHandler makes even a bit more sense). It may requires a bit more refactoring, but is easier to understand more flexible.

In my example I used craft queue/run . This works, but my intention was to handle a single job. I've tried to run craft queue/exec but I was unsure about the 2nd parameter. Maybe using run is fine, this way all (delayed) jobs get executed at some point.

Instead of making the path to the php binary configurable, we could make use of Process\PhpExecutableFinder. The PHP_BINDIRconstant is set properly in a common PHP-FPM environment, at least at fortrabbit.com.

[angrybrad]

@ostark Looks like that Symfony component requires PHP 7.1, which we wouldn't be able to require until Yii 2.1. https://github.com/symfony/process/blob/master/composer.json

[ostark]

#Nah, only the master branch requires 7.1
https://github.com/symfony/process/blob/3.4/composer.json

You depend on it already (indirectly)
https://github.com/composer/composer/blob/master/composer.json#L33

[khalwat]

This would be fantastic

[joshangell]

I'm liking this idea a lot ...