BACKEND: ClusterMQ as a new backend

[alexvorobiev]

I have recently discovered ClusterMQ which can run R code in SLURM/LSF/etc. jobs. The biggest advantage over batchtools is it uses ZMQ to transfer data directly to the distributed jobs. In my experience the most serious bottleneck in batchtools is using shared file system (NFS) for data transfer - especially if the data is large.

[HenrikBengtsson]

Yes, @mschubert's ClusterMQ is a great candidate for a future backend. I don't have the resources myself right now to work also on that. Having said that, and without having worked with ClusterMQ myself, I don't think it should be too much work to wrap it all up in a ClusterMQFuture - a future backend is mostly a thin layer on top of an existing API.

Related: I'm working on setting up a conformation test suite (e.g. future.tests pkg) that can be used by all future backend pkgs to make sure they got it correct. That is my number one priority before working on new backends.

[mschubert]

I fully support this, but unfortunately my time is also quite limited these days.

[wlandau]

Given that clustermq::Q() is synchronous, I am wondering what it would take to make an asynchronous ClusterMQFuture. Do we need local background processes to collect the results?

[wlandau]

Will future.clustermq somehow allow for heterogeneous transient workers? Some drake users such as @jennysjaarda prefer transient future-based workers over persistent clustermq-based workers, e.g. ropensci/drake#1083 (comment), but there is still the snag that batchtools is slower than clustermq.