I propose AWS Batch as a new clustermq scheduler. Batch has become extremely popular, especially as traditional HPC is waning. I have a strong personal interest in making Batch integrate nicely with R (ref: https://github.com/wlandau/targets/issues/152, https://github.com/wlandau/tarchetypes/issues/8, https://wlandau.github.io/targets-manual/cloud.html).
Batch is super easy to set up through the AWS web console, and I think it would fit nicely into future's ecosystem: maybe with something like future::plan(future.aws.batch::future_aws_batch, template = "batch.tmpl"), where batch.tmpl contains an AWS API call with the compute environment, job queue, job definition, and key pair. I think we could use curl directly instead of the much larger and rapidly developing paws package. The tricky part is how we retrieve the data back from an AWS Batch job. I'm not sure how to do that yet.
I propose AWS Batch as a new
clustermqscheduler. Batch has become extremely popular, especially as traditional HPC is waning. I have a strong personal interest in making Batch integrate nicely with R (ref: https://github.com/wlandau/targets/issues/152, https://github.com/wlandau/tarchetypes/issues/8, https://wlandau.github.io/targets-manual/cloud.html).Batch is super easy to set up through the AWS web console, and I think it would fit nicely into
future's ecosystem: maybe with something likefuture::plan(future.aws.batch::future_aws_batch, template = "batch.tmpl"), wherebatch.tmplcontains an AWS API call with the compute environment, job queue, job definition, and key pair. I think we could usecurldirectly instead of the much larger and rapidly developingpawspackage. The tricky part is how we retrieve the data back from an AWS Batch job. I'm not sure how to do that yet.