[PROPOSAL] More documentation on ec2 autoscaler

[drcrallen]

Description

The EC2 autoscaler is poorly documented in what it can and cannot do (Galaxy dependency!?). This ask is that better documentation be included.

Motivation

See #6953 for some details

[sascha-coenen]

more documentation would in deed be useful as the autoscaler is quite nice. Especially the different strategies are difficult for me to wrap my mind around.

By the way, I believe the off-the-shelf autoscaler is probably not for everybody because it is quite specific. I wrote an HTTP autoscaler extension if this is of any use. It basically delegates all autoscaling decisions to an external webservice.
For folks that are not 'brave' enough to fork a big code repo and put their own customization in, it might be a less daunting proposition to just implement a webservice in a technology of their own choosing.

We have currently only one webservice implementation that would spin up EMR clusters with middlemanagers deployed on the EMR master nodes. One could include a minimalistic autoscaler example within the Druid codebase - a single-file python script which would perhaps even do nothing else but to print out the scaling decisions relayed by Druid, might be very helpful for people to play around with and get a feeling for how the druid-autoscaler mechanism behaves. They might then feel empowered to extend this webservice to their own needs.
But I'm quite off topic now. Sorry. Just meant to mention it because the autoscaling feature in Druid is quite nice but I don't have the feeling that many people are using it which might be due to its specific nature (Galaxy and such)