Increased druid.worker.capacity in bin/start-druid script

[gianm]

Follow up to #13365.

People often want to evaluate Druid by installing it on a single server, ingesting a bunch of data, and then doing queries. In the common case, ingestion and querying are not happening simultaneously, unless of course it's realtime ingestion. But let's focus on batch for now.

There's a problem here: our out-of-box single-server configs, and the bin/start-druid script as well, are designed such that MiddleManager/Indexer occupies a relatively small portion of the resources of the server. On many common server sizes, druid.worker.capacity is set smaller than the number of processors on the machine, meaning we aren't able to use the full resources of the hardware to do ingestion. (Most ingestion tasks are single-threaded; parallelism comes from launching more tasks.)

I suggest we solve this using a combination of three approaches:

1. Use 512MB heap per task (or even 256MB) rather than 1GB when CPU-to-memory ratio is higher, so we can get more tasks and therefore use more processors. This would work for both MM and Indexer.
2. Allow TASKS to use some memory that we had set aside for segment page cache. Currently, that's half the memory of the server, as long as it's above a minimum threshold. The beauty of TASKS is that they only use memory if they are actually launched. So, it's OK to eat into that 50% buffer; we'll only actually use it when all task slots are actually full. Then, when they're not, it all becomes available for page cache again. (This doesn't apply to INDEXER, where the memory is always used. Can't pull the same trick here.)
3. Cap druid.worker.capacity at the number of processors; doesn't help much to have it be higher.

The goal should be that in most cases, for a single server setup, druid.worker.capacity ends up getting set to roughly the number of processors.

[findingrish]

Use 512MB heap per task (or even 256MB) rather than 1GB when CPU-to-memory ratio is higher, so we can get more tasks and therefore use more processors. This would work for both MM and Indexer.

Task memory is determined using the total memory allocated for tasks. It could be 512m/1024m/2048m as of now.
Ingestion of trips_60m requires min 1g task memory. So we allocate 512m for tasks only when tasks memory is very less (< 2g)

Allow TASKS to use some memory that we had set aside for segment page cache. Currently, that's half the memory of the server, as long as it's above a minimum threshold. The beauty of TASKS is that they only use memory if they are actually launched. So, it's OK to eat into that 50% buffer; we'll only actually use it when all task slots are actually full. Then, when they're not, it all becomes available for page cache again. (This doesn't apply to INDEXER, where the memory is always used. Can't pull the same trick here.)

We are using 80% of total memory for the services (up from 50% #13563).

Cap druid.worker.capacity at the number of processors; doesn't help much to have it be higher.

This is already done https://github.com/apache/druid/blob/master/examples/bin/start-druid-main.py#L467 .

[findingrish]

This pr #13581 further refines the logic to compute task memory and count.

mem=1GB-> 2 tasks of 512MB each
mem=1GB to 2GB -> 2 tasks of (mem/2) each
mem=2GB -> 2 tasks of 1GB each
mem=2GB to num_cpu * 1GB -> n tasks of (mem/n) each such that each task gets atleast 1GB
mem > num_cpu * 1GB -> num_cpu tasks of (mem/num_cpu) each with a max of 2GB

This is how task memory and count varies with increasing total memory.

Total memory for all services -> [task_count, task_memory]
6g -> [2, 512m]
12g -> [2, 766m]
16g -> [2, 1024m]
32g -> [4, 1024m]
64g -> [8, 1024m]
80g -> [10, 1040m]
100g -> [10, 1300m]
150g -> [10, 1950m]
200g -> [10, 2048m]

[kfaraz]

Thanks for the sugggestions, @gianm and for the fix, @findingrish !
I am closing this ticket as all 3 items have now been implemented.