Improve performance of transfering results from historical node to broker

[zhangxinyu1]

When we use groupby query, we find historical node only uses one cpu (almost 100% used) for a long time. This is because one thread of historical is sorting and compressing data.
So I'm thinking of that why not use two threads to do sorting and compressing respectively.
Besides, the default compressionCode is gzip, but gzip is slower than lz4 or some other compressionCodec, why not use a faster compressionCodec instead of gzip.

[gianm]

@zhangxinyu1 I think the simple answer is that it's because gzip is supported by jetty. You could try setting druid.broker.http.compressionCodec=identity and see if that helps your performance. Or, a patch to add lz4 support would be welcome (although I'm not sure what the best way is to do that)

[zhangxinyu1]

@gianm Yes, jetty only support gzip and identity. If we set druid.broker.http.compressionCodec=identity, the groupBy query will much faster, but the bandwidth will become the bottleneck. I have an idea as follows:

1. Set druid.broker.http.compressionCodec=identity as default
2. Compress result data with lz4 or other compressionCodec before historical return it.
3. Decompress result data after broker receive it.

[zhangxinyu1]

@gianm In DruidDefaultSerializersModule.java, JsonSerializer do sorting, serializing and compressing in one thread. That causes what I see (In historical node, only one cpu is used for a long time when I send a groupby query.)
Is it possible to parallelize these three operations?

[gianm]

@zhangxinyu1 it's possible, although I haven't done much thinking about how to do it. #4704 may be related. I haven't reviewed that patch yet to see how related it is, but I plan to review it soon.

One thing that comes to mind is it will be important to make sure that backpressure exists. We don't want a laggy top layer to lead to excessive memory use from queued up objects from deeper layers. This happens "automatically" when using a single thread but would need to be explicitly assured if we do multithreading.

[jihoonson]

Sounds good. I think it would be nice if users can choose their compressionCodec to use.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.