[Proposal] define a RPC protocol for querying data, support apache Arrow as data exchange interface

[weijietong]

In our scenario , a query of one Druid segment matched 400million rows from 900 million rows took 79ms to query out the matched data ,but took 19s to serialize all the data into json format, also with a high cpu load.Though json has a better readability, it's unacceptable for large data query.

I also mentioned that some other ones expected a binary transfer protocol ,as http is not good at high speed data transfer.

I will try to define a RPC protocol to substitute the http protocol . Also substitute the json output data format with apache Arrow ,as the output data to be expressed as ValueVector ,it doesn't need to be serialized or deserialized. Most of other systems likeKudu, Spark,Drill are collaborating to support Arrow as system exchange data standard.

I think a RPC and Arrow combination should be a candidate or a new interactive standard for the broker ,history ,realtime nodes to exchange data.

[jihoonson]

@weijietong, thanks for your proposal.

A binary protocol might be helpful for some users, but the http protocol will be still useful for others who don't want such a high performance interface. For example, some users don't want it because their results are always small. In this case, they will prefer to use a simple http protocol rather than a complex binary protocol.

I'm also not sure about substituting JSON with Arrow. Apache Arrow is an in-memory columnar data format optimized for efficient data processing and exchanging between data analytics systems. This columnar format will be helpful for other analytics systems if they need to read data from Druid, but not for users because humans and well-known BI tools are familiar with row-oriented formats. In the later case, the columnar output format needs an additional unnecessary conversion to a row-oriented format.

So, IMO, it would be better to support a binary protocol and the Arrow format as one of Druid's interfaces and output formats, respectively, rather than replacing the original ones. Actually, I believe it would be great if Druid supports various output formats.

[b-slim]

@weijietong have you tried using the Smile format ?

[fjy]

@jihoonson agree. I think we've known for awhile that for larger result sets, serde time can be the bottleneck. It'd be nice if we had a binary format, especially given recent changes to support larger result sets, but we have to look more into Arrow.

[drcrallen]

Note that druid does support SMILE for most http endpoints.

[drcrallen]

Another large bottleneck is serialization for cache. (just FYI)

[himanshug]

I don't think yet that HTTP is a problem and RPC is necessary. With HTTP you can totally transfer binary data , connections are persistent and there is great tooling available to work with HTTP.
More interestingly with HTTP 2.0 , it is gonna be even better and would allow parallel requests on same connection.

I can understand the sentiment around Json though and if Smile is also not good enough then adding another format is OK.

[weijietong]

Except avoiding the serde bottleneck,Arrow stores data as off heap DirectByteBuffer. By leveraging NIO RPC , we can achieve sending data with zero copy memory performance , that is what Servelet does not support .

btw I want to know how to enable Smile binary json format at current Druid version.

I agree that http and json is still useful for most common case. Substituting seems rash.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.