Avatica protobuf

[lkm]

Description

Druid hadn't implemented the AvaticaProtobufHandler and only supported Json serialisation for Avatica connections. Protocol buffers have much less overhead than Json and thus I would expect the transfer rate to be increased when using it. I'll try to build some benchmarking tools to test this, circumstantial evidence from test suites seems to indicate a speed increase:

... Time elapsed: 14.753 s - in org.apache.druid.sql.avatica.DruidAvaticaJsonHandlerTest
... Time elapsed: 11.482 s - in org.apache.druid.sql.avatica.DruidAvaticaProtobufHandlerTest

As a side note, for the officially supported Avatica Golang driver there isn't any Json serialisation support and thus will only work with this patch.

I was forced to add org.apache.calcite.avatica to druid-server in order to parse the pb messages and retrieve the connection id. The connection id is used to stabilise the choice of brokers when the request goes through the router.

One further note, I've chosen to add another url (/druid/v2/sql/avatica-pb/) for protocol buffers, ideally it would simply be controlled with a content-type, that is unfortunately not implemented in the JDBC driver so it will not work for the time being, however, the aforementioned golang driver does send content-type as application/x-google-protobuf.

Fixed the bug ...

Renamed the class ...

Added a forbidden-apis entry ...

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

---

[jihoonson]

Hi @lkm, thanks for the contribution. I left some comments around the query URL. In addition to them, would you please update doc here as well to explain how to use the new feature?

[jihoonson]

These if clause seems fragile from changes on Avatica request types (such as new request type), but I'm not sure if there is a better way..

[lkm]

Agreed, it's a bit fragile, however, that, I'm afraid, is the nature of protobufs. I'm not sure how likely they are to add new request types without significant overhaul to Calcite though

[jihoonson]

@lkm besides my comments above, would you please fix the conflicts?

[lkm]

Thanks for you time, @jihoonson. I've added the spotbugs exclude and documentation for it now.

[jihoonson]

The latest change LGTM. Thanks @lkm!

[clintropolis]

Sorry for the delayed review, overall LGTM 👍

We also have ITJdbcQueryTest, it would be useful I think to add coverage for the protobuf endpoint to the integration tests (and pretty easily) by adding additional connection strings, e.g. https://github.com/apache/druid/blob/master/integration-tests/src/test/java/org/apache/druid/tests/query/ITJdbcQueryTest.java#L56 but with the protobuf path and serialization=protobuf, which afaict should control the wire format: https://calcite.apache.org/avatica/docs/client_reference.html

[clintropolis]

side note, since we now have avatica stuff as dependencies to this project, it probably makes sense in a follow-up to modify the JSON version to use the types like this is doing instead of deserializing to a map.

[lkm]

I'm not sure how much there is to gain from that, what are you thinking would be the advantage? Also, looking through it I can't find an easy to use class like ProtobufTranslationImpl for json.

[clintropolis]

BTW, I guess the reason would be defensive for the most part, to more strongly type the check so that whenever we upgrade the library, stuff would fail at compile time instead of just fail in strange ways at run time if any of the requests ever change, however unlikely.

Some of the unit tests for the json path are using the avatica types for json to test this area already, so was just thinking since its used as a runtime depend now we could use the expected types for non tests too. But yeah, its definitely not necessary and doesn't provide a lot of gain, nor do I think we should do it in this PR or anything.

[clintropolis]

i wonder if this was causing some problem before... looking at other implementations some do check for null on this field, 👍

[clintropolis]

nit: I wonder if this test would be better using @RunWith(JUnitParamsRunner.class) with the different handler implementations instead of making it an abstract class

[lkm]

Would that not make the test harder to read? You mean parameterising each test case? I was going for a lighter touch approach, the abstracted methods are called in setUp, which would then need rewriting

[clintropolis]

very sorry for the delay @lkm, PR lgtm, could you fix up conflicts (again, sorry) and we can get this merged?

[lkm]

@clintropolis Sure. Pushed a bit too early there. Will try to get this fixed up before Friday

[lkm]

@clintropolis I apologise for dragging my feet on this one. I've properly rebased against most recent master now. So it should be good to go

[lkm]

@clintropolis will this be included in the upcoming 0.21 release? Should I provide a backport?

[clintropolis]

@clintropolis will this be included in the upcoming 0.21 release? Should I provide a backport?

0.21 branch is already created and we only backport bug and security fixes after the branch has been cut, so this will have to wait until the next release. Since 0.21 was created a few months ago and was delayed by 0.20.1 and 0.20.2 releases, it should be released very soon, but we will probably begin with the 0.22 release within the next month so hopefully it won't be too long.

[lkm]

Alright, I understand. Thanks for the quick response.