Fix lz4 library incompatibility in kafka-indexing-service extension

[sascha-coenen]

The druid-kafka-indexing-service extension contains the following two libraries:

• kafka-clients 0.9.0.1
• lz4 1.3.0

The kafka-clients lib itself has a dependency on the lz4 lib version 1.2.0.
This doesn't lead to a version conflict as long as the Kafka topic consumed from contains message-batches that are either uncompressed or use a different compression scheme.

If the kafka topic contains messages that have been lz4 compressed by the kafka-producer, then the kafka-indexing-service fails with the following exception which is due to the version conflict:

java.lang.NoSuchMethodError: net.jpountz.util.Utils.checkRange([BII)V
    at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.read(KafkaLZ4BlockInputStream.java:177) 

Possible options for a bugfix might be

• in the kafka-indexing-service extension POM, downgrade lz4 library version
• in the kafka-indexing-service extension POM, upgrade kafka-clients library to 0.10.0.0 which is built against lz4 version 1.3.0
  I haven't checked if the kafka-client v 0.10 would be backwards compatible with with kafka server 0.9

[sascha-coenen]

The kafka documentation states implicitly that it is not recommended to use a kafka-client v0.10 with a kafka 0.9 cluster.

[gianm]

@sascha-coenen, thanks for the report.

We chose to use kafka-clients 0.9 because it was reported that newer clients don't work with older brokers (see https://groups.google.com/d/msg/confluent-platform/jdv10HahRd8/l-5twegkIAAJ), and we didn't want to require 0.10 brokers just yet.

If downgrading the lz4 version in the extension pom works (s.t. main Druid still uses 1.3.0, but the extension uses 1.2.0 in its own classloader) then that seems like an ideal solution.

Any chance you could check that and confirm if it works or not?

[drcrallen]

@gianm the extension has the "default" libraries take priority over extension libraries. In this case it wouldn't help because it should pull the class definition from the application classloader rather than the extension one. You'd have to replace replace the application's version to get the change to stick properly.

[dpoldrugo]

Is there a way to force druid to use kafka-clients-0.10.0.0 ?
For the new indexing service or for Tranquility.

[gianm]

@dpoldrugo recompiling the kafka-indexing-service extension should work.

[dpoldrugo]

Ok, thank you Gian.

[dclim]

@gianm @drcrallen I tested two options which both seem to work:

1. Upgrade Kafka client to 0.10 - downside is that users will have to upgrade their brokers as the 0.10 client is not compatible with 0.9 brokers. Upside? Bug fixes? Kafka Stream client support?

2. Force Maven to use lz4 1.2.0 for the extension AND introduce a mechanism to remove lz4 1.3.0 from the classpath passed to the peon. This could be done by running a regex on druid.indexer.runner.classpath in ForkingTaskRunnerConfig. We could a) either have it be done manually (add another property like druid.indexer.runner.classpathExclusions) or b) we could do it automatically if the task type is a Kafka task. Downside to a) is that it's another config that users who want lz4 would have to know to do, and b) it's a little black magicky but maybe acceptable.

Thoughts?

[drcrallen]

If we really want to fix it, I think we should have a better classloader isolation story at the peon level. See #2502 and #2476 for more related items.

I haven't looked at the kafka client in depth, but how difficult would it be to support multiple versions of the kafka client, either in the same extension or multiple extensions?

[dclim]

@drcrallen hmm same extension - probably non-trivial and would require additional classloading work; different extensions - easy but ugly if you just cloned the extension, harder if you want to have a single copy of the source and use Maven configs to build two versions (since the API is not exactly the same between 0.9 and 0.10). I don't really love the idea of maintaining two versions just to support lz4 compression.

My main consideration in whether to move to 0.10 is how big of a nuisance broker updates would be for users. It can be done as a no-downtime rolling update so it's mainly the inconvenience of it I suppose. Again, it kinda feels like too much to ask for no benefit except for lz4 working (the other compression formats gzip and snappy work fine).

[dclim]

Related to #3426.

@gianm @drcrallen any thoughts about whether we should move to the 0.10 Kafka client? It would make this a non-issue and would bring us up to the latest version of Kafka but would require a Kafka broker migration by users.

[drcrallen]

@dclim How about not supporting lz4 in anything less than 0.10, and moving to kafka 0.10 when druid also goes to 0.10?

[s-bernard]

@dclim The kafka migration process is described there: http://docs.confluent.io/3.0.0/upgrade.html
A rolling upgrade seems possible.