BUG: publishing segments with kafka index service failed in druid 0.9.2

[hamlet-lee]

2016-12-17T22:46:33,776 WARN [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver - Failed publishAll (try 4), retrying in 14,822ms.
java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.FileSystem.rename(Lorg/apache/hadoop/fs/Path;Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/fs/Options$Rename;)V from class org.apache.hadoop.fs.HadoopFsWrapper
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
	at io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver.publishAll(FiniteAppenderatorDriver.java:417) [druid-server-0.9.2.jar:0.9.2]
	at io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver.finish(FiniteAppenderatorDriver.java:256) [druid-server-0.9.2.jar:0.9.2]
	at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:522) [druid-kafka-indexing-service-0.9.2.jar:0.9.2]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.FileSystem.rename(Lorg/apache/hadoop/fs/Path;Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/fs/Options$Rename;)V from class org.apache.hadoop.fs.HadoopFsWrapper
	at org.apache.hadoop.fs.HadoopFsWrapper.rename(HadoopFsWrapper.java:51) ~[?:?]
	at io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:119) ~[?:?]
	at io.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:571) ~[druid-server-0.9.2.jar:0.9.2]
	at io.druid.segment.realtime.appenderator.AppenderatorImpl.access$600(AppenderatorImpl.java:93) ~[druid-server-0.9.2.jar:0.9.2]
	at io.druid.segment.realtime.appenderator.AppenderatorImpl$3.apply(AppenderatorImpl.java:467) ~[druid-server-0.9.2.jar:0.9.2]
	at io.druid.segment.realtime.appenderator.AppenderatorImpl$3.apply(AppenderatorImpl.java:455) ~[druid-server-0.9.2.jar:0.9.2]
	at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
	... 3 more

this seems related to #3650.

[hamlet-lee]

in peon's log, I found my task used

2.7.1 org.apache.hadoop.fs.FileSystem class:

@Deprecated
  protected void rename(final Path src, final Path dst,
      final Rename... options) throws IOException {

but druid 0.9.2 it built with version hadoop 2.3.0, which has

@Deprecated
  protected void rename(Path src, Path dst, Options.Rename... options)
    throws IOException
  {

may be the "final" modifier caused the issue.

[gianm]

Do you have any idea why the peon would be using hadoop-hdfs 2.7.1? (Did you configure it specially or did you replace any jars?)

Druid's hdfs storage extension is built against hadoop 2.3.0 and I wouldn't necessarily expect it to work with other versions, since it uses some protected methods.

[gianm]

Btw, the idea is that even if your Hadoop cluster is a different version, it should be fine to use hadoop-hdfs client version 2.3.0, since the wire format hasn't changed and we're just talking to HDFS over the network.

[hamlet-lee]

When I start middleManager, I added hadoop classpath into -classpath jvm arg, This way seems to work in previous version of druid.

If not in this way, how should we specify hadoop classpath for middleManager?

[hamlet-lee]

Since we run on hadoop 2.7.1/2.7.2, and sometimes we may use some job properties which only work in newer hadoop. So I choosed to use 2.7.1 hadoop in classpath.

I worked around this problem by invoking rename by reflection. #3787

Its a temporary way, and I am not sure, if this way suitable to be merged into druid/master.