A couple of hdfs related fixes

[jasonxh]

• Class loading issue with hdfs-storage extension
  Even though the extension has hadoop-hdfs as a dependency, currently it's not able to load relevant classes during runtime because Hadoop is using thread context class loader by default.
• Exception when using hdfs with non-fully qualified segment path

@fjy can you take a look?

[drcrallen]

Hi @jasonxh thanks for the patch!

Do you happen to have more information on when setting the HdfsStorageDruidModule classloading is required? If you have some examples of failures that would be awesome.

[jasonxh]

This is the exception from path merging that happened during hadoop indexing task

Error: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: hdfs://hadoop:9000druid/segments/p13n_results
    at org.apache.hadoop.fs.Path.initialize(Path.java:206)
    at org.apache.hadoop.fs.Path.<init>(Path.java:116)
    at org.apache.hadoop.fs.Path.<init>(Path.java:89)
    at io.druid.indexer.JobHelper.prependFSIfNullScheme(JobHelper.java:521)
    at io.druid.indexer.JobHelper.makeSegmentOutputPath(JobHelper.java:420)
    at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:445)
    at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:272)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: hdfs://hadoop:9000druid/segments/p13n_results
    at java.net.URI.checkPath(URI.java:1804)
    at java.net.URI.<init>(URI.java:752)
    at org.apache.hadoop.fs.Path.initialize(Path.java:203)
    ... 14 more</code>

[jasonxh]

This is the exception where historical node failed to load segment from hdfs. I used to work around it by adding hadoop libs to class path, but was wondering why it should be necessary, considering that hdfs-storage-extension supposedly includes the dependency already.

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[p13n_results_2015-06-16T00:00:00.000Z_2015-06-17T00:00:00.000Z_2015-06-21T07:43:09.266Z]
        at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:146) ~[druid-services-0.8.0-rc2-SNAPSHOT-selfcontained.jar:0.8.0-rc2-SNAPSHOT]
        at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:171) [druid-services-0.8.0-rc2-SNAPSHOT-selfcontained.jar:0.8.0-rc2-SNAPSHOT]
        ...
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1882) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2298) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) ~[?:?]
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) ~[?:?]
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:360) ~[?:?]
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) ~[?:?]
        at io.druid.storage.hdfs.HdfsDataSegmentPuller.getSegmentFiles(HdfsDataSegmentPuller.java:177) ~[?:?]
        at io.druid.storage.hdfs.HdfsLoadSpec.loadSegment(HdfsLoadSpec.java:59) ~[?:?]
        at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-services-0.8.0-rc2-SNAPSHOT-selfcontained.jar:0.8.0-rc2-SNAPSHOT]
        at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-services-0.8.0-rc2-SNAPSHOT-selfcontained.jar:0.8.0-rc2-SNAPSHOT]
        at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:151) ~[druid-services-0.8.0-rc2-SNAPSHOT-selfcontained.jar:0.8.0-rc2-SNAPSHOT]
        at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:142) ~[druid-services-0.8.0-rc2-SNAPSHOT-selfcontained.jar:0.8.0-rc2-SNAPSHOT]

[nishantmonu51]

looks like it should fix the case mentioned in #713 ?

[himanshug]

we put the hadoop libs and extension jars explicitly on the classpath. please make sure that this change does not break other existing setups like those.

[jasonxh]

@himanshug the extension class loader uses the root loader as a fallback, so anything on your root CLASSPATH that doesn't exist in extension dependency hierarchy (e.g. your yarn configs) will still be discoverable. It impacts only the case where extension's hadoop dependency has a version (2.3.0) different than your hadoop version. Now that extension's own hadoop is being preferred over your hadoop, a version conflict is possible. However, it is recommended to recompile druid to match your hadoop version anyway.

[drcrallen]

@nishantmonu51 : We actually do that currently. That issue might be resolved.

[fjy]

@jasonxh do you mind filing out the CLA: http://druid.io/community/cla.html ?

[cheddar]

👍

[drcrallen]

Yeah, turns out this is needed for 0.8.0

[jasonxh]

@fjy I've filled out the CLA

[drcrallen]

@jasonxh Thanks!