druid-hdfs-storage errors when loaded via maven coordinates

[kevinludwig]

Briefly:
If I set druid-hdfs-storage via maven coordinates it gets an exception (running standalone realtime server):

java.io.IOException: No FileSystem for scheme: hdfs

The basic issue is that the configuration property fs.hdfs.impl is not set.

I can get things working in either of two ways:

1. put the druid-hdfs-storage jar and all dependency jars in the class path for the realtime server
2. put an hdfs-site.xml into one of the jars that get loaded via maven coordinates (e.g. create a local repository for maven dependencies and then jar uf to update one of the jars in place to include the hdfs-site.xml file)

Details here:
https://groups.google.com/forum/#!topic/druid-user/PEj3WlQlG0M

[drcrallen]

Root cause thread context is not set to be picked up in org.apache.hadoop.fs.FileSystem#loadFileSystems. Therefore the classes are not loaded from the isolated loader. Patch incoming

[drcrallen]

#1716 is proposed by setting the thread context in the critical region where classes are loaded. This has the disadvantage where the FileSystem static methods may be called by another part of the program, thereby clobbering any other loadings. It also may or may not change how hdfs-site.xml or other settings files are read as classpath resources.

[drcrallen]

I think @himanshug 's HDFS patch https://issues.apache.org/jira/browse/HDFS-8750 is the better way to go if you don't want to include all the hadoop stuff on the system classpath.

[himanshug]

@kevinludwig is it possible for you to test the fix suggested in #1721