indexing task not work properly with Hadoop HA

[aubdiy]

we use druid 0.12.0 version and hadoop 2.7.3 version

While using HDFS with Hadoop HA storage segments,

# conf-quickstart/druid/_common/common.runtime.properties
# For HDFS:
druid.storage.type=hdfs
druid.storage.storageDirectory=hdfs://jinzige/user/druid/segments

hdfs://jinzige is hadoop HA fs.defaultFS value,

core-site.xml:

<configuration>

    <property>
      <name>fs.defaultFS</name>
      <value>hdfs://jinzige</value>
      <final>true</final>
    </property>

    <property>
      <name>fs.trash.interval</name>
      <value>360</value>
    </property>
    

     ...

exception could be thrown while fetching or pushing task log:

2018-03-29T12:13:27,540 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Running job: job_1522308188942_0019
2018-03-29T12:13:32,649 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Job job_1522308188942_0019 running in uber mode : false
2018-03-29T12:13:32,649 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job -  map 0% reduce 0%
2018-03-29T12:13:39,369 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Task Id : attempt_1522308188942_0019_m_000000_0, Status : FAILED
Error: com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.IllegalArgumentException: java.net.UnknownHostException: jinzige
  at io.druid.storage.hdfs.HdfsDataSegmentPusher.<init>(HdfsDataSegmentPusher.java:65)
  while locating io.druid.storage.hdfs.HdfsDataSegmentPusher
  at io.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:97) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.storage.hdfs.HdfsStorageDruidModule)
  while locating io.druid.segment.loading.DataSegmentPusher annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=77, type=MAPBINDER, keyType=java.lang.String)
  at io.druid.guice.PolyBind.createChoice(PolyBind.java:70) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.LocalDataStorageDruidModule)
  while locating io.druid.segment.loading.DataSegmentPusher

1 error
	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028)
	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054)
	at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:126)
	at io.druid.indexer.HadoopDruidIndexerMapper.setup(HadoopDruidIndexerMapper.java:51)
	at io.druid.indexer.IndexGeneratorJob$IndexGeneratorMapper.setup(IndexGeneratorJob.java:276)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: jinzige
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:439)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:321)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:690)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:631)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829)
	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2817)
	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:437)
	at io.druid.storage.hdfs.HdfsDataSegmentPusher.<init>(HdfsDataSegmentPusher.java:70)
	at io.druid.storage.hdfs.HdfsDataSegmentPusher$$FastClassByGuice$$36c2ed09.newInstance(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
	at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104)
	at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104)
	at com.google.inject.multibindings.MapBinder$RealMapBinder$ValueProvider.get(MapBinder.java:821)
	at io.druid.guice.PolyBind$ConfiggedProvider.get(PolyBind.java:204)
	at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
	at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
	at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
	at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
	... 12 more
Caused by: java.net.UnknownHostException: jinzige
	... 49 more

[josephglanville]

It looks like it can't resolve jinzige, check that DNS is working correctly and search paths etc are properly configured.

[aubdiy]

@josephglanville thank you for your time， when we use it without hadoop HA, druid can work
when we use it wiht hadoop HA, druid throws that exception, hdfs://jinzige is hadoop HA fs.defaultFS value, in hdfs-sit.xml content

  
    <property>
      <name>dfs.ha.namenodes.jinzige</name>
      <value>nn1,nn2</value>
    </property>

    ...
 <property>
      <name>dfs.namenode.http-address.jinzige.nn1</name>
      <value>amb-117:50070</value>
    </property>
    
    <property>
      <name>dfs.namenode.http-address.jinzige.nn2</name>
      <value>amb-118:50070</value>
    </property>
    
    <property>
      <name>dfs.namenode.https-address.jinzige.nn1</name>
      <value>amb-117:50470</value>
    </property>
    
    <property>
      <name>dfs.namenode.https-address.jinzige.nn2</name>
      <value>amb-118:50470</value>
</property>

with this configs, druid 0.10 can work , I'm puzzled

[gianm]

Hi @aubdiy,

If you run the Hadoop CLI (or some other Hadoop HDFS client) using the same classpath as Druid, does that work?

Have you tried including dfs.nameservices in hdfs-site.xml too?

<property>
 <name>dfs.nameservices</name>
 <value>jinzige</value>
</property>

[jon-wei]

I was able to get a batch ingestion task working with namenode HA with the following .xml configs on the Druid side, testing against a sequenceiq hadoop 2.7.3 docker container (I faked out one of the name nodes and also didn't really set up the shared edits/fencing in a real way):

hdfs-site.xml

<property>
 <name>dfs.nameservices</name>
 <value>mycluster</value>
 <description>Logical name for this new nameservice</description>
</property>

<property>
 <name>dfs.ha.namenodes.mycluster</name>
 <value>nn1,nn2</value>
 <description>Unique identifiers for each NameNode in the nameservice</description>
</property>

<property>
 <name>dfs.namenode.rpc-address.mycluster.nn1</name>
 <value>hadoop273:8020</value>
</property>

<property>
 <name>dfs.namenode.http-address.mycluster.nn1</name>
 <value>hadoop273:50070</value>
</property>

<property>
 <name>dfs.namenode.rpc-address.mycluster.nn2</name>
 <value>xfake2:8020</value>
</property>

<property>
 <name>dfs.namenode.http-address.mycluster.nn2</name>
 <value>xfake2:50070</value>
</property>

<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>file:///tmp</value>
</property>

<property>
 <name>dfs.client.failover.proxy.provider.mycluster</name>
 <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

<property>
 <name>dfs.ha.fencing.methods</name>
 <value>shell(/bin/true)</value>
</property>

core-site.xml:

      <property>
          <name>fs.defaultFS</name>
            <value>hdfs://mycluster</value>
      </property>

@aubdiy Does the ingestion task work for you if you have the equivalent properties set in the hadoop .xml files on the Druid classpath?

[jon-wei]

@b-slim @nishantmonu51

Have you been able to successfully run batch ingestion with namenode HA on Druid 0.12.0 with HDP?

We've seen a customer hit this exception with Druid 0.12.0+ HDP, from investigating it seemed like the job.xml configuration was not picked up by the Druid mapper tasks (The job.xml in their mapper's local temp dir had the right configs, but those configs were not present in the Configuration object when that UnknownHostException is thrown in HdfsDataSegmentPusher)

It'd be interesting to know if this is a general Druid + Hadoop issue or one specific to Druid + HDP, or if everything is fine and this issue is caused purely by configuration problems.

[b-slim]

@jon-wei am sure we have positive runs Druid 12 + HDP 3.0 to be released soon (Hadoop 3) we do not have tests for Druid 12 with early version of Hadoop since we don't support such bundle (Our last release was based of Druid 10+some patches)

[jon-wei]

@b-slim got it, thanks

[aubdiy]

@jon-wei yes, i put the hadoop .xml files on the Druid classpath,
maybe I get the exception ，Because we use druid 0.12.0 version and Apache Ambari(HDP) Version2.6.1.5 (hadoop 2.7.3)，

thank for your help

[hellobabygogo]

@jon-wei yes, i put the hadoop .xml files on the Druid classpath,
maybe I get the exception ，Because we use druid 0.12.0 version and Apache Ambari(HDP) Version2.6.1.5 (hadoop 2.7.3)，

thank for your help

how did you fix this bug?