index task failed with storage type as "hdfs"

[flowbehappy]

java.lang.Exception: java.lang.IllegalArgumentException: Pathname /druid/segment_data/datasource1/datasource1/2013-08-20T03:00:00.000Z_2013-08-20T04:
00:00.000Z/2015-02-12T10:21:15.522Z/0 from hdfs://master.hadoop:9000/druid/segment_data/datasource1/datasource1/2013-08-20T03:00:00.000Z_2013-08-20T0
4:00:00.000Z/2015-02-12T10:21:15.522Z/0 is not a valid DFS filename.
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.IllegalArgumentException: Pathname /druid/segment_data/datasource1/datasource1/2013-08-20T03:00:00.000Z_2013-08-20T04:00:00.000Z
/2015-02-12T10:21:15.522Z/0 from hdfs://master.hadoop:9000/druid/segment_data/datasource1/datasource1/2013-08-20T03:00:00.000Z_2013-08-20T04:00:00.00
0Z/2015-02-12T10:21:15.522Z/0 is not a valid DFS filename.
        at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:187)
        at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:101)
        at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:820)
        at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:816)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:816)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:809)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1815)
        at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.serializeOutIndex(IndexGeneratorJob.java:408)
        at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:388)
        at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:253)
        at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)                                                                 
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)                                                    
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)                                                              
        at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)                               
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)                                                   
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)                                                                  
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)                                           
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)                                           
        at java.lang.Thread.run(Thread.java:745)   

The fileSystem here https://github.com/druid-io/druid/blob/master/indexing-hadoop/src/main/java/io/druid/indexer/IndexGeneratorJob.java#L435 supposed to be a DistributedFileSystem but it is not.

Note that it only go wrong in indexing service's remote mode. In local mode it is fine.

[drcrallen]

IIRC there's supposed to be code to sanitize distributed filesystem paths. (: bad) https://issues.apache.org/jira/browse/HADOOP-3257

[fjy]

@flowbehappy Did you get a chance to try #1121?

[flowbehappy]

@fjy Will try it this week.

[flowbehappy]

@fjy

It comes another problem before I can get there T_T:

2015-03-06T16:59:38,371 [ERROR] [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_dsp_campaign_only_2015-03-06T08:46:46.180Z, type=index_hadoop, dataSource=dsp_campaign_only}]
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_67]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_67]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_67]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_67]
    at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:256) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) [druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) [druid-services-0.7.0-selfcontained.jar:0.7.0]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_67]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_67]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_67]
    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_67]
Caused by: com.metamx.common.ISE: Job[class io.druid.indexer.LegacyIndexGeneratorJob] failed!
    at io.druid.indexer.JobHelper.runJobs(JobHelper.java:153) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:304) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    ... 11 more

After modified those lines (HadoopIndexTask.java 248~254),

//    final Class<?> indexGeneratorMainClass = loader.loadClass(HadoopIndexGeneratorInnerProcessing.class.getName());
//    final Method indexGeneratorMainMethod = indexGeneratorMainClass.getMethod("runTask", String[].class);
//    String[] indexGeneratorArgs = new String[]{
//        toolbox.getObjectMapper().writeValueAsString(indexerSchema),
//        version
//    };
//    String segments = (String) indexGeneratorMainMethod.invoke(null, new Object[]{indexGeneratorArgs});

    String[] indexGeneratorArgs = new String[]{
        toolbox.getObjectMapper().writeValueAsString(indexerSchema),
        version
    };
    String segments = HadoopIndexGeneratorInnerProcessing.runTask(indexGeneratorArgs);

I can see the real cause, JvmMonitor casing to Monitor failed:

2015-03-06T18:34:57,573 [ERROR] [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Uncaught Throwable while running task[HadoopIndexTask{id=index_hadoop_dsp_campaign_only_2015-03-06T10:34:45.822Z, type=index_hadoop, dataSource=dsp_campaign_only}]
java.lang.ExceptionInInitializerError
    at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:285) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:260) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) [druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) [druid-services-0.7.0-selfcontained.jar:0.7.0]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_67]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_67]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_67]
    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_67]
Caused by: com.google.inject.CreationException: Guice creation errors:

1) Error in custom provider, java.lang.ClassCastException: com.metamx.metrics.JvmMonitor cannot be cast to com.metamx.metrics.Monitor
  at io.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:78)
  at io.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:78)
  while locating com.metamx.metrics.MonitorScheduler
  at io.druid.server.metrics.MetricsModule.configure(MetricsModule.java:63)
  while locating com.metamx.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:448) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.Guice.createInjector(Guice.java:96) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.Guice.createInjector(Guice.java:73) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.Guice.createInjector(Guice.java:62) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:369) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:91) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    ... 8 more
Caused by: java.lang.ClassCastException: com.metamx.metrics.JvmMonitor cannot be cast to com.metamx.metrics.Monitor
    at io.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:84) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_67]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_67]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_67]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_67]
    at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:105) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:49) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.Guice.createInjector(Guice.java:96) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.Guice.createInjector(Guice.java:73) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at com.google.inject.Guice.createInjector(Guice.java:62) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:369) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:91) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]
    ... 8 more

That's so weird as I can see several logs like 2015-03-06T16:56:44,486 [INFO ] [main] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.JvmMonitor@304e3a2] before this exception comes in the same task log.

[flowbehappy]

Oh, I'm sorry!
I'm using 0.7.0 stable, and thought that #1121 already in it by mistake. After merging it, it works perfectly! Thanks!