Hadoop index task stopped working with 0.10.1-rc1

[erikdubbelboer]

With the previous version of Druid everything worked fine. But after the upgrade to 0.10.1-rc1 we are getting:

2017-07-12T10:14:01,315 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Task Id : attempt_1499852511486_0003_m_000000_0, Status : FAILED
Error: com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) null returned by binding at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:132) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.storage.google.GoogleStorageDruidModule)
 but the 2nd parameter of io.druid.storage.google.GoogleDataSegmentPusher.<init>(GoogleDataSegmentPusher.java:56) is not @Nullable
  while locating io.druid.storage.google.GoogleAccountConfig
    for the 2nd parameter of io.druid.storage.google.GoogleDataSegmentPusher.<init>(GoogleDataSegmentPusher.java:56)
  while locating io.druid.storage.google.GoogleDataSegmentPusher
  at io.druid.storage.google.GoogleStorageDruidModule.configure(GoogleStorageDruidModule.java:94) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.storage.google.GoogleStorageDruidModule)
  while locating io.druid.segment.loading.DataSegmentPusher annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=47, type=MAPBINDER, keyType=java.lang.String)
  at io.druid.guice.PolyBind.createChoice(PolyBind.java:67) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.LocalDataStorageDruidModule)
  while locating io.druid.segment.loading.DataSegmentPusher

That seems to be that somehow the druid.google.* aren't propagated to the hadoop task like they were before?

[erikdubbelboer]

Anyone have any idea what might cause this? I can try all kinds of things to fix it but I'm kinda lost right now.

[b-slim]

@erikdubbelboer after #4116 user can now propagate selected properties to hadoop workers.
Not sure how this was working for you before ?

[erikdubbelboer]

@b-slim It seems like this is broken. When I settuningConfig.allowedHadoopPrefix to [ "druid.storage.", "druid.javascript.", "druid.google." ] I still get:

...
2017-07-13T08:07:10,806 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: {
...
      "allowedHadoopPrefix" : [ "druid.storage.", "druid.javascript.", "druid.google." ]
...
2017-07-13T08:07:16,387 INFO [task-runner-0-priority-0] io.druid.indexer.HadoopDruidIndexerConfig - Running with config:
...
      "allowedHadoopPrefix" : [ "druid.storage.", "druid.javascript." ]
...

So it doesn't seem to propagate to the actual job.

[erikdubbelboer]

@b-slim Is it because these methods pass null instead of the actual value?: https://github.com/druid-io/druid/blob/3399d1a488896d675e54c1de90a596daea1c045f/indexing-hadoop/src/main/java/io/druid/indexer/HadoopTuningConfig.java#L272-L322

[erikdubbelboer]

This has fixed the issue for me:

diff --git a/indexing-hadoop/src/main/java/io/druid/indexer/HadoopTuningConfig.java b/indexing-hadoop/src/main/java/io/druid/indexer/HadoopTuningConfig.java
index e64c0e7..b131a5e 100644
--- a/indexing-hadoop/src/main/java/io/druid/indexer/HadoopTuningConfig.java
+++ b/indexing-hadoop/src/main/java/io/druid/indexer/HadoopTuningConfig.java
@@ -267,7 +267,7 @@ public class HadoopTuningConfig implements TuningConfig
         numBackgroundPersistThreads,
         forceExtendableShardSpecs,
         useExplicitVersion,
-        null
+        allowedHadoopPrefix
     );
   }

@@ -292,7 +292,7 @@ public class HadoopTuningConfig implements TuningConfig
         numBackgroundPersistThreads,
         forceExtendableShardSpecs,
         useExplicitVersion,
-        null
+        allowedHadoopPrefix
     );
   }

@@ -317,7 +317,7 @@ public class HadoopTuningConfig implements TuningConfig
         numBackgroundPersistThreads,
         forceExtendableShardSpecs,
         useExplicitVersion,
-        null
+        allowedHadoopPrefix
     );
   }

[erikdubbelboer]

You want me to make a pull request?

[b-slim]

yeah sure will review that, @erikdubbelboer thanks.

[gianm]

How were the properties passed down before? I thought that pre 0.10.1, properties were generally not passed down to the hadoop cluster. So it seems weird that now they'd need to be.

Or is the difference that now in 0.10.1, the druid.storage.type property is passed down, which means now druid.google. needs to be as well if you're using the google extension? If that's the case, I think we should either add druid.google. to the default list, or give the deep storage extensions some way to say what properties they need passed down; otherwise upgrading will be too much of a pain for people using the extension.

[b-slim]

@gianm i think you are right, now that we push down the storage properties, guice tries to bind things up, hence the issues. Am not sure what is the best fix but seems like we need to push all the properties of the selected storage module at least. Any way i will send a first PR to fix the null issue (guess @erikdubbelboer is in a different timezone) and then will focus on how to solve this. @gianm should this be part of 0.10.1 ?

[gianm]

@b-slim I think we should at least have something in the release notes explaining the situation. Pushing down the storage properties would be even better. Perhaps we could do this by adding a getPropertyPrefixForHadoop to DataSegmentPusher and then automatically adding that prefix to allowedHadoopPrefix for whatever the deep storage is.

[jon-wei]

@b-slim
Are you planning on working on a fix beyond #4541 for 0.10.1?

If so, let's keep this is 0.10.1 and we can bring the fix into rc2.

If not, let's move this issue to 0.11.0, and I'll mention the storage extension property push in the 0.10.1 release notes.

[b-slim]

i don't want to block the RC for this issue thought, i will get to work on it maybe next week.

[erikdubbelboer]

I'm trying to post at the druid-development list but my messages keep on being automatically deleted (even without any links).

I don't think this only affects the Google Storage extension. I think it affects all storage extensions as all of them use different prefixes for their properties and all of them inject their config into their SegmentPushers?