Error in coordinator/overlord when loading scan-query

[KenjiTakahashi]

I'm getting errors like this

2017-09-21T21:17:34.025034+00:00 monitowl-dev wh_druid_overlord[27859]: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:                                      
2017-09-21T21:17:34.025617+00:00 monitowl-dev wh_druid_overlord[27859]: 1) No implementation for io.druid.query.GenericQueryMetricsFactory was bound.                                
2017-09-21T21:17:34.026017+00:00 monitowl-dev wh_druid_overlord[27859]:   while locating io.druid.query.GenericQueryMetricsFactory                                                   
2017-09-21T21:17:34.026441+00:00 monitowl-dev wh_druid_overlord[27859]:     for the 1st parameter of io.druid.query.scan.ScanQueryQueryToolChest.<init>(ScanQueryQueryToolChest.java:48)                                          
2017-09-21T21:17:34.026811+00:00 monitowl-dev wh_druid_overlord[27859]:   at io.druid.query.scan.ScanQueryDruidModule.configure(ScanQueryDruidModule.java:36) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.query.scan.ScanQueryDruidModule)
2017-09-21T21:17:34.027141+00:00 monitowl-dev wh_druid_overlord[27859]: 2) No implementation for io.druid.query.GenericQueryMetricsFactory was bound.
2017-09-21T21:17:34.027372+00:00 monitowl-dev wh_druid_overlord[27859]:   while locating io.druid.query.GenericQueryMetricsFactory                                                   
2017-09-21T21:17:34.027621+00:00 monitowl-dev wh_druid_overlord[27859]:     for the 1st parameter of io.druid.query.scan.ScanQueryQueryToolChest.<init>(ScanQueryQueryToolChest.java:48)
2017-09-21T21:17:34.027988+00:00 monitowl-dev wh_druid_overlord[27859]:   while locating io.druid.query.scan.ScanQueryQueryToolChest
2017-09-21T21:17:34.028260+00:00 monitowl-dev wh_druid_overlord[27859]:     for the 1st parameter of io.druid.query.scan.ScanQueryRunnerFactory.<init>(ScanQueryRunnerFactory.java:52)
2017-09-21T21:17:34.028492+00:00 monitowl-dev wh_druid_overlord[27859]:   at io.druid.query.scan.ScanQueryDruidModule.configure(ScanQueryDruidModule.java:41) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.query.scan.ScanQueryDruidModule)
2017-09-21T21:17:34.028749+00:00 monitowl-dev wh_druid_overlord[27859]: 2 errors
2017-09-21T21:17:34.028975+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
2017-09-21T21:17:34.029211+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
2017-09-21T21:17:34.029438+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)            
2017-09-21T21:17:34.029665+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.Guice.createInjector(Guice.java:99)
2017-09-21T21:17:34.029944+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.Guice.createInjector(Guice.java:73)                                                 
2017-09-21T21:17:34.030185+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.Guice.createInjector(Guice.java:62)
2017-09-21T21:17:34.030399+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:390)               
2017-09-21T21:17:34.030678+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:63)
2017-09-21T21:17:34.030892+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.cli.ServerRunnable.run(ServerRunnable.java:40)                                               
2017-09-21T21:17:34.031104+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.cli.Main.main(Main.java:108)

while having scan-query extension in druid.extensions.loadList, which is in the _common/common.runtime.properties file.

I've confirmed that this does not affect historicals, nor brokers and that removing scan-query from extensions fixes the problem.

Is this expected? Should I be splitting the extensions definitions for different nodes now? What I'm worried about that our local tasks [or middle managers, or whatever actually serves incoming data :-)] (spawned by overlord) will not see it, then, which will be no good.

[KenjiTakahashi]

Oh, I forgot! This is on 0.10.1.

[stevenchen3]

Got the same here.

[almike]

yeah this is a major issue for us as well

[vogievetsky]

It appears that in Druid 0.10.1 (vs 0.10.0) non-query nodes (coordinator, overlord) can not load query extensions.
Previously in 0.10.0 they were simply ignored.
This is technically a regression I pray to the Druid gods that it will be fixed in 0.11.0
There is a very simple work-around for now: only load the scan query on the query nodes, broker, historical, etc.

[almike]

thanks @vogievetsky, we will try that out!

[KenjiTakahashi]

@vogievetsky Thanks, that's what I figured out as well. Still not sure about my question at the bottom of the first post, though. It is not clear to me what will happen in such scenario.

[leventov]

It appears that in Druid 0.10.1 (vs 0.10.0) non-query nodes (coordinator, overlord) can not load query extensions.
Previously in 0.10.0 they were simply ignored.
This is technically a regression I pray to the Druid gods that it will be fixed in 0.11.0
There is a very simple work-around for now: only load the scan query on the query nodes, broker, historical, etc.

This is caused by #4429 and IMO it's not a regression, it's a first step towards not loading everything everywhere. So not loading scan-query on coordinator and overlord is not a "workaround", it's "the right thing to do".

[gianm]

@leventov I don't agree, it's a real pain for users if they need to specify a different list of extensions for each node type. Especially since new users often don't fully understand what node types are doing what jobs. In the past, extensions could be loaded everywhere and they would just end up doing nothing if they don't apply to a particular node type.

I guess it does make sense to not actually load everything everywhere for efficiency sake, but for users' sanity, how about instead having extensions be able to declare what kinds of node types they should be loaded on.

[KenjiTakahashi]

I am fine with specifying different extensions for different nodes (although @gianm argument makes sense, too), I just need to know what will happen when a task spawned locally by overlord needs extension that overlord itself cannot load? My understanding is that currently, overlord's config is being passed down to local tasks.

BTW: The scan-query case should "fix itself" with the next release, because it got merged to core.

[gianm]

I just need to know what will happen when a task spawned locally by overlord needs extension that overlord itself cannot load?

@KenjiTakahashi, if the extension must be serialized as part of the task (for example a custom firehose) then the overlord must be able to load it. The scan query isn't needed to be serialized as part of the task, so, it doesn't need to be loaded on the overlord.

[KenjiTakahashi]

Yes, but how will a task know that it has to be loaded? Maybe I'm not being clear here, sorry. I mean we're using the index_realtime tasks spawned locally on overlord. There is no separate properties file for tasks/middlemanagers/<whatever is below overlord>. But still I should be able to query the incoming data, right? How would I use scan-query with that, if it's not specified in overlord config?

[leventov]

@KenjiTakahashi you can configure custom properties for forked task JVMs, but not overlord using druid.indexer.fork.property, see http://druid.io/docs/latest/configuration/indexing-service.html

[KenjiTakahashi]

So I can pass any property through this directly to task, bypassing overlord itself? That makes sense, gonna try it out, thanks.