Make possible to start Peon without DI loading of any querying-related stuff#4516
Conversation
| /** | ||
| * Using Provider, not {@link QueryRunnerFactoryConglomerate} directly, to not require {@link | ||
| * io.druid.indexing.overlord.TaskRunner} implementations that create TaskToolboxes to inject query stuff eagerly, | ||
| * because it may not be available, e. g. for batch tasks running in Spark or Hadooop. |
There was a problem hiding this comment.
minor: "Hadooop" -> "Hadoop"
|
@leventov does this really need to be in 0.10.1? |
|
@fjy if not |
|
Would you elaborate on how DI of querying-related stuffs blocks peons from running with Spark/Hadoop tasks? |
|
@jihoonson it's because we include our own extensions with configuration options for realtime peons, and they sneak into configuration for Spark peons. Those config options couldn't be instantiated on Spark peons, because we already exclude some other extensions/modules on them, I don't remember for what exact reason. |
|
@himanshug Did you want to review this PR? |
|
👍 looked at it today, i think its fine given that @leventov has a real use case for it. |
|
I'll create a backport |
…d stuff (apache#4516) * Make QueryRunnerFactoryConglomerate injection lazy in TaskToolbox/TaskToolboxFactory * Extract QueryablePeonModule and add druid.modules.excludeList config * Typo
6 participants
There are three steps:
QueryRunnerFactoryConglomeratelazily inTaskToolboxFactoryQueryablePeonModulewith the querying-related stuff loaded on Peon currentlydruid.extensions.moduleExcludeListconfig asdruid.modules.excludeListto be able to filter not only modules from extensions, but also "core" modules.Putting into
0.10.1becausedruid.extensions.moduleExcludeListwas added just in this version, to avoid compatibility problems with removing some configuration that was added just in the previous version. Also the lack of this feature may be blocking for other people who run Peons with Spark/Hadoop tasks.