Skip to content

enable lookup module for index_hadoop task#7304

Closed
FaxianZhao wants to merge 1 commit intoapache:masterfrom
FaxianZhao:enable_lookup_for_index_hadoop
Closed

enable lookup module for index_hadoop task#7304
FaxianZhao wants to merge 1 commit intoapache:masterfrom
FaxianZhao:enable_lookup_for_index_hadoop

Conversation

@FaxianZhao
Copy link
Copy Markdown
Contributor

Work around fix #5727.

There are some issues need to declare:

  1. Mapreduce job is out of GuiceRunnable's control, so the lifecycle is not avaliable. I work around solve it in Mapper.setup() function.
  2. Refer the issue 1, I think o.a.d.initialization.Initialization#makeInjectorWithModules is too expensive, it's better to split modules in different scenario. I don't have enough knowledge to do.
  3. Mapreduce job hasn't upload *.properties to MR classpath, and some lifecycle related modules need them.
  4. o.a.d.initialization.Initialization#makeInjectorWithModules always load extensions from filesystem, and MR job won't upload extensions to work dir. I work around fix it.
    I think we could declare the impl classes in the configuration. If all of them are loaded from classloader, then ignore the filesystem part.
  5. It seems there are some performance issues, I have no idea yet.

@FaxianZhao FaxianZhao closed this Apr 16, 2019
@FaxianZhao FaxianZhao reopened this Apr 16, 2019
@asdf2014
Copy link
Copy Markdown
Member

Hi, @FaxianZhao . The failure of travis has nothing to do with this PR, I will restart it. BTW, please resolve these conflicts.

@FaxianZhao FaxianZhao force-pushed the enable_lookup_for_index_hadoop branch from 4044078 to 773c655 Compare April 16, 2019 07:45
@FaxianZhao
Copy link
Copy Markdown
Contributor Author

FaxianZhao commented Apr 16, 2019

Hi, @asdf2014
I think current failure is caused by #7234 . I'm trying to understand the Lifecycle.Stage to fix this.

Hi, @clintropolis currently #7222 is merged in master branch. Do you have some suggestion for enable LookupExprMacro in hadoop mapreduce task.

By the way, there is a type in Lifecycle's javadoc. "- Stage.ANNOUNCENTS"

@clintropolis
Copy link
Copy Markdown
Member

Hi @FaxianZhao,
I'm unsure currently what problem you are running into and nothing obvious from the travis failure logs, but I can pull your branch and have a look as soon as I have the chance to try and get an idea of what is going on.

Also, thanks for spotting the typo, will fix 👍

@FaxianZhao
Copy link
Copy Markdown
Contributor Author

FaxianZhao commented Apr 19, 2019

Hi, @clintropolis , @asdf2014
Sorry to confuse you with previous comment.
Current failure is caused by org.apache.druid.curator.discovery.ServerDiscoverySelector cannot start with non-zookeeper environment(I trigger lifecycle in mapreduce job, and lifecycle will try to start it).
I'm wonder why the previous test was passed, it seems there is no zookeeper environment while org.apache.druid.indexer.IndexGeneratorJobTest running.

@FaxianZhao FaxianZhao closed this Apr 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lookup module not registered with overlord

3 participants