Register lookup module with overlord and middle manager.#7082
Closed
scrawfor wants to merge 3 commits intoapache:masterfrom
Closed
Register lookup module with overlord and middle manager.#7082scrawfor wants to merge 3 commits intoapache:masterfrom
scrawfor wants to merge 3 commits intoapache:masterfrom
Conversation
| new IndexingServiceFirehoseModule(), | ||
| new IndexingServiceTaskLogsModule() | ||
| new IndexingServiceTaskLogsModule(), | ||
| new LookupModule() |
Member
There was a problem hiding this comment.
#7222 introduces LookupSerdeModule which I think might be nicer behavior for middle manager and overlord than using LookupModule and configuring it not to load, but it will require this PR to wait until the other is merged.
Contributor
Contributor
Author
|
I will, but it will most likely be a few weeks until I am able to.
…On Fri, Apr 19, 2019, 8:59 PM Jonathan Wei ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In services/src/main/java/org/apache/druid/cli/CliMiddleManager.java
<#7082 (comment)>
:
> @@ -163,7 +164,8 @@ public WorkerNodeService getWorkerNodeService(WorkerConfig workerConfig)
}
},
new IndexingServiceFirehoseModule(),
- new IndexingServiceTaskLogsModule()
+ new IndexingServiceTaskLogsModule(),
+ new LookupModule()
@scrawfor <https://github.com/scrawfor> #7222
<#7222> was merged, can you
please rebase when you get a chance?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#7082 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABV4BOLQJQMHVW74R2PIDR3PRJTFZANCNFSM4GXXJY7Q>
.
|
Contributor
|
@scrawfor cool, thanks! |
Contributor
|
@scrawfor thanks for working on this. I'm untagging milestone since this issue is not necessarily a release blocker. Feel free to let me know if you think this should be. |
gianm
added a commit
to gianm/druid
that referenced
this pull request
May 24, 2019
This includes the router, overlord, middleManager, and coordinator. Does the following things: - Loads LookupSerdeModule on MM, overlord, and coordinator. - Adds LookupExprMacro to LookupSerdeModule, which allows these node types to understand that the 'lookup' function exists. - Adds a test to make sure that LookupSerdeModule works for virtual columns, filters, transforms, and dimension specs. This is implementing the technique discussed on these two issues: - apache#7724 (comment) - apache#7082 (comment)
fjy
pushed a commit
that referenced
this pull request
May 24, 2019
This includes the router, overlord, middleManager, and coordinator. Does the following things: - Loads LookupSerdeModule on MM, overlord, and coordinator. - Adds LookupExprMacro to LookupSerdeModule, which allows these node types to understand that the 'lookup' function exists. - Adds a test to make sure that LookupSerdeModule works for virtual columns, filters, transforms, and dimension specs. This is implementing the technique discussed on these two issues: - #7724 (comment) - #7082 (comment)
gianm
added a commit
to implydata/druid-public
that referenced
this pull request
May 24, 2019
This includes the router, overlord, middleManager, and coordinator. Does the following things: - Loads LookupSerdeModule on MM, overlord, and coordinator. - Adds LookupExprMacro to LookupSerdeModule, which allows these node types to understand that the 'lookup' function exists. - Adds a test to make sure that LookupSerdeModule works for virtual columns, filters, transforms, and dimension specs. This is implementing the technique discussed on these two issues: - apache#7724 (comment) - apache#7082 (comment)
gianm
added a commit
to gianm/druid
that referenced
this pull request
May 27, 2019
This includes the router, overlord, middleManager, and coordinator. Does the following things: - Loads LookupSerdeModule on MM, overlord, and coordinator. - Adds LookupExprMacro to LookupSerdeModule, which allows these node types to understand that the 'lookup' function exists. - Adds a test to make sure that LookupSerdeModule works for virtual columns, filters, transforms, and dimension specs. This is implementing the technique discussed on these two issues: - apache#7724 (comment) - apache#7082 (comment)
fjy
pushed a commit
that referenced
this pull request
May 27, 2019
This includes the router, overlord, middleManager, and coordinator. Does the following things: - Loads LookupSerdeModule on MM, overlord, and coordinator. - Adds LookupExprMacro to LookupSerdeModule, which allows these node types to understand that the 'lookup' function exists. - Adds a test to make sure that LookupSerdeModule works for virtual columns, filters, transforms, and dimension specs. This is implementing the technique discussed on these two issues: - #7724 (comment) - #7082 (comment)
jihoonson
pushed a commit
to implydata/druid-public
that referenced
this pull request
Jun 4, 2019
…apache#7767) This includes the router, overlord, middleManager, and coordinator. Does the following things: - Loads LookupSerdeModule on MM, overlord, and coordinator. - Adds LookupExprMacro to LookupSerdeModule, which allows these node types to understand that the 'lookup' function exists. - Adds a test to make sure that LookupSerdeModule works for virtual columns, filters, transforms, and dimension specs. This is implementing the technique discussed on these two issues: - apache#7724 (comment) - apache#7082 (comment)
Contributor
Author
|
@jihoonson Sorry for the lack of response, but that's fine. I haven't had much time these last few weeks, but I should now. I'll rebase and make the changes soon. |
Contributor
Author
|
Actually, it looks like this was resolved with #7752. I think this can be closed. |
jihoonson
pushed a commit
to implydata/druid-public
that referenced
this pull request
Jun 26, 2019
This includes the router, overlord, middleManager, and coordinator. Does the following things: - Loads LookupSerdeModule on MM, overlord, and coordinator. - Adds LookupExprMacro to LookupSerdeModule, which allows these node types to understand that the 'lookup' function exists. - Adds a test to make sure that LookupSerdeModule works for virtual columns, filters, transforms, and dimension specs. This is implementing the technique discussed on these two issues: - apache#7724 (comment) - apache#7082 (comment)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5727. Allows index documents with lookup functions.
One possible issue with this change is that with default settings, the middle manager attempts to load the lookup maps. With the low default memory settings, this very quickly causes out of memory errors. The following configuration items are necessary to prevent that from happening. I added this to the examples, but I feel like it might need a better fix.