Skip to content

Allow task to override ForkingTaskRunner tunings and jvm settings#1604

Merged
himanshug merged 1 commit intoapache:masterfrom
metamx:task-jvm-config
Sep 3, 2015
Merged

Allow task to override ForkingTaskRunner tunings and jvm settings#1604
himanshug merged 1 commit intoapache:masterfrom
metamx:task-jvm-config

Conversation

@nishantmonu51
Copy link
Copy Markdown
Member

Initial PR for discussion.

  1. Adds a context to tasks which can be used to specify environment configs for tasks (similar to query context)
  2. Allow tasks to override fork properties and jvm memory settings.

TODO:

  1. Add serde tests for task context
  2. UTs to verify forkingTaskRunner overriding properties specified in taskContext.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if we are gaining anything by using generics in this case?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree there. Can this just be Object?

or alternatively, have some other way of doing better type enforcing?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used the same semantics as used by druid queries.
I like this way so that user of this method need not worry about casting the returned value.
I can change it to Object as return type if you feel strongly about it ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding new comments to modified code

@himanshug
Copy link
Copy Markdown
Contributor

looks ok to me overall on first glance

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per prior comment. Suggest simply making Object and make the caller handle proper type checking.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changing to Object as return type.

@nishantmonu51
Copy link
Copy Markdown
Member Author

@himanshug @drcrallen handled the review comments.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a unit test for this case:

Have an interface which is expected from the context. But specify an implementation of that interface as the default value.

Does the casting and type detection work correctly? More specifically, in such a case is <ContextType> determined from the return or the parameter?

It may need something like public <ContextType, ContextTypeSub extends ContextType> ContextType getContextValue(String key, ContextTypeSub defaultValue)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added test.
the type detection is done on return type.

@drcrallen
Copy link
Copy Markdown
Contributor

Cool, I'm 👍 after travis

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java is funny with generics.
Can you try context = ImmutableMap.<String, Object>of("k1", 10)
then try getting getContextValue("k1","hello")

for me it failed and I believe to make it really work you have to do what @drcrallen suggested. However, at the same time, I don't know if so much complication is really needed here. Is getContextValue(key,default) really essential or we should just remove it.
For me getContext() and getContextValue(key) would already be enough.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed the method, it was not used anywhere anyways.

override javaOpts

fix compilation

review comments

Add Test for typecast

review comments - remove unused method.
@himanshug
Copy link
Copy Markdown
Contributor

LGTM (after Travis)

@nishantmonu51
Copy link
Copy Markdown
Member Author

bouncing for travis

@nishantmonu51 nishantmonu51 reopened this Sep 3, 2015
himanshug added a commit that referenced this pull request Sep 3, 2015
Allow task to override ForkingTaskRunner tunings and jvm settings
@himanshug himanshug merged commit 66a9a33 into apache:master Sep 3, 2015
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nishantmonu51 I think this will throw NPE if context is not provided.

@nishantmonu51
Copy link
Copy Markdown
Member Author

@guobingkun thanks for the catch, created #1703.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants