[Proposal] support for setting javaOpts per a task

[jaehc]

Currently, there is an setting druid.indexer.runner.javaOpts for a overlord node to specify java options for child processes it forks. However, this option has the effect on every tasks it makes.
I think it would be useful to support setting javaOpts for a specific task like as that of MapReduce or YARN. Then we can control task's jvm more finely. And we can also attach a remote debugger to a specific task to give an example of its usecase.
I guess tunningConfig is a good place to put that option under if possible.

"tuningConfig": {
      "type": "index",
      "targetPartitionSize": 0,
      "rowFlushBoundary": 0,
      "javaOpts": "-server -Xmx256m"
    }

[nishantmonu51]

this is already supported by passing in druid.indexer.runner.javaOpts in task context.

{
"type" : "index"
"tuningConfig" : {...}
"context" : {
"druid.indexer.runner.javaOpts" : "-server -Xmx256m"
}
}
Its still undocumented I think, its probably worth adding docs for it if the community finds it useful.

[jaehc]

@nishantmonu51 Thanks for your information.

[drcrallen]

What exactly are you trying to accomplish here?

My issue with adding or overriding java options is that it is very easy to screw up. Meaning that you have two configurations you have to keep in sync (default java options and task java options).

Even adding "extraJavaOpts" runs the risk of someone changing the origin javaOpts to have incompatible or weird flags.

Can you give an idea of the use cases you are trying to solve by exposing javaOpts?

[jaehc]

@drcrallen I totally agree with your point. We should use it very carefully not to make a mess. But, I believe there are still some useful usecases though.

Here is my usecase. I want to attach my debugger to an IndexTask which is forked from a overlord or middle manager. However, IndexTasks often finish its job and the jvm terminates too quickly to attach my debugger to it. So I want to put off the execution of IndexTask until my debugger has got a connection to the forked jvm. Giving suspend=y as a jvm option of a child process works out this issue as it delays the execution of code and makes the main thread(I guess) wait for a debugger attachment. But, if I set this option in the overlord configurations, all the tasks forked from the overlord will be stuck waiting for the debugger attachment, which is not the situation I want.

"context": {
    "druid.indexer.runner.javaOpts" : "-server -Xmx4g -Xms4g -server -Xdebug -Xrunjdwp:transport=dt_socket,address=0.0.0.0:0,suspend=y"
  }

[drcrallen]

@CHOIJAEHONG1 Cool. once #2246 is done, I was hoping an extra feature to be able to add JVM modules in a "better" way, such that you could add things like the Debugger properties you mentioned, or enable jmxremote or other JVM level stuff. And the modules would have basic capacities to append to the javaOpts to configure the properties, AND be able to communicate any job resource needs that it adds (ex: more ports, more memory, more cpu load)

[drcrallen]

since the context javaOpts solves this for the short term, would something like what I described solve it for the longer term?

[jaehc]

@drcrallen Context javaOpts is enough to solve the problem nicely. It sounds good to have that feature you mentioned, especially when it comes to dealing with YARN or MESOS and providing such functionalities based on them refering to #2246, I guess.

[suhassumukh]

@nishantmonu51 Is this documented already? I was once advised to set up different nodes and use affinity config in the task. My use case would be to use different options for different incoming density for datasources (like, one datasource gets about 5k rows per hour, another about 100k per hour).

[nishantmonu51]

@suhassumukhv: It still seems undocumented, A PR against the docs to add this info would be welcome.

[vsharathchandra]

Is there a way to do the same thing with kafka indexing service ? so that the real time ingestion peons of different datasources can be configured different memory.

[himanshug]

@vsharathchandra you can put same in kafka supervisor context and it would be propagated to all the launched KafkaIndexTasks

[vsharathchandra]

Thanks.Tried it it's working.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.