Use daemon thread pool for AsyncHttpClient in emitters #5057

xanec · 2017-11-08T18:20:31Z

This PR is to configure AsyncHttpClients to use daemon thread pool: #4973 (comment)

leventov · 2017-11-08T18:25:33Z

server/src/main/java/io/druid/server/emitter/ParametrizedUriEmitterModule.java

  )
  {
    final DefaultAsyncHttpClientConfig.Builder builder = new DefaultAsyncHttpClientConfig.Builder();
    if (sslContext != null) {


Could this common logic be shared between in HttpEmitterModule and ParametrizedUriEmitterModule?

Yeah, I think so. But I am having a hard time to come up with a proper place to put the common code. Previously, part of the logic was in java-util's HttpClientInit.createClient; maybe we should have a similar factory method for AsyncHttpClient in java-util?

it may be a static method in HttpEmitterModule, and ParametrizedUriEmitterModule uses it

You're right, I missed the other static method in HttpEmitterModule. Thanks 👍

himanshug · 2017-11-08T18:45:34Z

👍 @xanec were you able to reproduce the issue and verified this fix ?

himanshug · 2017-11-08T18:46:03Z

also, does this need to be backported to 0.11.0 ?

drcrallen · 2017-11-08T18:54:12Z

server/src/main/java/io/druid/server/emitter/HttpEmitterModule.java

    if (sslContext != null) {
      builder.setSslContext(new JdkSslContext(sslContext, true, ClientAuth.NONE));
    }
+    builder.setThreadFactory(new DefaultThreadFactory("emitter-http-client", true));


io.druid.java.util.common.concurrent.Execs#makeThreadFactory(java.lang.String)

drcrallen · 2017-11-08T18:54:33Z

server/src/main/java/io/druid/server/emitter/ParametrizedUriEmitterModule.java

    if (sslContext != null) {
      builder.setSslContext(new JdkSslContext(sslContext, true, ClientAuth.NONE));
    }
+    builder.setThreadFactory(new DefaultThreadFactory("parametrized-emitter-http-client", true));


io.druid.java.util.common.concurrent.Execs#makeThreadFactory(java.lang.String) should be used

xanec · 2017-11-08T20:27:38Z

@himanshug yeah, I believe we are facing similar issue on our side. I'll need to get back to you on the verification.

himanshug · 2017-11-08T21:47:23Z

ok, i think we can merge it once the build passes as we definitely want that pool to be non-daemon in any case.

…ttp-client

leventov · 2017-11-09T18:53:15Z

This build fails with some strange error:

[ERROR] COMPILATION ERROR : 
[ERROR] error reading /home/travis/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar; error in opening zip file
[ERROR] error reading /home/travis/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar; cannot read zip file
[ERROR] /home/travis/build/druid-io/druid/indexing-hadoop/src/test/java/io/druid/indexer/HdfsClasspathSetupTest.java:[35,30] cannot find symbol
  symbol:   class MiniDFSCluster
  location: package org.apache.hadoop.hdfs
[ERROR] /home/travis/build/druid-io/druid/indexing-hadoop/src/test/java/io/druid/indexer/HdfsClasspathSetupTest.java:[62,18] cannot find symbol
  symbol:   class MiniDFSCluster
  location: class io.druid.indexer.HdfsClasspathSetupTest
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:testCompile (default-testCompile) on project druid-indexing-hadoop: Compilation failure: Compilation failure: 
[ERROR] error reading /home/travis/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar; error in opening zip file
[ERROR] error reading /home/travis/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar; cannot read zip file
[ERROR] /home/travis/build/druid-io/druid/indexing-hadoop/src/test/java/io/druid/indexer/HdfsClasspathSetupTest.java:[35,30] cannot find symbol
[ERROR]   symbol:   class MiniDFSCluster
[ERROR]   location: package org.apache.hadoop.hdfs
[ERROR] /home/travis/build/druid-io/druid/indexing-hadoop/src/test/java/io/druid/indexer/HdfsClasspathSetupTest.java:[62,18] cannot find symbol
[ERROR]   symbol:   class MiniDFSCluster
[ERROR]   location: class io.druid.indexer.HdfsClasspathSetupTest
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :druid-indexing-hadoop

Making an experiment with updating Hadoop 2.7.3 -> 2.7.4.

FYI @nishantmonu51 @b-slim

gianm · 2017-11-09T21:36:22Z

@leventov seems like the weird error is due to this part:

error reading /home/travis/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.3/hadoop-hdfs-2.7.3-tests.jar; error in opening zip file

The hadoop-hdfs-2.7.3-tests.jar could be corrupt. The corruption could persist across builds due to the cache, enabled in our .travis.yml here:

cache:
  directories:
    - $HOME/.m2

Clearing the cache could help (https://docs.travis-ci.com/user/caching/).

* use daemon thread pool for AsyncHttpClient in emitters * changed to use existing helper methods * refactored creation of AsyncHttpClient

) * Use daemon thread pool for AsyncHttpClient in emitters (#5057) * use daemon thread pool for AsyncHttpClient in emitters * changed to use existing helper methods * refactored creation of AsyncHttpClient * Avoid Execs, it's not available in this branch.

…ache#5069) * Use daemon thread pool for AsyncHttpClient in emitters (apache#5057) * use daemon thread pool for AsyncHttpClient in emitters * changed to use existing helper methods * refactored creation of AsyncHttpClient * Avoid Execs, it's not available in this branch.

use daemon thread pool for AsyncHttpClient in emitters

5990779

leventov reviewed Nov 8, 2017

View reviewed changes

drcrallen requested changes Nov 8, 2017

View reviewed changes

changed to use existing helper methods

dd37a52

drcrallen approved these changes Nov 8, 2017

View reviewed changes

refactored creation of AsyncHttpClient

65722cd

leventov approved these changes Nov 8, 2017

View reviewed changes

leventov added this to the 0.11.0 milestone Nov 8, 2017

leventov added 2 commits November 9, 2017 01:26

Merge remote-tracking branch 'upstream/master' into daemon-threaded-h…

a3aa393

…ttp-client

Merge remote-tracking branch 'upstream/master' into daemon-threaded-h…

848663a

…ttp-client

leventov force-pushed the daemon-threaded-http-client branch from 0d03c23 to 848663a Compare November 9, 2017 21:41

gianm merged commit 81f2498 into apache:master Nov 10, 2017

gianm mentioned this pull request Nov 10, 2017

[Backport] Use daemon thread pool for AsyncHttpClient in emitters #5069

Merged

leventov deleted the daemon-threaded-http-client branch November 10, 2017 03:53

himanshug mentioned this pull request Nov 10, 2017

make AsyncHttpCLient instance timer thread daemon #5076

Merged

Use daemon thread pool for AsyncHttpClient in emitters #5057

Use daemon thread pool for AsyncHttpClient in emitters #5057

Uh oh!

Conversation

xanec commented Nov 8, 2017

Uh oh!

leventov Nov 8, 2017

Choose a reason for hiding this comment

Uh oh!

xanec Nov 8, 2017

Choose a reason for hiding this comment

Uh oh!

leventov Nov 8, 2017

Choose a reason for hiding this comment

Uh oh!

xanec Nov 8, 2017

Choose a reason for hiding this comment

Uh oh!

himanshug commented Nov 8, 2017

Uh oh!

himanshug commented Nov 8, 2017

Uh oh!

drcrallen Nov 8, 2017

Choose a reason for hiding this comment

Uh oh!

drcrallen Nov 8, 2017

Choose a reason for hiding this comment

Uh oh!

xanec commented Nov 8, 2017

Uh oh!

himanshug commented Nov 8, 2017

Uh oh!

leventov commented Nov 9, 2017

Uh oh!

gianm commented Nov 9, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants