Flaky test ITAppenderatorDriverRealtimeIndexTaskTest testRealtimeIndexTask()

[justinborromeo]

Affected Version

master branch

Description

Transient failures observed by both me and @jihoonson on Travis.

Steps to Replicate

Unsure as I don't run ITs on my local machine. Definitely fails inconsistently on Travis though (e.g. https://travis-ci.org/apache/incubator-druid/jobs/503819087)

Stack Trace

[ERROR] testRealtimeIndexTask(org.apache.druid.tests.indexer.ITAppenderatorDriverRealtimeIndexTaskTest)  Time elapsed: 10.178 s  <<< FAILURE!
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
	at org.apache.druid.tests.indexer.ITAppenderatorDriverRealtimeIndexTaskTest.postEvents(ITAppenderatorDriverRealtimeIndexTaskTest.java:108)
	at org.apache.druid.tests.indexer.ITAppenderatorDriverRealtimeIndexTaskTest.testRealtimeIndexTask(ITAppenderatorDriverRealtimeIndexTaskTest.java:57)
Caused by: java.util.concurrent.ExecutionException: org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
	at org.apache.druid.tests.indexer.ITAppenderatorDriverRealtimeIndexTaskTest.postEvents(ITAppenderatorDriverRealtimeIndexTaskTest.java:108)
	at org.apache.druid.tests.indexer.ITAppenderatorDriverRealtimeIndexTaskTest.testRealtimeIndexTask(ITAppenderatorDriverRealtimeIndexTaskTest.java:57)
Caused by: org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
	at org.apache.druid.tests.indexer.ITAppenderatorDriverRealtimeIndexTaskTest.postEvents(ITAppenderatorDriverRealtimeIndexTaskTest.java:108)
	at org.apache.druid.tests.indexer.ITAppenderatorDriverRealtimeIndexTaskTest.testRealtimeIndexTask(ITAppenderatorDriverRealtimeIndexTaskTest.java:57)
Caused by: org.jboss.netty.channel.ChannelException: Failed to handshake with host[https://127.0.0.1:8101]
Caused by: java.nio.channels.ClosedChannelException

[kamaci]

I think that problem is not just related to ITAppenderatorDriverRealtimeIndexTaskTest#testRealtimeIndexTask()

I get same error at my PR too:

https://travis-ci.org/apache/incubator-druid/jobs/503941925

Seems that root cause is related to java.util.concurrent.ExecutionException: org.jboss.netty.channel.ChannelException

This conversation may give us an insight about this problem: #3393

[justinborromeo]

@kamaci That's definitely the issue. The same error is popping up in #7017 .