Transient failures of integration tests

[leventov]

Please edit messages with failure "snapshots" by adding links to failing builds, to understand which tests fail most often.

[leventov]

ITKafkaTest - no output for 10m0s

1. https://travis-ci.org/druid-io/druid/jobs/238785328

[leventov]

ITUnionQueryTest, #4346

io.druid.java.util.common.ISE: one or more queries failed
	at io.druid.tests.indexer.ITUnionQueryTest.testUnionQuery(ITUnionQueryTest.java:94)

1. https://travis-ci.org/druid-io/druid/jobs/238169889
2. https://travis-ci.org/druid-io/druid/jobs/238809579
3. https://travis-ci.org/metamx/druid/jobs/238824345
4. https://travis-ci.org/druid-io/druid/jobs/238809579

Fixed

[leventov]

ITKafkaTest - infinite repetition of messages like

2017-06-02T17:44:17,098 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2017-06-02T17:44:17.098Z","service":"","host":"","metric":"jetty/numOpenConnections","value":0}]
2017-06-02T17:44:18,668 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://172.17.0.1:8090
2017-06-02T17:44:18,673 INFO [main] io.druid.testing.clients.OverlordResourceTestClient - Index status response{"task":"index_realtime_kafka_test_0_2017-06-02T17:38:58.540Z_laaaokep","status":{"id":"index_realtime_kafka_test_0_2017-06-02T17:38:58.540Z_laaaokep","status":"RUNNING","duration":-1}}
2017-06-02T17:44:18,674 INFO [main] io.druid.testing.utils.RetryUtil - Attempt[16]: Task index_realtime_kafka_test_0_2017-06-02T17:38:58.540Z_laaaokep still not complete. Next retry in 20000 ms

1. https://travis-ci.org/druid-io/druid/jobs/238785328
2. https://travis-ci.org/druid-io/druid/jobs/239036844
3. https://travis-ci.org/druid-io/druid/jobs/240075183
4. https://travis-ci.org/druid-io/druid/jobs/241367176

[leventov]

ITKafkaIndexingServiceTest - no output for 10m0s

1. https://travis-ci.org/druid-io/druid/jobs/239036844

[jihoonson]

The ConnectionException: Connection refused: /172.17.0.1:8081 error is usually seen while waiting for a coordinator is ready on a docker machine. In the above case, it looks like a docker machine for coordinator was not ready even after 10 attempts to connect. Maybe there was another problem blocking a coordinator or a docker machine starting up.

[leventov]

ITUnionQueryTest

[ERROR] testUnionQuery(io.druid.tests.indexer.ITUnionQueryTest)  Time elapsed: 350.097 s  <<< FAILURE!
io.druid.java.util.common.ISE: Error while making request to url[http://172.17.0.1:8081/druid/coordinator/v1/datasources/wikipedia_index_test3/intervals] status[204 No Content] content[]
	at io.druid.tests.indexer.ITUnionQueryTest.testUnionQuery(ITUnionQueryTest.java:141)

1. https://travis-ci.org/metamx/druid/jobs/241380082

[jihoonson]

ITUnionQueryTest - no output for 10m0s

https://travis-ci.org/druid-io/druid/jobs/247310893

This looks due to zookeeper timeout.

[jihoonson]

I think no output for 10m0s is due to exhaustion of memory. #4429 might be helpful.