Flaky UT: PrefetchableTextFilesFirehoseFactoryTest

[paul-rogers]

Test failed on Travis, works on a Mac. Test passed on previous runs. No code in the PR changed anything in the code covered by this test.

[ERROR] Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 73.209 s <<< FAILURE! - in org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactoryTest
[ERROR] org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactoryTest.testWithSmallCacheAndLargeFetch  Time elapsed: 60.066 s  <<< ERROR!
java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at org.apache.druid.data.input.impl.prefetch.Fetcher.openObjectFromLocal(Fetcher.java:226)
	at org.apache.druid.data.input.impl.prefetch.Fetcher.next(Fetcher.java:177)
	at org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactory$2.next(PrefetchableTextFilesFirehoseFactory.java:239)
	at org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactory$2.next(PrefetchableTextFilesFirehoseFactory.java:225)
	at org.apache.druid.data.input.impl.FileIteratingFirehose.getNextLineIterator(FileIteratingFirehose.java:105)
	at org.apache.druid.data.input.impl.FileIteratingFirehose.hasMore(FileIteratingFirehose.java:66)
	at org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactoryTest.testWithSmallCacheAndLargeFetch(PrefetchableTextFilesFirehoseFactoryTest.java:278)

I have observed other failures in "firehose" tests (can't recall if it is this same test), did some poking around, and noticed that the code, while using concurrency constructs, used them independently with no locking between the various objects. I suspect this leads to race conditions and occasional failures.

[paul-rogers]

Failed again in this run. The test passes when run on the same code branch from an IDE.

[paul-rogers]

Failed again in this run. The test passes when run on the same code branch from an IDE.

[paul-rogers]

Failed again in this run.

[paul-rogers]

Failed again in this run:

[INFO] Running org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactoryTest
[ERROR] Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 21.109 s <<< FAILURE! - in org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactoryTest
[ERROR] org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactoryTest.testReconnectWithCacheAndPrefetch  Time elapsed: 8.165 s  <<< ERROR!
java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at org.apache.druid.data.input.impl.prefetch.Fetcher.openObjectFromLocal(Fetcher.java:226)
	at org.apache.druid.data.input.impl.prefetch.Fetcher.next(Fetcher.java:177)
	at org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactory$2.next(PrefetchableTextFilesFirehoseFactory.java:239)
	at org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactory$2.next(PrefetchableTextFilesFirehoseFactory.java:225)
	at org.apache.druid.data.input.impl.FileIteratingFirehose.getNextLineIterator(FileIteratingFirehose.java:105)
	at org.apache.druid.data.input.impl.FileIteratingFirehose.hasMore(FileIteratingFirehose.java:66)
	at org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactoryTest.testReconnectWithCacheAndPrefetch(PrefetchableTextFilesFirehoseFactoryTest.java:352)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	...
Caused by: java.util.concurrent.TimeoutException
	... 39 more

[abhishekagarwal87]

I have started a thread to get rid of most of the firehose related code. If we don't end up deleting that code, we could ignore these test cases in code.

[paul-rogers]

Failed again in this build.

[paul-rogers]

PrefetchableTextFilesFirehoseFactoryTest.testWithCacheAndFetch failed again in this build.

PrefetchableTextFilesFirehoseFactoryTest.testRetry failed again in this build.

Since this test is preventing useful work from being done, can we just disable it until someone can fix it?

[paul-rogers]

PrefetchableTextFilesFirehoseFactoryTest.testReconnectWithCacheAndPrefetch failed again in this build. These failures are a real drag on productivity: I have to rerun the expensive Travis tests again and again so I can get to Phase 2 ITs.

[abhishekagarwal87]

These flaky test will be gone once #12852 is merged.

[paul-rogers]

PrefetchableTextFilesFirehoseFactoryTest.testReconnectWithCacheAndPrefetch failed again in this build. Looking forward to seeing #12852 be merged!

PrefetchableTextFilesFirehoseFactoryTest.testWithLargeCacheAndSmallFetch failed again in this build. AWS is making some serious money with all the retries that this test is causing. (These are full build retries since us non-committers can't retry just the failed test.)

Since these two tests are doomed, I've disabled them in my PR so I can get some work done. They will be disabled in master once that PR is merged. Then, they can be removed once #12852 is merged.