Add support for HttpFirehose

[jihoonson]

This change is 

[fjy]

👍

[himanshug]

will there be duplication in the case when this returns an InputStream ... which is partially read and then there is a temporary network glitch and consequently an IOException. parent class does the retry (which starts reading data from beginning again) and ends up introducing duplicates ?

[jihoonson]

I think it should be fine.

1. If prefetch is enabled,
   1.1) PrefetchableTextFilesFirehose can fetch http objects in background. When next() is called, it first checks that there are already fetched files in local storage. If exists, it simply chooses a fetched file and returns a LineIterator for that file.
   1.2) If there is no fetched files in local storage but some objects are still remained to be read, PrefetchableTextFilesFirehose fetches one of objects in background immediately. If an IOException occurs while downloading the object, it retries up to the maximum retry count. Finally, PrefetchableTextFilesFirehose returns a LineIterator only when the download operation is successfully finished.
2. If prefetch is disabled, PrefetchableTextFilesFirehose returns a LineIterator which directly reads the stream opened by openObjectStream(). If there is an IOException, it will throw it and the read will fail.

Does it make sense?

[himanshug]

yeah, I see from PrefetchTextFilesFirehoseFactory.download(..) that whole contents are downloaded irrespective of cache/prefetch capacity bytes , which is fair to keep implementation simpler . it will be nice to document it in some way though.

[jihoonson]

Thanks. I'll add a document about the behavior of PrefetchableTextFilesFirehose somewhere.

[jihoonson]

I added the behavior of PrefetchableTextFilesFirehose to its java doc.

[gianm]

👍 on code & design

[himanshug]

not clear why, as a user, i need to set 2 capacities. why would I not just have properties like...
long maxCacheCapacityBytes and boolean prefetchEnabled ?

[jihoonson]

Cache and fetch spaces are managed differently to achieve different goals. That is, cached files are not removed until the jvm terminates for future reads, while fetched files are removed when a connection on firehose is closed.

The goal of prefetching is to improve the throughput of the firehose, so maxFetchCapacityBytes should be tuned sometimes. For example, let me suppose an index task which reads data from an HttpFirehose and generates indexes.

• If the indexing speed is so fast but http connection is so slow, reading from the firehose will be a bottleneck. In this case, increasing maxFetchCapacityBytes will be helpful.
• Or, if the indexing speed is not bad but the amount of remaining disk is very little, decreasing maxFetchCapacityBytes might be helpful.

Maybe better to add a document about this?

[himanshug]

yeah more documentation could help.

however, as a user I would know how much disk space I am willing to spend on this task and it would be super nice if system automatically decides on the division of how much to use for cache vs prefetch space.
currently user needs to do the tuning that can potentially be done by the system itself.

[himanshug]

btw: this is not a blocker but something to be thought about and considered before finalizing the current model.

[jihoonson]

Sounds good. It would be great if we can make the PrefetchableTextFilesFirehose smarter in the future.

I added explanations on maxFetchCapacityBytes and maxCacheCapacityBytes.

[nishantmonu51]

👍