Use jna fadvise in order to prevent poisoining the page cache

[bolkedebruin]

Druid relies on the page cache of Linux in order to have memory segments. However when loading segments from deep storage or rebalancing the page cache can get poisoned by segments that should not be in memory yet. This can significantly slow down Druid in case rebalancing happens as data that might not be queried often is suddenly in the page cache.

It is possible to advise the kernel what to do when writing and reading data and what the expectancy is of the app how the data should be treated. This is done by using posix_fadvise which has a JnA equivalent.

I suggest using this as it will speed up Druid under load.

0. http://insights.oetiker.ch/linux/fadvise.html
1. https://github.com/java-native-access/jna/blob/master/README.md
2. https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/NativeIO.java

[drcrallen]

This topic is one I've been curious about for a while.

This can significantly slow down Druid in case rebalancing happens often as data that might not be queried often is suddenly in the page cache.

Do you have any measurable evidence of this?

We see a lot of segment load slow down on new nodes depending on the free memory or dirty ratio, but haven't made a reliable way to measure query performance impact of an ongoing node.

[bolkedebruin]

On a pre-production cluster 7x 32GB, ~15GB available for page cache, HDD (SAN)) we were having 450GB of data, segments of ~500MB. "Active" segments (85GB) fitted in page cache memory (105GB). We did not have any queries spanning beyond the active segments, nevertheless the queries can be quite unique (same dataset but different subset)

We noticed the following:

1. Constant rebalancing - multiple GBs of data were reschuffled every now and then
2. Slow query performance -even requerying resulted high loads and IO on the historical nodes. (This was partially solved with increasing the query cache on the broker)
3. When no rebalance happened, queries started to speed up, but when a rebalance happened again everything started to slow down to a crawl.

What 3 tells us is that we are starting to hit disk IO while all the data should already have been in memory/page cache.

We removed all historical data that wasn't needed yet and only allowed one segment to be rebalanced at a time. This solved the speed of the queries. Rebalancing still happens, but as this will always be data that is required for queries this is not such a big issue anymore.

Please note that writes to disk also end up in the page cache. So a rebalance of data can potentially poison the page cache of the sender and of the receiver, e.g. in case of data outside of the regularly queried range.

So ideally Druid should hint the kernel whether it thinks this data belongs in the page cache or not. This can be done by setting at least FADV_DONTNEED (at receiver) for segments that are rebalanced and maybe at the sender (that's harder as you don't know what is in the page cache).

We have grafana logs to look at (afaik).

[bolkedebruin]

@drcrallen If you have some hints where the actual files are read and written (rebalance, load from deep storage) I can try a patch myself.

[drcrallen]

The segment loading is done in io.druid.segment.loading.DataSegmentPuller implementations.

As an alternative, something I've suggested internally is to look at changing the file format so that the data that is pulled down is queried against as-is, without needing to decompress (write twice) that should, in theory, reduce the page cache thrashing from segment loading.

As far as "getting faster" the only way to really measure if page cache is affecting queries is to disable druid-level caching, which can be done at query time or as part of a node's configuration.

[bolkedebruin]

Thanks. I'll have a look.

Basically what you are saying is that the issue might be bigger than expected. I don't know enough about the code but I suspect the following happens:

1. Download segment and write to temporary file
2. Read temp and write to file

If this is the case it potentially poisons the cache twice, unfortunately. First with writing the temporary file (whole file in cache) and then the uncompressed file.

Ideally, you would read from the download stream and, currently, decompress on the fly and then write in 8mb chunks while setting FADV_DONTNEED on every chunk. 8mb is the recommended size for sequential files by Linus Torvalds (I need to lookup the reference couldn't find it right away). This could be an alternative to changing your file format I guess, but you could do that nevertheless.

I'll make some time to test properly, but that will probably after 2 weeks as we need the performance for user testing at the moment.

Update: I saw that uncompressing happens on the fly. So only if you need to convert to another format you poison the cache twice, which seems to be the case as you mentioned.

[gianm]

@bolkedebruin, thanks for looking into this. Some tidbits that may(?) be useful.

• Uncompress streams without having to download to tmp first #4364 changed the DataSegmentPuller behavior for a few implementations recently and seems related.
• You mentioned "the query cache on the broker". Just wanted to warn you about something there: activating broker query cache forces historical nodes to disable their merge step and send un-merged per-segment results to the broker. This is because the query cache is a per-query-per-segment cache, so the broker needs per-segment results to be able to use it. For this reason it usually scales better to disable query cache on the broker, and instead enable it on historicals.

[bolkedebruin]

@gianm

• Uncompress streams without having to download to tmp first #4364 reduces the issue for the affected Pullers. We use HDFS and its Puller still copies to local first. The HDFSPuller should be able to skip local cache as well. We might create a patch to a slow it to do so.
• Thanks for the pointer on the broker cache. We still have a small deployment at the moment, but we will keep it in mind.

We have an initial patch that we hope to test out in a few days when we get the chance.