In-app segment cache control instead of relying on Linux page cache subsystem (Direct IO instead of mmap)

[leventov]

This would allow to avoid cache trashing when a random query comes and pulls a lot of (otherwise unused) segments into memory. Also very simple heuristics, like giving preference to segments of more recent intervals, could improve the cache efficiency.

[leventov]

E. g. when a query comes to a historical node, it may have a flag "no cache", meaning that segments, that may need to be pulled into memory for processing of this query, should not be retained in the cache.

[leventov]

Another problem this thing could help with, is avoiding pushing a lot of segments though memory during initial loading from the local cache.

[leventov]

http://www.scylladb.com/2017/10/05/io-access-methods-scylla/ is a good article on this topic

[leventov]

Also, it would allow to limit the amount of memory devoted to page cache, if need anything different than "all available" memory.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.