handling 2x domain folder free space when merge #15343
Description
while merging, there's a period where the domain folder size becomes 2x (merged files done + existing smaller files not cleaned up).
If user doesn't have this space, NO_MERGE=true is perhaps the only option...
Ideas:
- prohibit merges > 1K steps for users?
- db does space check?
mix:
if need merge > 64 steps - and no space - warning, and allow only <64steps merges,
if need merge < 64 steps - and no space - error (can also be tolerant, can disable any merges in this case, but keep working - just print error messages periodically)
- compressor optimization for uncompressed modes
compress.NewCompressor does create 2 intermediate files. It necessary for compression. But it's unnecessary if we don't compress any thing (we have many files with disabled compression as i remember).
last time I had 500G free on gnosis machine and it was not enough to build 1 rcache.ef file (which creating 2 temporary files and not compressed).
we already have seg.NewReader(compresionMode) wrapper around seg.NewDecompressor which auto-magically using right methods. maybe we can have same wrapper for writes - where we know compressionMode in-advance.
- proactive cleanup of smaller files (need to check how much can be saved)...
right now, all entities get merged and only then cleanup of smaller files is done.
But that's only strictly needed for dependency groups (commitment/account/storage)so do cleanup before
Unlikely it will help for 0-4096.kv merge
- JIT compaction
another weird idea: we can merge 0-64.kv when 64-65.kv available. And during creation of 0-64.kv (during merge) skip all keys which exist in 64-65.kv.
I'm not sure - but likely it will remove most of popular keys (I mean after such merge: future manual compaction will show small skip-ratio).
I think it can be our version of "just-in-time compaction". (and it will work for all domains)
ToDo:
- Prohibit large merges on User's side