Off-Heap store memory utilization issues

[sheinbergon]

In the recent release (3.3.0) you've changed the in-memory mode of the database of the database to use the MVStore off-heap store. This mode utilizes direct buffers of course, but unmanaged direct-buffer pools behavior in the JVM is not that great.

We are seeing up to 10GB of off-heap memory direct buffer utilization, only to be GC'd (Full GC, which has a harsh impact on performance) down to a third of its size and then growing back again. Compaction also doesn't seem to behave as expected, and during load peaks we sometimes encounter Caused by: java.lang.IllegalStateException: java.lang.OutOfMemoryError: Direct buffer memory [1.4.200/3].

Now of course we can move to file based database or just change the direct memory limitation via -XX:MaxDirectMemorySize but I wonder how's the In-Memory mode before this change? Maybe we should have both modes enabled?

Direct buffers are known to problematic when it comes to memory management, and IMHO Heap buffers are preferable when memory is limited.

[anidotnet]

You are right. When incorporating OffHeapStorage, I should have kept two options available for off-heap and on-heap memory. I'll make these changes in next major version - 4.x, unless it is very urgent for you.

[sheinbergon]

It is urgent for us, we are encountering major memory constraints and considering replacing nitrite with sqlite all together.

[anidotnet]

@sheinbergon If you do

Nitrite db = Nitrite.builder()
        .openOrCreate();

it will create on-heap in-memory database. And to enable off-heap storage, use

Nitrite db = Nitrite.builder()
        .enableOffHeapStorage()
        .openOrCreate();

This fix has been released under 3.4.0