[VL] Use mixed memory (off-heap and on-heap) for native

[Yohahaha]

Description

Gluten only use off-heap memory for now, and I see some PR want extend spill with spill to heap. However, we could use mixed memory which means use off-heap first then use on-heap when off-heap will OOM to utilize memory as much as possible.

[zhztheplayer]

This is a big topic but indeed very helpful to the project. So it would be good if you can elaborate on the design if you have some, and update the PR description to make it more concrete.

What I could imagine is that we can hack into Spark's memory manager, then dynamically borrow some heap caps when JVM heap is shrinking to a number below -Xmx, when G1 GC or Parallel GC are used. CMS GC doesn't support heap shrinking AFAIK.

Hacking to deeper level of JVM is also a potential solution, but that may require some techniques like heap-pining of Hotspot JVM which seems to be a very old feature in its earlier versions. I am not sure whether the feature works well in JDK 8+ but probably we can do some path-findings around it anyway.

[Yohahaha]

I'm not ready to submit a PR, just use issue to record the idea.

[Yohahaha]

What I could imagine is that we can hack into Spark's memory manager, then dynamically borrow some heap caps when JVM heap is shrinking to a number below -Xmx, when G1 GC or Parallel GC are used. CMS GC doesn't support heap shrinking AFAIK.

What if we declare a MemoryConsumer with ON_HEAP mode? just report to Spark how much memory we need, but not do real allocation in JVM.

[zhztheplayer]

What I could imagine is that we can hack into Spark's memory manager, then dynamically borrow some heap caps when JVM heap is shrinking to a number below -Xmx, when G1 GC or Parallel GC are used. CMS GC doesn't support heap shrinking AFAIK.

What if we declare a MemoryConsumer with ON_HEAP mode? just report to Spark how much memory we need, but not do real allocation in JVM.

To me that would be a little bit dangerous: If you don't actually allocate from JVM, it will finally full then trigger GC. During which the process RSS will go beyond the limit.

But if we can somehow keep the GCs inside young generation, the approach could be comparatively safter however still be risky if we want to control the memory limit very accurately.

[Yohahaha]

Reservation and allocation are totally different things, reservation not trigger GC but allocation.
MemoryConsumer#allocatePage do reservation and allocation, MemoryConsumer#acquireMemory only do reservation that just notify Spark I need those number of bytes, when or how to allocate is not important.

[zhztheplayer]

Reservation and allocation are totally different things, reservation not trigger GC but allocation.

This is why it's dangerous: It doesn't allocate, and it doesn't trigger GC either. So any other part of the JVM could continue allocating from JVM until GC. So the total RSS will finally be beyond off-heap + on-heap, even before the actual GC happens.

JVM's heap size is not the size it allocated, it's allocated size + garbage size.

[Yohahaha]

So any other part of the JVM could continue allocating from JVM until GC. So the total RSS will finally be beyond off-heap + on-heap, even before the actual GC happens.

I suppose the other part is memory consumer, and all I am discussed are under Spark memory manager scope. Do you mean there will be the case which Spark on-heap pool used + of-heap pool used beyond limit?

[zhztheplayer]

Do you mean there will be the case which Spark on-heap pool used + of-heap pool used beyond limit?

No, it's the case that JVM heap size + off-heap size, which is the real RSS the process used, beyond the limit.

[zhztheplayer]

JEP 423 in JDK 22 may help. Should investigate later.