[VL] Cannot trigger spill when DynamicOffHeapSizingMemoryTarget is enabled

[wForget]

Backend

VL (Velox)

Bug description

#9336 removed the logic of passing the memory change of DynamicOffHeapSizingMemoryTarget to TaskMemoryManager, which will cause spark consumers to not be able to trigger spills from the native side.

Avoid chaining DynamicOffHeapSizingMemoryTarget with Spark consumer since it basically never triggers spills because of its design. The target will manage OOMs by itself.

I can't understand this description.

Gluten version

No response

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

[zhztheplayer]

The dynamic off-heap sizing code, before and after #9336, both don't trigger spill.

Before #9336, DynamicOffHeapSizingMemoryTarget essentially caps the available off-heap size within a number which is lower than the Spark off-heap size because of the co-accounted on-heap memory in this target. Because this number is lower than the Spark off-heap size, and when the number is reached, OOM will be reported, so from Spark memory manager's view, its used off-heap memory never reaches the total off-heap limit. That's why spill was never triggered.

#9336 just simplified the code, so this spill shortage can also be exposed to code reader clearly.

I had a similar question about the spill design in @zhli1142015's later rework of this feature, and he may have more insights.

[zhli1142015]

The formula defines the amount of memory available to each task — once this threshold is exceeded, the task memory manager triggers a spill.

Per-task memory = ((spark.executor.memory - 300MB) * spark.memory.fraction) 
                  / active tasks count

By default, spark.memory.fraction is 0.6, meaning that only about 60% of the executor memory (after subtracting 300 MB for overhead) is available for task computation.
As a result, the memory usage threshold that triggers a spill is much lower than the actual amount of memory required to cause an OOM(137).

The DynamicOffHeapSizingMemoryTarget does not change this logic. Its role is to let the JVM release free memory (Runtime.getRuntime().freeMemory()) back to the operating system when Velox needs more memory. Since Spark does not account for JVM free memory, this behavior is transparent to the TaskMemoryManager.
Under this architecture, Velox can theoretically utilize the entire memory allocated to the task, not just the configured off-heap portion.

@wForget In our case, we observed that the spill behavior was not affected by this feature.
Note that the JVM currently does not provide an API for ondemand memory shrinking — shrink only occurs as a side effect of a full GC, and its behavior may vary across different JVM versions.
Could you share your runtime environment, such as the JVM version, GC type, and memory configuration? Also, in what scenario did you observe that a spill should have occurred but instead resulted in an OOM?

[zhli1142015]

@zhztheplayer, I noticed that @wForget mentioned the TaskMemoryManager is not passed to DynamicOffHeapSizingMemoryTarget. I was wondering if this could potentially be related to the issue?
083ecfb#diff-b6234f870afb82ba142a4f4e3e358ddb30dc4d5f00b3b9f5b4e9afddc9b4a761R55

[wForget]

From this log, the offheap memory usage 2.3 GiB has exceeded the max memory of TaskMemoryManager (3 * 1024 MB -300 MB) * 0.6 = 1663.2 MB, so it should be enough to trigger spill.

25/10/24 09:35:25 INFO DynamicOffHeapSizingMemoryTarget: Updated VM flags: MaxHeapFreeRatio from 30 to 5.
25/10/24 09:35:25 WARN DynamicOffHeapSizingMemoryTarget: Starting full gc to shrink JVM memory: Total On-heap: 519110656, Free On-heap: 337648400, Total Off-heap: 2524971008, Used On-Heap: 181462256, Executor memory: 3113877504.
25/10/24 09:35:26 WARN DynamicOffHeapSizingMemoryTarget: Finished full gc to shrink JVM memory: Total On-heap: 519110656, Free On-heap: 414046152, Total Off-heap: 2524971008, Used On-Heap: 105064504, Executor memory: 3113877504, [GC Retry times: 3].
25/10/24 09:35:26 INFO DynamicOffHeapSizingMemoryTarget: Reverted VM flags back.
25/10/24 09:35:26 WARN DynamicOffHeapSizingMemoryTarget: Failing allocation as unified memory is OOM. Used Off-heap: 2524971008, Used On-Heap: 107549216, Free On-heap: 411561440, Total On-heap: 519110656, Max On-heap: 3113877504, Allocation: 486539264.
25/10/24 09:35:26 INFO TaskMemoryManager: Memory used in task 9695
25/10/24 09:35:26 INFO TaskMemoryManager: 0 bytes of memory were used by task 9695 but are not associated with specific consumers
25/10/24 09:35:26 INFO TaskMemoryManager: 0 bytes of memory are used for execution and 15127767 bytes of memory are used for storage
25/10/24 09:35:26 WARN ManagedReservationListener: Error reserving memory from target
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 464.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled). 
Current config settings: 
	spark.gluten.memory.offHeap.size.in.bytes=1663.2 MiB
	spark.gluten.memory.task.offHeap.size.in.bytes=1663.2 MiB
	spark.gluten.memory.conservative.task.offHeap.size.in.bytes=831.6 MiB
	spark.memory.offHeap.enabled=true
	spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
Dynamic off-heap sizing memory target stats: 
	DynamicOffHeapSizing.96: Current used bytes: 2.3 GiB, peak bytes: 2.3 GiB

spark conf:

spark.gluten.memory.dynamic.offHeap.sizing.enabled=true
spark.executor.memory=3g
spark.executor.extraJavaOptions=-XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=30 -Xms512m -XX:+UseSerialGC -Dfile.encoding=utf-8 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<LOG_DIR>/gc.log -XX:ErrorFile=<LOG_DIR>/hs_err_pid%p.log

[zhztheplayer]

I was wondering if this could potentially be related to the issue?
083ecfb#diff-b6234f870afb82ba142a4f4e3e358ddb30dc4d5f00b3b9f5b4e9afddc9b4a761R55

@zhli1142015 I was trying to understand why this issue is related to #9336. As I mentioned, #9336 was for optimizing the first version of this feature, which was implemented by #5439.

According to that version of the feature, DynamicOffHeapSizingMemoryTarget was using exactly the same capacity as Spark's off-heap memory.

https://github.com/apache/incubator-gluten/blob/55b2e92f54f95f980587d08f0beb430e0a922b15/gluten-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java#L33

So in #9336, we could safely do the simplification as spill was never triggered. I was once thinking about removing the whole feature because it was no longer maintained: #8600, #9336 was kind of, at the same time, preparing for that.

@zhli1142015 Would you confirm whether we are on the same page about the above? Thanks.

And after these actions, the good news was we got the feature reworked by #9585. Design was changed a lot so I think we can do anything we think is right to make sure spill works for the feature. Maybe by introducing the delegated target back, I am not sure.

By default, spark.memory.fraction is 0.6, meaning that only about 60% of the executor memory (after subtracting 300 MB for overhead) is available for task computation.

I would recommend taking more comprehensive factors into consideration when relying on such mechanism for spilling, like when on-heap usage exceeds 40%, how can we trigger spill prior to OOM, etc.

[zhli1142015]

@zhztheplayer my understanding is that when DynamicOffHeapSizingMemoryTarget is enabled, the following change causes the memory allocated by Velox not to be reported to the Spark Task Memory Manager, so spilling is not triggered — because DynamicOffHeapSizingMemoryTarget does not call the memoryTarget below. (Please correct me if I'm wrong.)

Even if the memory used by Velox is counted as on-heap memory, we still need to report the memory allocations to the Spark Task Memory Manager.

Why this is not discovered when doing PR #9585 ?
We’ve only tested PR #9585 extensively internally, but we are still using Gluten 1.3 internally, and RP #9336 has never been included. Spilling has not been an issue in our internal environment. I didn’t realize this issue until @wForget pointed it out.

[zhztheplayer]

We’ve only tested PR #9585 extensively internally, but we are still using Gluten 1.3 internally, and RP #9336 has never been included. Spilling has not been an issue in our internal environment. I didn’t realize this issue until @wForget pointed it out.

Now it gets clearer. That's also why I also raised a concern at #9585 (comment). Maybe for a solution like #9585, passing the memoryTarget to DynamicOffHeapSizingMemoryTarget for spilling could potentially make sense.

But assuming we have the delegate memory target set, as a non-blocker comment, more comprehensive consideration is needed to make the feature more robust, like how to trigger spill when on-heap memory already exceeds 40%, how to avoid process RSS overrun if there're constant on-heap memory allocations that DynamicOffHeapSizingMemoryTarget isn't aware, etc.