From ed0538c9f10b4cb605684ccd1bb701ddd83ba0a4 Mon Sep 17 00:00:00 2001 From: Yuan Zhou Date: Tue, 22 Apr 2025 10:02:45 +0100 Subject: [PATCH 1/5] [VL] Adding doc for dynamic offheap sizing feature Signed-off-by: Yuan Zhou --- docs/developers/VeloxDynamicSizingOffheap.md | 24 ++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 docs/developers/VeloxDynamicSizingOffheap.md diff --git a/docs/developers/VeloxDynamicSizingOffheap.md b/docs/developers/VeloxDynamicSizingOffheap.md new file mode 100644 index 000000000000..cc40a23438f9 --- /dev/null +++ b/docs/developers/VeloxDynamicSizingOffheap.md @@ -0,0 +1,24 @@ +## Dynamic Off-heap Sizing +Spark requires setting both on-heap and off-heap memory sizes, which initializes different memory layouts. Improper configuration of these settings can lead to lower performance. Dynamic off-heap sizing is an experimental feature designed to simplify this process. When enabled, off-heap settings are ignored, and Velox uses the on-heap size as the memory size. +## Detail implementations +To understand the details, it's essential to learn the basics of JVM memory management. There are many resources discussing JVM internals: +- https://exia.dev/blog/2019-12-10/JVM-Memory-Model/ +- https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/garbage_collect.html +- https://www.scaler.com/topics/memory-management-in-java/ +- https://developers.redhat.com/articles/2021/09/09/how-jvm-uses-and-allocates-memory# + +In general, the feature works as follows: + +- Spark first attempts to allocate memory based on the on-heap size. Note that the maximum memory size is controlled by spark.executor.memory. +- When Velox tries to allocate memory, Gluten attempts to allocate from system memory and records this in the memory allocator. +- If there is sufficient memory, allocations proceed normally. +- If memory is insufficient, Spark performs garbage collection (GC) to free on-heap memory, allowing Velox to allocate memory. +- If memory remains insufficient after GC, Spark reports an out-of-memory (OOM) issue. + +We then enforce a total memory quota, calculated as the sum of committed and in-use memory in the Java heap (using Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()) plus tracked off-heap memory in TreeMemoryConsumer. If an allocation exceeds this total committed memory, the allocation fails, triggering an OOM. + +With this change, the "quota check" is performed when an allocation in the native engine is informed to Gluten. In practice, this means the Java codebase can oversubscribe memory within the on-heap quota, even if off-heap usage is sufficient to fail the allocation. + +## Limitations + +This feature is still under heavy development. \ No newline at end of file From 1606cfb4c685292cc40a002302870ae09fcc9ec1 Mon Sep 17 00:00:00 2001 From: Yuan Date: Sun, 17 Aug 2025 12:53:39 +0100 Subject: [PATCH 2/5] refine Signed-off-by: Yuan --- docs/developers/VeloxDynamicSizingOffheap.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/developers/VeloxDynamicSizingOffheap.md b/docs/developers/VeloxDynamicSizingOffheap.md index cc40a23438f9..def723abd777 100644 --- a/docs/developers/VeloxDynamicSizingOffheap.md +++ b/docs/developers/VeloxDynamicSizingOffheap.md @@ -1,5 +1,7 @@ ## Dynamic Off-heap Sizing -Spark requires setting both on-heap and off-heap memory sizes, which initializes different memory layouts. Improper configuration of these settings can lead to lower performance. Dynamic off-heap sizing is an experimental feature designed to simplify this process. When enabled, off-heap settings are ignored, and Velox uses the on-heap size as the memory size. +Gluten requires setting both on-heap and off-heap memory sizes, which initializes different memory layouts. Improper configuration of these settings can lead to lower performance. + +To fix this issue, Dynamic off-heap sizing is an experimental feature designed to simplify this process. When enabled, off-heap settings are ignored, and Velox uses the on-heap size as the memory size. ## Detail implementations To understand the details, it's essential to learn the basics of JVM memory management. There are many resources discussing JVM internals: - https://exia.dev/blog/2019-12-10/JVM-Memory-Model/ @@ -21,4 +23,4 @@ With this change, the "quota check" is performed when an allocation in the nativ ## Limitations -This feature is still under heavy development. \ No newline at end of file +This feature is still in early development stage. \ No newline at end of file From 25c2174c8a38082a07aab7e03085b98b6120ddbf Mon Sep 17 00:00:00 2001 From: Yuan Date: Sun, 17 Aug 2025 13:06:45 +0100 Subject: [PATCH 3/5] adding nav order Signed-off-by: Yuan --- docs/developers/VeloxDynamicSizingOffheap.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/developers/VeloxDynamicSizingOffheap.md b/docs/developers/VeloxDynamicSizingOffheap.md index def723abd777..911b339a0ee5 100644 --- a/docs/developers/VeloxDynamicSizingOffheap.md +++ b/docs/developers/VeloxDynamicSizingOffheap.md @@ -1,3 +1,10 @@ +--- +layout: page +title: VeloxDynamicSizingOffheap +nav_order: 16 +parent: Developer Overview +--- + ## Dynamic Off-heap Sizing Gluten requires setting both on-heap and off-heap memory sizes, which initializes different memory layouts. Improper configuration of these settings can lead to lower performance. From 630e0ba2c4218dc3d247ac65bcd6ae2cbf102b1e Mon Sep 17 00:00:00 2001 From: Yuan Date: Sun, 17 Aug 2025 13:23:33 +0100 Subject: [PATCH 4/5] refine Signed-off-by: Yuan --- docs/developers/VeloxDynamicSizingOffheap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers/VeloxDynamicSizingOffheap.md b/docs/developers/VeloxDynamicSizingOffheap.md index 911b339a0ee5..d16f4f9d4a88 100644 --- a/docs/developers/VeloxDynamicSizingOffheap.md +++ b/docs/developers/VeloxDynamicSizingOffheap.md @@ -30,4 +30,4 @@ With this change, the "quota check" is performed when an allocation in the nativ ## Limitations -This feature is still in early development stage. \ No newline at end of file +This feature is in the preliminary stages of development and will be improved in future updates. \ No newline at end of file From 4060fa44a32ef762d294d24588a60900f6563745 Mon Sep 17 00:00:00 2001 From: Yuan Date: Sat, 23 Aug 2025 19:11:37 +0100 Subject: [PATCH 5/5] address comments Signed-off-by: Yuan --- docs/developers/VeloxDynamicSizingOffheap.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/developers/VeloxDynamicSizingOffheap.md b/docs/developers/VeloxDynamicSizingOffheap.md index d16f4f9d4a88..e6ff3b399dd9 100644 --- a/docs/developers/VeloxDynamicSizingOffheap.md +++ b/docs/developers/VeloxDynamicSizingOffheap.md @@ -8,7 +8,7 @@ parent: Developer Overview ## Dynamic Off-heap Sizing Gluten requires setting both on-heap and off-heap memory sizes, which initializes different memory layouts. Improper configuration of these settings can lead to lower performance. -To fix this issue, Dynamic off-heap sizing is an experimental feature designed to simplify this process. When enabled, off-heap settings are ignored, and Velox uses the on-heap size as the memory size. +To fix this issue, dynamic off-heap sizing is an experimental feature designed to simplify this process. When enabled, off-heap settings are ignored, and Velox uses the on-heap size as the memory size. ## Detail implementations To understand the details, it's essential to learn the basics of JVM memory management. There are many resources discussing JVM internals: - https://exia.dev/blog/2019-12-10/JVM-Memory-Model/ @@ -18,15 +18,15 @@ To understand the details, it's essential to learn the basics of JVM memory mana In general, the feature works as follows: -- Spark first attempts to allocate memory based on the on-heap size. Note that the maximum memory size is controlled by spark.executor.memory. +- Spark first attempts to allocate memory based on the on-heap size. Note that the maximum memory size is controlled by `spark.executor.memory`. - When Velox tries to allocate memory, Gluten attempts to allocate from system memory and records this in the memory allocator. - If there is sufficient memory, allocations proceed normally. - If memory is insufficient, Spark performs garbage collection (GC) to free on-heap memory, allowing Velox to allocate memory. - If memory remains insufficient after GC, Spark reports an out-of-memory (OOM) issue. -We then enforce a total memory quota, calculated as the sum of committed and in-use memory in the Java heap (using Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()) plus tracked off-heap memory in TreeMemoryConsumer. If an allocation exceeds this total committed memory, the allocation fails, triggering an OOM. +We then enforce a total memory quota, calculated as the sum of committed and in-use memory in the Java heap (using `Runtime.getRuntime().totalMemory()`) plus tracked off-heap memory in `TreeMemoryConsumer`. If an allocation exceeds this total committed memory, the allocation fails and triggers an OOM. -With this change, the "quota check" is performed when an allocation in the native engine is informed to Gluten. In practice, this means the Java codebase can oversubscribe memory within the on-heap quota, even if off-heap usage is sufficient to fail the allocation. +With this change, the "quota check" is performed when Gluten receives an memory allocation request. In practice, this means the Java codebase can oversubscribe memory within the on-heap quota, even if off-heap usage is sufficient to fail the allocation. ## Limitations