Skip to content

[GLUTEN-7802][CORE] Use original total memory size if enable dynamic offheap #7803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yikf wants to merge 2 commits into from
Closed

[GLUTEN-7802][CORE] Use original total memory size if enable dynamic offheap #7803

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d8163d9
Use original total memory size if enable dynamic offheap
yikf Nov 4, 2024
5a74ce8
Merge branch 'main' into total-memory-dynamic-offheap
yikf Nov 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,9 @@ private[gluten] class GlutenDriverPlugin extends DriverPlugin with Logging {
minOffHeapSize))
) {
throw new GlutenException(
s"Must set '$SPARK_OFFHEAP_ENABLED' to true " +
s"and set '$SPARK_OFFHEAP_SIZE_KEY' to be greater than $minOffHeapSize")
s"Must set '$SPARK_OFFHEAP_ENABLED' to true and set " +
s"'$SPARK_OFFHEAP_SIZE_KEY' to be greater than $minOffHeapSize, or set " +
s"'$GLUTEN_DYNAMIC_OFFHEAP_SIZING_ENABLED' to true to enable dynamic offheap feature.")
}

// Session's local time zone must be set. If not explicitly set by user, its default
Expand All @@ -160,8 +161,6 @@ private[gluten] class GlutenDriverPlugin extends DriverPlugin with Logging {
val taskSlots = SparkResourceUtil.getTaskSlots(conf)
conf.set(GLUTEN_NUM_TASK_SLOTS_PER_EXECUTOR_KEY, taskSlots.toString)

val onHeapSize: Long = conf.getSizeAsBytes(SPARK_ONHEAP_SIZE_KEY, 1024 * 1024 * 1024)

// If dynamic off-heap sizing is enabled, the off-heap size is calculated based on the on-heap
// size. Otherwise, the off-heap size is set to the value specified by the user (if any).
// Note that this means that we will IGNORE the off-heap size specified by the user if the
Expand All @@ -178,7 +177,11 @@ private[gluten] class GlutenDriverPlugin extends DriverPlugin with Logging {
// We will be careful to use the same configuration settings as Spark to ensure
// that we are sizing the off-heap memory in the same way as Spark sizes on-heap memory.
// The 300MB value, unfortunately, is hard-coded in Spark code.
((onHeapSize - (300 * 1024 * 1024)) *
val originalOnHeapSize: Long = conf.getSizeAsBytes(SPARK_ONHEAP_SIZE_KEY, 1024 * 1024 * 1024)
val originalOffHeapSize: Long = conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY, 0)
val totalMemorySize = originalOnHeapSize + originalOffHeapSize
conf.set(SPARK_ONHEAP_SIZE_KEY, totalMemorySize.toString)
((totalMemorySize - (300 * 1024 * 1024)) *
conf.getDouble(GLUTEN_DYNAMIC_OFFHEAP_SIZING_MEMORY_FRACTION, 0.6d)).toLong
} else {
// Optimistic off-heap sizes, assuming all storage memory can be borrowed into execution
Expand Down