[VL] Move pre-configuration code of dynamic off-heap sizing to its own place#9336
Conversation
zhztheplayer
force-pushed
the
wip-dynamic-sizing
branch
from
2a8c0fa to
d7a2dba
Compare
|
Run Gluten Clickhouse CI on x86 |
zhztheplayer
changed the title
|
Run Gluten Clickhouse CI on x86 |
zhouyuan
left a comment
zhouyuan
left a comment
There was a problem hiding this comment.
Thanks, the logic on dynamic off-heap sizing is cleaner.
|
|
||
| val DYNAMIC_OFFHEAP_SIZING_ENABLED = | ||
| buildConf("spark.gluten.memory.dynamic.offHeap.sizing.enabled") | ||
| buildStaticConf("spark.gluten.memory.dynamic.offHeap.sizing.enabled") |
There was a problem hiding this comment.
is this a behavior change by changing this to static config?
There was a problem hiding this comment.
Not really, it's just an improvement for configuration definitions.
| // Pessimistic off-heap sizes, with the assumption that all non-borrowable storage memory | ||
| // determined by spark.memory.storageFraction was used. | ||
| val fraction = 1.0d - conf.getDouble("spark.memory.storageFraction", 0.5d) | ||
| val conservativeOffHeapPerTask = (offHeapSize * fraction).toLong / taskSlots |
There was a problem hiding this comment.
it looks like changed the scope: conservativeOffHeapPerTask is effective on DYNAMIC_OFFHEAP_SIZING_ENABLED=false on previous code
There was a problem hiding this comment.
We don't have to deal with off-heap size in the future anymore since we no longer rely on off-heap memory manager when the feature is on.
We are just recovering the old code here before off-heap sizing was landed, so it's totally safe: https://github.com/apache/incubator-gluten/blame/022c208564dbb6dc59b7d64fbc4ad002c166e012/gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala
|
Cc: @zhli1142015 to be aware of this |
…n place (apache#9336) (cherry picked from commit 083ecfb) Change-Id: I1f03d59860a2a966cf4197de41ba6ce3dfd45e60 Reviewed-on: https://bigdataoss-internal-review.googlesource.com/c/third_party/apache/incubator-gluten/+/115779 Reviewed-by: Revanth Venkat Mikkilineni <revanthvenkat@google.com> Tested-by: Srinivas S T <srst@google.com> Reviewed-by: Preetesh Verma <preeteshverma@google.com>
2 participants
Dynamic off-heap sizing was introduced in #5439. The PR does a few of improvements against the implementation.
GlutenConfig.scalatoDynamicOffHeapSizingMemoryTarget.DynamicOffHeapSizingMemoryTargetwith Spark consumer since it basically never triggers spills because of its design. The target will manage OOMs by itself.