Description
In Gluten's query processing framework, we have applied the following Arrow-related data transitions that are actually implicit to users:
Spark-Arrow to Native-Arrow through Java API ColumnarBatches#ensureOffloaded
Native-Arrow / Native-Velox to Spark-Arrow through Java API ColumnarBatches#ensureLoaded
Native-Arrow to Native-Velox though C++ API VeloxColumnarBatch::from
These implicit transitions are developer-friendly but make performance tunning hard. As we have adopted transition API in Gluten's query planner, now it becomes possible to pull up the work of adding these transitions to query planning phase.
Description
In Gluten's query processing framework, we have applied the following Arrow-related data transitions that are actually implicit to users:
Spark-ArrowtoNative-Arrowthrough Java APIColumnarBatches#ensureOffloadedNative-Arrow/Native-VeloxtoSpark-Arrowthrough Java APIColumnarBatches#ensureLoadedNative-ArrowtoNative-Veloxthough C++ APIVeloxColumnarBatch::fromThese implicit transitions are developer-friendly but make performance tunning hard. As we have adopted transition API in Gluten's query planner, now it becomes possible to pull up the work of adding these transitions to query planning phase.