Skip to content

[VL] Explicit Arrow transitions that are viewable in query plan and Spark SQL UI #7313

@zhztheplayer

Description

@zhztheplayer

Description

In Gluten's query processing framework, we have applied the following Arrow-related data transitions that are actually implicit to users:

  1. Spark-Arrow to Native-Arrow through Java API ColumnarBatches#ensureOffloaded
  2. Native-Arrow / Native-Velox to Spark-Arrow through Java API ColumnarBatches#ensureLoaded
  3. Native-Arrow to Native-Velox though C++ API VeloxColumnarBatch::from

These implicit transitions are developer-friendly but make performance tunning hard. As we have adopted transition API in Gluten's query planner, now it becomes possible to pull up the work of adding these transitions to query planning phase.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions