Skip to content

[C++] Investigate batching filter node output #30994

@asfimport

Description

@asfimport

A highly selective filter could emit tiny batches. Running with tiny batches we have a lot of overhead.

On the other hand, merging batches together requires allocation and copies so it comes with overhead of its own.

When things are more stable we should investigate whether there is some threshold where we merge batches together after a filter.

Reporter: Weston Pace / @westonpace

Note: This issue was originally created as ARROW-15521. Please see the migration documentation for further details.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions