Skip to content

[C++][Python] Do not collect python iterators if not necessary #26030

@asfimport

Description

@asfimport

When converting python objects to arrow array currently we always collect the input to a sequence, but this may be memory consuming in certain cases.

For unknown sized iterators we could consume and temporarily store the seen items during inference potentially improving both the conversion time and peak memory usage.

Reporter: Krisztian Szucs / @kszucs

Related issues:

Note: This issue was originally created as ARROW-10006. Please see the migration documentation for further details.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions