Skip to content

Read out only the required columns from a Feather file on Disk #28827

@asfimport

Description

@asfimport

The LZ4 compressed feather format works really well and is quite nice and fast than Parquet when reading all the columns out. For single columns, looks like feather format does not yet support the capability to read out only the required columns from the disk. Are there any plans to add support for this? Here are some numbers to support my claim from an experiment with a table with 17 columns and 5 million rows in uncompressed parquet and LZ4 compressed feather format. No memory mapping involved.

 
pq_all_cols: 0.4179724836349487 ms
feather_all_cols: 0.26202451705932617 ms
pq_single_col: 0.10951032638549804 ms
feather_single_col: 0.2119576358795166 ms

Reporter: Jayjeet Chakraborty / @JayjeetAtGithub

Related issues:

Note: This issue was originally created as ARROW-13126. Please see the migration documentation for further details.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions