Skip to content

StatisticsConverter does not support nested fields (e.g. struct fields) #9539

@friendlymatthew

Description

@friendlymatthew

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

StatisticsConverter::try_new cannot extract Parquet statistics for nested fields. Internally it calls parquet_column() which explicitly bails on nested types:

if field.data_type().is_nested() {
// Nested fields are not supported and require non-trivial logic
// to correctly walk the parquet schema accounting for the
// logical type rules - <https://github.com/apache/parquet-format/blob/master/LogicalTypes.md>
//
// For example a ListArray could correspond to anything from 1 to 3 levels
// in the parquet schema
return None;
}

There is no alternative constructor that accepts a Parquet leaf column index directly, so callers that have already resolved the mapping (like Datafusion's row filter logic) have no way to use StatisticsConverter for nested fields

I'd like to propose adding StatisticsConverter::from_column_index(parquet_column_index, arrow_field, parquet_schema) constructor that bypasses schema resolution and accepts a pre-resolved leaf col index directly

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelog

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions