Skip to content

[Rust] [Parquet] Support v2 LogicalType #27654

@asfimport

Description

@asfimport

We currently do not read nor write the version 2 logical types. This is mainly because we do not have a mapping for it from parquet-format-rs.

To implement this, we can:

  • convert "parquet::basic::LogicalType" to "parquet::basic::ConvertedType"

  • implement "parquet::basic::LogicalType" which mirrors "parquet_format::LogicalType"

  • create a mapping between ConvertedType and LogicalType

  • write LogicalType to "parquet_format::SchemaElement" if v2 of the writer is used

    This would be a good starting point for implementing 2.6 types (UUID, NANOS precision time & timestamp).
    Follow-up work would be:

  • parsing v2 of the schema [ARROW-11365]

  • Using v2 in the Arrow writer (mostly schema conversion)

  • Supporting nanosecond precision time & timestamp

Reporter: Neville Dipale / @nevi-me
Assignee: Neville Dipale / @nevi-me

PRs and other links:

Note: This issue was originally created as ARROW-11803. Please see the migration documentation for further details.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions