[DISCUSSION] Parquet Metadata Improvements

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
As we work on various features of Parquet metadata it is becoming clear that working with the current code organization is challenging.

I just wanted to write down some of my thoughts about how it all fits together

Here are some challenges:

1. The naming is challenging Consistent naming for Parquet page index structures #6097
2. There is no way to easily write to bytes outside the context of a parquet file: Add ParquetMetadataWriter allow ad-hoc encoding of ParquetMetadata #6000
3. It is complicated to understand how to read optional parts of the metadata that are not inlined (e.g. OffsetIndexes) - Document when the ParquetRecordBatchReader will re-read metadata #5887
4. If we ever wanted to speed up (e.g. Use custom thrift decoder to improve speed of parsing parquet metadata #5854) it would be hard with the current structure
5. There is not always a 1-1 correspondence between file::metadata and the thrift structures in format::metadata,

Describe the solution you'd like
I would like to propose

1. We continue to clarify the distinction between file::metadata and format::metadata
2. Improve the API to translate back and forth between them and bytes and de-emphasize the conversion between thrift structures

Maybe this is clear to others but it is not to me

Here is how I see the structures involved:

                                ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐               ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─    
                                  ┌──────────────┐                         ┌───────────────────────┐ │   
                                │ │ ColumnIndex  │        │               ││    ParquetMetaData    │     
                                  └──────────────┘                         └───────────────────────┘ │   
  ┌──────────────┐              │ ┌────────────────┐      │               │┌───────────────────────┐     
  │   ..0x24..   │  ◀────────▶    │  OffsetIndex   │          ◀────────▶   │    ParquetMetaData    │ │   
  └──────────────┘              │ └────────────────┘      │               │└───────────────────────┘     
                                           ...                                       ...             │   
                                │ ┌──────────────────┐    │               │ ┌──────────────────┐         
bytes                             │  FileMetaData*   │                      │  FileMetaData*   │     │   
(thrift encoded)                │ └──────────────────┘    │               │ └──────────────────┘         
                                 ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─                 ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘   
                                                                                                         
                                     format::meta structures               file::metadata structures         
                                                                                                         
                                                                                                         
                                                     * Same name, different struct                       
                                                                                                         

I would like to focus on improving the API for going back/forth between bytes and the file::metadata structures

                                                  ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─    
                                                   ┌───────────────────────┐ │   
┌──────────────┐                                  ││    ParquetMetaData    │     
│   ..0x24..   │           ◀────────▶              └───────────────────────┘ │   
└──────────────┘                                  │┌───────────────────────┐     
                                                   │    ParquetMetaData    │ │   
                        Would like to focus       │└───────────────────────┘     
 bytes                  on this API to/from                                  │   
 (thrift encoded)       bytes and the             │ ┌──────────────────┐         
                        file::metadata              │  FileMetaData*   │     │   
                                                  │ └──────────────────┘         
                                                   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘   
                                                                                 
                                                   file::metadata structures     
                                                                                 

Describe alternatives you've considered
I think we probably need at least two different APIs:

Reading

1. One that writes to [u8] buffered in memory ( decode_footer and decode_metadata)
2. One that reads from an AsyncReader or something equivalent (MetadataLoader is enough / needs some more information)

Writing

1. Writes to [u8] API for encoding/decoding ParquetMetadata with more control #6002)
2. Writes to an AsyncWriter perhaps

Additional context

[alamb]

Here is a down payment for documentation #6184

[jp0317]

Thanks @alamb! IMHO the current file::metadata also lacks some features that might be helpful (and are available in c++ implementation). For instance, getting a complete Thrift-serialized representation of the FileMetaData, finding the index given the column dot-string path, finding the parent of a field.

EDITED: I just saw #6197 which feels relevant to the 1st one. For the last one, there's a Type struct in the codes which seems similar to the C++ Node. I'm currently am not sure how complex or whether it worths the effort to support a field-tree with parent info in the current codes? But a simple way might be maintaining a Vec<Option<TypePtr>> that marks the parent of each Type, while adding an index to each Type?

[alamb]

EDITED: I just saw #6197 which feels relevant to the 1st one.

I agree. We are also looking for help with the reading portion -- see comments on #6002 cc @adriangb

finding the index given the column dot-string path,

There is something similar here https://docs.rs/parquet/latest/parquet/arrow/fn.parquet_column.html but adding a real API that handles the field resolution logic for nested fields would be very nice. Perhaps you can file a ticket requesting this feature (I have found clearly worded tickets are very often picked up by people in this community)

For the last one, there's a Type struct in the codes which seems similar to the C++ Node. I'm currently am not sure how complex or whether it worths the effort to support a field-tree with parent info in the current codes? But a simple way might be maintaining a Vec<Option> that marks the parent of each Type, while adding an index to each Type?

I am not familiar with the usecase for finding the parent of a field so I don't have much to add to this

[alamb]

I think we have addressed all the challenges described in this ticket, so let's close it down for now