[Format] Add an Arrow Canonical Extension Type for Parquet Variant

[alamb]

Describe the enhancement requested

Parquet has added a new type for semi-structured data called Variant which is defined here:

• Variant encoding spec: https://github.com/apache/parquet-format/blob/main/VariantEncoding.md
• Variant shredding spec: https://github.com/apache/parquet-format/blob/main/VariantShredding.md

As it is common for engines to read data from Parquet into Arrow for in memory processing it is useful to have support for Variant in Arrow. @CurtHagenlocher proposes adding native Variant support in the Arrow format itself here:

• [Format] Consider adding an official variant type to Arrow #42069

An alternate approach is to add a Canonical Extension Type

@zeroshade wrote up a proposal

• Mailing List Discussion: https://lists.apache.org/thread/w06cxdojjcmry4m9vb0bo7owd1jsbtz5
• Google Document: https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing

And implemented an implementation in Go

• apache/arrow-go@5240503

This ticket tracks the idea of adding Variant as an official extension type

See also @neilechao 's PR to add variant read support to parquet

• [C++][Parquet] Add variant support #45937

Component(s)

Format

[alamb]

Most recent update from @zeroshade on the mailing list is:

I've
been waiting for one of the other implementations to implement the proposal
[2] before I go make a PR to add it to the docs in full. If you think it's
worthwhile for me to start drafting up a PR to add to the Canonical
Extensions right now then I'm happy to do so. I think most of the
objections to using an extension type instead of a real type are answered
or managed by the Proposal [2] and ensuring the extension type has
appropriate functional support and methods.

[ianmcook]

Closed by #47456