GH-46908: [Docs][Format] Add variant extension type docs

[zeroshade]

Rationale for this change

To support the addition of the Parquet Variant data type and the Iceberg adoption of the variant type, we need a defined way to pass this data through Arrow-compatible systems. As such, we need a specification for a canonical Arrow extension type to represent Variant data.

What changes are included in this PR?

Updates to the docs which define the Arrow Variant Extension type

• GitHub Issue: [Format] Add an Arrow Canonical Extension Type for Parquet Variant #46908

[github-actions]

⚠️ GitHub issue #46908 has been automatically assigned in GitHub to PR creator.

[zeroshade]

I'll wait for a few reviews on this before I send something on the mailing list to start a vote.

[ianmcook]

Higher up in this doc, can you please make this change:

The specification text to be added must follow these requirements:

1. It must define a well-defined extension name which must start with an allowed prefix. The currently allowed prefixes are:
   • "arrow." - For general-purpose canonical extension types.
   • "parquet." - For canonical extension types that are intended primarily for interoperability with Apache Parquet format.

[ianmcook]

For uses of the word "variant" outside of code, please standardize the case on "Variant" or "variant" consistently. Right now it's a mix.

[lidavidm]

We implicitly describe nulls here and then explicitly describe it below; would it make sense to explicitly introduce nulls first (and link to the docs) and then omit both of the other explanations?

[zeroshade]

Is the updated version better?

[amoeba]

Couple more comments.

[amoeba]

It sounds like applications don't have to reorder but may choose to and would need to be aware that they must access fields by name and not position. 1:1 compatibility with what Parquet is doing sounds like the better tradeoff (Option 2 above).

Could we extend the existing Note about field order or add another Note succinctly explaining this? i.e., answer @emkornfield question in the final document.

[alamb]

Sorry I have been out the last week -- I will review this shortly

[alamb]

For anyone who missed it this proposal was voted and approved: https://lists.apache.org/thread/44o0d3nvxx0y3fzoschny96k5f3mzvlb

(thank you @zeroshade for driving this)

[zeroshade]

Once we have consensus and approval here, we can merge this. Can everyone please take a look and try to comment on this (or give approval) by the end of the week?

[alamb]

looks good to me -- thank you @zeroshade for driving this

we have made good use of the extension type draft documentation in arrow-rs

FYI @scovich, @klion26 @codephage2020 @carpecodeum and @liamzwbao as you have contributed to the arrow-rs implementation and might be interested

[kou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 44da918

Submitted crossbow builds: ursacomputing/crossbow @ actions-fdb42d9677

Task	Status
preview-docs

[codephage2020]

Looks Good To Me! Thank you!

[kou]

Preview: http://crossbow.voltrondata.com/pr_docs/47456/format/CanonicalExtensions.html

[zeroshade]

Thanks everyone for the reviews and assistance making this happen!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6c0c3cd.

There weren't enough matching historic benchmark results to make a call on whether there were regressions.

The full Conbench report has more details.