[Variant] Support typed access for other types in `variant_get`

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736

We introduced the basic variant_get kernel in and now we need to expand its feature set

• [Variant] Add variant_get and Shredded VariantArray #8021

In particular, variant_get allows extracting variants (or subpaths) as typed arrays.

Describe the solution you'd like
I would like variant_get to support shredded arrays, along with being able to extract elements from them.

So roughly that means, implement VariantOutputBuilder::typed for the rest of the input variant types (Currently only Int32 is supported)

The type list is

1. PrimitiveTypes: DataType::Int32, etc
2. StringTypes: DataType::Utf8, Utf8View, etc
3. BinaryTypes: Binary, BinaryView, etc
4. Boolean
5. Time Types
6. Date Types
7. Timestamp Types

I recommend doing this as a few PRs adding different subsets of types

Describe alternatives you've considered

1. Add a test to create a VariantArray with the relevant Variant:: types, and then extract them as the desired type (e.g. make a VariantArray with Variant::Int32s and then extract it as a DataType::Int32

Note you should be able to extract to different types

So for example, a VariantArray with

Variant::Int32(23)
Variant::Int64(34)

SHould be extractable as both an Int64Array

Int64Array: [23, 34]

And an Int32Array

Int32Array: [23, 34]

Additional context
Reference

• Variant Spec
• Variant Shredding Spec
• Representing Variant In Arrow Proposal

[superserious-dev]

take

[alamb]

Note this is related to

• [Variant] Support Shredded Objects in variant_get: typed path access (STEP 1) #8150

Which covers implementing the path access

[superserious-dev]

@alamb, I think the casting behavior(eg. the type-widening of Int32 -> Int64) could make sense to implement in #8086 rather than this issue. Casting seems to be an extra step after typed access with the construction of a primitive output builder.

arrow-rs/parquet-variant-compute/src/variant_get/output/mod.rs

Lines 76 to 86 in 21a9a2a

	// handle typed output
	match as_type.data_type() {
	DataType::Int32 => Ok(Box::new(PrimitiveOutputBuilder::<Int32Type>::new(
	path,
	as_type,
	cast_options,
	))),
	dt => Err(ArrowError::NotYetImplemented(format!(
	"variant_get with as_type={dt} is not implemented yet",
	))),
	}

[alamb]

Makes sense to me

[sdf-jkl]

Seems like there are a lot of types that still need typed_value_to_variant support:

• - Unsigned ints (UInt8/16/32/64)
• - Decimal256
• - Timestamp(Second|Millisecond)
• - Date64
• - Time32(Second|Millisecond), Time64(Nanosecond)
• - Binary, LargeBinary
• - Nested/complex: List, LargeList, ListView, LargeListView, FixedSizeList, Struct, Map, Union, Dictionary, RunEndEncoded

We should look into a refactor to reuse existing arrow_to_variant builders and deduplicate the same logic.

[sdf-jkl]

We should look into a refactor to reuse existing arrow_to_variant builders and deduplicate the same logic.

This happens to be more painful that I thought it'd be