[Variant] WIP Tests for variant_get of shredded variants

[alamb]

Which issue does this PR close?

• Part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736
• Part of [Variant] Support variant_get kernel for shredded variants #7941

Rationale for this change

In order to make progress on accessing shredded variants, I think we need some tests showing how they are used and constructed. I am hoping some tests will help us along in figuring out how to implement this efficiently

I am hoping someone can use these tests to start implement efficient variant_get for shredded values

What changes are included in this PR?

Implement varaint_get tests based on the examples in the arrow variant doc from @zeroshade (see apache/arrow#46908)

Are these changes tested?

They are only tests

Are there any user-facing changes?

No

[alamb]

FYI @Samyak2 @scovich @friendlymatthew and @klion26 and @carpecodeum as I think you are interested in this feature

[Samyak2]

From what I understood of the spec, the value field needs to be present but it can have nulls. Did I understand it incorrectly? If not, this struct would be an invalid variant

[alamb]

My reading of https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/ was that value was optional in the case the variant shredded "perfectly"

I may also be misreading it.

Maybe @zeroshade has some time to coment

[scovich]

The shredding spec definitely says both columns are optional:

Both value and typed_value are optional fields used together to encode a single value.

Additionally:

Values in the two fields must be interpreted according to the following table:

value	typed_value	Meaning
null	null	The value is missing; only valid for shredded object fields
non-null	null	The value is present and may be any type, including null
null	non-null	The value is present and is the shredded type
non-null	non-null	The value is present and is a partially shredded object

As always in parquet, a physically missing column is interpreted as all-null.

[scovich]

CC @cashmand in case I totally misunderstood something

[cashmand]

Agreed, the intent is that a missing value column should be treated as equivalent to a present-but-all-null value column.

[zeroshade]

Correct, both are optional fields, but it's required to have at least one of them.

[Samyak2]

Got it. Thanks all!

[alamb]

The PR just adds tests here -- to show what I think the output of variant_get should be for different variant cases.

[carpecodeum]

Could we implement the approach @scovich suggested about invoking primitive append_xxx methods directly instead of using intermediate VariantBuilder?

[carpecodeum]

something like

pub fn append_variant<'m, 'v>(&mut self, variant: impl Into<Variant<'m, 'v>>) {
    let variant = variant.into();
    match variant {
        Variant::Int32(v) => self.append_int32(v),
        Variant::String(v) => self.append_string(v),
        Variant::Object(obj) => {
            // Build object directly into our buffers without intermediate VariantBuilder
            self.append_object_start();
            for field in obj.iter() {
                self.append_variant(field.value()); // recursive
            }
            self.append_object_end();
        }
        // ... other variants
    }
}

[alamb]

If you are concerned about the intermediate buffers, I think after the following two PRs are merged:

• Convert JSON to VariantArray without copying (8 - 32% faster) #7911
• [Variant] Avoid extra allocation in object builder #7935

There will be a way to append variant values to the final output buffer directly without copying to intermediate buffers

Is there another potential part of your question?

[carpecodeum]

Got it, will follow these PRs then

[klion26]

If the GetOptions points to the leaf node, will the result here be equal to the result in test get_variant_perfectly_shredded_int32_as_int32