Add support for FixedSizeList to variant_to_arrow by rishvin · Pull Request #9663 · apache/arrow-rs

rishvin · 2026-04-06T02:17:21Z

Closes [Variant] Add variant_to_arrow FixedSizeList type support #9531

Rationale for this change

Add support for FixedSizeList when invoking variant_to_arrow.

What changes are included in this PR?

Introduces a new builder VariantToFixedSizeListArrowRowBuilder.
Adds test cases for shredding and getting variant by FixedSizeList.

Are these changes tested?

By adding few test cases.

Are there any user-facing changes?

N/A.

alamb · 2026-04-06T19:47:47Z

@sdf-jkl could you help review this PR?

sdf-jkl

Thanks @rishvin, I left some suggestions.

rishvin · 2026-04-11T21:25:08Z

Thanks @sdf-jkl for reviewing changes. I have addressed the comments, could you re-review ?

sdf-jkl

Thanks for your work on this @rishvin! Everything looks great, I left some nits.

sdf-jkl · 2026-04-13T14:31:53Z

 }

 impl<'a> ListElementBuilder<'a> {
+    fn append_null(&mut self) -> Result<()> {


Maybe add a comment mentioning this is only used for FixedSizeList builder?

Co-authored-by: Konstantin Tarasov <33369833+sdf-jkl@users.noreply.github.com>

rishvin · 2026-04-13T19:24:37Z

Thanks @sdf-jkl addressed new comments.

sdf-jkl

One tiny change.

Co-authored-by: Konstantin Tarasov <33369833+sdf-jkl@users.noreply.github.com>

rishvin · 2026-04-14T04:25:39Z

Thanks again @sdf-jkl ! Addressed it.

sdf-jkl

Thanks @rishvin! LGTM.

@scovich could you please check this when available? Thanks!

scovich · 2026-04-14T15:33:25Z

+                // With `safe` cast option set to false, appending list of wrong size to
+                // `typed_value_builder` of type `FixedSizeList` will result in an error. In such a
+                // case, the provided list should be appended to the `value_builder.


I'm not sure the variant shredding spec allows for shredding as a fixed size list, if the resulting layout differs physically from a normal list?

Arrays can be shredded by using a 3-level Parquet list for typed_value.

If the value is not an array, typed_value must be null. If the value is an array, value must be null.

It looks to me like any attempt to shred as fixed-sized list must either succeed (if the size is correct) or hard-fail (because value as fallback is not allowed).

It differs physically on the Arrow side, but once we write it to Parquet it'd be same as other ListLikeArrays. But this leads to further discussion on adding FixedSizeList support for VariantArray as well as implementing other types, currently not supported in spec.

We're keeping value because we consider this a cast from Variant to FixedSizeList. The extra len check is there because there is no Variant::FixedSizeList enum to match to. If len is incorrect we consider the cast failed and proceed following the safe cast option as if typed_value is Null.

When casting from variant to arrow, we can do whatever we want.

But this code here is about going from binary variant to shredded variant. And the variant shredding spec directly forbids value to contain a variant array, when shredding as array.

True. I think the core issue is that Parquet currently has only one logical LIST type. If Parquet had a dedicated logical type for FixedSizeList, the spec wording could be more explicit.

Btw, there’s ongoing work on this too: apache/parquet-format#241 (recently revived).

Given the current spec text:

Arrays can be shredded by using a 3-level Parquet list for typed_value. If the value is not an array, typed_value must be null. If the value is an array, value must be null.

I read “array” as "a value matching the specific list shape we’re shredding into". For List/LargeList/ListView it's List values, for FixedSizeList array it's a FixedSizeList value.

From what I understand, the variant spec neither knows nor cares about the intricacies of arrow array types (it also doesn't care about spark or SQL). If we're shredding to a 3-level parquet list, and we encounter a variant array value, the resulting value column entry must be null.

I don't know...

My gut reaction is to forbid it completely because it's preferable to fail early and consistently rather than blow up unpredictably partway through shredding. The spec forbids us to fallback to value when the length is wrong, and we can't emit NULL either (regardless of safe casting options) because shredding is supposed to be lossless (it isn't a converting cast).

But I also don't have any intuition of actual use cases for this feature, to weigh its benefits against the downsides.

NOTE: The same dilemma applies to fixed length binary and fixed length string, except the spec technically doesn't forbid us to fallback to the value column. But how would we actually use it?

Thought experiment

Suppose we could use the value column as fallback in all three cases? Then the writer produces a shredded variant column where all wrong-length values are in value column (along with the wrong-type values) while all right-type-right-length values are in typed_value. But what good does it do? A reader who comes along later has no way to know about any of this, because there's nothing in the parquet that reliably indicates this was a fixed length array/binary/string column. Sure, we might embed arrow metadata, but non-arrow readers would ignore it and even arrow readers are on shaky ground trusting it because this is something entirely outside the variant spec.

Overall, enforcing list/binary/string length feels like a logical validation issue while shredding is a physical operation. And I suspect it's best to leave client code to enforce logical validity of all forms, not try to encode it in the type system. Additionally, It feels like fixed-len-xxx types fall in the same category as unsigned or non-zero types. True, arrow has unsigned int types but it lacks non-zero types. And parquet has neither.

True, the use case should be the first thing we think about before implementing.

FSL could be useful for embeddings and tensors but the question is why would anyone put them inside a Variant for such a heavy compute work? Maybe coordinates inside a log struct?

The fact that we can't store FSL in Parquet is a big reason to not do it.

FixedSizeBinary actually does have a Parquet representation - FIXED_LEN_BYTE_ARRAY and is used for shredding into a UUID, but other than UUID, what could be a use case for storing inside a Variant?

Overall, I think we'd not gain much by including FSL because it's so frail and the use cause is not defined. Isn't the whole point of Variant is to host semi-structured data, if we have fixed-len-something it'd be better to keep a separate column for it. If fixed-length is a logical constraint rather than a physical one, enforcement belongs downstream.

Overall, I think we'd not gain much by including FSL because it's so frail and the use cause is not defined. Isn't the whole point of Variant is to host semi-structured data, if we have fixed-len-something it'd be better to keep a separate column for it. If fixed-length is a logical constraint rather than a physical one, enforcement belongs downstream.

Agree. I just realized that somebody could create a FSL of [shredded] variant, if the use case just needs flexible typing of N list members. But that also seems somewhat contrived, to have such strongly-typed structure but sufficient uncertainty about the list element type to merit using variant.

Should we then keep it hard fail and leave it to the user?

Catching up late on this. I'm aligned with keeping it a hard fail and leaving it to the user.

Add support for FixedSizeList to variant_to_arrow

808b1b5

github-actions Bot added the parquet-variant parquet-variant* crates label Apr 6, 2026

Merge branch 'main' into fixed_size_list_variant_to_arrow

1d55ade

rishvin and others added 2 commits April 6, 2026 13:45

Fixed test case

11ffd40

Merge branch 'main' into fixed_size_list_variant_to_arrow

4a84d12

sdf-jkl suggested changes Apr 7, 2026

View reviewed changes

rishvin and others added 2 commits April 11, 2026 09:22

Merge branch 'main' into fixed_size_list_variant_to_arrow

64bf4aa

Addressed comments.

99c6cc5

rishvin requested a review from sdf-jkl April 11, 2026 21:24

Merge branch 'main' into fixed_size_list_variant_to_arrow

47f99c5

sdf-jkl reviewed Apr 13, 2026

View reviewed changes

rishvin and others added 2 commits April 13, 2026 11:53

Apply suggestions from code review

384ed29

Co-authored-by: Konstantin Tarasov <33369833+sdf-jkl@users.noreply.github.com>

Addressed remanining comments

220427f

rishvin requested a review from sdf-jkl April 13, 2026 19:24

Merge branch 'main' into fixed_size_list_variant_to_arrow

acd294a

sdf-jkl reviewed Apr 14, 2026

View reviewed changes

Comment thread parquet-variant-compute/src/shred_variant.rs Outdated

Update parquet-variant-compute/src/shred_variant.rs

d2e4dd6

Co-authored-by: Konstantin Tarasov <33369833+sdf-jkl@users.noreply.github.com>

rishvin requested a review from sdf-jkl April 14, 2026 04:25

sdf-jkl approved these changes Apr 14, 2026

View reviewed changes

scovich reviewed Apr 14, 2026

View reviewed changes

Conversation

rishvin commented Apr 6, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Apr 6, 2026

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rishvin commented Apr 11, 2026

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rishvin commented Apr 13, 2026

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rishvin commented Apr 14, 2026

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

scovich Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Thought experiment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rishvin commented Apr 6, 2026 •

edited by alamb

Loading

scovich Apr 14, 2026 •

edited

Loading

sdf-jkl Apr 14, 2026 •

edited

Loading