Skip to content

[Variant] Add variant_get and Shredded VariantArray#8021

Merged
alamb merged 18 commits intoapache:mainfrom
alamb:alamb/variant_shredding_strawman
Aug 7, 2025
Merged

[Variant] Add variant_get and Shredded VariantArray#8021
alamb merged 18 commits intoapache:mainfrom
alamb:alamb/variant_shredding_strawman

Conversation

@alamb
Copy link
Copy Markdown
Contributor

@alamb alamb commented Jul 29, 2025

Which issue does this PR close?

Rationale for this change

This is has a proposal for how to structure shredded VariantArrays and the variant_get kernel

If people like the basic idea I will file some more tickets to track additional follow on work

It is based on ideas ideas from @carpecodeum in #7946 and @scovich in #7915

I basically took the tests from #7965 and the conversation with @scovich recorded from #7941 (comment) and I bashed out how this might look

What changes are included in this PR?

  1. Update VariantArray to represent shredding
  2. Add code to variant_get to support extracting paths as both variants and typed fields
  3. A pattern that I think can represent shredding and extraction
  4. Tests for same

Note there are many things that are NOT in this PR that I envision doing as follow on PRs:

  1. Support and implementing Paths
  2. Support for shredded objects
  3. Support shredded lists
  4. Support nested objects / lists
  5. Full casting support
  6. Support for other output types: StringArray, StringViewArray, etc
  7. Many performance improvements

Are these changes tested?

Yes

Are there any user-facing changes?

New feature

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Variant] Support variant_get kernel for shredded variants

4 participants