GH-37876: [Format] Add list-view specification to arrow format #37877

felipecrv · 2023-09-26T15:58:12Z

Rationale for this change

More details in the draft implementations of this spec:

C++: GH-35344: [C++][Format] Implementation of the LIST_VIEW and LARGE_LIST_VIEW array formats #35345
Go: GH-35344: [Go][Format] Implementation of the LIST_VIEW and LARGE_LIST_VIEW array formats #37468

What changes are included in this PR?

Some unrelated fixes to the spec text (I can extract these to another PR if necessary)
Changes to the spec text
Additions to the Flatbuffers specifications of the Arrow format

Are these changes tested?

N/A.

Are there any user-facing changes?

Changes in documentation and backwards compatible additions to the format spec.

Closes: [Format] Add ListView to FlatBuffers and specification text #37876

github-actions · 2023-09-26T15:58:45Z

⚠️ GitHub issue #37876 has been automatically assigned in GitHub to PR creator.

…ach logical type

bkietz

Just a few nits in wording, otherwise looks good

docs/source/format/Columnar.rst

format/Schema.fbs

mapleFU

LGTM, thanks!

Co-authored-by: David Li <li.davidm96@gmail.com>

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

docs/source/format/Columnar.rst

felipecrv · 2023-10-03T15:05:11Z

Would it be enough to require that sizes[i] == 0 when i is null to call it a "valid empty list-view"

At least in Rust the rule is that a slice must have an end index less than or equal to the length of the data being sliced.

So in this case a slice would be valid iff sizes[i] + offsets[i] <= child_data[0].length().

It has been a while since I worked in C++, but if I recall correctly this is consistent with the way iterators work as well.

I will rewrite the text saying that non-empty nulls are allowed, then.

… sizes

zeroshade · 2023-10-04T20:00:16Z

The vote on the mailing list is officially passed, @bkietz you have an outstanding change requested can you take a look at the updates and update your review accordingly?

@pitrou @tustvold Any outstanding comments here or can we approve this?

tustvold · 2023-10-05T10:13:12Z

LGTM

pitrou · 2023-10-05T12:47:09Z

docs/source/format/Columnar.rst


-Each logical data type has a well-defined physical layout. Here are
-the different physical layouts defined by Arrow:
+Each logical data type has one or more well-defined physical layouts. Here


I would keep the singular. There is no disjunction in Arrow (unlike Parquet) between "logical" data type and physical layout. ListView and StringView are simply distinct types.

I will change this back to singular and all the other places I've changed it. But in the future, the "logical data type" terminology should probably be removed altogether because it's very confusing.

I definitely agree with that. The spec was often confusing to me at the start.

…outs

pitrou

Thanks a lot @felipecrv !

pitrou · 2023-10-05T14:40:40Z

@bkietz Any other comment?

conbench-apache-arrow · 2023-10-06T23:32:28Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 6d551aa.

There was 1 benchmark result indicating a performance regression:

Commit Run on ursa-i9-9960x at 2023-10-06 10:54:54Z
- dataframe-to-table (Python) with dataset=type_floats

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.

…pache#37877) ### Rationale for this change More details in the draft implementations of this spec: - C++: apache#35345 - Go: apache#37468 ### What changes are included in this PR? - Some unrelated fixes to the spec text (I can extract these to another PR if necessary) - Changes to the spec text - Additions to the Flatbuffers specifications of the Arrow format ### Are these changes tested? N/A. ### Are there any user-facing changes? Changes in documentation and backwards compatible additions to the format spec. * Closes: apache#37876 Lead-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com> Co-authored-by: David Li <li.davidm96@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr> Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>

adriangb · 2025-02-19T20:42:29Z

docs/source/format/Columnar.rst

+
+We illustrate an example of ``ListView<Int8>`` with length 4 having values::
+
+    [[12, -7, 25], null, [0, -127, 127, 50], []]


It would be nice to have an example that shows what happens with duplicate lists and duplicate values:

[[12, -7, 25], null, [0, -127, 127, 12], [], [12, -7, 25]]

@adriangb anything can happen: they can be duplicated in the data or entries can point to the same data.

Compact representation:

buffers: offsets: [0, _, 3, _, 0] sizes: [3, _, 4, 0, 3] children: values: [12, -7, 25, 0, -127, 127, 12]

Common representation:

buffers: offsets: [0, _, 3, _, 7] sizes: [3, _, 4, 0, 3] children: values: [12, -7, 25, 0, -127, 127, 12, 12, -7, 25]

using _ to indicate that the value doesn't matter

Doing de-duplication is an expensive operation, but you can imagine some kernel, by construction, producing a compact list-view array. Imagine a function that generates an array of prefixes of another array given sizes -- every offset of would be 0 and only the sizes would vary.

The main practical consequence of the ListViewArray is that lists can be written to the array in any random order. If you need to set array[i] to the logical value [a, b, c] all you have to do is append [a, b, c] to the child array and set offsets[i] and sizes[i] to the appropriate sizes. This is not possible with ListArray since an array at a random position i forces all the following values of the child array to move further.

Makes total sense, thanks!

klion26 · 2025-07-11T02:51:05Z

docs/source/format/Columnar.rst

+    * Offsets buffer (int32)
+
+      | Bytes 0-3  | Bytes 4-7   | Bytes 8-11  | Bytes 12-15 | Bytes 16-63           |
+      |------------|-------------|-------------|-------------|-----------------------|


Why does the value in Bytes 4-7 of the offset buffer is 7, does that mean the value does not matter because the validity bitmap is 0?

Yes. I tried to make it clear that you can't expect sizes or offsets to be zero on NULL lists. The only constraint is that offset+size < children[0].length

@felipecrv thanks for the reply!

felipecrv requested review from bkietz and pitrou September 26, 2023 15:58

felipecrv marked this pull request as ready for review September 26, 2023 15:58

github-actions bot added Component: Documentation awaiting review Awaiting review labels Sep 26, 2023

felipecrv changed the title ~~GH-37876: [Format] Add string-view to arrow format~~ GH-37876: [Format] Add list-view to arrow format Sep 26, 2023

felipecrv changed the title ~~GH-37876: [Format] Add list-view to arrow format~~ GH-37876: [Format] Add list-view specification to arrow format Sep 26, 2023

pitrou requested a review from wjones127 September 28, 2023 16:49

felipecrv added 5 commits September 28, 2023 17:59

Format: Point out that there might be multiple physical layouts for e…

907987c

…ach logical type

Format: Use singular like all the other items in this list

e1426cf

Format: Fix typos and remove trailing whitespace

bc4d8a4

Format: Introduce the ListView layout to the spec

f070ce6

Format: Add ListView and LargeListView to Flatbuffers spec

f384285

felipecrv force-pushed the format_list_view branch from 3fb1d4e to f384285 Compare September 28, 2023 20:59

bkietz requested changes Sep 29, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Sep 29, 2023

lidavidm reviewed Sep 29, 2023

View reviewed changes

format/Schema.fbs Outdated Show resolved Hide resolved

mapleFU approved these changes Sep 29, 2023

View reviewed changes

format/Schema.fbs: fix typo

077684a

Co-authored-by: David Li <li.davidm96@gmail.com>

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 30, 2023

Improve the text after bkietz suggestions

d88e00a

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

felipecrv force-pushed the format_list_view branch from dd6ed5f to d88e00a Compare September 30, 2023 03:53

felipecrv requested review from bkietz and lidavidm September 30, 2023 03:54

pitrou reviewed Oct 2, 2023

View reviewed changes

docs/source/format/Columnar.rst Outdated Show resolved Hide resolved

pitrou reviewed Oct 2, 2023

View reviewed changes

docs/source/format/Columnar.rst Outdated Show resolved Hide resolved

pitrou reviewed Oct 2, 2023

View reviewed changes

docs/source/format/Columnar.rst Outdated Show resolved Hide resolved

wjones127 reviewed Oct 2, 2023

View reviewed changes

docs/source/format/Columnar.rst Outdated Show resolved Hide resolved

Columnar.rst: Be more strict about offset values

42b8ec7

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 3, 2023

felipecrv added 3 commits October 3, 2023 14:44

Columnar.rst: Fix title level for 'Variadic Buffers'

4cf8201

Columnar.rst: Reinforce the message that sizes bit-width match offsets'

0d02bdc

Columnar.rst: Turn the invariants into a relation between offsets and…

c13b222

… sizes

felipecrv requested review from pitrou and tustvold October 3, 2023 18:29

pitrou reviewed Oct 5, 2023

View reviewed changes

Columnar.rst: Don't say that logical types have multiple physical lay…

dd8991a

…outs

felipecrv requested a review from pitrou October 5, 2023 14:05

pitrou approved these changes Oct 5, 2023

View reviewed changes

bkietz approved these changes Oct 5, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Oct 5, 2023

zeroshade merged commit 6d551aa into apache:main Oct 5, 2023

zeroshade removed the awaiting merge Awaiting merge label Oct 5, 2023

felipecrv deleted the format_list_view branch October 5, 2023 17:59

davisusanibar mentioned this pull request Feb 29, 2024

GH-40289: [Java] Proposal for ListViewVector implementation #40290

Closed

14 tasks

adriangb reviewed Feb 19, 2025

View reviewed changes

klion26 reviewed Jul 11, 2025

View reviewed changes


		We illustrate an example of ``ListView<Int8>`` with length 4 having values::

		[[12, -7, 25], null, [0, -127, 127, 50], []]

GH-37876: [Format] Add list-view specification to arrow format #37877

GH-37876: [Format] Add list-view specification to arrow format #37877

Uh oh!

Conversation

felipecrv commented Sep 26, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Sep 26, 2023

Uh oh!

bkietz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

felipecrv commented Oct 3, 2023

Uh oh!

zeroshade commented Oct 4, 2023

Uh oh!

tustvold commented Oct 5, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou commented Oct 5, 2023

Uh oh!

conbench-apache-arrow bot commented Oct 6, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felipecrv Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

felipecrv commented Sep 26, 2023 •

edited by github-actions bot

Loading

felipecrv Mar 11, 2025 •

edited

Loading