GH-39812: [Python] Add bindings for ListView and LargeListView #39813

danepitkin · 2024-01-26T22:58:37Z

Rationale for this change

Add bindings to the ListView and LargeListView array formats.

What changes are included in this PR?

Add initial implementation for ListView and LargeListView
Add basic unit tests

Are these changes tested?

Basic unit tests only (follow up PRs will be needed to implement full functionality)

Are there any user-facing changes?

Yes, documentation is updated in this PR to include the new PyArrow objects.

Closes: [Python] Add scaffolding for ListView and LargeListView array formats #39812

github-actions · 2024-01-26T22:59:22Z

⚠️ GitHub issue #39812 has been automatically assigned in GitHub to PR creator.

danepitkin · 2024-02-01T03:15:33Z

>>> import pyarrow as pa
>>>
>>> values = [1, 2, 3, None]
>>> offsets = [0, 1, 2]
>>> sizes = [2, 2, 2]
>>> 
>>> pa.ListViewArray.from_arrays(offsets, sizes, values)
<pyarrow.lib.ListViewArray object at 0x13ad6e200>
[
  [
    1,
    2
  ],
  [
    2,
    3
  ],
  [
    3,
    null
  ]
]
>>>
>>> pa.LargeListViewArray.from_arrays(offsets, sizes, values)
<pyarrow.lib.LargeListViewArray object at 0x13ad6e440>
[
  [
    1,
    2
  ],
  [
    2,
    3
  ],
  [
    3,
    null
  ]
]
>>>
>>> pa.list_view(pa.string())
ListViewType(list_view<item: string>)
>>> pa.large_list_view(pa.string())
LargeListViewType(large_list_view<item: string>)

jorisvandenbossche

Looks good!

jorisvandenbossche · 2024-02-01T14:10:50Z

python/pyarrow/array.pxi

I think here the additional disclaimer could be added that this also can return more values / values out of order if certain parts of the values array are not pointed to by the offsets.

In general, given the layout of the "views" list type, there is no guarantee about the content of the values here.

The ListArray.values version has a pointer to flatten(), which would be useful here as well, to explain the difference. But I assume Flatten is not yet implemented for ListView?

Flatten is not yet implemented, I figured the generic List / Array APIs can be tested and fixed in follow up PRs. I haven't marked that as a TODO anywhere.. let me know if you think it's worth adding as a comment!

I will update the comment, great idea.

jorisvandenbossche · 2024-02-01T14:12:02Z

python/pyarrow/array.pxi

Shouldn't this be a flat array?

Yes, you're right. I confused this with the logical value representation..

jorisvandenbossche · 2024-02-01T14:16:33Z

python/pyarrow/array.pxi

Ah, I hadn't expected this to work. So those get filled by ListViewArray::FromArrays? In which case this is not zero-copy?

What happens if this is passed together with a mask? EDIT: I see that is checked and raises an error in the C++ code

Hmm, yeah good point, maybe I should leave this example out to deter users from accidentally creating copies.

I will update this to use a null mask instead of None

python/pyarrow/array.pxi

jorisvandenbossche · 2024-02-01T14:18:13Z

python/pyarrow/array.pxi

This might be a left-over from copy pasting this from offsets. Or can you also pass a null for the sizes to FromArrays?

It does technically work the same. If you use None in either the offsets or sizes arrays, the value will become null.

>>> import pyarrow as pa >>> offsets = [0] >>> values = [1] >>> sizes = [None] >>> pa.ListViewArray.from_arrays(offsets, sizes, values) <pyarrow.lib.ListViewArray object at 0x13509a1a0> [ null ] >>> sizes = [0] >>> pa.ListViewArray.from_arrays(offsets, sizes, values) <pyarrow.lib.ListViewArray object at 0x13509a3e0> [ [] ]

I added this to the unit tests in test_array.py

jorisvandenbossche · 2024-02-01T14:18:43Z

python/pyarrow/array.pxi

Suggested change

>>> array.offsets

>>> array.sizes

Will need to update the below output as well.

Good catch!

python/pyarrow/tests/test_array.py

jorisvandenbossche · 2024-02-01T14:50:05Z

python/pyarrow/types.pxi

This comment can probably be generalized, mentioning that the "view" type in general might not be supported by all Arrow implementations (regardless of the size?)

danepitkin · 2024-02-06T16:55:40Z

The next PR will focus on converting python objects to ListView arrays. That will enable ListView to be added to the existing ListArray tests. For now, some temporary basic unit testing is included.

jorisvandenbossche

Looks good!

jorisvandenbossche · 2024-02-07T14:13:33Z

python/pyarrow/array.pxi

Suggested change

values taking into consideration the array's offset.

values taking into consideration the array's order and offset.

+1 I also updated LargeListViewArray.values() docstrings

jorisvandenbossche · 2024-02-07T14:14:15Z

python/pyarrow/array.pxi

Maybe also show array itself first (so you can compare with that output)?

jorisvandenbossche · 2024-02-07T14:16:29Z

python/pyarrow/array.pxi

Maybe here show both array and array.values outputs as well so it's easier to compare the different ones?

jorisvandenbossche · 2024-02-07T14:18:24Z

python/pyarrow/array.pxi

Suggested change

def flatten(self, pool=None):

def flatten(self, memory_pool=None):

We are not very consistent about it, but at least in this file we use memory_pool more than pool .. (something to clean up at some point)

jorisvandenbossche · 2024-02-07T14:19:09Z

python/pyarrow/array.pxi

Suggested change

def flatten(self, pool=None):

def flatten(self, memory_pool=None):

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

jorisvandenbossche

Thanks!

…pache#39813) ### Rationale for this change Add bindings to the ListView and LargeListView array formats. ### What changes are included in this PR? * Add initial implementation for ListView and LargeListView * Add basic unit tests ### Are these changes tested? * Basic unit tests only (follow up PRs will be needed to implement full functionality) ### Are there any user-facing changes? Yes, documentation is updated in this PR to include the new PyArrow objects. * Closes: apache#39812 Lead-authored-by: Dane Pitkin <dane@voltrondata.com> Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

github-actions bot added the awaiting review Awaiting review label Jan 26, 2024

github-actions bot added Component: Python Component: Documentation labels Jan 29, 2024

danepitkin changed the title ~~GH-39812: [Python] Add ListView and LargeListView~~ GH-39812: [Python] Add scaffolding for ListView and LargeListView Jan 30, 2024

danepitkin requested review from AlenkaF and jorisvandenbossche January 31, 2024 22:45

jorisvandenbossche reviewed Feb 1, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Feb 1, 2024

danepitkin requested a review from jorisvandenbossche February 6, 2024 16:52

jorisvandenbossche changed the title ~~GH-39812: [Python] Add scaffolding for ListView and LargeListView~~ GH-39812: [Python] Add bindings for ListView and LargeListView Feb 7, 2024

jorisvandenbossche reviewed Feb 7, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Feb 7, 2024

Dane Pitkin added 9 commits February 7, 2024 16:38

apacheGH-39812: [Python] Add ListView and LargeListView

13e0e40

Add ListViewType and LargeListViewType

f8fa96b

Implement ListViewScalar and LargeListViewScalar

0394963

Remove debug statement

c03abe2

Remove debug include

bd81932

Remove debug statement

6248f6b

Remove breakpoint

d1f6ab1

Fix doctests

bfe5670

Add ListViewArray and LargeListViewArray

ce1e3da

Dane Pitkin and others added 9 commits February 7, 2024 16:38

Fix linter and remove unnecessary whitespace edit

e2bd202

Fix docstrings

0bdf95a

Update python/pyarrow/array.pxi

3e2bba3

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

Update docstrings, update comments, fix test cases

79c0698

Add flatten() api

8131123

Fix linter

5e87ec1

Fix docstrings

2990362

Fix docstrings properly..

6a53242

Improve docstring examples

9bf1673

danepitkin force-pushed the danepitkin/python-listview branch from 3155cea to 9bf1673 Compare February 7, 2024 21:39

jorisvandenbossche approved these changes Feb 8, 2024

View reviewed changes

jorisvandenbossche merged commit 42e35f1 into apache:main Feb 8, 2024

jorisvandenbossche removed the awaiting change review Awaiting change review label Feb 8, 2024

github-actions bot added the awaiting merge Awaiting merge label Feb 8, 2024

This comment was marked as resolved.

Sign in to view

	values taking into consideration the array's offset.
	values taking into consideration the array's order and offset.

	def flatten(self, pool=None):
	def flatten(self, memory_pool=None):

GH-39812: [Python] Add bindings for ListView and LargeListView #39813

GH-39812: [Python] Add bindings for ListView and LargeListView #39813

Uh oh!

Conversation

danepitkin commented Jan 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Jan 26, 2024

Uh oh!

danepitkin commented Feb 1, 2024

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danepitkin commented Feb 6, 2024

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

danepitkin commented Jan 26, 2024 •

edited

Loading