[Python] Limit the size of the repr for large Tables

[asfimport]

The new repr is nice that it shows a preview of the data, but this can also become very long flooding your console output for larger tables.

We already default to 10 preview cols, but each column can still consist of many chunks. So it might be good to also limit it to 2 chunks?

The ChunkedArray.to_string method already has a window keyword, but that seems to control both the number of elements to show per chunk as the number of chunks (while it would be nice to limit eg to 2 chunks but show up to 10 elements for each chunk).

cc @amol-

Reporter: Will Jones / @wjones127
Assignee: Will Jones / @wjones127

Related issues:

• [Python] Add character limit to ChunkedArray repr (is a parent of)

PRs and other links:

• GitHub Pull Request #12091

_{Note: This issue was originally created as ARROW-14798. Please see the migration documentation for further details.}

[asfimport]

Will Jones / @wjones127:
What do you think of adding an inner_window field to {}PrettyPrintOptions{}? Then in the case of chunked_array, window would refer to number of chunks to show and inner_window would refer to elements to show within those chunks.

cc @xhochy 

Example of table with repr that's a little too long as-is:

import pyarrow
import string

def print_table(size: int, chunks: int):
    t = pyarrow.table({
        name: pyarrow.chunked_array([range(size) for _ in range(chunks)])
        for name in string.ascii_lowercase
    })
    print(t)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Yes, it indeed sounds as a good start to distinguish both aspects with separate options

[asfimport]

Will Jones / @wjones127:
The thing about inner window is it only helps at one level. Another way to think about the problem is that we care more about seeing primitive elements than about container elements. So we could instead have container_window and {}window{}, where the latter only applies to primitives.

So the window and inner_window would work like:

// window: 1, inner_window: 3
>>> pa.chunked_array([[list(range(10))] * 10] * 10)# 10 chunks, 10 elements, 10 list-elements
[[[0, 1, 2,...,7, 8, 9], [0, 1, 2,...,7, 8, 9], [0, 1, 2,...,7, 8, 9],...,[0, 1, 2,...,7, 8, 9], [0, 1, 2,...,7, 8, 9], [0, 1, 2,...,7, 8, 9]],
 ...,
[[0, 1, 2,...,7, 8, 9], [0, 1, 2,...,7, 8, 9], [0, 1, 2,...,7, 8, 9],...,[0, 1, 2,...,7, 8, 9], [0, 1, 2,...,7, 8, 9], [0, 1, 2,...,7, 8, 9]]]
// container_window: 1, window: 4
>>> pa.chunked_array([[['a'] * 10] * 10] * 10) # 10 chunks, 10 elements, 10 list-elements
[[[0, 1, 2, 3, ..., 6, 7, 8, 9], ...,[0, 1, 2, 3, ..., 6, 7, 8, 9]], ..., [[[0, 1, 2, 3, ..., 6, 7, 8, 9], ...,[0, 1, 2, 3, ..., 6, 7, 8, 9]]]

[asfimport]

Will Jones / @wjones127:
Another issue I notice is that most column types will appear as one line in a Table repr, except a struct column will display child arrays across multiple lines (like it would when the array is printed on its own).

>>> pa.table({'arr': pa.array([{'x': 1}, {'x': 2}, {'x': 3}]),
...           'arr2': pa.array([1, 2, 3])})
pyarrow.Table
arr: struct<x: int64>
  child 0, x: int64
arr2: int64
----
arr: [  -- is_valid: all not null  -- child 0 type: int64
    [
      1,
      2,
      3
    ]]
arr2: [[1,2,3]]

 

[asfimport]

Krisztian Szucs / @kszucs:
Postponing to 8.0.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Can we keep this on 7.0.0? (or keep ARROW-15329 on 7.0.0?)

The table repr is basically unusable right now unless you are using small test data

[asfimport]

Krisztian Szucs / @kszucs:
We can, though I wouldn't like to block the release on this issue since it is an improvement (it could even count as a new feature).

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 12091
#12091