ARROW-15329: [Python] Add character limit to Table.to_string()

[wjones127]

This prevents table column preview from being too long.

The main disadvantage is there is not way to guarantee that we don't truncate the preview in the middle of a value. For example, [20, 30, 40] could be truncated to [20, 30, 4..., which might be a little misleadning.

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-15329

[edponce]

Ideally, we could always truncate "correctly" and consistently, that is, truncate at a delimiter and show closing brackets: [[1,2,3,4,...]]

But several questions first:

• Is the delimiter always a comma or known in a variable?
• Is the table representation always encoded as a list of lists? Or can there be a less/more nesting?

Assuming "yes" for the questions above. There are several cases that can occur:

1. No truncation: [[1,2,3,4]]
2. Truncate does not reaches the final end bracket (# of brackets is arbitrary): [[1,2,3,4]...
3. Truncate mid-value: [10,20,30,4...
4. Truncate at a value delimiter: [[1,2,3,...
5. Truncate at a list delimiter: [[1,2,3,4],...
6. Truncate at a list ending bracket: [[1,2,3,4]...

To display all truncated cases "correctly" and consistently, after slicing the cols_char_limit, you would need

• a stack to keep track of open/close bracket pairs
• check which symbol is truncation occurring on.

Then resolve as follows:

• If it is a delimiter, then add ... and match with closing brackets
• If it is a closing bracket
  • and there are only closing bracket left, then add them to obtain the full representation.
  • add ... and match with closing brackets
• Else it is truncating a value, so remove (or add) characters until reaching a delimiter/bracket

Most of these checks can be identified with regex.

[jorisvandenbossche]

@edponce I fully agree that, ideally, this truncation is "smart" about where to cut off and add ....
But in general it's also the question to what extent this is worth the extra complexity (depending on how complex it would be of course). Instead of parsing the string, another option could also be to slice the number of elements before converting to string (although for nested data types that won't necessarily work as desired).

Now, on the short-term (for 0.7.0), I personally find it more important that we at least do some truncation (because currently the repr can be completely useful / annoying by flooding your terminal)

[edponce]

I understand and agree with the current approach. Simply leaving notes as food for thought or future reference.

[wjones127]

Thanks for the feedback. I implemented a very basic version of this for now. This looks pretty good for this example:

>>> from random import sample, choice
>>> import pyarrow as pa
>>> arr_int = pa.array(range(50))
>>> tree_parts = ["roots", "trunk", "crown", "seeds"]
>>> arr_list = pa.array([sample(tree_parts, k=choice(range(len(tree_parts)))) for _ in range(50)])
>>> arr_struct = pa.StructArray.from_arrays([arr_int, arr_list], names=['int_nested', 'list_nested'])
>>> arr_map = pa.array(
...     [
...         [(part, choice(range(10))) for part in sample(tree_parts, k=choice(range(len(tree_parts))))]
...         for _ in range(50)
...     ],
...     type=pa.map_(pa.utf8(), pa.int64())
... )
>>> table = pa.table({
...     'int': pa.chunked_array([arr_int] * 10),
...     'list': pa.chunked_array([arr_list] * 10),
...     'struct': pa.chunked_array([arr_struct] * 10),
...     'map': pa.chunked_array([arr_map] * 10),
... })
>>> print(table)
pyarrow.Table
int: int64
list: list<item: string>
  child 0, item: string
struct: struct<int_nested: int64, list_nested: list<item: string>>
  child 0, int_nested: int64
  child 1, list_nested: list<item: string>
      child 0, item: string
map: map<string, int64>
  child 0, entries: struct<key: string not null, value: int64> not null
      child 0, key: string not null
      child 1, value: int64
----
int: [[0,1,2,3,4,5,6,7,8,9,...,40,41,42,43,44,45,46,47,48,49],[0,1,2,3,4,5,6,7,8,9,...,40,41,42,43,44,45,46,47,48,49],[0,1,2,3,...]...]
list: [[["seeds","trunk","roots"],["trunk","crown"],["crown"],["trunk"],["crown"],[],["roots","seeds"],["roots"],["trunk","roots"]...]...]
struct: [  -- is_valid: all not null  -- child 0 type: int64
    [
      0,
      1,
      2,
      3,
      4,
      5,
      6,...]...]
map: [[    keys:["seeds","crown","trunk"]values:[7,8,7],    keys:["roots","crown"]values:[8,4],    keys:["crown","roots","trunk"]...]...]

The unfortunate thing is it will have bad behavior in the case of string columns containing [. For example,

>>> pa.table({'x': pa.array(["[" * 100]* 500)})
x: [["[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[","[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[",...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]

I think that kind of behavior is pretty unavoidable until we push this limit into the PrettyPrinter implementation itself.

[edponce]

Is there a JIRA/PR for implementing similar functionality in C++ Pretty Printer?

[wjones127]

Just created one: https://issues.apache.org/jira/browse/ARROW-15363

[edponce]

The default in the docstring does not matches the value in function arguments.

[wjones127]

Thanks.

[pitrou]

@jorisvandenbossche Do you want to take a look at this PR?

[amol-]

Closing because it has been untouched for a while, in case it's still relevant feel free to reopen and move it forward 👍