[C++][Dataset] Assert integer overflow in partitioning falls back to string

[asfimport]

From https://stackoverflow.com/questions/64137664/how-to-override-type-inference-for-partition-columns-in-hive-partitioned-dataset

Small reproducer:

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({'part': [3760212050]*10, 'col': range(10)})
pq.write_to_dataset(table, "test_int64_partition", partition_cols=['part'])

In [35]: pq.read_table("test_int64_partition/")
...
ArrowInvalid: error parsing '3760212050' as scalar of type int32
In ../src/arrow/scalar.cc, line 333, code: VisitTypeInline(*type_, this)
In ../src/arrow/dataset/partition.cc, line 218, code: (_error_or_value26).status()
In ../src/arrow/dataset/partition.cc, line 229, code: (_error_or_value27).status()
In ../src/arrow/dataset/discovery.cc, line 256, code: (_error_or_value17).status()

In [36]: pq.read_table("test_int64_partition/", use_legacy_dataset=True)
Out[36]: 
pyarrow.Table
col: int64
part: dictionary<values=int64, indices=int32, ordered=0>

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Ben Kietzman / @bkietz

PRs and other links:

• GitHub Pull Request #8462

_{Note: This issue was originally created as ARROW-10145. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I think at least we should fall back to string instead of raising an error like this (as long as we only support reading partition fields as string or int32), but long term we should expand the supported types.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Note that reading it with pyarrow.dataset interface and with specifying a schema manually, it seems to work correctly:

In [38]: import pyarrow.dataset as ds

In [40]: dataset = ds.dataset("test_int64_partition/", schema=pa.schema([("part", pa.int64()), ("col", pa.int64())]), partitioning="hive")

In [42]: dataset.to_table()
Out[42]: 
pyarrow.Table
part: int64
col: int64

In [43]: dataset.to_table().to_pandas()
Out[43]: 
         part  col
0  3760212050    0
1  3760212050    1
2  3760212050    2
3  3760212050    3
4  3760212050    4
5  3760212050    5
6  3760212050    6
7  3760212050    7
8  3760212050    8
9  3760212050    9

At the moment, this schema keyword can not yet be passed to the pq.read_table function (something we should add!)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Issue resolved by pull request 8462
#8462