[C++][Dataset] Automatically detect boolean partition columns

[asfimport]

Saving a ParquetDataset using a boolean column as a partitioning column will store True/False as the values in the path. On reload these columns will then be string columns with the values 'True' and 'False'.

Reporter: Uwe Korn / @xhochy

Related issues:

• [Python] Datatypes not preserved for partition fields in roundtrip to partitioned parquet dataset (relates to)
• [Python][Dataset] Support using dataset API in pyarrow.parquet with a minimal ParquetDataset shim (depends upon)

_{Note: This issue was originally created as ARROW-3388. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
An update on this issue: also with the new datasets API, they get read back as strings.

Now, I am not fully sure we necessarily should automatically infer True/False string values. With the new API you can also specify the schema of the partition columns, which seems the general solution for this. However, this also does not work yet, as it just gives nulls (will open a separate issue about that, since it is not specific to booleans)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Correction: it actually does work with specifying a schema (I just forgot to specify flavor="hive" for my example)

# create partitioned table with boolean partition keys
path = pathlib.Path(".") / "dataset_boolean_partition"
path.mkdir(exist_ok=True)
table = pa.table({"part": [True, True, False, False], "values": range(4)})
pq.write_to_dataset(table, str(path), partition_cols=["part"])

# legacy -> gives dictionary column of string values
pq.ParquetDataset(case).read()

# new API -> gives string column
pq.ParquetDataset(case, use_legacy_dataset=False).read()

# specify schema -> actually get a column of bools
partitioning = ds.partitioning(pa.schema([('part', pa.bool_())]), flavor="hive")
pq.ParquetDataset(case, use_legacy_dataset=False, partitioning=partitioning).read()
pq.ParquetDataset(case, use_legacy_dataset=False, partitioning=partitioning).read().to_pandas()

Are we fine with requiring the user to pass a manual schema here? Or do we actually still want to automatically infer True/False to be boolean type?

cc @fsaintjacques @bkietz

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
I think it's normal to require a schema.

[asfimport]

Neal Richardson / @nealrichardson:
I agree that requiring a schema isn't bad here, though the magic behavior might be nice. I'll rewrite the title to make this about C++ datasets and make it an idea for an enhancement, not a bug report.

[asfimport]

Ben Kietzman / @bkietz:
@xhochy @nealrichardson @jorisvandenbossche Is this issue still desired? Partition columns are inferred with dictionary type by default, so in the case where the only keys present were "True" and "False" we'd infer a string dictionary with those two values. The int32 indices are not as efficient as a boolean array, but perhaps it is sufficient

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I am fine with requiring a schema from the user to get boolean data.

Partition columns are inferred with dictionary type by default

Not in C++, I think? (it's an option to do it?)

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.