[Python] Categorical boolean column saved as regular boolean in parquet

[asfimport]

When saving a pandas dataframe to parquet, if there is a categorical column where the categories are boolean, the column is saved as regular boolean.

This causes an issue because, when reading back the parquet file, I expect the column to still be categorical.

 
Reproducible example:

import pandas as pd
import pyarrow

# Create dataframe with boolean column that is then converted to categorical
df = pd.DataFrame({'a': [True, True, False, True, False]})
df['a'] = df['a'].astype('category')

# Convert to arrow Table and save to disk
table = pyarrow.Table.from_pandas(df)
pyarrow.parquet.write_table(table, 'test.parquet')

# Reload data and convert back to pandas
table_rel = pyarrow.parquet.read_table('test.parquet')
df_rel = table_rel.to_pandas()

The arrow table variable correctly converts the column to an arrow DICTIONARY type:

>>> df['a']
0     True
1     True
2    False
3     True
4    False
Name: a, dtype: category
Categories (2, object): [False, True]
>>>
>>> table
pyarrow.Table
a: dictionary<values=bool, indices=int8, ordered=0>

However, the reloaded column is now a regular boolean:

>>> table_rel
pyarrow.Table
a: bool
>>>
>>> df_rel['a']
0     True
1     True
2    False
3     True
4    False
Name: a, dtype: bool

I would have expected the column to be read back as categorical.

Reporter: Joao Moreira

Related issues:

• [C++][Parquet] Support direct dictionary decoding of types other than BYTE_ARRAY (is blocked by)
• [Python] Consistent handling of categoricals (is related to)

_{Note: This issue was originally created as ARROW-13342. Please see the migration documentation for further details.}

[asfimport]

Alessandro Molina / @amol-:
FYI, this seems to happen for numeric types. The same is true for int and double

import pyarrow as pa
import pyarrow.parquet

# Convert to arrow Table and save to disk
table = pa.Table.from_arrays([
    pa.DictionaryArray.from_arrays(indices=pa.array([0, 1, 1, 0, 1, 1, 0], type=pa.int8()), 
                                   dictionary=[10000000000000000, 20000000000000000])
], names=["data"])
print(table)
pa.parquet.write_table(table, 'test.parquet')

# Reload data and convert back to pandas
table_rel = pa.parquet.read_table('test.parquet')
print(table_rel)

The exception seem to be unicode which preserves the dictionary form.

In general, from my understanding, it seems arrow currently writes in dictionary form only the binary types (

arrow/cpp/src/parquet/column_writer.cc

Line 1008 in 14b75ee

return ::arrow::is_base_binary_like(dict_type.value_type()->id());

)

This in general seems a fairly reasonable behaviour to me, because in general binary data/text is where you get most saving due to the delta in size between the data and the indices.

On the other side, I don't think that "preserving same exact type" that was provided to a writer is an expectation that can be always satisfied or that it makes sense to enforce. In some cases enforcing dictionary encoding because the input was dictionary encoded might lead to bigger parquet files, or for some types for example that might not even be possible, think of CSV or JSON, not all types can be represented in those formats and thus the data you read back might have a different type

[asfimport]

Weston Pace / @westonpace:
Judging by ARROW-3246 I believe the original intent was to support more types than just the binary types.  Also, Arrow is supposed to be storing custom metadata (arrow:SCHEMA) in the parquet file with details on the original Arrow schema.  This way, if we need to store the data more efficiently then we can do so while still restoring the original types upon deserialization.  I checked and store_schema is set to true and it does properly decode and restore the original schema which does report the column "a" as dictionary<values=bool, indices=int8, ordered=0>.  I'm not sure if some linkage is broken or if it only tries to restore the metadata on binary types.

 

However, I think the resolution for this issue is (although maybe not easy), the existing mechanism should be extended to handle more types.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Directly reading dictionary encoded data in Parquet as dictionary typed arrays in Arrow is currently indeed only supported for BYTE_ARRAY storage (i.e. string, binary). The issue tracking the follow-up to expand this support to other data types is ARROW-6140.

The documentation of pq.read_table also somewhat mentions this:

read_dictionary : list, default None
    List of names or column paths (for nested types) to read directly
    as DictionaryArray. Only supported for BYTE_ARRAY storage. 
    ...

This parameter allows to explicitly specify which columns to read as dictionary arrays. But it doesn't mention the default behaviour, which is to infer those columns from the stored arrow:SCHEMA.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

But it doesn't mention the default behaviour, which is to infer those columns from the stored arrow:SCHEMA.

This is handled in ApplyOriginalStorageMetadata at

arrow/cpp/src/parquet/arrow/schema.cc

Lines 879 to 889 in 14b75ee

	if (origin_type->id() == ::arrow::Type::DICTIONARY &&
	inferred_type->id() != ::arrow::Type::DICTIONARY &&
	IsDictionaryReadSupported(*inferred_type)) {
	// Direct dictionary reads are only suppored for a couple primitive types,
	// so no need to recurse on value types.
	const auto& dict_origin_type =
	checked_cast<const ::arrow::DictionaryType&>(*origin_type);
	inferred->field = inferred->field->WithType(
	::arrow::dictionary(::arrow::int32(), inferred_type, dict_origin_type.ordered()));
	modified = true;
	}

And it is coded in https://github.com/apache/arrow/blob/14b75ee71d770ba86999e0e7a0e0b94629b91968/cpp/src/parquet/arrow/schema.cc#L410-L413 for which types this is currently supported to read as dictionary type:

arrow/cpp/src/parquet/arrow/schema.cc

Lines 410 to 413 in 14b75ee

	bool IsDictionaryReadSupported(const ArrowType& type) {
	// Only supported currently for BYTE_ARRAY types
	return type.id() == ::arrow::Type::BINARY || type.id() == ::arrow::Type::STRING;
	}

—

In summary, I think the long term fix is ARROW-6140, and short term for this issue there are two options:

• better document the current behaviour
• cast afterwards to dictionary type, based on the ARROW:schema or pandas metadata (but this can be costly, after first materializing the full column, and not necessarily always wanted)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Some more explanation of the situation also in ARROW-11157

[asfimport]

Alessandro Molina / @amol-:
I'm interested in taking a shot at ARROW-6140 unless it becomes urgent and someone more experience than me with the C++/Parquet side of things wants to take over. I'll take the ticket for the moment, anyone should feel free to take it over if there is extra urgency.