[Python] More graceful reading of empty String columns in ParquetDataset

[asfimport]

When currently saving a ParquetDataset from Pandas, we don't get consistent schemas, even if the source was a single DataFrame. This is due to the fact that in some partitions object columns like string can become empty. Then the resulting Arrow schema will differ. In the central metadata, we will store this column as pa.string whereas in the partition file with the empty columns, this columns will be stored as pa.null.

The two schemas are still a valid match in terms of schema evolution and we should respect that in

arrow/python/pyarrow/parquet.py

Line 754 in 79a2207

def validate_schemas(self):

Instead of doing a pa.Schema.equals in

arrow/python/pyarrow/parquet.py

Line 778 in 79a2207

if not dataset_schema.equals(file_schema, check_metadata=False):

we should introduce a new method pa.Schema.can_evolve_to that is more graceful and returns True if a dataset piece has a null column where the main metadata states a nullable column of any type.

Reporter: Uwe Korn / @xhochy

Related issues:

• [Python][Parquet][C++] Null values in a single partition of Parquet dataset, results in invalid schema on read (is related to)
• [Python][C++][Parquet] Support reading Parquet files having a permutation of column order (is related to)
• [Python][Dataset] Support using dataset API in pyarrow.parquet with a minimal ParquetDataset shim (depends upon)
• [C++][Dataset] Support null -> other type promotion in Dataset scanning (depends upon)

Original Issue Attachments:

• read_parquet_dataset.error.read_table.novalidation.txt
• read_parquet_dataset.error.read_table.txt

_{Note: This issue was originally created as ARROW-2659. Please see the migration documentation for further details.}

[asfimport]

Aldrin Montana / @drin:
In trying to read from a parquet dataset using validate_schema=False, I then get the following exception (see read_parquet_dataset.error.read_table.novalidation.txt):

Traceback (most recent call last):
  File "toolbox/read_parquet.py", line 85, in <module>
    dsv_data_table = read_parquet_data(parsed_args.input_path)
  File "toolbox/read_parquet.py", line 64, in read_parquet_data
    use_pandas_metadata=use_pandas_metadata
  File "/Users/amontana/.local/share/virtualenvs/kitsi-zdrw075I/lib/python3.6/site-packages/pyarrow/parquet.py", line 806, in read
    all_data = lib.concat_tables(tables)
  File "table.pxi", line 1285, in pyarrow.lib.concat_tables
  File "error.pxi", line 77, in pyarrow.lib.check_status

pyarrow.lib.ArrowInvalid: Schema at index 3 was different:

 

Given this error, it seems that validating the schema is just one piece of the overall difficulty, and some way of pushing down the valid schema evolution into concat_tables in table.pxi will also be necessary.

@xhochy, I'm not sure if this impression is correct, and if it is, whether we would want a related Jira to track changes in (or underlying) the concat_tables function.

[asfimport]

Aldrin Montana / @drin:
It seems to me that the following locations are the CPP code underlying the cython code:

• arrow::Table in cpp/src/arrow/table.cc

• arrow::Schema in cpp/src/arrow/type.cc

  but since I don't actually know cython, I can't tell if this is the code that is eventually invoked when calling pyarrow.table.pxi:concat_table()

[asfimport]

Python User:
I have also run into this when trying to convert from a pandas dataframe in chunks. Here is code and data to reproduce.

schema-bug.csv

a,b
1,x
2,y
3,
4,

schema_bug.py

#!/usr/bin/env python
import sys

import pandas as pd
import pyarrow.parquet

writer = None
for chunk in pd.read_csv(sys.stdin, dtype={'a': int, 'b': str}, chunksize=2):
    print('Pandas dtypes', list(chunk.dtypes.astype(str)), file=sys.stderr)
    table = pyarrow.Table.from_pandas(chunk)
    writer = writer or pyarrow.parquet.ParquetWriter(sys.stdout.buffer, table.schema)
    writer.write_table(table)

writer.close()

Here are the results:

$ ./schema_bug.py < schema-bug.csv > /dev/null
Pandas dtypes ['int64', 'object']
Pandas dtypes ['int64', 'object']
Traceback (most recent call last):
  File "schema_bug.py", line 13, in <module>
    writer.write_table(table)
  File "python/lib/python3.6/site-packages/pyarrow/parquet.py", line 335, in write_table
    raise ValueError(msg)
ValueError: Table schema does not match schema used to create file:
table:
a: int64
b: null
__index_level_0__: int64
metadata
--------
{b'pandas': b'{"index_columns": ["__index_level_0__"], "column_indexes": [{"na'
            b'me": null, "field_name": null, "pandas_type": "unicode", "numpy_'
            b'type": "object", "metadata": {"encoding": "UTF-8"}}], "columns":'
            b' [{"name": "a", "field_name": "a", "pandas_type": "int64", "nump'
            b'y_type": "int64", "metadata": null}, {"name": "b", "field_name":'
            b' "b", "pandas_type": "empty", "numpy_type": "object", "metadata"'
            b': null}, {"name": null, "field_name": "__index_level_0__", "pand'
            b'as_type": "int64", "numpy_type": "int64", "metadata": null}], "p'
            b'andas_version": "0.23.4"}'} vs.
file:
a: int64
b: string
__index_level_0__: int64
metadata
--------
{b'pandas': b'{"index_columns": ["__index_level_0__"], "column_indexes": [{"na'
            b'me": null, "field_name": null, "pandas_type": "unicode", "numpy_'
            b'type": "object", "metadata": {"encoding": "UTF-8"}}], "columns":'
            b' [{"name": "a", "field_name": "a", "pandas_type": "int64", "nump'
            b'y_type": "int64", "metadata": null}, {"name": "b", "field_name":'
            b' "b", "pandas_type": "unicode", "numpy_type": "object", "metadat'
            b'a": null}, {"name": null, "field_name": "__index_level_0__", "pa'
            b'ndas_type": "int64", "numpy_type": "int64", "metadata": null}], '
            b'"pandas_version": "0.23.4"}'}

(The difference is that the first chunk (shown second) has pandas_type: unicode, where the second chunk has pandas_type: empty.)

Even if you save the initial schema and try to use it writing subsequent tables, you get this:

pyarrow.lib.ArrowNotImplementedError: ('No cast implemented from null to null', 'Conversion failed for column b with type object')

This is with Pandas 0.23.4, pyarrow 0.10.0.

[asfimport]

Wes McKinney / @wesm:
We should implement schema normalization at the Parquet level. I don't think it would be appropriate to push this functionality lower into the Arrow data structures

[asfimport]

Wes McKinney / @wesm:
I'm moving this to 0.13 as unfortunately I don't think we have the time to do this properly for 0.12

I suggest we implement a couple of different things to help us:

• "Schema-normalized concatenate tables" – perform safe casts and determine the merged schema for a collection of smaller tables, or attempt to safely cast tables to a fixed schema. As null will safely cast to anything this will solve the problem one way

• Additionally implement partitioned writes natively against Arrow tables without going through pandas, to avoid the issues in ARROW-2860

[asfimport]

Wes McKinney / @wesm:
It doesn't appear this is going to get done for 0.14 – I don't have the time to personally do it.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
The original root cause of this issue (write_to_dataset outputting parquet files with different schemas) has already been solved a while ago in ARROW-2891

The other mentioned issues (when reading a dataset, we should allow for basic schema evolution like allowing null -> any type promotion) are in the meantime solved in the Arrow Datasets project.

So once we use the datasets API under the hood in pyarrow.parquet (ARROW-8039), this issue should be solved (we can still add a test for it to close this issue)

[asfimport]

Wes McKinney / @wesm:
I think the issue can be resolved once Datasets is the default implementation / engine

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
ARROW-9147 is now fixed, allowing a "null -> any" type promotion, so that should fix the remaining issues mentioned here.

[asfimport]

Antoine Pitrou / @pitrou:
@jorisvandenbossche What should be the fate of this issue?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
No urgent action to be taken (so moved to next milestone). This issue will eventually be solved when switching the default in ParquetDataset to use the new implementation (ARROW-9720). And it would be good to add some explicit tests (assigned to myself to do so).

[asfimport]

Steven Anton:
This is still an issue for me. The ParquetDataset implementation won't help in the case where you're creating a partitioned dataset one partition at a time. Suppose one partition has all nulls in a column; that partition will be written with the null datatype. In another partition, the data could contain strings.

There are a number of options that could help:

• At the very least, I feel that Schema.from_pandas should generate a warning when encountering object columns that are all null. Something along the lines of "Unable to infer data type for ". Yes, there's a note in the docstring, but that's not visible when using something like pd.DataFrame.to_parquet.
• Another idea is to add a kwarg to Schema.from_pandas to explicitly specify the data type for these null columns where the data type is ambiguous. The function could allow either a single type or a dictionary mapping the column name to the correct type.
• Lastly, it seems reasonable to allow some simple schema merging. For example, if you have a partitioned dataset and some partitions have the data type as null for one column in one partition but string in another, it seems we should be able to merge the two to string. (Of course, null, int, and string could not be merged and should still raise an exception.)

[asfimport]

Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.