Some logical types not supported when loading Parquet

[asfimport]

When I try to read a parquet file with some logical types in it, pyarrow says they are not supported:

table = pq.read_table('t.parquet')
---------------------------------------------------------------------------
ArrowException                            Traceback (most recent call last)
<ipython-input-14-b7190e66bcb5> in <module>()
----> 1 table = pq.read_table('parquet/t')

/opt/conda/lib/python3.5/site-packages/pyarrow/parquet.py in read_table(source, columns, nthreads, metadata)
    113 
    114     pf = ParquetFile(source, metadata=metadata)
--> 115     return pf.read(columns=columns, nthreads=nthreads)
    116 
    117 

/opt/conda/lib/python3.5/site-packages/pyarrow/parquet.py in read(self, nrows, columns, nthreads)
     78 
     79         return self.reader.read(column_indices=column_indices,
---> 80                                 nthreads=nthreads)
     81 
     82 

/opt/conda/lib/python3.5/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.ParquetReader.read (/feedstock_root/build_artefacts/pyarrow_1488133203047/work/arrow-f6924ad83bc95741f003830892ad4815ca3b70fd/python/build/temp.linux-x86_64-3.5/_parquet.cxx:7706)()

/opt/conda/lib/python3.5/site-packages/pyarrow/error.pyx in pyarrow.error.check_status (/feedstock_root/build_artefacts/pyarrow_1488133203047/work/arrow-f6924ad83bc95741f003830892ad4815ca3b70fd/python/build/temp.linux-x86_64-3.5/error.cxx:1197)()

ArrowException: NotImplemented: Unhandled logical type for int32

This is the schema of the parquet file (see attached):

optional group root {
  optional int64 instant (TIMESTAMP_MILLIS);
  optional int32 time (TIME_MILLIS);
  optional double a-double;
  optional int64 another-int;
  optional binary a-string (UTF8);
  optional group list (LIST) {
    repeated group list {
      optional int64 element;
    }
  }
  optional boolean a-boolean;
  optional group a-group {
    optional boolean bool;
    optional int64 another;
  }
  optional int64 an-int;
  optional int32 a-date (DATE);
}

I assume this is because not pyarrow doesn't support loading all the parquet logical types yet. Is there someplace I can look (even if it's not documented, just in the codebase), where I can find what types are supported currently and which are not?

Reporter: Saul Shanabrook
Assignee: Wes McKinney / @wesm

Related issues:

• Arrow 0.5.0 release (blocks)
• C++: Support TIME logical type in parquet_arrow (is blocked by)
• C++: Support nested structs in parquet_arrow (is blocked by)
• [Python] Implement time type conversions in to_pandas (requires)

Original Issue Attachments:

• t.parquet

_{Note: This issue was originally created as ARROW-601. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
hi [~saulshanabrook] – all of the schema mappings are contained in https://github.com/apache/parquet-cpp/blob/master/src/parquet/arrow/schema.cc

what system did you use to create the file? The two problematic types are the DATE (see PARQUET-890) and the struct "a-group" (the support for nested types is not yet complete – any interest in helping with this?). @xhochy may have some other comments

[asfimport]

Uwe Korn / @xhochy:
DATE support is now in parquet-cpp/pyarrow master and the error messages are now more informative. Support for TIME and STRUCT are still missing in the path, I'll add tickets for those.

[asfimport]

Miki Tebeka / @tebeka:
Small patch at PR #270.

Note that when running a small test on the file provided with the JIRA issue I get a segfault. Tracking it down got me to see that the number of fields calculated in FromParquetSchema is 9 where the actual number of fields in the node is 8.
See parquet/arrow/schema.cc line 283. This IMO stems from ReadTable (parquet/arrow/reader.cc)

[asfimport]

Wes McKinney / @wesm:
@tebeka if I had to guess, the segfault is because of the struct type

optional group a-group {
    optional boolean bool;
    optional int64 another;
  }

I don't think we have any tests yet for structs (PARQUET-911).

[asfimport]

Wes McKinney / @wesm:
Added this as a blocker for 0.3 to at least see if the behavior is better now (skipping the unsupported fields) or anything else we should do until we have full struct support

[asfimport]

Wes McKinney / @wesm:
Now we have

In [5]: table = pq.read_table('/home/wesm/Downloads/t.parquet')

In [6]: table
Out[6]: 
pyarrow.Table
an-int: int64
another-int: int64
a-double: double
a-boolean: bool
a-group: struct<bool: bool, another: int64>
a-string: string
a-date: date64[ms]
time: time32[ms]

In [7]: table.to_pandas()
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
<ipython-input-7-cad6e023c888> in <module>()
----> 1 table.to_pandas()

/home/wesm/code/arrow/python/pyarrow/_table.pyx in pyarrow._table.Table.to_pandas (/home/wesm/code/arrow/python/build/temp.linux-x86_64-3.5/_table.cxx:9726)()
    745             nthreads = pyarrow._config.cpu_count()
    746 
--> 747         mgr = table_to_blockmanager(self.sp_table, nthreads)
    748         return _pandas().DataFrame(mgr)
    749 

/home/wesm/code/arrow/python/pyarrow/_table.pyx in pyarrow._table.table_to_blockmanager (/home/wesm/code/arrow/python/build/temp.linux-x86_64-3.5/_table.cxx:7763)()
    539 
    540     with nogil:
--> 541         check_status(pyarrow.ConvertTableToPandas(table, nthreads,
    542                                                   &result_obj))
    543 

/home/wesm/code/arrow/python/pyarrow/_error.pyx in pyarrow._error.check_status (/home/wesm/code/arrow/python/build/temp.linux-x86_64-3.5/_error.cxx:1542)()
     64             raise ArrowKeyError(message)
     65         elif status.IsNotImplemented():
---> 66             raise ArrowNotImplementedError(message)
     67         elif status.IsTypeError():
     68             raise ArrowTypeError(message)

ArrowNotImplementedError: NotImplemented: struct<bool: bool, another: int64>

The attached file has schema

message root {
  optional int64 an-int;
  optional int64 another-int;
  optional double a-double;
  optional boolean a-boolean;
  optional group a-group {
    optional boolean bool;
    optional int64 another;
  }
  optional binary a-string (UTF8);
  optional int32 a-date (DATE);
  optional int32 time (TIME_MILLIS);
}

So the Arrow schema looks OK (we should switch to date32 now), but the struct should be skipped on read because it isn't supported. @xhochy can you take a look?

[asfimport]

Uwe Korn / @xhochy:
Schema is ok but I really don't like that we would skip columns that we don't support. In this case, I would hope that the user explicitly selects the columns he wants to read and exclude the struct columns from that.

[asfimport]

Wes McKinney / @wesm:
That sounds fine. Instead we should raise an exception that a column is unsupported (the read should not have succeeded here)?

[asfimport]

Uwe Korn / @xhochy:
Oh, I only just now realised that we have read the data already into Arrow. Yes, the read should have failed earlier, probably at some point we're not checking a status.

[asfimport]

Wes McKinney / @wesm:
I opened a patch for PARQUET-963, taking this off the 0.3 release and will leave this issue open until we can read the file completely

[asfimport]

Wes McKinney / @wesm:
Well, we can read and convert the structs now, but not the time32[ms]:

In [12]: pq.read_table('/home/wesm/Downloads/t.parquet', columns=['a-group.bool', 'a-group.another']).to_pandas()
Out[12]: 
                          a-group
0  {'bool': True, 'another': 343}

In [13]: pq.read_table('/home/wesm/Downloads/t.parquet').to_pandas()
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
<ipython-input-13-7a965403c9d1> in <module>()
----> 1 pq.read_table('/home/wesm/Downloads/t.parquet').to_pandas()

/home/wesm/code/arrow/python/pyarrow/table.pxi in pyarrow.lib.Table.to_pandas (/home/wesm/code/arrow/python/build/temp.linux-x86_64-3.5/lib.cxx:32758)()
    768             nthreads = cpu_count()
    769 
--> 770         mgr = pdcompat.table_to_blockmanager(self, nthreads)
    771         return pd.DataFrame(mgr)
    772 

/home/wesm/code/arrow/python/pyarrow/pandas_compat.py in table_to_blockmanager(table, nthreads)
    141             )
    142 
--> 143     result = lib.table_to_blocks(block_table, nthreads)
    144 
    145     blocks = []

/home/wesm/code/arrow/python/pyarrow/table.pxi in pyarrow.lib.table_to_blocks (/home/wesm/code/arrow/python/build/temp.linux-x86_64-3.5/lib.cxx:31292)()
    570 
    571     with nogil:
--> 572         check_status(
    573             libarrow.ConvertTableToPandas(
    574                 c_table, nthreads, &result_obj

/home/wesm/code/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status (/home/wesm/code/arrow/python/build/temp.linux-x86_64-3.5/lib.cxx:7378)()
     64             raise ArrowKeyError(message)
     65         elif status.IsNotImplemented():
---> 66             raise ArrowNotImplementedError(message)
     67         elif status.IsTypeError():
     68             raise ArrowTypeError(message)

ArrowNotImplementedError: NotImplemented: No known equivalent Pandas block for Arrow data of type time32[ms] is known.

See ARROW-1183

[asfimport]

Wes McKinney / @wesm:
This is finally resolved by ARROW-1183: #849

In [1]: import datetime

In [2]: import pyarrow.parquet as pq

In [3]: t = pq.read_table('/home/wesm/Downloads/t.parquet')

In [4]: t.to_pandas()
Out[4]: 
   an-int  another-int  a-double a-boolean                         a-group  \
0     123         2332  232.2323      True  {'another': 343, 'bool': True}   

  a-string     a-date             time  
0      hie 2017-03-03  13:37:34.721000  

[asfimport]

Wes McKinney / @wesm:
Resolved by 50b518a and prior patches implementing support for list and struct reads