[Python] Map data types doesn't work from Arrow to Parquet

[asfimport]

Hi,

I'm having problems using 'map' data type in Arrow/parquet/pandas.

I'm able to convert a pandas data frame to Arrow with a map data type.

When I write Arrow to Parquet, it seems to work, but I'm not sure if the data type is written correctly.

When I read back Parquet to Arrow, it fails saying "reading list of structs" is not supported. It seems that map is stored as list of structs.

There are two problems here:

1. Map data type doesn't work from Arrow -> Pandas. Fixed in ARROW-10151

2. Map data type doesn't get written to or read from Arrow -> Parquet.

   Questions:

   1. Am I doing something wrong? Is there a way to get these to work? 

   2. If these are unsupported features, will this be fixed in a future version? Do you plans or ETA?

   The following code example (followed by output) should demonstrate the issues:

   I'm using Arrow 1.0.0 and Pandas 1.0.5.

   Thanks!

   Mayur

   $ cat arrowtest.py
   
   import pyarrow as pa
   import pandas as pd
   import pyarrow.parquet as pq
   import traceback as tb
   import io
   
   print(f'PyArrow Version = {pa.__version__}')
   print(f'Pandas Version = {pd.__version__}')
   
   df1 = pd.DataFrame({'a': [[('b', '2')]]})
   print(f'df1')
   print(f'{df1}')
   
   print(f'Pandas -> Arrow')
   try:
       t1 = pa.Table.from_pandas(df1, schema=pa.schema([pa.field('a', pa.map_(pa.string(), pa.string()))]))
       print('PASSED')
       print(t1)
   except:
       print(f'FAILED')
       tb.print_exc()
   
   print(f'Arrow -> Pandas')
   try:
       t1.to_pandas()
       print('PASSED')
   except:
       print(f'FAILED')
       tb.print_exc()print(f'Arrow -> Parquet')
   
   fh = io.BytesIO()
   try:
       pq.write_table(t1, fh)
       print('PASSED')
   except:
       print('FAILED')
       tb.print_exc()
       
   print(f'Parquet -> Arrow')
   try:
       t2 = pq.read_table(source=fh)
       print('PASSED')
       print(t2)
   except:
       print('FAILED')
       tb.print_exc()

   $ python3.6 arrowtest.py
   PyArrow Version = 1.0.0 
   Pandas Version = 1.0.5 
   df1 
   a 0 [(b, 2)] 
    
   Pandas -> Arrow 
   PASSED 
   pyarrow.Table 
   a: map<string, string>
    child 0, entries: struct<key: string not null, value: string> not null
    child 0, key: string not null
    child 1, value: string 
    
   Arrow -> Pandas 
   FAILED 
   Traceback (most recent call last):
   File "arrowtest.py", line 26, in <module> t1.to_pandas() 
   File "pyarrow/array.pxi", line 715, in pyarrow.lib._PandasConvertible.to_pandas 
   File "pyarrow/table.pxi", line 1565, in pyarrow.lib.Table._to_pandas File "XXX/pyarrow/1/0/x/dist/lib/python3.6/pyarrow/pandas_compat.py", line 779, in table_to_blockmanager blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes) 
   File "XXX/pyarrow/1/0/x/dist/lib/python3.6/pyarrow/pandas_compat.py", line 1115, in _table_to_blocks list(extension_columns.keys())) 
   File "pyarrow/table.pxi", line 1028, in pyarrow.lib.table_to_blocks File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status pyarrow.lib.ArrowNotImplementedError: No known equivalent Pandas block for Arrow data of type map<string, string> is known. 
    
   Arrow -> Parquet 
   PASSED 
    
   Parquet -> Arrow 
   FAILED 
   Traceback (most recent call last): File "arrowtest.py", line 43, in <module> t2 = pq.read_table(source=fh) 
   File "XXX/pyarrow/1/0/x/dist/lib/python3.6/pyarrow/parquet.py", line 1586, in read_table use_pandas_metadata=use_pandas_metadata) 
   File "XXX/pyarrow/1/0/x/dist/lib/python3.6/pyarrow/parquet.py", line 1474, in read use_threads=use_threads 
   File "pyarrow/_dataset.pyx", line 399, in pyarrow._dataset.Dataset.to_table 
   File "pyarrow/_dataset.pyx", line 1994, in pyarrow._dataset.Scanner.to_table 
   File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status 
   File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status pyarrow.lib.ArrowNotImplementedError: Reading lists of structs from Parquet files not yet supported: key_value: list<key_value: struct<key: string not null, value: string> not null> not null

   Updated to indicate to Pandas conversion done, but not yet for Parquet.

Reporter: Mayur Srivastava / @mayuropensource

Related issues:

• [Python] Add support MapArray to_pandas conversion (is related to)

_{Note: This issue was originally created as ARROW-9812. Please see the migration documentation for further details.}

[asfimport]

Mayur Srivastava / @mayuropensource:
If I use python dict in pandas and convert to arrow, I get a struct data type in arrow. 

Another question is that how pandas type should get converted to arrow?

# Try with Python dict                                                                                                                                        
print('Try with Python dict:')
df2 = pd.DataFrame({'a': [{'b': 2}]})
print(df2)

t3 = pa.Table.from_pandas(df2)
print(t3)

print(t3.to_pandas())

Try with Python dict:
          a
0  {'b': 2}
pyarrow.Table
a: struct<b: int64>
  child 0, b: int64
          a
0  {'b': 2}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@mayuropensource thanks for the issue. As you correctly noted, there are two different issues at play:

1. Map data type doesn't work from Arrow -> Pandas.

As the error message indicates, this conversion is not yet implemented.
I don't think someone is actively working on this, and contributions in this area are certainly welcome.

2. Map data type doesn't get written to or read from Arrow -> Parquet.

It does get written (as a list of structs, which is how the Map Type is represented, see

arrow/format/Schema.fbs

Lines 98 to 125 in 3fb1356

	/// A Map is a logical nested type that is represented as
	///
	/// List<entries: Struct<key: K, value: V>>
	///
	/// In this layout, the keys and values are each respectively contiguous. We do
	/// not constrain the key and value types, so the application is responsible
	/// for ensuring that the keys are hashable and unique. Whether the keys are sorted
	/// may be set in the metadata for this field.
	///
	/// In a field with Map type, the field has a child Struct field, which then
	/// has two children: key type and the second the value type. The names of the
	/// child fields may be respectively "entries", "key", and "value", but this is
	/// not enforced.
	///
	/// Map
	/// - child[0] entries: Struct
	/// - child[0] key: K
	/// - child[1] value: V
	///
	/// Neither the "entries" field nor the "key" field may be nullable.
	///
	/// The metadata is structured so that Arrow systems without special handling
	/// for Map can make Map an alias for List. The "layout" attribute for the Map
	/// field must have the same contents as a List.
	table Map {
	/// Set to true if the keys within each value are sorted
	keysSorted: bool;
	}

). However, such mixture of nested lists and structs is not yet supported on the read side. This is actively being worked on (see ARROW-1644), and hopefully this will work in the next Arrow version.

So you are not directly doing something wrong, but the Map type is not yet very well supported in the Parquet and pandas conversions. Using a struct type, as in your last example, is typically better supported (simple structs are supported both in Parquet IO as in conversion to pandas)

[asfimport]

Mayur Srivastava / @mayuropensource:
Thanks @jorisvandenbossche

When ARROW-1644 is done, we can start using for non-pandas use cases.

 

 

[asfimport]

Bryan Cutler / @BryanCutler:
I started work on ARROW-10151 for the Pandas conversion. Let's keep this open for Parquet conversion after ARROW-1644.

[asfimport]

Derek Marsh / @dmarsh19:
Hi all,

I've searched existing issues as best I can and this issue mentions "1. Map data type doesn't work from Arrow -> Pandas."

I built the project from master at roughly 15:00 UTC today (October 9) and added one line before I built pyarrow:

    _Type_MAP: np.object_,

after this line: types.pxi

This enables Table.to_pandas() to convert a MapType to List[Tuple[...]]

>>> import pyarrow as pa
>>> d = pa.map_(pa.int64(), pa.float64())
>>> d.to_pandas_dtype()
<class 'numpy.object_'>

>>> tbl
pyarrow.Table
stored_on: double
vals: map<int64, double>
 child 0, entries: struct<key: int64 not null, value: double> not null
 child 0, key: int64 not null
 child 1, value: double
>>> tbl.to_pydict()
{'stored_on': [1585347700.204351], 'vals': [[(514, 12.0), (515, 1300.0), (519, 125.0), (2978, 126.0), (3236, 13107.0), (3237, 1.0), (3238, 1.0), (3239, 3.0), (3240, 3.0)]]}
>>> df = tbl.to_pandas()
>>> df.vals
0 [(514, 12.0), (515, 1300.0), (519, 125.0), (29...
Name: vals, dtype: object
>>> df.vals.iloc[0]
[(514, 12.0), (515, 1300.0), (519, 125.0), (2978, 126.0), (3236, 13107.0), (3237, 1.0), (3238, 1.0), (3239, 3.0), (3240, 3.0)]
>>> df.vals.iloc[0][0]
(514, 12.0)

I understand this is a very trivial working example, but am interested what any maintainers think about this solution and if it merits further testing/consideration.

Thanks.

[asfimport]

Bryan Cutler / @BryanCutler:
Hi @dmarsh19 , I implemented Map types to Pandas conversion recently in ARROW-10151, but looks like I forgot that line you pointed out in types.pxi. That should be in for the upcoming release, if you are able to do a PR for it's cut - likely today or tomorrow - that would be great. Otherwise, I can go ahead and add it. I will update this Jira to reflect Pandas conversion is complete. I made ARROW-10260 to add to_pandas_dtype. Thanks!

[asfimport]

Antoine Pitrou / @pitrou:
The test script in the issue description works on git master. I'm going to close this issue.