[Python] Support conversion from python sequence to dictionary type

[asfimport]

Converting from a python sequence with specified target type isn't implemented yet for dictionary type:

In [1]: pa.array(['a', 'b', 'a'], pa.dictionary(pa.int32(), pa.string()))                                                                                                                                          
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
<ipython-input-1-bda8628a4917> in <module>
----> 1 pa.array(['a', 'b', 'a'], pa.dictionary(pa.int32(), pa.string()))

~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()

~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

~/scipy/repos/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Sequence converter for type dictionary<values=string, indices=int32, ordered=0> not implemented

---

Original report

Hello, I am trying to do the following (please correct me if I am doing some non-sense):

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

fields = [pa.field("object", pa.dictionary(pa.int64(), pa.string()))]
data = {"object": { 
                               "a": "a",
                               "b": "b",
                               "c": "c", 
                               "s": "d" }}
df = pd.DataFrame(data)
table = pa.Table.from_pandas(df, pa.schema(fields))
pq.write_table(table, "test.parquet") 

and I am getting:

Traceback (most recent call last):
  File "pa_test.py", line 17, in <module>
    table = pa.Table.from_pandas(df, pa.schema(fields))
  File "pyarrow/table.pxi", line 1451, in pyarrow.lib.Table.from_pandas
  File "/home/tremes/GITHUB/data-pipeline/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py", line 575, in dataframe_to_arrays
    for c, f in zip(columns_to_convert, convert_fields)]
  File "/home/tremes/GITHUB/data-pipeline/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py", line 575, in <listcomp>
    for c, f in zip(columns_to_convert, convert_fields)]
  File "/home/tremes/GITHUB/data-pipeline/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py", line 566, in convert_column
    raise e
  File "/home/tremes/GITHUB/data-pipeline/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py", line 560, in convert_column
    result = pa.array(col, type=type_, from_pandas=True, safe=safe)
  File "pyarrow/array.pxi", line 265, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 80, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 106, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: ('Sequence converter for type dictionary<values=string, indices=int64, ordered=0> not implemented', 'Conversion failed for column object with type object')

Workaround is to use df.to_parquet("test.parquet")

Reporter: Tomas Remes
Assignee: Krisztian Szucs / @kszucs

PRs and other links:

• GitHub Pull Request #8008

_{Note: This issue was originally created as ARROW-9369. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
The dictionary type maps to pandas' categorical type, so you can first convert the dataframe column to a categorical, and then this works:

In [17]: df["object"] = df["object"].astype("category")                                                                                                                                                            

In [19]: table = pa.Table.from_pandas(df, pa.schema(fields))                                                                                                                                                       

In [20]: table                                                                                                                                                                                                     
Out[20]: 
pyarrow.Table
object: dictionary<values=string, indices=int64, ordered=0>

In principle, the converter for the dictionary type could be expanded to take a non-categorical type, but that is right now not yet implemented.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I renamed the issue to reflect the possible enhancement to add a converter from a general python sequence to a dictionary type.

[asfimport]

ARF / @ARF1:
I think this issue has been fixed and can be closed:

pa.array(['a', 'b', 'a'], pa.dictionary(pa.int32(), pa.string()))

Output:

<pyarrow.lib.DictionaryArray object at 0x00000172FAABEBA0>

-- dictionary:
  [
    "a",
    "b"
  ]
-- indices:
  [
    0,
    1,
    0
  ]

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Yes, think this was fixed by ARROW-9992