[Python] Decimal conversion not working for NaN values

[asfimport]

import pyarrow as pa
import pandas as pd
import decimal

pa.Table.from_pandas(pd.DataFrame({'a': [decimal.Decimal('1.1'), decimal.Decimal('NaN')]}))

throws following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/table.pxi", line 875, in pyarrow.lib.Table.from_pandas (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:44927)
  File "/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 350, in dataframe_to_arrays
    convert_types)]
  File "/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 349, in <listcomp>
    for c, t in zip(columns_to_convert,
  File "/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 345, in convert_column
    return pa.array(col, from_pandas=True, type=ty)
  File "pyarrow/array.pxi", line 170, in pyarrow.lib.array (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:29224)
  File "pyarrow/array.pxi", line 70, in pyarrow.lib._ndarray_to_array (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:28465)
  File "pyarrow/error.pxi", line 98, in pyarrow.lib.check_status (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:9068)
pyarrow.lib.ArrowException: Unknown error: an integer is required (got type str)

Same problem with other special decimal values like infinity.

Reporter: Antony Mayi / @antonymayi
Assignee: Phillip Cloud / @cpcloud

Related issues:

• [C++/Python] Decimal conversion not working for exponential notation (is related to)

PRs and other links:

• GitHub Pull Request #1610
• GitHub Pull Request #1651

_{Note: This issue was originally created as ARROW-2145. Please see the migration documentation for further details.}

[asfimport]

Phillip Cloud / @cpcloud:
Thanks for the report, taking a look now.

[asfimport]

Phillip Cloud / @cpcloud:
@antonymayi Do you have a specific use case for this, or were you tinkering around and trying a few things?

[asfimport]

Antony Mayi / @antonymayi:
I am trying to use it for a real system where an event with zero value (decimal.Decimal('0')) is distinct from no event (decimal.Decimal('nan')) while both cases need to be stored. Being able to store decimal nan within the same column spares me from introducing another column just to flag the no-event cases.

This is the same case like with floats - nan/inf is valid float value (and supported by pyarrow/parquet) - but here I need to use decimal because of the precision...

[asfimport]

Wes McKinney / @wesm:
Can you use null instead?

[asfimport]

Antony Mayi / @antonymayi:

Can you use null instead?

I guess the current implementation looks at the first cell of the column, finds instance of Decimal and then expects the whole column contains just decimals - so the answer is no, at least based on this observation:

>>> pa.Table.from_pandas(pd.DataFrame({'a': [decimal.Decimal('1.1'), None]}))
...
pyarrow.lib.ArrowException: Unknown error: 'NoneType' object has no attribute 'as_tuple'

or for numpy.nan:

>>> pa.Table.from_pandas(pd.DataFrame({'a': [decimal.Decimal('1.1'), np.nan]}))
...
pyarrow.lib.ArrowException: Unknown error: 'float' object has no attribute 'as_tuple'

[asfimport]

Wes McKinney / @wesm:
Those both look like bugs that should be fixed. If those worked (or if Decimal('NaN') became null), would that work?

[asfimport]

Phillip Cloud / @cpcloud:
Both are definitely bugs, working on a fix.

[asfimport]

Antony Mayi / @antonymayi:
yes, if null was accepted that would solve my usecase perfectly (in the same way Decimal('NaN') would but I guess null is even better as arrow is sparse for nulls while NaN takes space...)

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 1651
#1651