pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column X with type int64')

[jarandaf]

We have started to get runtime errors when saving model predictions as parquet files in AzureML compute instances. Could any new AzureML release break something? We did not change anything from our side so it seems some non-pinned dependency is now resolved differently.

[lostmygithubaccount]

it looks like pyarrow==3.0.0 released last week, could that be the issue?

what code are you calling that produces this? can you provide the full traceback?

[jarandaf]

I have been able to reproduce this, both from the specific compute instance image and from a brand new docker image with the following commands:

docker pull mcr.microsoft.com/azureml/base:intelmpi2018.3-ubuntu16.04
docker run -it `docker images | grep intelmpi2018.3-ubuntu16.04 | tr -s ' '| cut -d' ' -f 3` /bin/bash

Inside the docker container (this just reproduces what is inside our environment.yml file):

root@xxx> conda update conda
root@xxx> conda install -c conda-forge pip=20.1.1 lightgbm=3.0.0 scikit-learn=0.23.2 pyarrow=2.0.0 pandas=1.1.3 pyspark=2.4.0 matplotlib=3.3.2 joblib=1.0.0 seaborn=0.11.1 ipython
root@xxx> pip install azureml-sdk==1.21.0

From an ipython shell:

In [1]: import pandas as pd
In [2]: df = pd.DataFrame({'x': [1,2]})
In [3]: df.to_parquet('test.parquet', engine='pyarrow')

which raises:

---------------------------------------------------------------------------
ArrowTypeError                            Traceback (most recent call last)
<ipython-input-3-bebaa85a69fa> in <module>
----> 1 df.to_parquet('test.parquet', engine='pyarrow')

/opt/miniconda/lib/python3.7/site-packages/pandas/util/_decorators.py in wrapper(*args, **kwargs)
    197                 else:
    198                     kwargs[new_arg_name] = new_arg_value
--> 199             return func(*args, **kwargs)
    200
    201         return cast(F, wrapper)

/opt/miniconda/lib/python3.7/site-packages/pandas/core/frame.py in to_parquet(self, path, engine, compression, index, partition_cols, **kwargs)
   2370             index=index,
   2371             partition_cols=partition_cols,
-> 2372             **kwargs,
   2373         )
   2374

/opt/miniconda/lib/python3.7/site-packages/pandas/io/parquet.py in to_parquet(df, path, engine, compression, index, partition_cols, **kwargs)
    274         index=index,
    275         partition_cols=partition_cols,
--> 276         **kwargs,
    277     )
    278

/opt/miniconda/lib/python3.7/site-packages/pandas/io/parquet.py in write(self, df, path, compression, index, partition_cols, **kwargs)
     99             from_pandas_kwargs["preserve_index"] = index
    100
--> 101         table = self.api.Table.from_pandas(df, **from_pandas_kwargs)
    102
    103         if is_fsspec_url(path) and "filesystem" not in kwargs:

/opt/miniconda/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_pandas()

/opt/miniconda/lib/python3.7/site-packages/pyarrow/pandas_compat.py in dataframe_to_arrays(df, schema, preserve_index, nthreads, columns, safe)
    586     if nthreads == 1:
    587         arrays = [convert_column(c, f)
--> 588                   for c, f in zip(columns_to_convert, convert_fields)]
    589     else:
    590         from concurrent import futures

/opt/miniconda/lib/python3.7/site-packages/pyarrow/pandas_compat.py in <listcomp>(.0)
    586     if nthreads == 1:
    587         arrays = [convert_column(c, f)
--> 588                   for c, f in zip(columns_to_convert, convert_fields)]
    589     else:
    590         from concurrent import futures

/opt/miniconda/lib/python3.7/site-packages/pyarrow/pandas_compat.py in convert_column(col, field)
    572             e.args += ("Conversion failed for column {!s} with type {!s}"
    573                        .format(col.name, col.dtype),)
--> 574             raise e
    575         if not field_nullable and result.null_count > 0:
    576             raise ValueError("Field {} was non-nullable but pandas column "

/opt/miniconda/lib/python3.7/site-packages/pyarrow/pandas_compat.py in convert_column(col, field)
    566
    567         try:
--> 568             result = pa.array(col, type=type_, from_pandas=True, safe=safe)
    569         except (pa.ArrowInvalid,
    570                 pa.ArrowNotImplementedError,

/opt/miniconda/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

/opt/miniconda/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._ndarray_to_array()

/opt/miniconda/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._ndarray_to_type()

/opt/miniconda/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column x with type int64')

I observed that the package azureml-dataset-runtime[fuse] (azureml-sdk dependency) actually requires pyarrow<2.0.0,>=0.17.0 and downgrades the pyarrow version to 1.0.1 but I am not sure if this is actually the reason of the error.

EDIT: for some reason, this does not work without the azureml-sdk dependency either. If I create a conda environment locally without the azureml-sdk dependency I don't get any errors which makes me think the problem might be more related to the base image used instead. What I fail to understand is why this worked before and now it does not.

[crmcpherson]

I believe this issue is caused by an update in numpy=1.20.0 released a few days ago. I don't know what the exact cause of the issue is, but it appears to cause an incompatibility within pyarrow. Possibly due to some of the depreciated types in NumPy

A temporary fix for me seemed to be fixing numpy==1.19.5 for the time being. Alternatively you can convert the column types to object before running the export.

[jarandaf]

I can confirm reverting to numpy<1.20.0 fixes the issue (pandas==1.1.3 has as requirement numpy>=1.15.4, this is why the new version 1.20.0 released this last Saturday was now picked). Thank you @crmcpherson for the heads up, good catch!

[basilvetas]

I ran into the same error using the snowflake-connector-python package, which was converting to parquet files under the hood, and likewise was able to fix it by reverting to numpy<1.20.0 (converting all the types to object while maintaining numpy==1.20.0 did not work for me however).

[newTypeGeek]

it looks like pyarrow==3.0.0 released last week, could that be the issue?

what code are you calling that produces this? can you provide the full traceback?

Upgraded pyarrow works for me

[markokole]

If it helps:
I was getting this error: pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column IN_MU_user_fee with type bool')

pyarrow==2.0.0
numpy==1.20.1
Downgraded to 1.19.1 and it worked.

Upgraded pyarrow to 3.0.0 and numpy 1.20.1 also worked well.