[Python] pa.lib.tzinfo_to_string(tz) throws ValueError: Unable to convert timezone `tzoffset(None, -14400)` to string

[asfimport]

Overview:

When trying to save DataFrame to parquet error is thrown while parsing a column with the following properties:

 

dtype: datetime64[ns, tzoffset(None, -14400)]
dtype.tz: tzoffset(None, -14400)

 

 

Error:

ValueError: Unable to convert timezone `tzoffset(None, -14400)` to string

 

Error stack:

File "pyarrow/table.pxi", line 1139, in pyarrow.lib.Table.from_pandas
File "/home/koojav/projects/toptal/teftel/.venv/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 480, in dataframe_to_arrays
types)
File "/home/koojav/projects/toptal/teftel/.venv/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 209, in construct_metadata
field_name=sanitized_name)
File "/home/koojav/projects/toptal/teftel/.venv/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 153, in get_column_metadata
string_dtype, extra_metadata = get_extension_dtype_info(column)
File "/home/koojav/projects/toptal/teftel/.venv/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 126, in get_extension_dtype_info
metadata = {'timezone': pa.lib.tzinfo_to_string(dtype.tz)}
File "pyarrow/types.pxi", line 1149, in pyarrow.lib.tzinfo_to_string
ValueError: Unable to convert timezone `tzoffset(None, -14400)` to string

Libraries:

• pandas 0.24.2
• pyarrow 0.13.0

Reporter: Michał Kujawski

Related issues:

• [Python] support zoneinfo / dateutil timezones (is duplicated by)

_{Note: This issue was originally created as ARROW-5540. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
@jorisvandenbossche or @kszucs can you take a look?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~Koojav] Thanks for the report.

Going from the output, I assume that you are using a dateutil timezone, is that correct? (type(dtype.tz) should show it)

At the moment, pyarrow only supports pytz timezones and the standard library's fixed offset timezone. See ARROW-5248 for that.

[asfimport]

Michał Kujawski:
@jorisvandenbossche thank you for a quick reply.

The types of my data are:

type(dtype.tz) : <class 'dateutil.tz.tz.tzoffset'>

and

type(dtype) : <class 'pandas.core.dtypes.dtypes.DatetimeTZDtype'>

 

The issue you have linked seems the same as mine.

 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Thanks for the follow-up. OK, since it is due to not supporting dateutil at the moment, going to close this issue as a duplicate of ARROW-5248.

To prevent this issue, you should be able to prevent that pandas creates a dateutil timezone instead of a pytz timezone (not sure how that happened in your case).