[Python] support zoneinfo / dateutil timezones

[asfimport]

The dateutil packages also provides a set of timezone objects (https://dateutil.readthedocs.io/en/stable/tz.html) in addition to pytz. In pyarrow, we only support pytz timezones (and the stdlib datetime.timezone fixed offset):

In [2]: import dateutil.tz                                                                                                                                    

In [3]: import pyarrow as pa                                                                                                                                  

In [5]: pa.timestamp('us', dateutil.tz.gettz('Europe/Brussels'))                                                                                              
...
~/miniconda3/envs/dev37/lib/python3.7/site-packages/pyarrow/types.pxi in pyarrow.lib.tzinfo_to_string()

ValueError: Unable to convert timezone `tzfile('/usr/share/zoneinfo/Europe/Brussels')` to string

But pandas also supports dateutil timezones. As a consequence, when having a pandas DataFrame that uses a dateutil timezone, you get an error when converting to an arrow table.

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Alenka Frim / @AlenkaF
Watchers: Rok Mihevc / @rok

Related issues:

• [Python] pa.lib.tzinfo_to_string(tz) throws ValueError: Unable to convert timezone tzoffset(None, -14400) to string (duplicates)
• [Python] support dateutil timezones (duplicates)
• [Python] Recognize datetime.timezone.utc as UTC on conversion python->pyarrow (relates to)

PRs and other links:

• GitHub Pull Request #12421

_{Note: This issue was originally created as ARROW-5248. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Another example of dateutil timezone was reported in ARROW-5540.

Reproducible example:

In [32]: import dateutil

In [33]: tz = dateutil.tz.tzoffset(None, -14400)

In [34]: pd.Timestamp("2019-01-01", tz=tz)
Out[34]: Timestamp('2019-01-01 00:00:00-0400', tz='tzoffset(None, -14400)')

In [39]: pa.array(pd.Series([pd.Timestamp("2019-01-01", tz=tz)]))
...
ValueError: Unable to convert timezone `tzoffset(None, -14400)` to string

[asfimport]

mafrosis:
I believe another instance of the same bug can be seen rather simply with the following:

In [1]: import pyarrow as pa

In [2]: from dateutil.tz import tzlocal

In [3]: pa.timestamp('ns', tzlocal())
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-95e5f76d00ff> in <module>
----> 1 pa.timestamp('ns', tzlocal())

~/Development/jiracli/venv/lib/python3.8/site-packages/pyarrow/types.pxi in pyarrow.lib.timestamp()

~/Development/jiracli/venv/lib/python3.8/site-packages/pyarrow/types.pxi in pyarrow.lib.tzinfo_to_string()

~/Development/jiracli/venv/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/Development/jiracli/venv/lib/python3.8/site-packages/dateutil/tz/tz.py in tzname(self, dt)
    236     @tzname_in_python2
    237     def tzname(self, dt):
--> 238         return self._tznames[self._isdst(dt)]
    239
    240     def is_ambiguous(self, dt):

~/Development/jiracli/venv/lib/python3.8/site-packages/dateutil/tz/tz.py in _isdst(self, dt, fold_naive)
    289
    290         # Check for ambiguous times:
--> 291         dstval = self._naive_is_dst(dt)
    292         fold = getattr(dt, 'fold', None)
    293

~/Development/jiracli/venv/lib/python3.8/site-packages/dateutil/tz/tz.py in _naive_is_dst(self, dt)
    257
    258     def _naive_is_dst(self, dt):
--> 259         timestamp = _datetime_to_timestamp(dt)
    260         return time.localtime(timestamp + time.timezone).tm_isdst
    261

~/Development/jiracli/venv/lib/python3.8/site-packages/dateutil/tz/tz.py in _datetime_to_timestamp(dt)
   1812     seconds since January 1, 1970, ignoring the time zone.
   1813     """
-> 1814     return (dt.replace(tzinfo=None) - EPOCH).total_seconds()
   1815
   1816

AttributeError: 'NoneType' object has no attribute 'replace'

[asfimport]

Antoine Pitrou / @pitrou:
The problem is that, once you have tz = dateutil.tz.gettz('Europe/Brussels'), it seems impossible to get back the "Europe/Brussels" string from tz.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
In addition, we should probably also support the Python 3.9+ zoneinfo timezones:

>>> import zoneinfo
>>> tz = zoneinfo.ZoneInfo("Europe/Brussels")
>>> tz
zoneinfo.ZoneInfo(key='Europe/Brussels')
>>> tz.key
'Europe/Brussels'

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
For dateutil, I indeed don't directly know how to get the zone name. You can get the "displayed name" with:

>>> import dateutil
>>> tz = dateutil.tz.gettz('Europe/Brussels')
>>> tz
tzfile('/usr/share/zoneinfo/Europe/Brussels')
>>> import datetime
>>> dt = datetime.datetime(2012, 1, 1)
>>> tz.tzname(dt)
'CET'

but that's not the name we want to store in the tz metadata.

There seems to be a long-time issue about it: dateutil/dateutil#76

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Issue resolved by pull request 12421
#12421