[Python] Write nanosecond timestamps using new NANO LogicalType Parquet unit

[asfimport]

The following code

import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd

n=3
df = pd.DataFrame({'x': range(n)}, index=pd.DatetimeIndex(start='2017-01-01', freq='1n', periods=n))
pq.write_table(pa.Table.from_pandas(df), '/tmp/t.parquet')

results in:

ArrowInvalid: Casting from timestamp[ns] to timestamp[us] would lose data: 1483228800000000001

The desired effect is that we can save nanosecond resolution without losing precision (e.g. conversion to ms). Note that if freq='1u' is used, the code runs properly.

Environment: Python 3.6.4. Mac OSX and CentOS Linux release 7.3.1611. Pandas 0.21.1 .
Reporter: Jordan Samuels
Assignee: TP Boudreau / @tpboudreau

Related issues:

• [Python] Cast all timestamp resolutions to INT96 use_deprecated_int96_timestamps=True (is related to)
• [C++] Support for writing TIMESTAMP_NANOS Parquet metadata (is related to)
• [C++] Upgrade to use LogicalType annotations instead of ConvertedType (depends upon)

_{Note: This issue was originally created as ARROW-1957. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Nanosecond resolution timestamps are deprecated in the Parquet format. You can use the use_deprecated_int96_timestamps option to use deprecated 96-bit nanosecond timestamps

http://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow.parquet.write_table

[asfimport]

Jordan Samuels:
@wesm Thanks for your response and explanation. I'm still not clear on why nanosecond resolution timestamps are deprecated - is that for performance reasons? What are the medium-/long-term ramifications of using the deprecated option you mention?

[asfimport]

Wes McKinney / @wesm:
This is a question for the Parquet project, I recommend reaching out to dev@parquet.apache.org to discuss further

[asfimport]

Krisztian Szucs / @kszucs:
@wesm Invalid?

[asfimport]

Wes McKinney / @wesm:
Nanoseconds are being added as a supported timestamp resolution for INT64 storage, see

https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L261

Upgrading to make use of this in Parquet C++ is not trivial (umbrella JIRA for this is PARQUET-1411), so I suggest leaving this issue open until this is supported in Arrow writes

[asfimport]

Wes McKinney / @wesm:
@tpboudreau I assume this is on your critical path

[asfimport]

TP Boudreau / @tpboudreau:
Yes, thanks for assigning it.

[asfimport]

TP Boudreau / @tpboudreau:
There's a test resembling the reporter's code confirming that this issue will be fixed when ARROW-3729 is resolved:

https://github.com/apache/arrow/pull/4421/files#diff-348d66d69aec7aedf0a6f753613acb43R2052

[asfimport]

Wes McKinney / @wesm:
Resolved in d54425d

[asfimport]

Daniel Nugent / @nugend:
@tpboudreau Perhaps this issue should be reopened. This error is present today in 0.16.0 when attempting to write parquet files if the deprecated 96 bit timestamps are not explicitly enabled. I am not sure if this is intentional or not but from the previous comments that would seem to not be the case.

[asfimport]

Wes McKinney / @wesm:
Want to open a new issue? Or if it's definitely the same issue (and was not fixed last time) you can reopen

[asfimport]

TP Boudreau / @tpboudreau:
I'm fairly confident this issue was resolved in the 0.14.x release. I won't have a chance to look closer or attempt to reproduce this for a few days. If a new issue is opened on this, please tag me.

[asfimport]

Daniel Nugent / @nugend:
Ah, my apologies. I see where my error was. I failed to specify the 2.0 parquet file version.