[Format] Define more generic Interval logical type

[asfimport]

Per discussion in e7e399d#commitcomment-18711366, we can create an Interval type with a unit to be more general.

Reporter: Wes McKinney / @wesm
Assignee: Julien Le Dem / @julienledem

Related issues:

• Arrow release 0.1 (blocks)

PRs and other links:

• PR

_{Note: This issue was originally created as ARROW-270. Please see the migration documentation for further details.}

[asfimport]

Julien Le Dem / @julienledem:
How about the following? @StevenMPhillips @jacques-n @wesm

enum IntervalUnit: short { YEAR_MONTH, DAY_TIME}
table interval {
  IntervalUnit: unit;
}

References:
http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt section 4.5.2
https://msdn.microsoft.com/en-us/library/ms716506(v=vs.85).aspx
http://www.ibm.com/support/knowledgecenter/SSGU8G_12.1.0/com.ibm.esqlc.doc/ids_esqlc_0190.htm

[asfimport]

Uwe Korn / @xhochy:
Given that there are discussion to change the timestamp type to be based on Decimals and the unit introduced by the scale, shouldn't we also do this with the Interval type then?

[asfimport]

Jacques Nadeau / @jacques-n:
IntervalUnit seems fine to me.

As far as timestamp/decimal, I'm not inclined to change. I think most of the processing engines and storage formats that we work with use epoch in either millis, micros or nanos.

[asfimport]

Wes McKinney / @wesm:
Some other systems define an absolute "timedelta" type consisting of a particular number of days, seconds, milliseconds, microseconds, etc. The unit is fixed, and the timedelta is stored in int64 format

In [9]: import pandas as pd

In [10]: ts = pd.Timedelta(1000, unit='s')

In [11]: ts
Out[11]: Timedelta('0 days 00:16:40')

In [12]: ts.seconds
Out[12]: 1000

In [13]: ts.asm8 # Internal representation
Out[13]: numpy.timedelta64(1000000000000,'ns')

What do you think about this kind of data (it would share the same absolute time units as timestamp, basically)?

[asfimport]

Jacques Nadeau / @jacques-n:
This matches DAY_TIME I believe. The difference is that we are currently fixed to four bytes, right?

[asfimport]

Wes McKinney / @wesm:
Right – some SQL implementations only have hours/minutes/seconds. Based on the unit metadata (which we need to add I suppose) the storage could be int32 or int64.

[asfimport]

Julien Le Dem / @julienledem:
PR: #144

[asfimport]

Julien Le Dem / @julienledem:
Issue resolved by pull request 144
#144