Wrong units displayed in data.table with POSIXct arithmetic

[boethian]

Recently posted on StackOverflow because didn't know how to report bug report / make feature request here.

http://stackoverflow.com/questions/25214170/wrong-units-displayed-in-data-table-with-posixct-arithmetic

Problem reproduced here for completeness, I can take down SO "question" if this bug is deemed legitimate here:

When durations are computed in data.table (v1.9.2), the wrong units can be printed with POSIXct arithmetic. It seems the first units are chosen.

require("data.table")
dt <- data.table(id=c(1,1,2,2), 
                  event=rep(c("start", "end"), times=2), 
                  time=c(as.POSIXct(c("2014-01-31 06:05:30", 
                                      "2014-01-31 06:45:30", 
                                      "2014-01-31 08:10:00", 
                                      "2014-01-31 09:30:00"))))
dt$time[2] - dt$time[1]  # in minutes
dt$time[4] - dt$time[3]  # in hours
dt[ , max(time) - min(time), by=id]  # wrong units printed for id 2

I realize that one of these is the correct way to do it to get expected behavior, but wanted to report this behavior. Not sure if it is really a data.table problem or POSIXct problem.

dt[ , difftime(max(time), min(time), units="mins"), by=id]  # both mins
dt[ , difftime(max(time), min(time), units="hours"), by=id]  # both hours

[mattdowle]

Agreed it's a bug. Thanks for the nice report. I tested in latest dev just now and it occurs there too.

Good to have the S.O. question as well i.e. don't take it down. More chance people will find it and see the workarounds etc.

As a first step, the attributes on each group result could be checked for consistency and a warning issued if not. I thought it did that already but clearly not. Tagged High for the this detection and warning aspect which should be relatively straightforward and could be an issue in other cases unrelated to POSIXct. Making it return the right result without speed penalty could be hard, so likely best to have user call difftime(,units=), and that could be a suggestion in the warning message.

[MichaelChirico]

Was just trying to figure out if I couldn't fix this, but it seems the mistake is done entirely in C (everything's fine up until .Call(Cdogroups, ...) AFAICT), so it's above my weight class for now... basically the group concatenation must be ignoring the attr of the difftime objects

[MichaelChirico]

Some more details copied as this re-surfaced for me:

library(data.table)
DT = data.table(
  a = c(1, 1, 2, 2),
  b = structure(
    c(0, 10, 0, 100000000), 
    class = c('POSIXct', 'POSIXt')
  )
)
DT[ , diff(b), keyby = a]
#    a            V1
# 1: 1   10.000 secs
# 2: 2 1157.407 secs

The problem is that diff.POSIXt silently invokes -.POSIXt, which chooses units automatically given its inputs.

These units are ignored on group wrapup:

DT[ , {
  x = diff(b)
  dput(x)
  x
}, keyby = a]
# structure(10, units = "secs", class = "difftime")
# structure(1157.40740740741, units = "days", class = "difftime")
#    a            V1
# 1: 1   10.000 secs
# 2: 2 1157.407 secs