[Request] IDateTime should use the time zone of a POSIXct argument consistently on the date and time

[e-mu-pi]

Edit: I changed the issue title based on my current understanding. Previously it read: Pass along ..in IDateTime or remove from documentation. Here's the quick version. My two posts propose a solution, but it may violate some goals of the package.

# create a POSIXct variable with non-UTC time zone. Problem will
# also occur without tz argument if your system time is not UTC.
posix_tz <- as.POSIXct("2014-12-02 18:30:00", tz = "US/Central")
IDateTime(posix_tz)
# date converts to UTC, time remains local.
#        idate    itime
#1: 2014-12-03 18:30:00

The ... in IDateTime are not passed along to as.IDate and as.ITime. This would make it easier to handle time zones. If there are reasons not to do that, maybe it should be removed from the function and the documentation (I don't know enough about R to say one way or the other; it seems conventional to have ... everywhere).

I realize that ITime does not really support time zones, and I'm not trying to suggest it should. I would just like a way to manage data when I do not have the luxury of forcing UTC (data must be delivered downstream in local time zones for Shiny app). I take my UTC data, convert it to the local time zone, then try to convert to IDateTime for use in a data.table.

As I hope my example shows, the issue can be worked around with as.IDate and as.ITime individually. It took me a long time to realize that when I used IDateTime with a tz argument, it wasn't getting used. Maybe this could save someone else from getting confused on this point.

As a side note, the help(IDateTime) includes the following example.

datetime <- seq(as.POSIXct("2001-01-01"), as.POSIXct("2001-01-03"), by = "5 hour")    
(af <- data.table(IDateTime(datetime), a = rep(1:2, 5), key = "a,idate,itime"))

af[, mean(a), by = list(wday = wday(idate))] 

If the user has not set her time zone to UTC, this will produce unexpected results (at least they are unexpected if you don't understand the details of how this works). For example, US time zones will not see '2001-01-01 20:00:00' because it mixes the UTC converted date '2001-01-02' with the local time '20:00:00'.

library(data.table)
sessionInfo()
# R version 3.1.1 (2014-07-10)
# Platform: x86_64-redhat-linux-gnu (64-bit)
# 
# locale:
#   [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
# [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
# [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
# [10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
# 
# attached base packages:
#   [1] stats     graphics  grDevices utils     datasets  methods   base     
# 
# other attached packages:
#   [1] data.table_1.9.4
# 
# loaded via a namespace (and not attached):
#   [1] chron_2.3-45  plyr_1.8.1    Rcpp_0.11.3   reshape2_1.4  stringr_0.6.2 tools_3.1.1  
my_tz <- 'US/Central'
utc <- 'UTC'
#Local time crosses UTC date change
posixct_with_tz <- as.POSIXct(c("2014-12-01 17:30:00",
                                "2014-12-01 18:30:00"), 
                              tz = my_tz)

# as.IDate (from as.Date) by default will convert 
# POSIXct to UTC.
# You can force local timezone using tz, which you
# need to do to agree with ITime default.
as.IDate(posixct_with_tz) # returns UTC dates
# [1] "2014-12-01" "2014-12-02"
as.IDate(posixct_with_tz, tz = my_tz)
# [1] "2014-12-01" "2014-12-01"
as.IDate(posixct_with_tz, tz = utc)
# [1] "2014-12-01" "2014-12-02"

# Default behavior for as.ITime is contrary to what as.IDate 
# does. That's because as.ITime passes its argument to
# as.POSIXlt, which recognizes and retains the timezone.
# For agreement with as.IDate default, you should convert to UTC.
as.ITime(posixct_with_tz) #returns local time
# [1] "17:30:00" "18:30:00"
as.ITime(posixct_with_tz, tz = my_tz)
# [1] "17:30:00" "18:30:00"
as.ITime(posixct_with_tz, tz = utc)
# [1] "23:30:00" "00:30:00"


# Despite the documented '...' in IDateTime, additional 
# arguments are not passed to as.IDate and as.ITime.
IDateTime(posixct_with_tz) # UTC Dates, Local Times
IDateTime(posixct_with_tz, tz = my_tz) # UTC Dates, Local Times
IDateTime(posixct_with_tz, tz = utc) # UTC Dates, Local Times
# All return hybrid utc date, local time
#         idate    itime
#1: 2014-12-01 17:30:00
#2: 2014-12-02 18:30:00


# Pass along tz to avoid IDate, ITime disagreement
myIDateTime <- function(x,...) {
  data.table(idate = as.IDate(x,...), itime = as.ITime(x,...))
}
myIDateTime(posixct_with_tz, tz = my_tz) # Local Dates, Local Times
#         idate    itime
#1: 2014-12-01 17:30:00
#2: 2014-12-01 18:30:00
myIDateTime(posixct_with_tz, tz = utc) # UTC Dates, UTC Times
#         idate    itime
#1: 2014-12-01 23:30:00
#2: 2014-12-02 00:30:00

[e-mu-pi]

For my project, I created a specialization of as.IDate for POSIXct along with about 20 tests that it works as I expect. Unlike as.Date, this will not convert POSIXct arguments to UTC, so this may not fit the goals about how directly IDate can replace Date.

I like this approach because the user will have a what you see (in POSIXct) is what you get (in IDate), without any auto-conversion to UTC. An alternative would be to specialize as.ITime for POSIXct to automatically convert times to UTC before casting them to ITime.

In particular, the example from help(IDateTime) will give the weekdays corresponding to dates stored in the POSIXct arguments, not the weekdays after converting the POSIXct arguments to UTC. In the example from help, you won't see a change if your time zone is set to UTC. Because my use case is specifically wanting to use local times, I added a time zone to the example below.

as.IDate.POSIXct <- function(x,...) {
    if( hasArg("tz") ) {
        as.IDate( as.Date(x, ...) )
    }
    else if( "tzone" %in% names(attributes(x)) ) {
         as.IDate( as.Date(x, tz = attr(x,"tzone"), ...) )
    }
    else {
        attr(x,"tzone") <- Sys.timezone() # make system time zone explicit,
        as.IDate( x, ...) # then call this again
    }
}

# example from help(IDateTime) with explicit time zone
tz <- "US/Central"
datetime <- seq(as.POSIXct("2001-01-01",tz=tz), as.POSIXct("2001-01-03",tz=tz), by = "5 hour")    
(af <- data.table(IDateTime(datetime), a = rep(1:2, 5), key = "a,idate,itime"))

af[, mean(a), by = list(wday = wday(idate))] 
# Without redefining for POSIXct, this produced weekdays in UTC
#   wday  V1
#1:    2 1.5
#2:    3 1.4
#3:    4 2.0
# With specialization of IDate for POSIXct, the local weekdays are used
#   wday  V1
#1:    2 1.4
#2:    3 1.6