dt[TRUE] no longer performs shallow copy

[renkun-ken]

In previous versions of data.table, dt1 <- dt[TRUE] creates a shallow copy of dt so that dt and dt1 have different memory addresses. It would be safe to add columns to dt1 without influencing dt with no cost of copying any column in it. This is particularly useful when dt is extremely large and different scripts need to use it to compute different columns without copying it.

library(data.table)

dt <- data.table(id = 1:10)
dt1 <- dt[TRUE]
dt1[, x := 1]
dt2 <- dt[TRUE]
dt2[, x := 2]
dt
#>     id
#>  1:  1
#>  2:  2
#>  3:  3
#>  4:  4
#>  5:  5
#>  6:  6
#>  7:  7
#>  8:  8
#>  9:  9
#> 10: 10
dt1
#>     id x
#>  1:  1 1
#>  2:  2 1
#>  3:  3 1
#>  4:  4 1
#>  5:  5 1
#>  6:  6 1
#>  7:  7 1
#>  8:  8 1
#>  9:  9 1
#> 10: 10 1
dt2
#>     id x
#>  1:  1 2
#>  2:  2 2
#>  3:  3 2
#>  4:  4 2
#>  5:  5 2
#>  6:  6 2
#>  7:  7 2
#>  8:  8 2
#>  9:  9 2
#> 10: 10 2

address(dt)
#> [1] "0x7f8655ef4200"
address(dt1)
#> [1] "0x7f8655f40000"
address(dt2)
#> [1] "0x7f8655eae200"

3937881 changes this behavior and dt[TRUE] will not shallow copy dt so that the following code does not work any more.

library(data.table)

dt <- data.table(id = 1:10)
dt1 <- dt[TRUE]
dt1[, x := 1]
dt2 <- dt[TRUE]
dt2[, x := 2]
dt
#>     id x
#>  1:  1 2
#>  2:  2 2
#>  3:  3 2
#>  4:  4 2
#>  5:  5 2
#>  6:  6 2
#>  7:  7 2
#>  8:  8 2
#>  9:  9 2
#> 10: 10 2
dt1
#>     id x
#>  1:  1 2
#>  2:  2 2
#>  3:  3 2
#>  4:  4 2
#>  5:  5 2
#>  6:  6 2
#>  7:  7 2
#>  8:  8 2
#>  9:  9 2
#> 10: 10 2
dt2
#>     id x
#>  1:  1 2
#>  2:  2 2
#>  3:  3 2
#>  4:  4 2
#>  5:  5 2
#>  6:  6 2
#>  7:  7 2
#>  8:  8 2
#>  9:  9 2
#> 10: 10 2

address(dt)
#> [1] "0x7fb92930be00"
address(dt1)
#> [1] "0x7fb92930be00"
address(dt2)
#> [1] "0x7fb92930be00"

Currently data.table:::shallow is not exported so there's no way to use public API to shallow copy a data.table without losing its key. (.subset(dt, ...) and then setDT will shallow copy dt but its key will lose).

[mattdowle]

This is amazing you found so quick after the change. It helps a lot with #3213 I was just looking at too : test 1542 is exactly this ...

[renkun-ken]

Thanks for the quick fix!

Just notice I filed a minor issue #2254 long time ago related to such shallow on subsetting behavior.

[mattdowle]

Thanks! I linked up #2254 to #3215 and will look at the same time.

[jangorecki]

@mattdowle Is it actually desired behaviour to perform shallow copy on dt[TRUE]? In my opinion it is not.

[mattdowle]

@jangorecki But then what is desirable behavior: a deep copy or no copy? And why.

[jangorecki]

@mattdowle IMO deep copy. DT[TRUE] - I see passing TRUE to i as analogous to select * from tbl where 1 = 1, so filter on always true condition.

[mattdowle]

@jangorecki Note that TRUE has to be explicitly written just like that: DT[TRUE] (which is strange anyway) not DT[1==1]. I agree in general but Kun is using this unadvertised feature and depending on it. So until we can export shallow() I figured just to leave it as-is for now and come back to it when we can be sure what Kun should change to. Otherwise we break his workflow.

[renkun-ken]

@mattdowle agreed. I only need shallow(). Since it is not yet exported, I can only rely on dt[TRUE] at the moment. If shallow() is exported, I'm okay with that dt[...] always performs deep copy.