Add a `copy` argument to `[`

[brodieG]

This would allow mimicking of transform / mutate behavior:

dt.a <- data.table(a=1:3)
dt.b <- dt.a[, b:=5:7, cp=T]

dt.b is now a new object with two columns, whereas dt.a remains a one column dt. While the reference nature of data.table is fantastic most of the times sometimes it can be a little annoying.

Simple work arounds exist so this is not high priority:

dt.a[][, b:=5:7]

Although maybe the ideal solution would be if data.table detects it is being used as part of an assignment, not to do the modification by reference (though that will likely break lots of existing code...).

[jangorecki]

similarly I'm already using kind of dt[,.SD][, b:=5:7].
I'm not sure if adding another arg to [ is a good idea because we would have soon tons of useful args there.
Surely the copy feature control is extremely important for data.table, at the moment I don't have any better ideas other than additional arg proposed by brodieG.
My use case of that is well described here: https://github.com/danielkrizian/rChartsDygraphs/pull/26/files#diff-6bfa0be678597a52f9e0c185c3bda239R75

[arunsrinivasan]

I'm not following this perhaps, but there's a copy() function specifically exported for this purpose, isn't it?

dt.b <- copy(dt.a)[, col := value]

[jangorecki]

Arun, correct, but...
when using more complex transformations it can get a little bit ugly
dt2 <- copy(dt[,fun_before_copy1(.SD)][,fun_before_copy2(.SD)])[,fun_after_copy3(.SD), by=grp_by_vars]
might also impact the performance(?):
above command would copy dt after fun_before_copy1 and fun_before_copy2
while the mentioned workaround:
dt2 <- dt[,fun_before_copy1(.SD)][,fun_before_copy2(.SD)][,.SD][,fun_after_copy3(.SD), by=grp_by_vars]
would copy only the final results which is already aggregated (smaller set), is my way of thinking correct?

[arunsrinivasan]

Jan, dt[, .SD] will already return a copy. And that'll be chained along with your other statements. It's equivalent to copying dt using copy() explicitly: copy(dt)[][][].

As long as you're chaining, you need to copy only the data.table you start with, and copy() is great for that. I can't think of a situation this gets ugly or where we need another argument to [.data.table.

I'm happy to be persuaded with examples :).

[brodieG]

Arun, regarding:

dt.b <- copy(dt.a)[, col := value]

Yes, certainly that's an option, but I see it as equivalent to:

dt.b <- dt.a[][, col:=value]

though you could argue that's really no more typing than

dt.b <- dt.a[, col:=value, cp=T]

Really, I think the problem is that j is too overburdened (I'm getting philosophical now) and it leads to at times awkward expressions. Here is a more complete FR (perhaps needs to be submitted separately) that would address all my concerns:

DT[
  i=..., 
  j=list(...),              # Current select from .SD behavior
  k=list(a=exp, b=exp...),  # Sets cols a, b, by reference, if only this is used returns invisibly
  l=c(...),                 # Current `j` with `with=F` behavior
]

This then allows for some pretty neat manipulations that are a little awkward now:

DT <- data.table(a=1:10, b=1:10)
DT[k=list(a=rev(a), b=1:5, c=letters[1:10], d=LETTERS[1:10])]   # modifies a, b, by reference, adds c, d, you can do this today
DT.B <- DT[j=list(e=a * 2)]                                     # unchanged from today
DT.C <- DT[j=list(e=a * 2), k=list(b=b^2), l=-3]                # modifies b in DT, returns with columns a, b, d, e (c skipped)

The last one might seem a little like .SDcols, but the main thing is it allows you to add columns:

DT[j=sapply(.SD, rev), l=1:2, .SDcols=3:4]   # returms columns 1:4 of DT, with 3:4 reversed

One problem is that order of l columns and j columns isn't defined...

[arunsrinivasan]

@brodieG, on the first part, again, what's the difference (other than syntax) between:

dt2 = copy(dt1)[, a := b]
dt2 = dt1[][, a := b]
dt2 = dt1[, a := b, copy=TRUE]

If nothing else is different (which I think is the case), what's good about the last syntax or bad about the first syntax?

And if the last syntax should first add the column and then copy, then:

dt2 = copy(dt1[, a := b])

But this use case is quite rare IMO.

---

On your point about i,j,k,l - this is quite subjective, and is likely to introduce another level of complexity to users who're used to the current syntax, not to mention a drastic change that'll break most user code, and perhaps deviating from data.table's philosophy, which is quite different from dplyr's - having a verb for every common operation and the rest as a do.

j can select, aggregate and add/update columns. This is quite natural to follow when coupled with i. The idea of i,j,k,l doesn't seem appealing to me than the current syntax.

But it's nice to have these discussions. IIRC some of the major changes to data.table have come through such discussions (and at times heated) ;).

[brodieG]

Arun, I think you're right that the copy argument isn't worth it.

Re ijkl, I don't think this would break anything. You could still use j with := and not even know k exists. Same with l. I am not proposing changing j behavior at first since the ijkl proposal does not require it (assuming general adoption, deprecation of := in j could be revisited, but again, not necessary).

Also, whether ijkl is good being subjective is something I don't want to give up on quite yet. To achieve:

DT[j=sapply(.SD, rev), l=1:2, .SDcols=3:4]

Currently, I think you have to resort to stuff like:

cbind(DT[j=sapply(.SD, rev), .SDcols=3:4], DT[j=1:2, with=FALSE])

or

DT[j=cbind(list(a, b), sapply(.SD, rev)), .SDcols=3:4]  # or some such, untested

and gets more annoying if you have large data tables with dozens of columns that you want returned unchanged in addition to the ones that are added (this is a very common use case for me: I have lots of columns, I want to compute new columns and use sapply(.SD) in doing so, and often using by as well).

Or how about this (note sapply should return names, which would then allow data.table to figure out what columns need to be over-written):

DT[k=sapply(.kSD, rev), .kSD:=-c(1, 3, 8)]   # reverse by reference all columns except 1, 3, and 8

Maybe I'm just not thinking about the right way to do this in current syntax, but this seems very natural to me and a lot easier, and also very much in keeping with the non-reference modifcation semantics of j.

By adding this k we can now leverage all the cool stuff you can do in j AND use it to modify by reference.

[arunsrinivasan]

It'd be great to explain with an example, so that we can better understand the use case.

[brodieG]

I'll put some together. Right now I have to stop procrastinating and do some work! Thanks for being up for an open discussion.

[eantonya]

For comparison, I think these are the current equivalent expressions for your examples:

DT <- data.table(a=1:10, b=1:10)

# proposed
DT[k=list(a=rev(a), b=1:5, c=letters[1:10], d=LETTERS[1:10])]   # modifies a, b, by reference, adds c, d, you can do this today
# current
DT[, `:=`(a = rev(a), b = 1:5, c = letters[1:10], d = LETTERS[1:10])]

# proposed
DT.C <- DT[j=list(e=a * 2), k=list(b=b^2), l=-3]                # modifies b in DT, returns with columns a, b, d, e (c skipped)
# current
DT.C <- DT[, b := b^2][, cbind(.SD, e = a * 2), .SDcols = -3]

[arunsrinivasan]

@eantonya thanks. I'd write that final syntax as follows:

DT.C <- DT[, b := b^2][, .SD, .SDcols=-3][, e := a^2]

In fact, I'd write DT[, b := b^2] separately as follows:

DT[, b := b^2]
DT.C <- DT[, .SD, .SDcols=-3][, e := a^2]
# or 
DT.C <- DT[, -"c", with=FALSE][, e := a^2]

But that's my personal preference.

The data.table syntax is DT[where, select | update, group by] + chaining, as explained quite nicely by Matt on slide 57 here. I'm guessing this feature is looking towards blurring the line between selecting and updating so that they don't have to be chained, and be used simultaneously.

I find the syntax still cleaner the current way, and there's no big speed loss. I for one never use data.tables i,j by specifying i= and j=. So, having k,l would result in code like: DT[,,list(...),, by=], where list() corresponds to k here.

IMO, it doesn't seem a compelling enough argument, still.

[brodieG]

@eantonya , @arunsrinivasan , I think the examples you put forth show that what I'm requesting is more or less covered and perhaps really just a matter of personal preference (I find it ungainly to have to type .SDcols=-3 to drop a column when using .SD, for example, but that's just me).

Also, I hadn't thought of using the form :=( (a.k.a. snotty unhappy child) of :=.

One that I can't quite figure out still: how do you deal with things like so:

DT[k=lapply(.kSD, rev), .kSD=-c(1, 3, 8)]   # reverse by reference all columns except 1, 3, and 8

Trying things like:

DT[, do.call(`:=`, lapply(.SD, rev)), .SDcols=-c(1, 3, 8)]

Don't seem to work (and even if they did I'd argue that's starting to get borderline cryptic). Is there a solution for this type of thing?

[eantonya]

I think the following syntax should work and warrants an FR:

DT[, names(.SD) := lapply(.SD, rev), .SDcols = -c(1,3,8)]

but right now you'll have to get the names manually beforehand.

[brodieG]

@eantonya , added as #795.