Allow the use of `..var` in i

[markfairbanks]

It would be useful if this was possible for use in i, much like it is possible to use in j:

library(data.table)

test_df <- data.table(x = 1:3, y = 1:3)
x <- 3

# Works in j
copy(test_df)[, new := y + ..x][]
#>    x y new
#> 1: 1 1   4
#> 2: 2 2   5
#> 3: 3 3   6

# Fails in i
test_df[y < ..x]
#> Error in .checkTypos(e, names_x): Object '..x' not found amongst x, y

[avimallu]

I suppose this can be now closed given the integration of #4304:

> library(data.table)
> test_df <- data.table(x = 1:3, y = 1:3)
> x <- 3
> test_df[y < x, env = list(x = x)]
   x y
1: 1 1
2: 2 2

[markfairbanks]

I still think this would be worth adding even with the new addition. While the new env argument is useful, the ..var syntax allows you to differentiate between the column var and the global var. For example:

library(data.table)

test_df <- data.table(x = 1:3, y = 1:3)
x <- 3

# Can refer to both column x and global x in j
copy(test_df)[, new := x + ..x][]
#>    x y new
#> 1: 1 1   4
#> 2: 2 2   5
#> 3: 3 3   6

# Can now only refer to global x
test_df[x < x, env = list(x = x)]
#> Empty data.table (0 rows and 2 cols): x,y

In this case it would be useful to be able to do:

test_df[x < ..x]

Edit
Actually as I think about this this could be solved by using a different name for the global x inside the env arg

library(data.table)

test_df <- data.table(x = 1:3, y = 1:3)
x <- 3

test_df[x < env_x, env = list(env_x = x)]
#>    x y
#> 1: 1 1
#> 2: 2 2

I would still prefer the ..var syntax (it seems much cleaner to me), but the issue could be closed if the env argument was made to be the recommended way moving forward.

[avimallu]

The env argument is the recommended method that replaces get, mget and eval, although not the .. operator, as will be mentioned in the datatable-programming vignette when it is released:

data.table/vignettes/datatable-programming.Rmd

Line 369 in 79c3d3e

In `[.data.table` it is also possible to use other interfaces for variable substitution, or passing quoted expressions. Those are `get` and `mget` for inline injection of variables by providing their names as character, and `eval` that tells `[.data.table` that expression we passed into an argument is a quoted expression, so should be handled differently. Those interfaces should be considered as retired and we recommended to use new `env` argument instead.

[markfairbanks]

although not the .. operator

If the .. operator isn't being retired, wouldn't that allow it to be expanded to work in i?

It's up to the data.table team of course, but I still think this functionality would be useful.

[jangorecki]

now you can have DIY prefixes
this should work as well

test_df[x < .x, env = list(.x = x)]
test_df[x < ..x, env = list(..x = x)]
test_df[x < ...x, env = list(...x = x)]

[jangorecki]

AFAIK .. is not retired, but I can imagine that .. development can slow down now when there is alternative approach.
I think we could even try to pre-process expression for handling .. at the very beginning, where we do handle now new interface.

[dvg-p4]

I find test_df[x < ..x, env = list(..x = x)] to be pretty ugly and verbose, and not a lot better than just

..x = x
df[x == ..x]

It would be very nice if .. just worked out-of-the-box in i like it does in j.

[jangorecki]

I understand your point well.

New meta programming interface was developed having in mind it will be consumed by downstream packages and other software, like R scripts or shiny apps. Therefore more important than fast typing and reading was to be unambiguous (not being affected by scoping of overlapping names) and to be deeply customizable. ..var interface fails in both of those cases.

[dvg-p4]

True, you'd have the ambiguity right back if someone named a variable ..var in the calling scope...but isn't that just as much a problem for j? I'm not seeing how the current behavior (when not explicitly using env) avoids "being affected by scoping of overlapping names".

[jangorecki]

Yes it is

possibly(?) you can use ..var in i already now, but not nested inside an expression yet

[dvg-p4]

Nope, doesn't seem to be recognized at all (unless that's changed in dev? I'm on 1.16.0)

[jangorecki]

No, then must not be there yet