Inconsistency for sanity, being data.frame like for easy transition

[arunsrinivasan]

(After a brief discussion with Matt)

The behaviour with=FALSE:

require(data.table)
DT = data.table(x=1:5, y=6:10, z=11:15)

DT[, c("y", "z"), with=FALSE]

In talking to colleagues, and at meetings or over emails, it seems that restoring the data.frame behaviour only for those cases where j is integer/character vector can only bring more sanity (trading inconsistency).

The issue is that data.table usage revolves around [ a lot, and therefore users are confronted with having to learn this difference quite early, and having to learn new syntax for a known basic operation doesn't sit well. It also doesn't seem to help in explaining how a data.table is a data.frame with this basic operation.

AFAICT, there's no real usage to having just character/integer vectors in j. Therefore, it'd be great to have with=FALSE being unnecessary and be able to subset columns the data.frame way:

DT[, c("y", "z")]
DT[, 2:3]

The default return of vector in case of only one column and use of drop=FALSE should also be restored. This'll help get over the basic data.frame like usage very quickly without having to wonder "why", and start learning the actual essential enhanced-ness data.table provides.

It'd be great to hear thoughts from other users as well.

This has come up before (raised by Matt) : http://r.789695.n4.nabble.com/with-FALSE-td4589266.html but 'leave it as it is' was the response more or less.

[DavidArenburg]

+1MM

[jrowen]

I like this idea too. From the earlier responses, it seems the biggest drawback could be introducing inconsistency, as a new user would expect the two approaches below to return the same result.

DT[,c("colA","colB")] 

colvars = c("colA","colB") 
DT[,colvars]

Is there a way they both could return the same result?

[arunsrinivasan]

@jrowen thanks. Yes they both should return the same result (as data.frame would).
Will have to think a bit more about this though.

x <- "z"
DT[, x]

It'd be ambiguous in this case, isn't it?

One way off the top of my head is for the enhanced-ness to kick in j, only when it is wrapped with .() or list(), but perhaps that's too big a design change...

Hm, now I'm thinking if this'd only create more problems instead :-(

[markdanese]

Perhaps you could make the error message more friendly and help the user. Or even find the cases and add "with = FALSE" and advise the user that the change was made (like with setting column names the "old" way). I have been using data.table for a year and a half, and I periodically want to use column numbers for some quick interactive work and get an error. Not a big deal to type with = FALSE, but a nice reminder would be welcome. This would serve to teach new users as well.

[franknarf1]

I don't know. It might just make it harder for people to learn. I agree with Mark that adding a discouraging warning would help with that.

If you allow too much prominence to this way of accessing columns, it may prove something of a slippery slope. Can you really do this without also doing these?

• allowing numeric vectors (which are truncated to floor(j) in data.frames)
• making DT[int_or_char] match the data.frame analogue (where it subsets DT like a list)

Aside: If you do this, perhaps you could add some faster accessor for j (in terms of shorter code), analogous to the list subsetting in my last bullet point. I find with=FALSE awkward and verbose and so had been doing workarounds like [.listof`(DT,int_or_char)` (broken in R 3.2.0 onward) and [.noquote(DT,int_or_char). A function like this would allow experienced users of the new functionality to sidestep the warning Mark suggested and to write clearer, more readable code (since, on reviewing their code, they wouldn't have to wonder whether they were looking at data.table- or data.frame-style j).

EDIT: I'm trying to explain what I mean over here: http://chat.stackoverflow.com/transcript/message/24012297#24012297

[eantonya]

I quite like the automatic with=FALSE guessing, but not the drop
reinstatement - I don't want to see that terrible option resurrected and
muddying the waters of data.table.

[raubreywhite]

I agree with eduard, drop= true is one of the worst parts of data.frame. I think it makes sense to implement with= false, as this improves consistency and doesn't materially degrade the quality of data table, but drop= true would just be implementing a bad idea for the sake of consistency.

Sent from my iPhone

On 22 Jun 2015, at 5:22 am, eduard notifications@github.com wrote:

I quite like the automatic with=FALSE guessing, but not the drop
reinstatement - I don't want to see that terrible option resurrected and
muddying the waters of data.table.

On Sun, Jun 21, 2015, 10:37 AM franknarf1 notifications@github.com wrote:

I don't know. It might just make it harder for people to learn. I agree
with Mark that adding a discouraging warning would help with that.

If you allow too much prominence to this way of accessing columns, it may
prove something of a slippery slope. Can you really do this without also
doing these?

• setting drop=TRUE as default
• allowing numeric vectors (which are truncated to floor(j) in
  data.frames)
• making DT[int_or_char] match the data.frame analogue (where it
  subsets DT like a list)

Aside: If you do this, perhaps you could add some faster accessor for j
(in terms of shorter code), analogous to the list subsetting in my last
bullet point. I find with=FALSE awkward and verbose and so had been doing
workarounds like [.listof(DT,int_or_char)(broken in R 3.2.0 onward) and [.noquote(DT,int_or_char). A function like this would allow experienced users of the new functionality to sidestep the warning Mark suggested and to write clearer, more readable code (since, on reviewing their code, they wouldn't have to wonder whether they were looking at data.table- or data.frame-stylej).

—
Reply to this email directly or view it on GitHub
#1188 (comment)
.

—
Reply to this email directly or view it on GitHub.

[franknarf1]

I think it defeats the purpose of the change if you don't use drop=TRUE for these character-or-integer cases. If using data.table syntax DT[,.(mycol)], retain drop=FALSE, sure; I don't think changing that case would help anything.

[eantonya]

@franknarf1 I disagree. drop argument is only relevant for single column retrievals, so not having it only affects part of the cases, and the effect it has on those cases is one of consistency, and not the strange sometimes this sometimes that behavior of data.frame.

[franknarf1]

@eantonya Yeah, I guess we do disagree; sorry if I'm repeating myself, but I'll try to clarify. I'm not crazy about the sometimes-this-sometimes-that behavior of data.frame either, but the premise of this proposed enhancement is that data.frame syntax should be supported to some limited extent.

Within that limited scope (when j (1) does not use any columns of DT and (2) evaluates to character or integer... or something like that), we should give people what they expect. It's not like you or I are going to use it, so what harm? And if we don't give them what they expect, why bother giving them the concession to begin with? They'll still have grounds to complain about inconsistency. (I won't use it because I want to be able to read my code without the mental overhead of figuring out whether data.frame syntax is being used.)

[arunsrinivasan]

@franknarf1 perhaps I should clarify.

Ideally, what I'd like is for data.frame syntax in j to do everything that data.frame syntax does as shown below:

DT[, 1:2]
DT[, c("x", "y")]

cols = c("x", "y")
DT[, cols]

all of these should return two column data.table.

However, as @jrowen pointed out from the old post, the last case is tricky (for cases like the one I've shown in the previous post). Unless this case can be taken care of quite nicely, I personally don't see a huge advantage of implementing this feature. I can imagine myself explaining the behaviour to beginners (or in a talk) with too many ifs-and-buts.. and that's not helping.

So, what would be great is to figure out whether there's a way around the last scenario without breaking too many things. And whether it's worth it.

I don't feel strongly about drop = . being present or not. And IMO that's not the main part of this discussion, at least until it's clear that we are going to implement this functionality.

I'm also fully aware of the case DT[3:4] vs DF[3:4], but this doesn't seem to come up at all as an issue.. (on SO, or r-help or here or data.table-help) AFAICT.

[franknarf1]

@arunsrinivasan Yeah, I also don't see a benefit from the feature change. As you say, it seems like it would make explaining the syntax harder and lead to messier code everywhere (as people start using data.frame syntax as a crutch).

Back to my aside (mentioned in your last sentence). Yeah, I've never seen anyone else complain about DT[1:3] vs DF[1:3], but maybe they should! Really, if we had the functionality mentioned in this thread so that DT[.SDcols=1:3] and DT[.SDcols=c("a","b")] worked as my intuition suggests they should, it would be really handy. It's off-topic here, because that change wouldn't be any sort of crutch for people who don't want to learn data.table syntax, though. Not sure if that's already a FR... Oh, just found it: #1149