Changig column classes is slow.

[juanpide]

I'm working with large datasets (around 4GB), mixing dates, numbers and factors.

And I want to perform operations such as changing all column classes to character.

mydata[, names(mydata) := lapply(.SD, as.character)]

But it's quite slow.

If you just had the data on the disk and reload it with
my <- fread("mydata.csv", stringsAsFactors=F, colClasses=list(character=1:3775))

it's much faster.

Is there alternative to the first method?
Or maybe it could be improved in some way.

[MichaelChirico]

What's the problem with the second method? colClasses = 'character' will suffice by the way; and you're much better off asking this on SO

[MichaelChirico]

Also, try Google:

http://stackoverflow.com/questions/28447014/the-fastest-way-to-convert-numeric-to-character-in-r

[juanpide]

The second method crashes with my data with the latest data.table version.

As the second method is much faster I think the first one could be improved as well.
Anyway using the second method, saving the file and reloading it again doesn't seem to be an elegant way of doing things.

[pstoyanov]

If your data has relatively few unique values, you can benefit substantially by using by = ... inside the DT[] -- this way you only call as.character once for each unique value. (I've had a 50+ times speedup when doing the same with as.yearmon.)

But then you have to create a temp variable (can't change the type of a subset of a column), and to loop over variable names outside the DT call. Then again, if you have the RAM, you can split the DT into chunks and work on them in separate threads for further speedup.

PS: Recently the docs said something about keyby now being much faster than by but I have not had the chance to play with it...

[juanpide]

@pstoyanov could you post the full syntax for example, please?

[pstoyanov]

Hi, here's a proof of concept for the by=... part. Maybe move the question to SO for more details & recommendations.

On my machine the first version takes ~25s, the one with by= takes about 2.5s.

Note the extra verbose part (printing of column names) is to underscore the need to use copy(names(DT)) -- try it without copy() and see what happens :)

library(data.table)

set.seed(1)
n <- 10^7
DT <- data.table(a = round(runif(n), 2), b = round(runif(n), 3), c = round(runif(n), 4))

str(DT)
DT1 <- copy(DT)
DT2 <- copy(DT)
tables()

system.time(DT1[, names(DT1) := lapply(.SD, as.character)])

system.time({
  dt2_names <- copy(names(DT2))
  cat(dt2_names, "\n")
  for(vn in dt2_names) {
    cat("\n", vn)
    DT2[, temp_var := as.character(get(vn)), by = vn]
    cat("\t", dt2_names, "\t", names(DT2))
    DT2[, (vn) := NULL]
    setnames(x = DT2, old = "temp_var", new = vn)
    cat("\t\t\t", dt2_names, "\t", names(DT2))
  }
})

setcolorder(DT1, names(DT))
setcolorder(DT2, names(DT))
identical(DT1, DT2)

[juanpide]

It works very well.
It would be great to have something like this implemented inside data.table.

It's strange if you don't use copy(names(DT2)) the result is disordered and the first column is not coverted to character.

[MichaelChirico]

Similar: #3279