Release checks

[mattdowle]

Hi @renkun-ken.
Could you run latest data.table dev through your systems please.
There are some significant internal code changes that it would be good to run past your checks early.
Please check that more cores are being used more of the time, and whether your test suite total time is any faster or not.
Thanks!!
Matt

[renkun-ken]

I notice that significant changes are made recently so I keep up with those on a daily basis. So far the system runs perfectly with 4c9a1f0.

As for the total time, it is unfortunate that fork parallelism is heavily used so that multi-threading falls to single thread and does not recover when fork is ended. Therefore I haven't observed significant difference in computing time so far.

I'll test against the latest commit very soon.

[mattdowle]

Thanks @renkun-ken!
We could maybe have a look at fork again. See past problems here :
https://github.com/Rdatatable/data.table/blob/master/src/openmp-utils.c#L73
Are you on Linux/Mac/WIndows? ( I didn't think Windows had fork, but somebody told me recently that it does.)
As those comments say, we used to have an after_fork callback that restored omp to using more than 1 thread. But the problems were on Mac and/or Intel's iomp. It may be possible to try it again just on Linux, if you're on Linux.
Does setDTthreads(0) restore multi-threaded after a parallel fork for you? If so, there's a good chance the after_fork could turn it on automatically if we put it back.

[renkun-ken]

I'm on Linux.

I test a small part of system against 0daefad and the computing time reduced 30%. I'll do a more thorough test soon.

I filed #2885 on auto-restore multi-threading after fork. fst adopts this behavior for quite a while and it works for me perfectly so far.

[renkun-ken]

Latest commit 0daefad is broken here. I'll take a closer look and file an issue about it soon.

[renkun-ken]

The problem lies on an assumed behavior that dt[TRUE] always creates a shallow copy of dt so that adding columns to the shallow copy does not influence the original data.table. See #3214.

[mattdowle]

Thanks @renkun-ken. #3214 now fixed by PR #3213 and merged.

[renkun-ken]

I filed another issue #3216 with the latest commit 5b16cc5.

[mattdowle]

@renkun-ken Many thanks for this invaluable testing!!
#3216 fixed, and #2885 (restore multi threading after fork) done too.
Please rerun whenever convenient.

[renkun-ken]

A full run with 76d4a62 is in progress...

[renkun-ken]

Just found another issue which seems to be breaking. The updated subsetting tends to under-allocating. See #3228.

[renkun-ken]

Empty .SDcols now produces an error (which I think is by design in recent changes?), following code no longer works:

dt <- data.table(x = rnorm(10), y = rnorm(10))
cols <- intersect(c("p", "q"), names(dt))
dt[, x := rowSums(.SD), .SDcols = cols]

As #3229 has just been merged, no other issues are found so far.

[mattdowle]

Thanks again for the quick turnaround!

Looks like that failed in 1.11.8 too but with a better message. Will make the error message same as before but still an error, right?

> dt <- data.table(x = rnorm(10), y = rnorm(10))
> cols <- intersect(c("p", "q"), names(dt))

v1.11.8
======
> dt[, x := rowSums(.SD), .SDcols = cols]
Error in `[.data.table`(dt, , `:=`(x, rowSums(.SD)), .SDcols = cols) : 
  RHS of assignment to existing column 'x' is zero length but not NULL. If you intend to
delete the column use NULL. Otherwise, the RHS must have length > 0; e.g., NA_integer_.
If you are trying to change the column type to be an empty list column then, as with all
column type changes, provide a full length RHS vector such as vector('list',nrow(DT));
i.e., 'plonk' in the new column.

dev error not intended
======
> dt[, x := rowSums(.SD), .SDcols = cols]
Error in `[.data.table`(dt, , `:=`(x, rowSums(.SD)), .SDcols = cols) : 
  Empty .SDcols is not yet supported.

[renkun-ken]

Sorry, I just noticed that there might be a bug in 1.11.8 on empty .SDcols that I happended to take advantage of unintentionally.

The following code works with 1.11.8:

library(data.table)

dt <- data.table(x = rnorm(10), y = rnorm(10))
gs <- list(a = c("x", "y", "z"), b = c("p", "q"))

for (g in names(gs)) {
  dt[, (g) := rowSums(.SD), .SDcols = intersect(colnames(dt), gs[[g]])]
}

But if the for-loop is run again, the RHS ... error occurs.

I think I should change my code here.