R 3.4.3 -> 3.5 performance drop

[s-Nick-s]

I've noticed a significant drop in data.table performance after upgrading to R 3.5.
For example:

set.seed(1)
types <- c("A", "B", "C", "D", "E", "F")
obs <- 4e7
dt1 <- data.table(percent = round(runif(obs, min = 0, max = 1), digits = 2),
                  type = as.factor(sample(types, obs, replace = TRUE)))
microbenchmark::microbenchmark(
  test1 <- one[, list(percent_total = sum(percent)), by = type]
, times = 30)

On 3.5 gives me:

Unit: milliseconds
                                                          expr    min      lq     mean   median       uq     max neval
 test1 <- dt1[, list(percent_total = sum(percent)), by = type] 522.53 539.668 576.2352 572.8301 602.0069 677.892    30
>sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)

Matrix products: default
BLAS: /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.19.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.11.4

loaded via a namespace (and not attached):
[1] microbenchmark_1.4-4 compiler_3.5.0       tools_3.5.0  

But on 3.4.3 it's ~20% faster:

Unit: milliseconds
                                                          expr      min       lq     mean   median       uq      max neval
 test1 <- dt1[, list(percent_total = sum(percent)), by = type] 415.5938 445.8137 477.6562 470.0536 499.8974 576.9203    30
> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)

Matrix products: default
BLAS: /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.19.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
 [6] LC_MESSAGES=C              LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.11.4

loaded via a namespace (and not attached):
[1] microbenchmark_1.4-4 compiler_3.4.3       tools_3.4.3   

[jangorecki]

Thank you for reporting. Confirmed on latest devel.
Smaller example.

library(data.table)
set.seed(1)
types <- c("A", "B", "C", "D", "E", "F")
obs <- 4e7
dt1 <- data.table(percent = round(runif(obs, min = 0, max = 1), digits = 2),
                  type = as.factor(sample(types, obs, replace = TRUE)))
system.time(dt1[, list(percent_total = sum(percent)), by = type])

3.4.4 timings are same as in 3.4.3.
3.5.1 timings are same as in 3.5.0.

[jangorecki]

I tested latest devel on 3.5.1 and 3.4.4 and there is no performance difference anymore. It is likely to be fixed by recent rework of many C functions. Please re-open if issue is still valid for you on latest devel.

[s-Nick-s]

@jangorecki
Unfortunately, the issue is still valid. After rerunning tests (with 4e6 observations) I get:

#Platform: x86_64-pc-linux-gnu (64-bit)
#R3.5.1
#1.11.8
Unit: milliseconds
                                                          expr      min       lq     mean   median       uq      max neval
 test1 <- dt1[, list(percent_total = sum(percent)), by = type] 50.22021 53.60446 56.84797 54.67532 57.61063 118.4352   100
 
#R3.4.4
#1.11.8
Unit: milliseconds
                                                          expr      min       lq     mean   median       uq      max neval
 test1 <- dt1[, list(percent_total = sum(percent)), by = type] 43.01133 44.16637 46.24535 44.63486 44.84428 89.37498   100

And 1.12.0 gives me x2 time increase on both R versions. (I think this is due to #3395):

#1.12.0
Unit: milliseconds
                                                          expr      min       lq    mean   median     uq      max neval
 test1 <- dt1[, list(percent_total = sum(percent)), by = type] 90.60795 93.11161 119.436 109.1086 146.69 178.8412    30

It looks like I don't have permissions to re-open this issue. Please re-open it if you can confirm my tests.

[jangorecki]

@s-Nick-s I think it is related to number of threads (as in the link you mentioned). Below is R 3.5.2 vs R 3.4.4 using only single thread in both cases, data.table 1.12.0.
Please confirm that setting threads to 1 makes R 3.5+ faster than R 3.4.

library(data.table)
set.seed(1)
types <- c("A", "B", "C", "D", "E", "F")
setDTthreads(1L)
obs <- 4e6
dt1 <- data.table(percent = round(runif(obs, min = 0, max = 1), digits = 2),
                  type = as.factor(sample(types, obs, replace = TRUE)))
system.time(dt1[, list(percent_total = sum(percent)), by = type])
microbenchmark::microbenchmark(
  test1 <- dt1[, list(percent_total = sum(percent)), by = type]
, times = 30)

3.5.2

   user  system elapsed 
  0.095   0.012   0.107 
> microbenchmark::microbenchmark(
+   test1 <- dt1[, list(percent_total = sum(percent)), by = type]
+ , times = 30)
Unit: milliseconds
                                                          expr      min
 test1 <- dt1[, list(percent_total = sum(percent)), by = type] 85.36912
       lq    mean   median       uq      max neval
 87.16382 99.1758 98.38073 105.0603 120.9579    30

3.4.4

   user  system elapsed 
  0.103   0.004   0.107 
> microbenchmark::microbenchmark(
+   test1 <- dt1[, list(percent_total = sum(percent)), by = type]
+ , times = 30)

Unit: milliseconds
                                                          expr      min
 test1 <- dt1[, list(percent_total = sum(percent)), by = type] 101.8106
       lq     mean   median       uq      max neval
 106.1996 115.2929 114.8559 122.1709 136.3099    30

[s-Nick-s]

@jangorecki
For me the 1.12.0 with setDTthreads(1L) has about the same speed on both R versions.
I guess that means that this issue is fixed in 1.12.0
Thank you for testing this. I will re-test and confirm once the thread number issue is fixed.

[RaynorJim]

Hi @jangorecki

I worked with @s-Nick-s a while back when he discovered the performance drop. I did some tests these days, and although I find 1.12.+ to give same or better time on R3.5+ vs R.3.4.4, i do find that running R3.4.4 with data.table 1.11.8 is about 2x as quick as running this example on anything newer than that combination (any R version beginning R3.5 all the way to R4.0.0) with any version of data.table 1.12.x (I did use setDTthreads(1L))

R3.4.4 + data.table 1.11.8
Unit: milliseconds expr min lq mean median test1 <- dt1[, list(percent_total = sum(percent)), by = type] 51.30958 55.41111 73.35613 59.93117 uq max neval 87.61779 154.0714 30

R3.6.3 + data.table 1.12.8

Unit: milliseconds expr min lq mean median test1 <- dt1[, list(percent_total = sum(percent)), by = type] 121.8025 135.3461 178.0072 178.2529 uq max neval 195.0769 347.6791 30

Please if you have the time, make a clean install of R3.4.4 + data.table 1.11.8 and verify this.

Looking forward to your reply,
Cheers.