Not using multi-threading after a mclapply call

[renkun-ken]

I'm using the latest development version of fst and find that it stops using multi-threading as long as mclapply is called to fork multiple processes.

library(fst)
library(parallel)
library(data.table)

threads_fst()
data <- mclapply(1:5, function(x) rnorm(50000000), mc.cores = 5)
setDT(data)

threads_fst()
write_fst(data, "~/data/data.fst", compress = 100)

The output is as follows:

> library(fst)
fst 0.7.3 IN DEVELOPMENT built 2017-12-06 09:52:24 UTC
(OpenMP detected, using 20 cores)
> library(parallel)
> library(data.table)
data.table 1.10.4.3
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
> fst::threads_fst()
[1] 20
> data <- mclapply(1:5, function(x) rnorm(50000000), mc.cores = 5)
> setDT(data)
> fst::threads_fst()
[1] 1
> write_fst(data, "~/data/data.fst", compress = 100)

I'm not sure why the number of threads to use by fst in the master process would decrease to 1, rather than only the child processes decrease multi-threading to 1?

My session info:

> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.3 LTS

Matrix products: default
BLAS: /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.18.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.10.4-3 fst_0.7.3          

loaded via a namespace (and not attached):
[1] compiler_3.4.3 tools_3.4.3    yaml_2.1.15    Rcpp_0.12.14  

[MarcusKlik]

Hi @renkun-ken, thanks for reporting!

Indeed, the number of threads used by the master thread should not drop back to one.
I think I should include a method that returns the previously allocated number of threads in this code segment.

Good find, thanks!

[MarcusKlik]

Hi @renkun-ken, this issue was caused by fst switching into single threaded mode on a fork (to prevent problems) but not switching back to the original number of threads at forks-end. That default behavior has been changed now and fst will restore the original number of threads at the end of a fork. This default behavior can be changed by using parameter reset_after_fork in threads_fst():

fst::threads_fst(nr_of_threads = 8, reset_after_fork = FALSE)
#> [1] 8

Alternatively, option fst_restore_after_fork can be set to FALSE. Both methods will leave fst in single-threaded mode after a fork and can be used in case restoring threads creates problems (some issues have been reported on the data.table repository when the Intel compiler is used).

Thanks again for submitting your issue!