Protect against using forked processes (aka "multicore") in RStudio

[HenrikBengtsson]

Issue

Parallelization via forked processing should be avoided when running R from RStudio, cf.
rstudio/rstudio#2597 (comment). There are simply too many things that may break RStudio's interaction with R. Several RStudio users report on R sessions hanging and stray R processes that do not terminate. This is a problem that has been there from the beginning and it is a hard problem to solve so to the best of my knowledge it's unlikely that this will be "fixed" in RStudio any time soon.

In terms of the future ecosystem, this means that we should avoid using plan(multicore) when running R via RStudio.

Tasks

• Make it possible to enable/disable forking when running RStudio:
  • environment variable R_FUTURE_FORK_ENABLE=true/false
  • R option future.fork.enable = TRUE/FALSE
  • Have supportsMulticore() acknowledge these. When unset, try to infer automatically, e.g. disable when running on MS Windows or via RStudio. This will cause multicore to fall back to sequential when running in RStudio similarly to how it works when on MS Windows.
• Comment about this in the help/docs, e.g.
  • help(multicore)
  • supportsMulticore()

See also

• Settings to disable forked processing in R, e.g. parallel::mclapply() HenrikBengtsson/Wishlist-for-R#94

[HenrikBengtsson]

This has now been implemented in the develop branch. Starting with the next release, multicore futures will be disabled when running R from RStudio.

[grantmcdermott]

I know this issue is closed — I'm late to the party — but is there any technical reason why you want to default to sequential rather than multisession?

To the best of my knowledge, there are no real inherent risks to switching from multicore (forking) to multisession (PSOCKs)... apart from the obvious additional memory consumption involved in copying over environments. If that is correct then, sure, some users may run out of memory in certain edge cases, but I think that's an acceptable tradeoff for executing the expected behaviour of parallelization.

(See also Simon Urbanek's comment here. You could make a similar argument about working with really big data files that some machines can handle, but others can't...)

Going parallel is complicated and involves tradeoffs. From my perspective, future struck the right bargain in opting for ease and consistency of implementation over protecting against every possible eventuality.

PS — This is the subject of an ongoing Twitter thread, but I think it's good practice to move the discussion to a GitHub issue.

[HenrikBengtsson]

Reopening to show this is not a done deal and discussion is open

[HenrikBengtsson]

Only got a 15 minutes right now;

In lieu of an official(*) R options/env var for disabling forked processing, I invented future.fork.enable and R_FUTURE_FORK_ENABLE. (*) I posted 'SUGGESTION: Settings to disable forked processing in R, e.g. parallel::mclapply()' to the R-devel list on 2019-04-11 suggesting to adding enable.forks and R_ENABLE_FORKS for a R-wise standard that can disable also parallel::mclapply().

The most forward way to disable forked processing in the future framework, that is, to disable 'multicore', was to have future::supportsMulticore() to acknowledge future.fork.enable/R_FUTURE_FORK_ENABLE:

https://github.com/HenrikBengtsson/future/blob/f87feed6bb4e543d5b59d48b3b49e7ef667f5366/R/supportsMulticore.R#L46-L50

That implementation automatically turns 'multicore' into 'sequential'. This is what already happens on MS Windows.

For consistency with MS Windows, where we have 'multiprocess' -> 'multisession', one can argue that we want the same type of rule to apply for 'RStudio', and possibly other environments to come. I'm not against this solution.

In the bigger picture, maybe 'multiprocess' should be retired? Since I added it (early on), I always felt it is a potential source of problems. It's a big bit "magic", it complicated the documentation, it behaves differently depending on OSes (although most users won't notice until they run into issues, cf. doParallel::registerDoParallel(ncores)). Main argument was so that there could be one goto parallel plan that would "work the best" on the current OS, which helps in vignettes, blogs, ..., and avoid *nix folds from putting plan(multicore) in their examples and then having Windows people complaining "it does not work!". But, maybe it's time to promote 'multisession' instead.

[DavisVaughan]

In the bigger picture, maybe 'multiprocess' should be retired?

I actually came to this conclusion too while thinking through this problem. I don't think it would be such a bad thing since multicore has so many issues, and it does feel a bit magical. It's also "one more thing" to understand when getting started with future. What if plan(multisession) was promoted, and plan(multiprocess) was slowly deprecated. Maybe a deprecation strategy of:

current behavior ->
fall back to multisession on windows/rstudio (unless overridden by the env var allowing multicore) ->
become an alias of multisession with a warning ->
deprecate

I do think there would be some annoyance and push back if in the next release multiprocess turned into sequential on Mac/Linux while in RStudio. If it wasn't the main recommended entry point into plan() then I think it would be less of an issue.