httpgd+plumber hangs R

[r2evans]

Describe the bug

A combination of this and (as an example) plumber::plumb() puts R into a state where it cannot be interrupted. httpgd alone and plumber alone do not trigger the non-interruptibility, it's the combination, and it's in R-4.3 and newer, R-4.2.2 does not have this problem.

First reported on emacs/ESS #1297, I have since confirmed that the bug is still present on the R console without an IDE.

To Reproduce

Demonstration of basic functionality with interruptibility:

while (TRUE) { Sys.sleep(1); message(format(Sys.time())); }
# 2025-02-11 09:18:17
# 2025-02-11 09:18:18
# 2025-02-11 09:18:19
#   C-c C-c
httpgd::hgd()
# httpgd server running at:
#   http://127.0.0.1:60946/live?token=6qZHAhGb
dev.list()
# unigd 
#     2 
while (TRUE) { Sys.sleep(1); message(format(Sys.time())); }
# 2025-02-11 09:18:23
# 2025-02-11 09:18:24
#   C-c C-c
41+1
# [1] 42

A simple plumber file:

#* @get /test
function(req, res, ...) { Sys.sleep(1); pi; }

Using that plumber.R file in a fresh R session, this does NOT hang:

plumber::plumb("plumber.R")$run(port=9999)
# Running plumber API at http://127.0.0.1:9999
# Running swagger Docs at http://127.0.0.1:9999/__docs__/
#   C-c C-c
41+1
# [1] 42

Same plumber.R file, fresh R session, this DOES hang, every time:

Sys.getpid()
# [1] 38659
httpgd::hgd()
# httpgd server running at:
#   http://127.0.0.1:60978/live?token=f5a6qLEH
dev.list()
# unigd 
#     2 
plumber::plumb("plumber.R")$run(port=9999)
# Running plumber API at http://127.0.0.1:9999
# Running swagger Docs at http://127.0.0.1:9999/__docs__/
#   C-c C-c
#   C-c C-c
#   C-c C-c
### in a shell: `kill -HUP 38659`
# Process R:1 hangup: 1 at Tue Feb 11 09:21:24 2025

Using just R consoles (no IDEs, no emacs, no nothin'),

At no time was the graphics device used.

Expected behavior

R remains interruptible.

Environment

• MacOS 15.3, R-4.4.3, httpgd-2.0.1 through -2.0.4, hangs every time
• Ubuntu 24.04, R-4.3.3, httpgd-2.0.4, plumber-1.2.2, hangs every time
• windows, R-4.2.3, httpgd-2.0.1 and -2.0.4, plumber-1.3.0, DOES NOT HANG (fresh install of packages, I tested both binary and compile-from-source)
• windows, R-4.4.3, httpgd-2.0.4, plumber-1.3.0, hangs every time

Additional context

I'm not convinced this is immune to changes in the R version or perhaps compiler/flags. The "win R-4.2.2" that doesn't hang perplexes me.

None of this is tested in RStudio (I don't have it installed, sorry).

[HenrikBengtsson]

Just in case it's relevant. I'm observing httpgd getting when checking it with R CMD check --as-cran on a host with a very large number of CPU cores. In my case it happens on a 256-core machine, but if check it on a machine with 38-cores, it works fine. Both run the Rocky 8 with identical setups.

Here is what I observe when I check on the 256-core machine;

$ pstree -l -a 622448
sh /path/to/R-4.4.3-gcc13/lib64/R/bin/Rcmd check httpgd_2.0.4.tar.gz --no-manual -o /c4/home/henrik/futureverse/future/revdep/checks/httpgd/new
  └─R --no-restore --no-echo --args nextArghttpgd_2.0.4.tar.gznextArg--no-manualnextArg-onextArg/c4/home/henrik/futureverse/future/revdep/checks/httpgd/new
     └─sh -c LANGUAGE=en _R_CHECK_INTERNALS2_=1 '/path/to/R-4.4.3-gcc13/lib64/R/bin/R' --vanilla --no-echo < '/scratch/henrik/RtmpWUAgAh/file97f7a8d4cff7'
         └─R --vanilla --no-echo
            └─sh /path/to/R-4.4.3-gcc13/lib64/R/bin/Rcmd BATCH --vanilla testthat.R testthat.Rout
                └─R -f testthat.R --restore --save --no-readline --vanilla
                   └─258*[{R}]

Among those 258 child processes, there is one process that runs - the others are at 0% CPU use. Using strace -p <pid> on that process reveals it's stuck in an infinite loop pooling:

sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
...

while its 258-1 siblings are stuck at:

$ strace -p 648068
strace: Process 648068 attached
futex(0x146c7a9a19d0, FUTEX_WAIT, 648329, NULL

It looks like there's (a) some multi-threading going wild, (b) it uses "detectCores()" number of threads by default, and there is (c) possibly some deadlock happening. If I terminate the running child process (e.g. kill -S SIGTERM 622448), it appears to release it and the other child processes get resolved, but then there is one more test process stuck a bit later with I have to kill (SIGKILL). I don't know if this works consistently, but it's in line with a deadlock taking place. Thus far, it's not obvious to me where the multi-threading processes kicks in.

After killing the remain test process, R CMD check completes, which also reveals:

* checking tests ...
  Running ‘testthat.R’/path/to/R-4.4.3-gcc13/lib64/R/bin/BATCH: line 60: 1200393 Killed                  ${R_HOME}/bin/R -f ${in} ${opts} ${R_BATCH_OPTIONS} > ${out} 2>&1
 [146s/175s]
 [146s/176s] ERROR
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
  > library(testthat)
  > library(httpgd)
  >
  > test_check("httpgd")
  [ FAIL 12 | WARN 25 | SKIP 0 | PASS 2 ]

  Error in gzfile(file, mode) : cannot open the connection
  Calls: test_check ... o_apply -> lapply -> FUN -> <Anonymous> -> saveRDS -> gzfile
  In addition: Warning message:
  In gzfile(file, mode) :
    cannot open compressed file 'testthat-problems.rds', probable reason 'Too many open files'                                                                                     
  Execution halted

The "Too many open files" problem can step from something is trying to open, say, 256 files (=256 CPU cores), and R has only 128-3 connections available. I'm not sure, but at least a guess.

I hope this is a bit of help.

PS. I used terms "threads" and "processes" a bit sloppy about; should probably use "threads" in more places.

[benz0li]

The "Too many open files" problem can step from something is trying to open, say, 256 files (=256 CPU cores), and R has only 128-3 connections available. I'm not sure, but at least a guess.

@HenrikBengtsson What does prlimit¹ return (on those machines)? What is the SOFT and HARD limit for NOFILE (max number of open files)?

Footnotes

1. Alternative: ulimit -Sa (SOFT limits), ulimit -Ha (HARD limits) ↩

[HenrikBengtsson]

The "Too many open files" problem can step from something is trying to open, say, 256 files (=256 CPU cores), and R has only 128-3 connections available. I'm not sure, but at least a guess.

@HenrikBengtsson What does prlimit1 return (on those machines)? What is the SOFT and HARD limit for NOFILE (max number of open files)?

$ prlimit
RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited bytes
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited bytes
LOCKS      max number of file locks held      unlimited unlimited locks
MEMLOCK    max locked-in-memory address space     65536     65536 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files                1024    262144 files     <===
NPROC      max number of processes              4124691   4124691 processes
RSS        max resident set size              unlimited unlimited bytes
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals        4124691   4124691 signals
STACK      max stack size                     unlimited unlimited bytes

$ Rscript --vanilla -e "parallelly::availableCores(which = 'all')"
           system /proc/self/status    cgroups.cpuset             nproc 
              256               256               256               256 

[benz0li]

@HenrikBengtsson 1024 is not that many open files...

Set the SOFT limit to the HARD limit (for the current process) using command prlimit --pid $$ --nofile=262144 and try again.

[HenrikBengtsson]

@benz0li, thanks.

The main reason I reported back here is that I think httpgd, or whatever underlying dependency is responsible here, should at a minimum detect and protect against this, e.g. giving an error. We can just guess how many end-users are out there that run into this without even knowing httpgd is involved, or has the skills and experience how to track this one down, less so knowing how to work around it.

PS. I think the main culprit is that it's trying to hijack all CPU cores for its multi-threading, regardless how many CPU cores there are. I'm a strong believer in not doing this (e.g. https://www.jottr.org/2022/12/05/avoid-detectcores/). I advocate for sequential processing by default, because that's the only one we as a developer can make assumptions about. Or, at a minimum, use a constant number of threads/processes and never let it scale as a function of the number of cores available on the host (e.g. the developer might only test with 8-16 cores, and don't even consider what's happening on hosts with 128, 256, ... cores).

PS2. I haven't checked if httpgd attempts to respect CGroups settings or not.

[benz0li]

The main reason I reported back here is that I think httpgd, or whatever underlying dependency is responsible here, should at a minimum detect and protect against this, e.g. giving an error.

R CMD check --as-cran involves testthat. Maybe testthat is the cause – not httpgd:

• https://testthat.r-lib.org/articles/parallel.html
• https://github.com/r-lib/testthat/blob/30f5b119875852005f2a02e6adf96276dbd1ef26/DESCRIPTION#L55

[r2evans]

R CMD check --as-cran involves testthat. Maybe testthat is the cause – not httpgd:

Whether or not Henrik's comment is related to testthat, the origin of this issue is not.

@benz0li

[benz0li]

R CMD check --as-cran involves testthat. Maybe testthat is the cause – not httpgd:

Whether or not Henrik's comment is related to testthat, the origin of this issue is not.

Yes. @HenrikBengtsson should open a separate issue about that.

[benz0li]

@r2evans Out of curiousity: Are you able to reproduce this issue at https://demo.jupyter.b-data.ch?

Follow the steps in your description. Then, you should be able to open https://demo.jupyter.b-data.ch/user/r2evans/proxy/9999/__docs__/

[r2evans]

I haven't tried. Why is testing in a different framework clearer than testing in base R with no scaffolding or wrappers?

Regardless, the issue is not about whether httpgd is still responsive. It's that it gets to a point that it never yields or quits.

[benz0li]

Same plumber.R file, fresh R session, this DOES hang, every time:

Sys.getpid()
# [1] 38659
httpgd::hgd()
# httpgd server running at:
#   http://127.0.0.1:60978/live?token=f5a6qLEH
dev.list()
# unigd 
#     2 
plumber::plumb("plumber.R")$run(port=9999)
# Running plumber API at http://127.0.0.1:9999
# Running swagger Docs at http://127.0.0.1:9999/__docs__/
#   C-c C-c
#   C-c C-c
#   C-c C-c

@r2evans I can reproduce at https://demo.jupyter.b-data.ch.

FYI @nx10