Warning on CRAN: cannot form a team with 24 threads, using 2 instead

[mattdowle]

From Prof Ripley :

---

The CRAN policy says:

'If running a package uses multiple threads/cores it must never use more
than two simultaneously: ...'

Packages

RNiftyReg Rfast data.table quanteda sentometrics sylcount

attempt to use more, often all the cores of a 24-core machine and this
shows under clang's or Intel's OMP runtime with messages like

OMP: Warning #96: Cannot form a team with 24 threads, using 2 instead.

when we have set OMP_THREAD_LIMIT=2 as a defensive measure. See the
clang-fedora log or clang-UBSAN additional issue on the CRAN result page
for your package.

How to control OpenMP thread usage is discussed in 'Writing R
Extensions' §1.2.1.1.

It is possible this comes from another package you use, in which case
take it up with the maintainer of that package and set OMP_THREAD_LIMIT
yourself.

Please correct ASAP and before Feb 20 to safely retain the package on CRAN.

---

[mattdowle]

My reply :

---

How do we know when we are being run by CRAN? Hence I rely on you setting OMP_THREAD_LIMIT=2. What's wrong with that? There is no way that I know of for the package to know its tests are being run by CRAN.
The full sentence from CRAN policy is "If running a package uses multiple threads/cores it must never use more than two simultaneously: the check farm is a shared resource and will typically be running many checks simultaneously."
Well, that's true, because I know you set OMP_THREAD_LIMIT=2. Now it's only a defensive mechanism? That's news to me.
data.table is careful to follow §1.2.1.1: it always uses num_threads() and passes its own getDTthreads() to it. I have greps in CRAN_release.cmd procedures which ensure each and every parallel regions does this. Just as §1.2.1.1 advises. This getDTthreads() limits the user request by omp_get_max_threads() which respects OMP_THREAD_LIMIT.

There might be some nested parallelism for the first time in the latest data.table release (again in the froll.c new code) and I can see how that might possibly end up requesting more than OMP_THREAD_LIMIT within a nested parallel region by calling getDTthreads() twice. So I can look into that. But I don't see any warning locally. I tried just now. You say it's just some compilers. Either way, it's going to be some work for me to reproduce, understand and I need time.

Given all this, I don't think I deserve the one-month-or-else treatment. It's not even as if these packages really are using 24 cores. The limit is working, they're using 2 cores and CRAN is healthy. There's a new warning. That's all. It's new, right? If it's old then you haven't told us before and data.table has been using OpenMP in this way for quite a while now.

---

[jangorecki]

Looks like getDTthreads ignores OMP_THREAD_LIMIT env var. Try

echo $OMP_THREAD_LIMIT
R --vanilla
library(data.table)
Sys.setenv("OMP_THREAD_LIMIT"=1)
Sys.getenv("OMP_THREAD_LIMIT")
getDTthreads(verbose=TRUE)
q("no")

echo $OMP_THREAD_LIMIT
export OMP_THREAD_LIMIT=1
R --vanilla
library(data.table)
Sys.getenv("OMP_THREAD_LIMIT")
getDTthreads(verbose=TRUE)
q("no")

same behavior between solaris and master branches

I checked OMP_NUM_THREADS env var also, and here it does work, but only when it is being set before starting R, if it was set from inside R then it is ignored.

echo $OMP_NUM_THREADS
R --vanilla
library(data.table)
Sys.setenv("OMP_NUM_THREADS"=1)
Sys.getenv("OMP_NUM_THREADS")
getDTthreads(verbose=TRUE)
q("no")

echo $OMP_NUM_THREADS
export OMP_NUM_THREADS=1
R --vanilla
library(data.table)
Sys.getenv("OMP_NUM_THREADS")
getDTthreads(verbose=TRUE)
q("no")

[mattdowle]

Hm. I see the same. That means omp_get_max_threads() doesn't account for OMP_THREAD_LIMIT.

I've read the OpenMP specification before when I was writing getDTthreads() w.r.t. to the R-ext manual §1.2.1.1. Section 3.2.3 of OpenMP 4.5 specification states :

Summary: The omp_get_max_threads routine returns an upper bound on the number of threads that could be used to form a new team if a parallel construct without a num_threads clause were encountered after execution returns from this routine.
Binding: The binding task set for an omp_get_max_threads region is the generating task.
Effect: The value returned by omp_get_max_threads is the value of the first element of the nthreads-var ICV of the current task. This value is also an upper bound on the number of threads that could be used to form a new team if a parallel region without a num_threads clause were encountered after execution returns from this routine.

Which I had read as "it returns the upper bound"; i.e. if user has limited threads then this routine would return that limit. But that's not what it actually says is it. It says "it returns an upper bound". What I thought I was getting there was the user limited number. It's more like any old upper bound.

Seems like getDTthreads needs to use omp_get_thread_limit() not omp_get_max_threads(). Section 3.2.14 of the spec says :

Summary: The omp_get_thread_limit routine returns the maximum number of OpenMP threads available to participate in the current contention group.
Binding: The binding thread set for an omp_get_thread_limit region is all threads on the device. The effect of executing this routine is not related to any specific region corresponding to any construct or API routine.
Effect: The omp_get_thread_limit routine returns the value of the thread-limit-var ICV.
Cross References:

• thread-limit-var ICV, see Section 2.3 on page 36.
• OMP_THREAD_LIMIT environment variable, see Section 4.10 on page 300.

Why it's useful for OpenMP to have both omp_get_max_threads() as well as omp_get_thread_limit(), beats me. The cross references for neither one link to the other directly in version 4.5 of the spec that I had read. I checked the recent version 5.0 of the OpenMP Spec (Nov 2018) and the cross-references of both functions have now been updated to include links to each other but I don't see any other clarification as to why both exist. Algorithm 2.1 in section 2.5.1 of OpenMP 4.5 ( section 2.6.1 of OpenMP 5.0) alludes to more complexity, so maybe it's something to do with that.

omp_get_max_threads() has a note:

Note: The return value of the omp_get_max_threads routine can be used to dynamically allocate sufficient storage for all threads in the team formed at the subsequent active parallel region.

But equally, omp_get_thread_limit() could be used to allocate storage. It would be more appropriate too when omp_get_thread_limit() < omp_get_max_threads() otherwise storage would be allocated wastefully.

Maybe we should use MIN(omp_get_max_threads(), omp_get_thread_limit()) but doing that seems a bit onerous and inconvenient. At least for us it would be only once in the central getDTthreads().

We could also detect and warn about ineffective attempts to use Sys.setenv(). And even apply their values so they were effective would probably be more convenient to user.

[MarcusKlik]

Hi @mattdowle and @jangorecki,

following your discussion, I did a small experiment to check the behavior of omp_get_thread_limit() for different settings of OMP_THREAD_LIMIT. It turns out that for invalid settings of OMP_THREAD_LIMIT, a value of INT_MAX is returned. An equivalent example:

thread_limit <- function(omp_thread_limit) {

  Sys.setenv(OMP_THREAD_LIMIT = omp_thread_limit)

  Rcpp::cppFunction('

      #include <omp.h>
      int get_thread_limit() {
      return omp_get_thread_limit();}

    ', plugins = "openmp")

  get_thread_limit()
}


thread_limit(1)  # ok
#> [1] 1

thread_limit(2)  # ok
#> [1] 2

thread_limit(-1)  # INT_MAX
#> [1] 2147483647

thread_limit("A")  # INT_MAX
#> libgomp: Invalid value for environment variable OMP_THREAD_LIMIT
#> [1] 2147483647

thread_limit("20000")  # ok (but inefficiently high)
#> libgomp: Invalid value for environment variable OMP_THREAD_LIMIT
#> [1] 20000

thread_limit(0)  # INT_MAX
#> libgomp: Invalid value for environment variable OMP_THREAD_LIMIT
#> [1] 2147483647

So, if OMP_THREAD_LIMIT is set to a string, zero or a negative value before compiling the OMP code, INT_MAX will be returned upon calling omp_get_thread_limit(). I believe @mattdowle's suggestion to use

std::min(omp_get_max_threads(), omp_get_thread_limit())

for calculating the number of threads is definitely the safest solution! There are use-cases for having more threads than (logical) cores (for example if the threads are doing a lot of waiting) but I don't think that applies to data.table.

I hope you have some use for this info, thanks for the great work on data.table!

[kbenoit]

We're having trouble with this too (see quanteda/quanteda#1581 and the linked issue) but it seems to happen on macOS and not Linux. I got the same "30-day or else" email from BDR a few days ago too.

Our fix quanteda/quanteda@492cb4c to the package default settings (where value = 2)

Sys.setenv("OMP_THREAD_LIMIT" = value)

did not make the problem disappear.

[mattdowle]

@kbenoit The fact you can reproduce is good. quanteda uses data.table iirc. So we'll make the change to data.table and then ask you to rerun to see if that fixes it. We don't have macOS in CI so it's hard for us to reproduce although we can have a good guess.

[kbenoit]

@mattdowle I did even better than that, I set up a reproducible example package for you to try out, see https://github.com/kbenoit/testomp. Interesting that not every function triggers this. I don't know if it was := or the sample that was doing it.

Thanks!!

ps and yes of course we use data.table! 😄