Stress-test threading/batchSize is always on

[mattdowle]

Two current limitations to overcome :

• getDTthreads() doesn't return more than omp_get_max_threads(). This prevents stress testing logic and bounds on small data by increasing number of threads. I wasn't able to reproduce segfault in parallel gforce #3204 until I mimicked getDTthreads() returning 20 (greater than omp_get_max_threads()) on my machine.

> setDTthreads(20)
> getDTthreads(verbose=TRUE)
omp_get_max_threads() = 8
omp_get_thread_limit() = 2147483647
DTthreads = 20
[1] 8

• I'm currently deliberately allowing very small batch sizes (sometimes 2 rows) in order to stress-test the test-suite. Otherwise we have to add a lot of large data tests before threading kicks in. This is good, and segfault in parallel gforce #3204 quickly came to light because of this good strategy. However, this stress-testing is left in for release. So it could be that overhead on small data is larger. Need to add a flag that is on for stress-testing in the test-suite, but turned off by default in release. It may be that overhead is not actually increased on small data and that it can be left on -- need to test.
  Update: throttle added in throttle threads for iterated small data tasks #4484 solves this 2nd part.

[jangorecki]

was it switched off before 1.12.0?

[mattdowle]

Not as far as I know. I just tested now and I still see getDTthreads(verbose=TRUE) returning the same output as above and returning 8. I'll follow up to your findings in the other issue: #3300 (comment)

[mattdowle]

PR #3435 changed setDTthreads() to allow up to omp_get_num_procs() which is the number of logical CPUs on the machine. On my laptop with 4 cores, this limit is 8 for example. This should be plenty to do the types of stress-mentioned in this issue.

[mattdowle]

The 2nd part needs a flag adding, or a wrapper, to turn off openMP for small data. Related to #3438 and #3743.