Running parallel comparefits

[scarlehoff]

I've updated to the latest version of reportengine and tried to use --parallel but had the same problem that I had previously, it uses too much memory and crashes before finishing. Is there a recommended way to run it? Or is the (at least double) memory usage a feature of it?

(in both cases we should reflect that in the docs)

[comane]

Hi @scarlehoff, thanks for the comment.
The recommended way of using it is

1. Open a scheduler (possibly in a separate screen) using the command:
   dask-scheduler

2. assign to the scheduler some workers and specify at the same time the amount of memory they can use (you might need quite a bit of memory for something like vp-comoarefits), for example:
   dask-worker <scheduler address> --nworkers 10 --nthreads 1 --memory-limit "14 GiB"

3. Run the validphys process using --parallel and specifying the use of the scheduler with --scheduler

Sorry for the possibly weird formatting of the comment as I am writing from my phone.
Also, I will add this to the docs as soon as I am back from holidays.

[scarlehoff]

No worries!

It might well be that for comparefits the gain is very little, I guess we should have some defaults per script (depending on our own experience with them)

[comane]

Hi @scarlehoff, you can find some documentation here (#1804).
I have added two examples, namely, plot_pdfs.yaml and vp-comparefits.

Let me know in case you think we would want other standard examples to be included too.