Fit many replicas in parallel

[scarlehoff]

A rebase for #1039 seemed way more complicated than manually porting the changes so I did that.

Missing: a few review comments that I still need to implement and ensuring I haven't broken anything along the way.

[github-actions]

Greetings from your nice fit 🤖 !
I have good news for you, I just finished my tasks:

• Fit Name: NNBOT-c2e30882f-2021-03-16
• Fit Report: https://vp.nnpdf.science/Z79LCwGgRTeEYTlxgrrU7A==
• Fit Data: https://data.nnpdf.science/fits/NNBOT-c2e30882f-2021-03-16.tar.gz

Check the report carefully, and please buy me a ☕ , or better, a GPU 😉!

[wilsonmr]

Was this to save time because the generation of reps is slow? Or was it some other reason? I worry about this propagating if it's set in the example.

[scarlehoff]

No, just because I didn't do it in the original.
It shouldn't be complicated because the logic is the same as the one with 1 -r 30 but now in parallel. But doing it while wasting as little memory as possible might need some work (or might not, but need to check).

[scarlehoff]

This is now ready. I'm running things like hyperopt to ensure nothing is broken. Nothing was in #1039 but this merge/rebase was not trivial so I want to check.
It addresses the review comments from @RoyStegeman in #1039

The main limitation is that at the moment one cannot run in parallel while generating pseudodata (so it is useless for a fit!). It shouldn't be difficult (specially now with #1081), just telling the provider we want X replicas, but wasn't part of the original PR that was already reviewed so it will go to a separate one.

[scarlehoff]

The failed fit: https://vp.nnpdf.science/ccTgYKZbRdO7l97w8RAShg==/
Hyperopt: https://vp.nnpdf.science/RRzuKZfeRl2B47MsJoC0oQ==/

[RoyStegeman]

This looks good and I haven't noticed any problems while running the code. I left two comments for things I would personally change, but otherwise it's fine.

[scarlehoff]

Given that we are pushing back a bit NNPDF I'd rather have this merged, but in doing the final tests I've noticed that while for TF 2.2 I didn't notice a difference on speed, with 2.4.1 (the current Conda default) this branch is about 10/15% slower, so I need to investigate that (as the main way of running the code will still be in CPU).

[scarlehoff]

Ok, it was the usage of einsum in a few places (/cc @scarrazza your horror stories with qibo were useful :P) which is good for GPU but not for CPU.
I'll continue with the tests so it hopefully can be merged next week.

[RoyStegeman]

That's actually good to know! I like einsum but wasn't aware it's significantly slower on cpu than some alternatives.

But how does that explain the difference between tf2.2 and tf2.4.1 (or related numpy version)?

[scarlehoff]

Probably the speed difference was drowned by something else so I didn't notice it (both are faster now in Boogiepop than before).

[scarlehoff]

Ok, tests performed:

Standard:

• Travis
• Bot

Non standard fits:

25 replicas with only a few datasets, to see that 1) it runs 2) positivity work well 3) nothing obviously break

• 1 net per flavour
• pch
• feature scaling
• l2 and diagonalization

Benchmarks

On doing this I noted the einsum problem.

• Speed
• Memory

If I haven't forgotten anything important, this can be merged.

[scarlehoff]

First I need to have a good fit bot with the new data (#1218) and then test the bot in this branch and then it can merged.

[github-actions]

Greetings from your nice fit 🤖 !
I have good news for you, I just finished my tasks:

• Fit Name: NNBOT-c0912ed3f-2021-05-13
• Fit Report: https://vp.nnpdf.science/s8nAEnIVRU-nzFYvshE0zQ==
• Fit Data: https://data.nnpdf.science/fits/NNBOT-c0912ed3f-2021-05-13.tar.gz

Check the report carefully, and please buy me a ☕ , or better, a GPU 😉!

[scarlehoff]

I'll merge this as soon as #1211 is green-lighted