vp-comparefits blows up memory

[Zaharid]

If I run comparefits like

vp-comparefits NNPDF40_nnlo_as_01180_corr_s2 NNPDF40_nnlo_as_01180_corr_s2 --title="test" --author="Zahari Kassabov" --keywords="test" --debug

with a fully updated conda environment I see that the memory usage blows up to over 32 Gb, causing it to swap and never finish. I think that is because of the inclusion of the overfitting metric. This causes a lot of overhead by things like

[DEBUG]: Found sep_mult for spec (('dataspecs', 0), ('t0_info', 0), ('t0_info', 0), ('pdfreplicas', 25), 'fitenvironment', ('group_dataset_inputs_by_experiment', 0)) at 6

which involves blowing many things over replicas.

Collecting over replicas is not a very good idea.

[Zaharid]

See #1503.

[scarlehoff]

Let's take it out from comparefits for the time being.

We've also found a few problems and, more importantly, as soon as #1636 is merged we lose (again) the possibility of reusing all fits.

That said, I don't think I've seen this? (I've been definitely being using the latest overfit metric but not the latest conda, I've been updating just vp)

[Zaharid]

Agree that its best to take it out ASAP. I haven't exactly found what is blowing in memory, but pretty sure it is this (among other things because it worked fine with earlier versions).

[RoyStegeman]

I generate all comparefit reports on the cluster for a reason. The overfit metric is expensive to evaluate and we can reduce the memory usage a bit, but I think we should accept that the comparefit report is just not something that necessarily has to be possible to generate on a laptop.

We could consider addressing #1553 and exclude (among other things) the overfit metric.

[scarlehoff]

We could consider addressing #1553 and exclude (among other things) the overfit metric.

Yes, this would be great. But I would like to be able to run comparefit in my desktop computer and 32GB would be too much (another reason not to use conda I guess :P)

[Zaharid]

I think the pdfs I am trying have way more than 100 replicas. Nothing to do with conda.

[scarlehoff]

Ah, ok, that might be it then.

[RoyStegeman]

No the overfit metric currently just uses all other replicas that have been generated as part of the fit to evaluate the R_O. So kind of only works for a fit with O(10) replicas. I suppose we may as well generate a number of pseudodatasets on the fly where the number can be set in the runcard.