Failing `test_restart_from_pickle` in `n3fit/tests/test_hyperopt.py`

[Cmurilochem]

It seems that the test test_restart_from_pickle created by me fails occasionally. The reason is that, because we calculate too few trials (2 and then 4) in the hyperopt, it may happen that none of the trials end up with a hyper_loss below the threshold; this is certainly not in the scope of the test itself and might be slightly annoying.

To correct for that I suggest:

• increasing the number of trials and/or epochs (which might not directly solve the problem a priori with added timings)
• increasing the k-fold hyper opt threshold here in the hyper-quickcard.yml file
• create a specific runcard for this test and add a high threshold since here we are only interested in asserting that the final json files are equivalent and show the same number (regardless their magnitude).

Which one would be the best alternative ? Please, also feel free to suggest any other alternative.

[scarlehoff]

increasing the k-fold hyper opt threshold
create a specific runcard for this test

Any of these two would work for me since indeed, we are only interested in the final .json, it doesn't matter if the results are not great.

However,

it may happen that none of the trials end up with a hyper_loss below the threshold;

I think the actual problem then is that the results are not reproducible? Two runs of the same runcard should produce the same result if the seeds are the same.

[RoyStegeman]

To be honest, I can't say I've noticed it failing yet, so I guess it doesn't happen that often. Since this is not a regression test, and the main point of this test is to see if the restart functionality works I'd say increasing the threshold is fine.

I think the actual problem then is that the results are not reproducible? Two runs of the same runcard should produce the same result if the seeds are the same.

As the fitbot also shows, fits may change numerically over time (due to change in dependencies or whatever. I'm not sure this is the test we want to fail due to such things though.

[scarlehoff]

But the changes are very small. Passing below a threshold should be safe.

[RoyStegeman]

This is a hyperopt using three datasets (and thus fits to one and two datasets), I could imagine that can make it quite unstable compared to the other regression tests we do.

Having said that, perhaps it's indeed worth checking if two runs with the same environment produce exactly the same results.

[Cmurilochem]

I agree. I remember @APJansen having some issues with it this week and I am also having it in #1726 now after updating my branch with master. However, it was ok before.

I will check carefully as suggested and report the results here soon.

[Cmurilochem]

Here is the result. For this I have used the test_restart_from_pickle test to print the restart_json[i]['result'] and direct_json[i]['result'] and have to increase the threshold from threshold: 2.0 to threshold: 500.0 in hyper-quickcard.yml file.

I have run 2 experiments on my Mac:

Run 1

Trial 0
Restart result {'status': 'ok', 'loss': 20.70434057433898, 'validation_loss': '3.3594213', 'experimental_loss': '12.276893', 'kfold_meta': {'validation_losses': '[[2.299371 ]\n [4.4194713]]', 'experimental_losses': '[[10.660989 ]\n [13.8927965]]', 'hyper_losses': '[19.66948573 21.73919542]'}}
Direct result {'status': 'ok', 'loss': 20.70434057433898, 'validation_loss': '3.3594213', 'experimental_loss': '12.276893', 'kfold_meta': {'validation_losses': '[[2.299371 ]\n [4.4194713]]', 'experimental_losses': '[[10.660989 ]\n [13.8927965]]', 'hyper_losses': '[19.66948573 21.73919542]'}}
Trial 1
Restart result {'status': 'ok', 'loss': 49.88519278394024, 'validation_loss': '14.405686', 'experimental_loss': '20.125654', 'kfold_meta': {'validation_losses': '[[10.667139]\n [18.144234]]', 'experimental_losses': '[[14.569938]\n [25.681372]]', 'hyper_losses': '[46.88904478 52.88134079]'}}
Direct result {'status': 'ok', 'loss': 49.88519278394024, 'validation_loss': '14.405686', 'experimental_loss': '20.125654', 'kfold_meta': {'validation_losses': '[[10.667139]\n [18.144234]]', 'experimental_losses': '[[14.569938]\n [25.681372]]', 'hyper_losses': '[46.88904478 52.88134079]'}}
Trial 2
Restart result {'status': 'ok', 'loss': 124.90943584826874, 'validation_loss': '35.37278', 'experimental_loss': '35.428932', 'kfold_meta': {'validation_losses': '[[18.18835]\n [52.55721]]', 'experimental_losses': '[[21.345316]\n [49.51255 ]]', 'hyper_losses': '[144.61109819 105.2077735 ]'}}
Direct result {'status': 'ok', 'loss': 124.90943584826874, 'validation_loss': '35.37278', 'experimental_loss': '35.428932', 'kfold_meta': {'validation_losses': '[[18.18835]\n [52.55721]]', 'experimental_losses': '[[21.345316]\n [49.51255 ]]', 'hyper_losses': '[144.61109819 105.2077735 ]'}}
Trial 3
Restart result {'status': 'fail', 'loss': inf, 'validation_loss': '26.753922', 'experimental_loss': '52.710144', 'kfold_meta': {'validation_losses': '[[26.753922]]', 'experimental_losses': '[[52.710144]]', 'hyper_losses': '[82.31994232]'}}
Direct result {'status': 'fail', 'loss': inf, 'validation_loss': '26.753922', 'experimental_loss': '52.710144', 'kfold_meta': {'validation_losses': '[[26.753922]]', 'experimental_losses': '[[52.710144]]', 'hyper_losses': '[82.31994232]'}}

Run 2

Trial 0
Restart result {'status': 'ok', 'loss': 20.70434057433898, 'validation_loss': '3.3594213', 'experimental_loss': '12.276893', 'kfold_meta': {'validation_losses': '[[2.299371 ]\n [4.4194713]]', 'experimental_losses': '[[10.660989 ]\n [13.8927965]]', 'hyper_losses': '[19.66948573 21.73919542]'}}
Direct result {'status': 'ok', 'loss': 20.70434057433898, 'validation_loss': '3.3594213', 'experimental_loss': '12.276893', 'kfold_meta': {'validation_losses': '[[2.299371 ]\n [4.4194713]]', 'experimental_losses': '[[10.660989 ]\n [13.8927965]]', 'hyper_losses': '[19.66948573 21.73919542]'}}
Trial 1
Restart result {'status': 'ok', 'loss': 49.88519278394024, 'validation_loss': '14.405686', 'experimental_loss': '20.125654', 'kfold_meta': {'validation_losses': '[[10.667139]\n [18.144234]]', 'experimental_losses': '[[14.569938]\n [25.681372]]', 'hyper_losses': '[46.88904478 52.88134079]'}}
Direct result {'status': 'ok', 'loss': 49.88519278394024, 'validation_loss': '14.405686', 'experimental_loss': '20.125654', 'kfold_meta': {'validation_losses': '[[10.667139]\n [18.144234]]', 'experimental_losses': '[[14.569938]\n [25.681372]]', 'hyper_losses': '[46.88904478 52.88134079]'}}
Trial 2
Restart result {'status': 'ok', 'loss': 124.90943584826874, 'validation_loss': '35.37278', 'experimental_loss': '35.428932', 'kfold_meta': {'validation_losses': '[[18.18835]\n [52.55721]]', 'experimental_losses': '[[21.345316]\n [49.51255 ]]', 'hyper_losses': '[144.61109819 105.2077735 ]'}}
Direct result {'status': 'ok', 'loss': 124.90943584826874, 'validation_loss': '35.37278', 'experimental_loss': '35.428932', 'kfold_meta': {'validation_losses': '[[18.18835]\n [52.55721]]', 'experimental_losses': '[[21.345316]\n [49.51255 ]]', 'hyper_losses': '[144.61109819 105.2077735 ]'}}
Trial 3
Restart result {'status': 'fail', 'loss': inf, 'validation_loss': '26.753922', 'experimental_loss': '52.710144', 'kfold_meta': {'validation_losses': '[[26.753922]]', 'experimental_losses': '[[52.710144]]', 'hyper_losses': '[82.31994232]'}}
Direct result {'status': 'fail', 'loss': inf, 'validation_loss': '26.753922', 'experimental_loss': '52.710144', 'kfold_meta': {'validation_losses': '[[26.753922]]', 'experimental_losses': '[[52.710144]]', 'hyper_losses': '[82.31994232]'}}

As expected they give exact the same results. Not sure if these would be the same if using ubuntu for instance. When I leave the threshold: 2.0 in hyper-quickcard.yml file. I have the error:

 Bug in n3fit ocurred. Please report it.
Traceback (most recent call last):
  File "/Users/murilo/Library/CloudStorage/OneDrive-NetherlandseScienceCenter/Documents/NLeSC_Projects/Proton_structure/nnpdfgit/nnpdf/n3fit/src/n3fit/scripts/n3fit_exec.py", line 290, in run
    super().run()
  File "/Users/murilo/Library/CloudStorage/OneDrive-NetherlandseScienceCenter/Documents/NLeSC_Projects/Proton_structure/nnpdfgit/nnpdf/validphys2/src/validphys/app.py", line 152, in run
    super().run()
  File "/Users/murilo/opt/anaconda3/envs/nnpdf-dev/lib/python3.9/site-packages/reportengine/app.py", line 380, in run
    rb.execute_sequential()
  File "/Users/murilo/opt/anaconda3/envs/nnpdf-dev/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 166, in execute_sequential
    result = self.get_result(callspec.function,
  File "/Users/murilo/opt/anaconda3/envs/nnpdf-dev/lib/python3.9/site-packages/reportengine/resourcebuilder.py", line 175, in get_result
    fres =  function(**kwdict)
  File "/Users/murilo/Library/CloudStorage/OneDrive-NetherlandseScienceCenter/Documents/NLeSC_Projects/Proton_structure/nnpdfgit/nnpdf/n3fit/src/n3fit/performfit.py", line 229, in performfit
    true_best = hyper_scan_wrapper(
  File "/Users/murilo/Library/CloudStorage/OneDrive-NetherlandseScienceCenter/Documents/NLeSC_Projects/Proton_structure/nnpdfgit/nnpdf/n3fit/src/n3fit/hyper_optimization/hyper_scan.py", line 135, in hyper_scan_wrapper
    best = hyperopt.fmin(
  File "/Users/murilo/opt/anaconda3/envs/nnpdf-dev/lib/python3.9/site-packages/hyperopt/fmin.py", line 540, in fmin
    return trials.fmin(
  File "/Users/murilo/opt/anaconda3/envs/nnpdf-dev/lib/python3.9/site-packages/hyperopt/base.py", line 671, in fmin
    return fmin(
  File "/Users/murilo/opt/anaconda3/envs/nnpdf-dev/lib/python3.9/site-packages/hyperopt/fmin.py", line 593, in fmin
    return trials.argmin
  File "/Users/murilo/opt/anaconda3/envs/nnpdf-dev/lib/python3.9/site-packages/hyperopt/base.py", line 620, in argmin
    best_trial = self.best_trial
  File "/Users/murilo/opt/anaconda3/envs/nnpdf-dev/lib/python3.9/site-packages/hyperopt/base.py", line 611, in best_trial
    raise AllTrialsFailed
hyperopt.exceptions.AllTrialsFailed

which means that all losses were equal to inf.

[Cmurilochem]

Note: I was curious and switched to the master branch and performed the test there. Conclusion: the test is running normally in the master

Results

Trial 0
Restart result {'status': 'ok', 'loss': 39.339472212806186, 'validation_loss': '2.2993183', 'experimental_loss': 10.660690008425245, 'kfold_meta': {'validation_losses': ['2.2993183'], 'experimental_losses': [10.660690008425245], 'hyper_losses': ['[39.33947221]']}}
Direct result {'status': 'ok', 'loss': 39.339472212806186, 'validation_loss': '2.2993183', 'experimental_loss': 10.660690008425245, 'kfold_meta': {'validation_losses': ['2.2993183'], 'experimental_losses': [10.660690008425245], 'hyper_losses': ['[39.33947221]']}}
Trial 1
Restart result {'status': 'ok', 'loss': 93.77809403932388, 'validation_loss': '10.667141', 'experimental_loss': 14.569936714920344, 'kfold_meta': {'validation_losses': ['10.667141'], 'experimental_losses': [14.569936714920344], 'hyper_losses': ['[93.77809404]']}}
Direct result {'status': 'ok', 'loss': 93.77809403932388, 'validation_loss': '10.667141', 'experimental_loss': 14.569936714920344, 'kfold_meta': {'validation_losses': ['10.667141'], 'experimental_losses': [14.569936714920344], 'hyper_losses': ['[93.77809404]']}}
Trial 2
Restart result {'status': 'ok', 'loss': 289.2196659784379, 'validation_loss': '18.18834', 'experimental_loss': 21.345310585171568, 'kfold_meta': {'validation_losses': ['18.18834'], 'experimental_losses': [21.345310585171568], 'hyper_losses': ['[289.21966598]']}}
Direct result {'status': 'ok', 'loss': 289.2196659784379, 'validation_loss': '18.18834', 'experimental_loss': 21.345310585171568, 'kfold_meta': {'validation_losses': ['18.18834'], 'experimental_losses': [21.345310585171568], 'hyper_losses': ['[289.21966598]']}}
Trial 3
Restart result {'status': 'ok', 'loss': 164.6398822553389, 'validation_loss': '26.753922', 'experimental_loss': 52.71014284620098, 'kfold_meta': {'validation_losses': ['26.753922'], 'experimental_losses': [52.71014284620098], 'hyper_losses': ['[164.63988226]']}}
Direct result {'status': 'ok', 'loss': 164.6398822553389, 'validation_loss': '26.753922', 'experimental_loss': 52.71014284620098, 'kfold_meta': {'validation_losses': ['26.753922'], 'experimental_losses': [52.71014284620098], 'hyper_losses': ['[164.63988226]']}}

Sorry, I will investigate my hyperopt_loss branch.

[RoyStegeman]

Ah so when you said the test failed occasionally you had just observed it in the hyperopt_loss branch (where I indeed now notice the same error in the CI log)?

It seems that the loss is always above 2, even in master, though this indeed shouldn't mean we can't still use these negative results to restart hyperopt.

[Cmurilochem]

That is true. I agree @RoyStegeman. Yes, I think I was biased by only looking at hyperopt_loss and also remember Aron with a similar problem this week. I am investigating what is going on in hyperopt_loss. It should pass with the same config as master (I suppose so!). Thanks once again!