Skip to content

Implementation of hyperopt model selection#1976

Closed
Cmurilochem wants to merge 8 commits into
masterfrom
dev_hyperopt_model_selection
Closed

Implementation of hyperopt model selection#1976
Cmurilochem wants to merge 8 commits into
masterfrom
dev_hyperopt_model_selection

Conversation

@Cmurilochem
Copy link
Copy Markdown
Collaborator

@Cmurilochem Cmurilochem commented Mar 4, 2024

This PR aims to implement #1962 into validphys vp-hyperoptplot.py.

Updates

The model selection following to the min_chi2_max_phi algorithm (refactored from best_chi2_worst_phi2 in #1962) could be done in two ways:

1. via a basic script that imports validphys hyperopt_dataframe

An example is shown below:

from pathlib import Path

from validphys.hyperoptplot import hyperopt_dataframe

hyperopt_name = 'hyperopt_10_rep_chi2_average'

# Get the current directory as a Path object
current_directory = Path.cwd()
path = current_directory / hyperopt_name

args = {
    'debug': True,
    'filter': [],
    'hyperopt_folder': path,
    'loss_target': 'min_chi2_max_phi',       # select Juan & Roy's algorithm
    'max_phi_n_models': 10,                       # select the n lowest values of 1/phi
    'val_multiplier': 0.0,
    'threshold': 3.0,
    'combine': False,
    'autofilter': [],
    'include_failures': False
}

all_data, best_setup, best_models = hyperopt_dataframe(args)

where best_setup corresponds to the model which shows the lowest 1/phi among those with the lowest chi2, while best_models is a pandas DataFrame containing all max_phi_n_models.

2. via vp-hyperoptplot

by running:

vp-hyperoptplot -l min_chi2_max_phi --max_phi_n_models 10 hyperopt_10_rep_chi2_average -t 3

which would launch an html file in your browser with some statistics and plots (I thinks this could be shared within the NNPDF server later on)

Screenshot 2024-03-08 at 16 54 36 (2)

TODO: add a table in the html file containing the detailed specs of best_models.

@Cmurilochem Cmurilochem force-pushed the dev_hyperopt_model_selection branch from fe55608 to 8679dca Compare March 8, 2024 08:36
@Cmurilochem Cmurilochem added validphys enhancement New feature or request labels Mar 8, 2024
@Cmurilochem Cmurilochem force-pushed the dev_hyperopt_model_selection branch 2 times, most recently from d7a3873 to e5294e8 Compare March 12, 2024 16:48
@Cmurilochem Cmurilochem force-pushed the dev_hyperopt_model_selection branch from e5294e8 to 14efc0c Compare March 18, 2024 08:16
@Cmurilochem Cmurilochem force-pushed the dev_hyperopt_model_selection branch from 14efc0c to 21c5a06 Compare April 19, 2024 07:48
@Cmurilochem Cmurilochem force-pushed the dev_hyperopt_model_selection branch from 21c5a06 to 695c722 Compare May 27, 2024 06:43
@Cmurilochem Cmurilochem force-pushed the dev_hyperopt_model_selection branch 2 times, most recently from 2ea055a to aa61cb7 Compare June 12, 2024 07:12
@Radonirinaunimi Radonirinaunimi force-pushed the dev_hyperopt_model_selection branch from b44c2ee to 535ea80 Compare October 2, 2024 18:37
@scarlehoff
Copy link
Copy Markdown
Member

Can we review plus merge this now @Radonirinaunimi or do we want to wait for ref. replies?

@scarlehoff
Copy link
Copy Markdown
Member

Now that the paper is accepted, should we merge this as is?

@scarlehoff
Copy link
Copy Markdown
Member

@Radonirinaunimi what do we do with this PR?

@Radonirinaunimi
Copy link
Copy Markdown
Member

This is now overly deprecated so let's close it (and in any case it will be removed in favor of the new sampling).

@scarlehoff scarlehoff closed this Sep 30, 2025
@scarlehoff scarlehoff deleted the dev_hyperopt_model_selection branch September 30, 2025 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Automated selection of models via a best_chi2_worse_phi2 algorithm

3 participants