ESMValTool requires access to the web to retrieve the recipes

[rbeucher]

https://github.com/Climate-REF/climate-ref/blob/01eabb53ecf47caf5d42f69e3e3601ac002c33ff/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/recipe.py#L150C12-L150C17

Hi,

I have been running some tests on the HPC and ran into this issue today. It is a bit of a problem because, on many HPC systems, internet access is restricted to specific queues with very limited resources. It also feels sub-optimal in general. The recipes are already shipped with the ESMValTool package, so I would expect them to be retrieved from the installed package rather than fetched online.

An alternative could be to support an environment variable that points to one or more local paths to search for recipes.

@bouweandela, keen to hear your thoughts. At the moment this is quite limiting for us, and it means we cannot use REF on Gadi.

[bouweandela]

Thanks for reporting @rbeucher! @minxu74 Also complained about this, so I will take a look at how we can solve this. I expect there will be a similar issue with downloading the data needed for running the fire diagnostic.

[bouweandela]

An alternative could be to support an environment variable that points to one or more local paths to search for recipes.

This is implemented in the current code you linked above, you can use REF_METRICS_ESMVALTOOL_DATA_DIR. I had a go at implementing a more user-friendly solution in #471.

[lewisjared]

Perhaps we should add an optional hook for a provider to run some setup steps. This setup step is common across all providers and a common place for providers to add this logic. This could then include fetching extra provider-specific datasets (which is already an explicit command to run) or generating a local cache so we might be able to consolidate.

Users will then run each of the providers setup hooks via a common CLI command (ref setup?) which will replace all the different CLI steps for fetching data. HPC users will need to run this on a node which has internet as we can't get around the need to fetch data, but we can make it happen at a known time. This should also be run after upgrading versions.