Feedback as a non-ESGF user: discoverability of diagnostics, opaque provider config, and internet reliance

[rbeucher]

Hi all,

I have recently been diving a bit more into the REF (sorry it has been a while, I had to switch back to other projects for a bit). What follows is feedback purely from the perspective of someone using REF as a tool for running evaluation outside of the ESGF workflow, so it might or might not apply broadly.

1. Hard to discover available diagnostics

At the moment it is a bit difficult to figure out what diagnostics are actually available to me. I am constantly jumping between:

• the diagnostic providers’ documentation (e.g. ESMValTool recipes, PMP docs, etc.)
• the data I have already ingested into REF
• the REF documentation

It would help a lot to have something like:

ref diagnostic --list

Ideally showing the available diagnostics based on what I have ingested. Without this, discoverability feels guess-and-check, and I am not sure this will be practical in normal usage.

2. Provider configuration is opaque in REF

When using REF, the provider environment (e.g. ESMValTool) is installed, but it is not clear how it is configured inside REF. On HPC systems this will be problematic, especially for centres that want to:

• use their own data pools
• rely on site-specific configuration files

Since we do not see the config, it’s not obvious how to override it.

Suggestion: allow optional additional configuration paths in ref.toml that point to local provider settings (like ESMValTool config-user.yml). Maybe something like:

[providers.esmvaltool]
config = "/path/to/esmvaltool/config-user.yml"

Having some way to explicitly manage this would make HPC deployment much more realistic.

3. Internet reliance for retrieving diagnostics

I raised this in a previous issue #468 , but I want to reinforce it: REF should not rely on internet access to download diagnostics, documentation, or recipes. Many HPC centres restrict external network access, or only allow it on very small queues with limited resources. The current approach is going to be a blocker for many sites.

I know there is no “easy” solution here, but for HPC use it might be essential to:

• vendor or mirror diagnostic metadata
• ship diagnostics with REF releases
• provide offline plugin bundles

Anything that removes network reliance would help.

---

Happy to clarify any of the points above. Thanks again for all the work — really impressive progress overall!

[lewisjared]

Thanks for the feedback @rbeucher. All useful points.

The discoverability has been a weakness that we haven't addressed yet in multiple layers (datasets, diagnostics, executions). We also need to describe the experiments/variables that the REF actually cares about.

Your 2nd and 3rd points suggest that we need to add more hooks for ways that users can interact with the diagnostic providers. Perhaps there is too much magic currently which is fine for simple setups but falls over when a more complex use-case arrives

[lewisjared]

REF should not rely on internet access to download diagnostics, documentation, or recipes. Many HPC centres restrict external network access, or only allow it on very small queues with limited resources. The current approach is going to be a blocker for many sites.

Can this statement be narrowed to REF should not rely on internet access to download diagnostics, documentation, or recipes when executing?

Is the requirement that when running a solve as a compute job it can't access the internet, or is it that the application has to be able to be installed completely air-gapped. The lack of internet access on compute nodes is more common than I expected so is intended to be supported (but obviously missed something). Being completely air-gapped would require extra engineering effort to bake all of the recipes into a release.

We should test that there are no fetches during the solve and document where/when internet access is required