Lazy implementation of multi_model_statistics and ensemble_statistics preprocessors

[Peter9192]

Description

Make the multi_model_statistics and ensemble_statistics preprocessors lazy.

Note that they are only lazy if

• at least one of the input cubes is lazy
• the statistics operator is lazy (all but MEDIAN)
• the input cubes do not have to be interpolated to get the right time points

Closes #781

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

• PR has a descriptive title for the changelog
• Code follows the style guide
• Documentation is available for new functionality
• YAML files pass pre-commit or yamllint checks
• Circle/CI tests pass
• Codacy code quality checks pass
• Documentation builds successfully on readthedocs
• Unit tests are available

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[stefsmeets]

Hi @Peter9192 , I just noticed #1009 got fixed in a separate PR (#1010) which is causing a merge conflict. Just wanted to check with you if this is still an issue in the changes we are making here (meaning can we ignore it)?

[Peter9192]

Hi @Peter9192 , I just noticed #1009 got fixed in a separate PR (#1010) which is causing a merge conflict. Just wanted to check with you if this is still an issue in the changes we are making here (meaning can we ignore it)?

If this is a feature that's needed than there should be a test for it. I think the new implementation will work only if either no or all cubes have this 'altitude' coordinate, not sure what the current master does...

[bouweandela]

No. So far, we have not been able to find additional funding to work on this, unfortunately.

[rcomer]

SciTools/iris#3901 has now been merged, so the lazy percentile aggregation will be available at Iris v3.3, which is due out in the summer.

[valeriupredoi]

many thanks for the heads up @rcomer

[sloosvel]

No. So far, we have not been able to find additional funding to work on this, unfortunately.

Should this PR be removed from the 2.6 milestone then?

[bouweandela]

Example recipe for testing:

---
documentation:
  description: Test recipe for multimodel statistics
  authors:
    - andela_bouwe

  references:
    - gleckler08jgr

  projects:
    - c3s-magic

preprocessors:

  preproc:
    regrid:
      target_grid: 0.5x0.5
      scheme: linear
    multi_model_statistics:
      span: overlap
      statistics: [mean]
      keep_input_datasets: true

diagnostics:

  tas:
    variables:
      tas:
        preprocessor: preproc
        product: output1
        mip: Amon
        project: CMIP5
        exp: historical
        ensemble: r1i1p1
        start_year: 1850
        end_year: 2005
    additional_datasets:
      - {dataset: ACCESS1-0}
      - {dataset: ACCESS1-3}
      - {dataset: bcc-csm1-1}
      - {dataset: bcc-csm1-1-m}
      - {dataset: BNU-ESM}
      - {dataset: CanESM2}
      - {dataset: CCSM4}
      - {dataset: CESM1-BGC}
      - {dataset: CESM1-CAM5}
      - {dataset: CESM1-FASTCHEM}
      - {dataset: CESM1-WACCM}
      - {dataset: CSIRO-Mk3-6-0}
      - {dataset: FIO-ESM}
      - {dataset: inmcm4}
      - {dataset: IPSL-CM5A-LR}
      - {dataset: IPSL-CM5A-MR}
      - {dataset: IPSL-CM5B-LR}
      - {dataset: MPI-ESM-LR}
      - {dataset: MPI-ESM-MR}
      - {dataset: MPI-ESM-P}
      - {dataset: NorESM1-M}
      - {dataset: NorESM1-ME}
      - {project: CMIP6, dataset: TaiESM1, ensemble: r1i1p1f1, grid: gn}
    scripts: null

[bouweandela]

@fnattino and I have some time to work on this now, but it seems unlikely that it will be ready in time for v2.7.

[bouweandela]

Tests by @fnattino appear to show that this works quite well when used with #1714.

[valeriupredoi]

many thanks to all who contributed to this - it looks great and I bet it's also adding quite a bit of performance improvement (I've not tested though); I am approving, but I have a few, minor questions. Cheers 🍺