Support tests for multimodel statistics preprocessor function

[bouweandela]

To support testing the multimodel statistics preprocessor function with the data from this repository, the following still needs to be done

Importable data loader #3:

• A function for loading the data conveniently, e.g. load_timeseries() which returns a cubelist (use cube_helper.load for loading the data?)
• Make installable module esmvaltool_sample_data

Download script in #4:

• Replace the . in the paths with a / for all subdirectories of esmvaltool_sample_data/data/timeseries
• Download all timeseries datasets, but only save small ones (e.g. skip AWI data for now, because they save one file per year and this becomes big even with subsetting)
• Improve download script so the subsetting is done for equal geographical regions/vertical levels instead of just by slicing the first 2 steps
• Select also an ocean variable for subsetting (item 2 in Populate repository with data #1), e.g. with a daily frequency (if multimodel statistics supports it) -> this needs proper support from iris first
• Clean up the script for downloading data (now in esmvaltool_sample_data/sample_data.py)

Add metadata #11 :

• Add license information
• Improve README
• Improve CONTRIBUTING
• Update setup.py

Other:

• Move download script to root directory #9 Rename download script to download_sample_data.py and move it to root

[bouweandela]

@stefsmeets This is the script I used for loading all files of a dataset into a single cube initially, might be of help:

import warnings
import glob
import os
from pathlib import Path
import iris
import cf_units

def strip_attributes(cube):
    """Remove attributes that cause issues with merging and concatenation."""
    for attr in ['creation_date', 'tracking_id', 'history']:
        if attr in cube.attributes:
            cube.attributes.pop(attr)

def simplify_time(cube):
    coord = cube.coord('time')
    coord.convert_units(
        cf_units.Unit(
            'days since 1850-1-1 00:00:00',
            calendar=coord.units.calendar))


def main():

    rootpath = Path("/mnt/lustre02/work/ik1017/CMIP6/data")

    project = 'CMIP6'
    mip_table = 'Amon'
    var_name = 'ta'

    activity = 'CMIP'
    institute = '*'
    model = '*'
    experiment = 'historical'
    ensemble = 'r1i1p1f1'
    grid = '*'
    version = '*'
    
    dirglob = str(
        rootpath
        / project
        / activity
        / institute
        / model
        / experiment
        / ensemble
        / mip_table
        / var_name
        / grid
        / version
    )

    fileglob = '_'.join([
        var_name,
        mip_table,
        model,
        experiment,
        ensemble,
        grid,
        '*.nc',
    ])


    for dirname in glob.glob(dirglob):
        print(dirname)
        files = [str(f) for f in Path(dirname).glob(fileglob)]
        with warnings.catch_warnings():
            warnings.filterwarnings(
                'ignore',
                message="Missing CF-netCDF measure variable .*",
                category=UserWarning,
                module='iris',
            )
            cubes = iris.load(files)
        for cube in cubes:
            strip_attributes(cube)
            simplify_time(cube)

        cubes = cubes.concatenate()
        print(cubes)

        outdir = Path('out') / os.path.relpath(dirname, rootpath)

        outdir.mkdir(parents=True, exist_ok=True)
        filename = str(outdir / os.path.basename(files[0]))

        cube = cubes[0]
        iris.save(cube[:, :2, :2, :2], target=filename)

if __name__ == '__main__':
    main()

[stefsmeets]

Thanks, this code is quite simple and much faster than using cube_helper. Will use this then to avoid relying on a dependency.