Support for ERA5 data in native format

[bouweandela]

This pull request makes it possible to use ERA5 data directly in ESMValCore. CMORization is implemented as a fix.

Related to issue ESMValGroup/ESMValTool#1120.

Related ESMValTool PR ESMValGroup/ESMValTool#1432

Tasks

• Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
• This pull request has a descriptive title that can be used in a changelog
• Add unit tests
• Public functions should have a numpy-style docstring so they appear properly in the API documentation. For all other functions a one line docstring is sufficient.
• If writing a new/modified preprocessor function, please update the documentation
• Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
• Codacy code quality checks pass. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
• Please use yamllint to check that your YAML files do not contain mistakes
• If you make backward incompatible changes to the recipe format, make a new pull request in the ESMValTool repository and add the link below

---

Closes #447
Also closes #311

[bouweandela]

TODO:

• Check that latitude and vertical coordinate are correctly inverted
• If original unit is m of water, convert to kg/m^2 (read original units from netcdf file, filename in cube attributes) using that 1 m^3 of water is 1000 kg.
• if original unit is (0 - 1), set to 1 (read original units from netcdf file, filename in cube attributes)
• Ignore iris warnings about invalid units at load time, these will be found by cmor checker anyway
• Create separate classes for variable specific fixes
• Take Consistently use dimension/coordinate names in VariableInfo.coordinates #413 into account
• Update corresponding recipe in ERA5 CMORization by recipe ESMValTool#1432 and check that output filenames can be used with project set to OBS6
• Also fix hourly time coordinate, should be shifted back half an hour for accumulated/mean/min/max variables, see https://confluence.ecmwf.int/display/CKB/ERA5+data+documentation#ERA5datadocumentation-Meanratesandaccumulations
• Add unit tests
• Add documentation
• Optionally, support per dataset DRS. Currently this is defined per project, but this does not make sense for the datasets in the Native6 project
• Fix Make datafinder understand dates in the middle of the file name #311 (make sure to use this information: Handling of time ranges (mostly in datafinder) #345 (comment))
• Change all tests for esmvalcore/cmor/_fixes/fix.py: get_fixes because its arguments are changed in this PR.
• Similarly, the Fix class now takes vardef as input, which breaks many tests. Add None or a vardef from the cmor tables.
• Add fix for orography (present in the old cmorizer)

[Peter9192]

@bouweandela and I addressed the remaining issues above, and this PR is now ready for review.

Verifying the output against old cmorizer, I found some differences that I think are actually better in the new version than in the old cmorizer:

• The time units in old are not changed (hours since 1990.....). In new (days since 1850...), they agree with the cmip6 coordinate definition from the cmor table.
• long_names for lon and lat are written without capitals in the old cmorizer. The new cmorizer uses capitals, in accordance with the CMIP table.
• uas van vas are not in the CMIP6_E1hr table. Their variable definition gets looked up in another table, but the time coordinate is inferred from the E1hr table (time1 in this case, instead of 'time'). The result is that the new cmorizers does not add bounds (the old cmorizer just adds bounds to all coordinates). I think the new behaviour makes more sense, because uas and vas are instantaneous variables, but we can change this (e.g. just add bounds to all vars like the old cmorizer) if people disagree.
• The new cmorizer addresses time values after daily_statistics (or monthly_statistics) #398 by shifting the time points half an hour back, but the result is that we end up with a time coordinate that has 1 point less, because 1st of january 00:00 is moved to the previous year, and the last point 31st december 00:30 cannot be inferred because it should have come from the next year.

PS: Failing circleci tests are addressed in #452

[mattiarighi]

Asking @valeriupredoi to have a look at the changes in _data_finder.py. 🍺

[mattiarighi]

All variables successfully tested, except monthly/orog, see main_log.txt.

[Peter9192]

All variables successfully tested, except monthly/orog, see main_log.txt.

Thanks @mattiarighi ! I removed monthly/orog because all we use it for is to create an fx variable out of it, and since we're doing that for hourly already, we don't need it in daily and monthly as well.

[mattiarighi]

@valeriupredoi are you ok with this?

[valeriupredoi]

@valeriupredoi are you ok with this?

looking at it right now, hold yer horses 🐴

[valeriupredoi]

OK a few minor comments - has @jvegasbsc reviewed this PR yet?

[Peter9192]

I ran again with Bouwe's most recent changes and everything works fine. Think it can be merged now.

[bascrezee]

Nice work @bouweandela, @Peter9192 and others. I want to get started with this in combination with the variable soil moisture (monthly data as provided by the CDS). Could you very briefly help me get started. E.g.:

-where to put the data? (we used to put download instructions at top of the cmorization scripts, where to put them for native6? --> I would be a big fan of having this included in ESMValCore)
-how to specify the dataset in the recipe?

Thanks.

[bouweandela]

Hi Bas,

where to put the data? (we used to put download instructions at top of the cmorization scripts, where to put them for native6? --> I would be a big fan of having this included in ESMValCore)

I guess we (again) failed to provide enough documentation ;-( The native6 project can be configured in the config-user.yml and config-developer.yml files with a DRS, similar to other projects. Still to do is to make it possible to have a dataset specific DRS, so we can have more than just ERA5 under the native6 project:
https://github.com/ESMValGroup/ESMValTool/blob/master/config-user-example.yml#L15

ESMValCore/esmvalcore/config-developer.yml

Lines 191 to 201 in a9312a7

	native6:
	cmor_strict: false
	input_dir:
	default: 'Tier{tier}/{dataset}'
	era5cli: 'Tier{tier}/{dataset}'
	input_file:
	default: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc'
	era5cli: 'era5_{era5_name}*{era5_freq}.nc'
	output_file: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}'
	cmor_type: 'CMIP6'
	cmor_default_table_prefix: 'CMIP6_'

how to specify the dataset in the recipe?

For an example recipe using ERA5, see here: https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/recipes/cmorizers/recipe_era5.yml