Add displaying and loading of recipe output to the notebook API

[stefsmeets]

Description

Hi everyone, this PR extends the API to run recipes in an interactive notebook.

It adds functionality to obtain the output data (data or plots), and display or load them directly in the notebook.

• For more discussion, see issue ESMValCore (and ESMValTool) Python/Jupyter Notebook API #498
• Follows up on Add loading and running recipes to the notebook API #907, Add Listing and finding recipes to the experimental notebook API #901, Implement importable config object in experimental API submodule #868
• Link to documentation: https://esmvaltool--957.org.readthedocs.build/projects/ESMValCore/en/907/api/esmvalcore.api.html
• Demo notebook: https://gist.github.com/stefsmeets/064c86a83a0490589203eee6cc51e31b

Todo

• Wrap recipe_output dict and add convenience methods
• Add shorthand to get items of a type, i.e. recipe_output.get_output_of_type('data')
• Elaborate/update on documentation
• Fine-tune TaskOutput usage (__repr__ / rename .filename attribute)

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

• PR has a descriptive title for the changelog
• Code follows the style guide
• Documentation is available for new functionality
• YAML files pass pre-commit or yamllint checks
• Circle/CI tests pass
• Codacy code quality checks pass
• Documentation builds successfully on readthedocs
• Unit tests are available

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[stefsmeets]

I just realized we do not depend on xarray in ESMValCore. Just wondering if having a .load_xarray method warrants adding it to the list of dependencies. Afaik ESMValTool has it in its dependencies after all.

[Peter9192]

Very cool stuff @stefsmeets, this is really impressive already! Here's a few general comments:

• The documentation (before the API reference) is a bit limited, and I'm not sure if I could readily use it. The notebook helped me to understand how to the items from the output. A slightly more elaborate example would be nice.
• Would it be simple enough to make shorthand methods for these intimidating lines in the example notebook like data_items = [item for tasks in recipe_output.values() for item in tasks if item.kind == 'data']? Then I think that's worth adding.
• How much effort would it be to have a nice markdown rendering of the output, like you did for the Recipe Info? I think that could potentially be awesome as well.

[stefsmeets]

Thanks @Peter9192

* The documentation (before the API reference) is a bit limited, and I'm not sure if I could readily use it. The notebook helped me to understand how to the items from the output. A slightly more elaborate example would be nice.

Yeah, I agree. I am collecting the notebooks I'm developing so that they can be used to support the documentation. I will try to see what I can do to add some more information.

* Would it be simple enough to make shorthand methods for these intimidating lines in the example notebook like `data_items = [item for tasks in recipe_output.values() for item in tasks if item.kind == 'data']`? Then I think that's worth adding.

This would mean turning recipe output into a class, and then add a method like .get_output_of_type('data'). I can do that 👍

* How much effort would it be to have a nice markdown rendering of the output, like you did for the Recipe Info? I think that could potentially be awesome as well.

Technically it is not too difficult. I'm already using HTML so that I can render the caption as well as the image. This can be extended with other attributes. I think you can do inline html in markdown, which may be a another way to do it.

I actually started adding the authors, but I stopped because I was worried of losing scope for this PR. It would require going through the attributes and presenting them in a nice way, including stuff like authors and references (which we already have), but also realms and domains. I think this PR puts a good baseline to continue from.

[stefsmeets]

It was a bit of work, but I added wrappers for the task and recipe output to make it a bit easier to work with these objects. This also lays the foundation to do stuff like recipe_output.report() and get a nice html representation. I think this is something we may want in the future (but not in this PR!).

[stefsmeets]

Hi @Peter9192 , I implemented all your feedback. If you approve, I think this PR is ready to be merged!

[Peter9192]

Hey Stef! This is starting to be very very cool! I have a couple of follow-up comments/questions, but all-in-all I'm very happy with this PR.

On codacy: shall we just move the imports to the top?

[stefsmeets]

On codacy: shall we just move the imports to the top?

I moved iris, but I can't do that for xarray, because it is imported as a convenience function, but we don't have it in our dependencies (see #957 (comment)).

[stefsmeets]

This PR is ready to be merged @nielsdrost !