Enhance error when input files are not found

[schlunma]

This addresses an issue of @ledm's presentation at the workshop: Right now, the tool simply outputs

No input files found for variable {...}

if it does not find input data. To get an idea of what went wrong, user's have to look at the debug log. Also, I think the attribute filename in the variable dictionary is very confusing, since it does only indicate the location of the file after preprocessing, which is definitely not useful here.

The main changes are in the following code snippet. The text of the actual error messages themselves is of course open for discussion.

ESMValCore/esmvalcore/_recipe_checks.py

Lines 95 to 110 in b9d82b0

	if not input_files:
	var.pop('filename', None)
	logger.error("No input files found for variable %s", var)
	if dirnames and filenames:
	logger.error("Looked for files matching %s in %s", filenames,
	dirnames)
	elif dirnames and not filenames:
	logger.error(
	"Looked for files in %s, but did not find any file pattern "
	"to match against", dirnames)
	elif filenames and not dirnames:
	logger.error(
	"Looked for files matching %s, but did not find any existing "
	"input directory", filenames)
	logger.error("Set 'log_level' to 'debug' to get more information")
	raise RecipeError("Missing data")

[ledm]

Good work @schlunma. The PR runs as expected! I ran it and it outputs better error messages.

I was thinking that the message:

Looked for files matching ['msftmyz_Omon_CanESM25_historical_r1i1p1_*.nc'] in ['/users/modellers/ledm/workspace/ESMValToolTest/KIT_data/']

would be better if the filename and directory were combined. Ie:

Looked for files matching: '/users/modellers/ledm/workspace/ESMValToolTest/KIT_data/msftmyz_Omon_CanESM25_historical_r1i1p1_*.nc'

You could then copy and paste it into the command line and figure out whats the problem. Does that make sense?

[schlunma]

I was thinking that the message:

Looked for files matching ['msftmyz_Omon_CanESM25_historical_r1i1p1_*.nc'] in ['/users/modellers/ledm/workspace/ESMValToolTest/KIT_data/']

would be better if the filename and directory were combined. Ie:

Looked for files matching: '/users/modellers/ledm/workspace/ESMValToolTest/KIT_data/msftmyz_Omon_CanESM25_historical_r1i1p1_*.nc'

You could then copy and paste it into the command line and figure out whats the problem. Does that make sense?

Yeah, makes perfectly sense, this is way better 😁 I will change it right now.

[mattiarighi]

Can we merge this?

[bouweandela]

No, I want to have a look at the intake library (see #31) before doing major work on data_finder.py.

[valeriupredoi]

@bouweandela and meself had a discussion at the Workshop about intake and we both agreed that it'd be a good thing but not just yet since it'll involve a major overhaul of all the data finding and logging functionalities; so I say let's get this in since it'll be very helpful especially in conunction with the dryrun capability #307

[valeriupredoi]

quite a few conflicts tho, wouldn't want to be the one fixing them 🤣

[valeriupredoi]

@schlunma would you be a peach and fix the conflicts pls, then I'll approve and it'll be good to go 🍺

[valeriupredoi]

good to go by meself 🍺

[schlunma]

Ahhh, now you were too fast, I wanted to add a test for the Check.data_availability feature 😬 I also haven't tested it yet, I'm really not sure if that broke anything 😄

[valeriupredoi]

should be fine - can you add the test in a separate PR? That makes more sense since this dealt with printing messages only 🍺