Preprocessor chain is not being applied to cell measures

[ledm]

Basically, if your preprocessor changes the shape of the diagnostic data cube (for instance a spatial or temporal slice, such as the annual_statistics or extract_region preprocesors), before using the area_statistics, zonal_meanandvolume_statistics` preprocessors, then the resulting cube will no longer have the same shape as the FX data.

This means that we can only use the mean operation in area_statistics, zonal_meanandvolume_statistics` can only be applied on th global scale. It is not currently possible to use them on a regional scale.

Many of the preprocessors need to be applied to both the diagnostic dataset, but also the FX data, in the cases when the mean operator is used in area_statistics, zonal_meanandvolume_statistics`.

This issue is related to the recent fixes merged by @mattiarighi, @schlunma and @valeriupredoi. The PR's #429 and #433 have closed some FX related issues recently, but I don't want this issue to get lost with the other closed issues and PR.

[ledm]

I think that this issue had been raised elsewhere, but never had a dedicate issue number.

[ledm]

@schlunma @mattiarighi @valeriupredoi @zklaus, do any of you have any thoughts on how to implement this fix?

I don't mind trying to do it myself, but it would be good to have a discussion on how to do it first. Cheers!

[ledm]

This fix needs:

• Apply the same preprocessor functions in the same order to the fx data.

However, there are lots of caveats, because nothing is ever easy:

• only apply when the fx files are used by a preprocessor function (typically in a *_statistics function). Remember that not all statistics operator need the fx files.
• Only apply when a preprocessor before the statistics preprocessor changes the shape of the diagnostic cube.
• Does not apply any of the preocessor functions that would occur after the _statistics preprocessor that uses the FX file.
• Also applies any preprocessor to the FX file that would make the FX and the diagnostic data incompatible (even if the operation maintains the same shape). For instance, regrid_time.
• Keep in mind that some FX files are time-dependent and some are time-independent. This means that should only apply time preprocessing functions to fx data with a time component.
• Presumably some preprocessors should never need to be applied to the FX (mask_fillvalues, convert_units, perhaps?)

Feel free to edit the comment for clarity of if we encounter additional requirements/caveats.

[valeriupredoi]

cool cheers @ledm - will have a stab at this later on today, first fixing tests 🔢

[mattiarighi]

Thank you @ledm for listing the caveats.
One question: can you give me an example of time-dependent fx? I only have experience with the masks and the area files.

[ledm]

A time dependent FX file is fairly common in the Ofx volcello. The ocean grid varies with time (for instance, due to tides, storm air pressure changes or sea level rise) and so the cell volume needs to change.

The CMIP6 models: GFDL-CM4, UKESM1 and HadGEM3 all have time varying volcellos. Here's a recipe that shows this problem:

preprocessors:
    prep:
        custom_order: true
        extract_volume:
            z_min: 0
            z_max: 100
        annual_statistics:
            operator: mean
        volume_statistics:
            operator: mean
            fx_files: [volcello, ]

diagnostics:
  diag:
      variables:
        thetao700_Omon:
          short_name: thetao
          preprocessor: prep
          additional_datasets:
            - {dataset: GFDL-CM4,        project: CMIP6, mip: Omon, exp: historical, ensemble: r1i1p1f1, grid: gn, start_year: 1960, end_year: 2014}
            - {dataset: HadGEM3-GC31-LL, project: CMIP6, mip: Omon, exp: historical, ensemble: r1i1p1f3, grid: gn, start_year: 1960, end_year: 2014}
            - {dataset: UKESM1-0-LL,     project: CMIP6, mip: Omon, exp: historical, ensemble: r2i1p1f2, grid: gn, start_year: 1960, end_year: 2014}

In this case, we'd need to apply both the extract_volume and the annual_statistics operators to the volcello FX data, before volume_statistics can correctly use the volcello data to calculate the volume-weighted mean temperature.

[zklaus]

@ledm is right about the changing nature of volcello. Other affected variables are thkcello and masscello. However, note that if they are time-dependent they live in Omon (and Odec), not Ofx.

[valeriupredoi]

started the work in #439 - nearly there but not ready yet, I'll let you guys know when done, most prob in a couple hours 🍺

[ledm]

Just noticed that zonal_statistics and meridional_statistics don't even have the fx_files argument at the moment (but they probably should!) I'm making an issue about it now.

[valeriupredoi]

Another thing I need to fix in #439 is to retrieve and use only the fx files with the same mip as the master variable since we dont want to use the eg mip=Ofx ones, just realized that

[zklaus]

What do you mean by that? Whether we must use eg (Omon, volcello) or (Ofx, volcello) only depends on the model, after all.

[ledm]

I think he means this: #440.