PI-3473: Add Ancillary Variable support to merge

[stephenworsley]

Addresses half of #3552. Now merge treats Ancillary Variables the same as Cell Measures.

[pp-mo]

Now merge treats Ancillary Variables the same as Cell Measures.

Unfortunately I think this approach is probably wrong, as discussed offline with @bjlittle ...

NOTE related prior discussions #3603 #3084 #3234

We think that not ever merging (stacking) cell-measures does make ok sense, as these generally describe the "data spatial sampling grid" in dims X, Y and possibly Z.
( So, we expect that "grid" to be probably complete within each source file, and not extend into "extra" dimensions seen only by merging multiple netcdf files ).

However in contrast, Auxiliary Variables can easily map over any and all dimensions of the data, particularly when they are used as status flags : they could have a value at every single datapoint.
So Ancillaries should be stacked up by merging in a manner analogous to the data itself.

( Actually, I had thought that merge would do this with AuxCoords too, but it seems we have chosen not to.
I only just found this out 😉 )

---

On the other hand ...

I see this was not in the original Acceptance Criteria on #3552, as opposed to the equivalent for concatenate :

• cubes with similar ancillary variables which span the axis of concatenation, will concatenate to form a new ancillary variable which spans the concatenated cube
• cubes with differing ancillary variables (i.e. same names but some difference) will not merge;

So, having seen that we can't currently do this with AuxCoords either, I wonder..
can we leave this as extended functionality for later delivery ?

@stephenworsley has shown that we can work-around these cases via concatenate :

• .. using first iris.util.new_axis to create the new ("merge") dimension ..
• .. then a fairly short utility to "promote" Ancillaries (?and AuxCoords) to mapping the 'new axis',
  • ( roughly : replace with reshaped version, with extra length-1 dimension and new-dim added into dims-map )
• .. then resulting cubes will concatenate to the required result

See proposed extension to new_axis : #3600

[stephenworsley]

can we leave this as extended functionality for later delivery ?

@pp-mo Does this mean that iris 3.0 should discard/ignore ancillary variables on merge? If this is the case, perhaps a warning should be given when this happens.
Or are you suggesting we keep the current functionality of the pull request with a view that this can be changed/extended in the future?

[pp-mo]

@stephenworsley Does this mean that iris 3.0 should discard/ignore ancillary variables on merge? ... Or are you suggesting we keep the current functionality of the pull request

The latter I think: That is, not discard but only carry over matching versions. A statement of that would be useful, somewhere but frankly I'm not sure where. Suggest if you like..

I still need to check this out further, including test coverage, but I will need a bit longer. Sorry to be so slow today.

[pp-mo]

Ok, after long thinks about the concatenate #3566, I've convinced myself that this one really is much simpler. Thanks @stephenworsley !