Support netcdf variable emulation

[pp-mo]

The basic idea : when loading from / saving to a dataset-like object (as provided by #5214), Iris can spot objects which "emulate" a netCDF4.Variable and provide direct access to a content array (i.e. not via file access).
This allows us to directly transfer array data (real or lazy) without copying.

TODO: in retrospect, _in_memory_array is really not a good choice, as it seems to imply numpy data, whereas crucially it will often be dask. We should change the name to something better.
Maybe "_data_array".

This, along with #5214 (formerly #5024), should provide all we need to support the intended "Xarray bridge" - see #4994 .
Scheduled for Iris 3.6

Currently draft/WIP, because this will need adjustment after #5191, and it's far easier to resolve if that goes in first !

Also note comment here

we may have a bit more work to successfully integrate these changes with those from #5095

[pp-mo]

Ping @ESadek-MO @trexfeathers. Ideally I'd like this to squeeze into Iris 3.5.

Included in Iris 3.6 project

Although we've now bumped #4994 to Iris 3.7, the remainder of that work will happen in a separate "ncdata" package.
So if we can get these preliminaries into Iris itself, we can go live with that, without requiring a further Iris release to support it (I think 🤞 !)

[codecov]

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (a3931f6) 89.31% compared to head (1518756) 89.32%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5212   +/-   ##
=======================================
  Coverage   89.31%   89.32%           
=======================================
  Files          89       89           
  Lines       22375    22379    +4     
  Branches     5368     5370    +2     
=======================================
+ Hits        19985    19989    +4     
  Misses       1640     1640           
  Partials      750      750           

Impacted Files	Coverage Δ
lib/iris/fileformats/netcdf/loader.py	80.51% <100.00%> (+0.14%)	⬆️
lib/iris/fileformats/netcdf/saver.py	89.02% <100.00%> (+0.02%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[trexfeathers]

Ping @ESadek-MO @trexfeathers. Ideally I'd like this to squeeze into Iris 3.5.

IMHO this should be a stretch goal, since we've already committed to the work that we will do for 3.5.

[pp-mo]

Ping @ESadek-MO @trexfeathers. Ideally I'd like this to squeeze into Iris 3.5.

IMHO this should be a stretch goal, since we've already committed to the work that we will do for 3.5.

It was a mistake ! now in Iris 3.6

[pp-mo]

Update : rebased

Actually more re-written, but it parallels what went before.
Been out rather too long 😬 🕚

BTW still in draft.

1. may need more work to get tests passing
2. want to get this working with prototype ncdata (which now needs some naming updates) before committing to it.

[pp-mo]

Update : Ready for Review

Existing tests now passing.
Tested against a revised WIP 'ncdata' + seems to function.

Ping @ESadek-MO is this for you to review also ?
( when done with #5214 !! )

[ESadek-MO]

Looks good to me in it's current state, is this waiting on threadsafety as in #5214?

[pp-mo]

Looks good to me in it's current state, is this waiting on threadsafety as in #5214?

These changes are supposedly ~independent, but yes I think it's probably best to get #5214 in first.
I'm just making a merge of the 2 for re-testing against ncdata, and I'm finding conflicts.

Thanks for looking!

[pp-mo]

I think I was wrong about the conflicts -- using an older version.
I am just re-spinning the tests to be sure it's good ...

[pp-mo]

Ok thanks @ESadek-MO , I'm happy this is good to go !

( btw I tested a combination of this+#5414 against prototype ncdata, and it seems good 👍 )

[ESadek-MO]

Thanks @pp-mo, looks good to me!