Optimise unit conversion and ops

[znicholls]

At the moment our unit conversion is fairly custom. It also relies on pandas in the ops, which is probably pretty slow given you have to make the dataframe before operating on it. Given we now have ScmRun, which relies on xarray, there is probably going to be a better way. There is much talk of integrating Pint with xarray so maybe that's the best route. From what I can tell, the jumping off points/relevant issues are: pydata/xarray#525 and pydata/xarray#3594

[znicholls]

@lewisjared fyi

[znicholls]

@rgieseke and @swillner I have no idea if this is of interest to you, but just fyi

[znicholls]

We might even be able to skip straight to using pint in our internal handling thanks to pydata/xarray#3975

[znicholls]

Now that we've swapped back to storing internals as dataframes, the xarray route probably isn't the right one. (This is very cool though for other projects https://xarray.dev/blog/introducing-pint-xarray, and xarray's unit aware support for many common operations does make me wonder if we've attached our horse to the wrong cart by using pandas as our backend)

[lewisjared]

That is what PRIMAP2 uses for unit handling. We shouldn't feel too bad as that didn't exist when we were playing with it

[znicholls]

Ye but I remember we switched back to pandas for our internals for a different reason. I think it was the sparsity issue from memory...