DataArray.clip() no longer supports the out argument

[seth-p]

As of xarray 0.18.0, DataArray.clip() no longer supports the out argument. This is due to #5184. Could you please restore out support?

@max-sixty

[max-sixty]

FYI I'm out on paternity leave for a few weeks.

Generally IIRC out hasn't been part of the public API, and only present because of our delegating to numpy for some functions.

What is the case for having out kwargs?

[seth-p]

What is the case for having out kwargs?

It lets you reuse memory you already have. In particular for a simple operation like clip, you can do it in-place: da.clip(..., out=da.values). Very useful if you deal with lots of data and memory is a concern.

[raybellwaves]

FWIW. This showed up in xskillscore as we were doing np.clip(xarray object, min, min). We updated the code to do xarray object.clip(min, max) which we probably should have been doing in the first place (xarray-contrib/xskillscore#309 (comment))

[max-sixty]

Thanks for the example @raybellwaves

Supporting np.{func} would be good — I have only partially followed the interop with numpy — e.g. __array_ufunc__ (edit: __array_function__) — is anyone familiar with whether we're violating the standards by not having an out arg with clip?

FYI, we don't generally have out args, but we do have them in nanops.py:

xarray/core/nanops.py:
   78: def nanmin(a, axis=None, out=None):
   86: def nanmax(a, axis=None, out=None):
  112: def nansum(a, axis=None, dtype=None, out=None, min_count=None):
  137: def nanmean(a, axis=None, dtype=None, out=None):
  151: def nanmedian(a, axis=None, out=None):
  169: def nanvar(a, axis=None, dtype=None, out=None, ddof=0):
  178: def nanstd(a, axis=None, dtype=None, out=None, ddof=0):
  184: def nanprod(a, axis=None, dtype=None, out=None, min_count=None):
  193: def nancumsum(a, axis=None, dtype=None, out=None):
  199: def nancumprod(a, axis=None, dtype=None, out=None):

[keewis]

clip dispatches through __array_function__ (it's not a real "ufunc"), so we don't really support that (there's a issue for adding __array_function__, but no progress yet). In general I think we should only support functions which don't have a axis parameter and return NotImplemented for all others, which means clip would be a good candidate.

We currently do have some support for numpy.clip(arr, ...) through __array_wrap__ (so no support for Dataset), not sure if that even allows deviating from a standard?

[max-sixty]

Thanks @keewis , that makes sense.

And ensuring I return to @seth-p 's original issue — I would be up for adding out if we're violating the __array_wrap__ standard by not having all its kwargs.

Otherwise I would vote weakly against out throughout xarray's API, and strongly against it out on an arbitrary subset of methods.

[keewis]

__array_wrap__ doesn't allow specifying any kwargs (as far as I can tell), it has two parameters: obj and context. I'd have to read up on this, but obj seems to be the data (from asarray or __array_prepare__?) and context seems to be None most of the time.

In any case, I think we should try to replace __array_wrap__ with __array_function__ (or __array_module__ / __array_namespace__?) soon.

Also, I agree on being consistent regarding the out parameter.

[seth-p]

I'm not familiar at all with the various numpy interfaces, so I can't offer any input implementation-wise. But as a user, being able to do operations in place (via out or otherwise) is extremely useful when dealing with large arrays under memory constraints. In fact my one "philosophical" beef with xarray is that it seems almost no attention is paid to minimizing memory consumption (whether through in-place operations, or more generally minimizing temporary memory usage).

[Illviljan]

xarray allows using dask arrays for data that don’t fit into memory and dask doesn't support out, https://docs.dask.org/en/latest/array-api.html#dask.array.clip.

[dcherian]

almost no attention is paid to minimizing memory consumption (whether through in-place operations, or more generally minimizing temporary memory usage).

I think we'd be open to fixing this when it doesn't compromise readability. Can you open a new issue with some particularly bad examples?

[seth-p]

almost no attention is paid to minimizing memory consumption (whether through in-place operations, or more generally minimizing temporary memory usage).

I think we'd be open to fixing this when it doesn't compromise readability. Can you open a new issue with some particularly bad examples?

I wouldn't necessarily say that it's particularly bad, but see the discussion following #2922 (comment).

[max-sixty]

Closing as stale. If someone wanted to implement out, we could discuss it — but it would need some work