BUG: Cube equality boolean data

[cpelley]

Issue introduced in numpy 1.14 I think where subtracting two boolean arrays was deprecated.

  File "/path/to/iris/cube.py", line 3024, in __eq__
    result = np.all(np.abs(self.data - other.data) < 1e-8)
TypeError: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

This may be better placed as a more general numpy comparison operator in iris but I don't see there is anywhere suitable for such a thing to live currently.

[rcomer]

Why not just use numpy.allclose?

[rcomer]

Why not just use numpy.allclose?

Or even dask.array.allclose.

[rcomer]

da is already dask.array and the function returns a dask array. So I think you need:

da.allclose(self.core_data(), other.core_data()).compute()

(edited following @pp-mo's comment)

[rcomer]

If we're happy with using da.allclose, I guess this comment about loading the data can be removed?

[pp-mo]

If this is using 'self.data', 'other.data' then of course it is still realising the cubes.

In fact I don't think we can use dask here, as the result needs to be a concrete boolean, whereas da.allclose returns a lazy scalar value, even when the args are concrete ...

>>> da.allclose(np.array([1,2]), np.array([1,1]))
dask.array<all-aggregate, shape=(), dtype=bool, chunksize=()>
>>> _.compute()
False
>>> 

[rcomer]

If this is using 'self.data', 'other.data' then of course it is still realising the cubes.

Ah yes of course, you'd want to use core_data() or lazy_data()

Definitely in learning mode here...

Is there no advantage to using da.allclose with the compute call? I was thinking it would access the files in chunks and therefore avoid busting memory.

Aside: that thing you did with the underscore, I did not know was a thing 😮 .

[pp-mo]

Is there no advantage to using da.allclose with the compute call? I was thinking it would access the files in chunks and therefore avoid busting memory.

Actually that is a really good point 👍
It won't realise any more, which it used to, but I think we always view that as a good thing 😉

you'd want to use core_data()

I think it makes sense to use 'core_data' as it involves the minimum of extra work.

My key point was : the routine itself must still return a regular boolean, not a lazy value.
... but I see now you have just added the ".compute()" above ! 👍

the underscore

Yes, it's where the interpreter puts unassigned results (in addition to printing them).
Also very handy when you forgot to save the result of a calculation !

>>> long_and_complicated()
BigObject(...)
>>> keep_that = _

[bjlittle]

@pp-mo Fancy taking this on?

@cpelley I think that you're going to have to re-base to fold in the latest Travis CI changes

[pp-mo]

Thanks @bjlittle
@cpelley I think this is good to go, if you can get a green tick on it.
🐴 🥕
🙏

[lbdreyer]

@cpelley Sorry about this! The travis tests are failing but I believe the fix for those tests went in already. If you rebase we should be able to get a green tick.

Do you mind rebasing and then this can go in?

[pp-mo]

Hi @cpelley !
Jog : can you get this rebased in a day or so, and maybe add a whatsnew item (not essential: we can make something up) ?
Then it can go into Iris 2.3.

[pp-mo]

#3431 replaced : closing.