BUG: assert_array_almost_equal fails to compare array with masked array

[zbruick]

Reproducing code example:

The following code sample passes, but is expected to fail:

from numpy.testing import assert_array_almost_equal
assert_array_almost_equal(np.array([10, 20]), np.ma.array([10, np.nan], mask=[False, True]), 2)

Numpy/Python version information:

python 3.7.3
numpy 1.17.2

[mattip]

Why is this an issue? You are masking the value, so it is does not participate in the assertion.

[kalvdans]

I'm with the reporter here, a masked value never equals any unmasked value. Not even almost.

[dopplershift]

@mattip The problem IMO is this:

# Assert passes
assert_array_almost_equal(np.array([10, 20]),
                          np.ma.array([10, 20], mask=[False, False]), 2)

# Also passes--Yikes!
assert_array_almost_equal(np.array([10, 20]),
                          np.ma.array([10, 20], mask=[True, True]), 2)

I have now completely changed the mask, but the assert still passes. In what sense, in the second case, should those arrays be considered "almost equal"? I understand in normal calculations, that masking means "ignore this element". But in the case of an assert helper, used in testing and development, almost equal to me means "almost same values" and "same mask".

The problem as originally stated is insidious because our test was written to expect, and test against, a value of np.array([10, 20]). When the function with a bug (incorrectly) returned a masked array of np.array([10, np.nan], mask=[False, True]), our test did not catch it. That seems incredibly bad, even dangerous, behavior to me.

[WarrenWeckesser]

Apparently the comparison functions in numpy.testing ignore elements where either operand is masked, and not just those where both are masked. This behavior is common to most of the functions:

In [84]: import numpy as np

In [85]: from numpy import testing

In [86]: a = np.array([1.0, 2.0, 3.4])

In [87]: b = np.ma.array([1.0, 0, 0], mask=[False, True, True])

In [88]: np.allclose(a, b)
Out[88]: True

In [89]: testing.assert_allclose(a, b)

In [90]: testing.assert_array_equal(a, b)

In [91]: testing.assert_equal(a, b)

In [95]: testing.assert_almost_equal(a, b)

In [96]: testing.assert_array_almost_equal(a, b)

In [97]: testing.assert_array_almost_equal_nulp(a, b)

Two functions that behave differently are array_equal and array_equiv:

In [114]: np.array_equal(a, b)
Out[114]: False

In [115]: np.array_equiv(a, b)
Out[115]: False

I see two important issues here:

• The docstring of each of the testing and comparison functions should explain how masked arrays are handled.
• There is a need for testing and comparison functions that require equivalent masks in both operands. That is, a masked element is ignored only if it is masked in both operands.

[dopplershift]

For a testing function called array*equal, it would seem to me that the proper behavior is to only ignore masked values if both are masked. I can't think of any semantic meaning to "equal" where one array is allowed to have a value masked and the other isn't. I think the problem comes down to:

a = np.array([10, 20])
b = np.ma.array([10, 20], mask=[True, True])
a == b

prints

masked_array(data=[--, --],
             mask=[ True,  True],
       fill_value=True,
            dtype=bool)

I'm guessing __eq__ ends up delegating to masked_array's implementation, which as described above, ignores anything involving a masked value.

[arobrien]

We struck this bug today and I agree that it is insidious; if our dev's hadn't been practicing test driven development and expecting a failed test before fixing it then we might not have caught it.

The Masked arrays documentation appears to encourage a few different use cases, so different users (and developers) may have different expectations when it comes to the subtleties of the behaviour. In my mind (considering two arrays mx and my which have the same shape but different masks):

• where the documentation talks about ignoring masked values for operations such as mx.mean() or mx.sum() this is has a well understood meaning
• when performing arithmetic between differently-masked arrays such as mx + my then the documentation is at least clear that the masks propagate (the mask of the result is the union of the input masks)
• when comparing element-wise such as mx < my and mx == my then mask propagation still makes sense
• BUT when comparing for array equality, such as testing.assert_equal(mx, my) and ma.allequal(mx, my) then there are differing concepts:
  • testing.assert_equal(mx, my) effectively takes the union of the masks (mask propagation) before comparing, and does not allow this behaviour to be modified. ma.allequal(mx, my, True) behaves the same way.
  • ma.allequal(mx, my, False) returns False if either of the arguments have any masked values. This leads to the strange result that ma.allequal(mx, mx, False) returns False, although it is consistent with np.nan != np.nan.
  • the behaviour @zbruick and @dopplershift described (where the masks must be equal for the comparison to return True) does not appear to be available from either method, unless you explicitly test np.testing.assert_equal(mx.mask, my.mask)

In my opinion, when it comes to the ma.allequal() and ma.allclose(), all three comparison strategies are probably valid, depending on the application, and should be supported. This might mean adding an extra argument, or expanding the accepted values for the 3rd argument.

For testing.assert_equal() and other testing methods, however, I strongly believe that the behaviour @zbruick and @dopplershift describe should be the default unless explicitly overridden by an argument. When testing we should generally be more defensive and, when in doubt, raise.

For now, the work around in our tests is to explicitly call np.testing.assert_equal(mx.mask, my.mask) every time.

[edmundhenley]

As a minor/cosmetic addition to what's been reported above, note also that masked arrays also don't play nicely with the helpful repr for np.testing.assert_allclose() - see below.

• And is the masked array also playing havoc with the mismatched element count?
• I've not looked to see if any of the other np.testing tools have similar issues.
• Reporting this here as doesn't seem to be worth a standalone issue (please say if you disagree!)

Click to see evidence that repr doesn't show up nicely (Python v3.12.0; numpy v1.26.0)

>>> import numpy as np
>>> import numpy.ma as ma

>>> mx = ma.masked_array([1,2,3], mask=[0, 0, 1])
>>> my = ma.masked_array([1,2,4], mask=[0, 0, 1])
>>> mz = ma.masked_array([0,2,3], mask=[0, 0, 1])

>>> np.testing.assert_allclose(mx, my, rtol=0.01, atol=0.5)  # "unexpectedly" (YMMV!) equal, as reported above

>>> np.testing.assert_allclose(mx, mz, rtol=0.01, atol=0.5)  # unmasked difference: you shall not pass! But repr fails :(
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "$me/.conda/envs/latest_numpy/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "$me/.conda/envs/latest_numpy/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "$me/.conda/envs/latest_numpy/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.01, atol=0.5

Mismatched elements: 2 / 3 (66.7%)
Max absolute difference: 1
Max relative difference: 0.
 x: [repr failed for <MaskedArray>: Cannot convert masked element to a Python int.]
 y: [repr failed for <MaskedArray>: Cannot convert masked element to a Python int.]
>>> # also, why is above reporting 2 mismatched elements? Surely only 1 differs?!

>>> np.testing.assert_allclose(mx.data, mz.data, rtol=0.01, atol=0.5)  # what repr looks like for unmasked arrays
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "$me/.conda/envs/latest_numpy/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "$me/.conda/envs/latest_numpy/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "$me/.conda/envs/latest_numpy/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.01, atol=0.5

Mismatched elements: 1 / 3 (33.3%)
Max absolute difference: 1
Max relative difference: 0.
 x: array([1, 2, 3])
 y: array([0, 2, 3])
>>> # that's more like it - both for repr, and for mismatched element count!

>>> np.__version__
'1.26.0'
>>> import sys; print(sys.version)
3.12.0 | packaged by conda-forge | (main, Oct  3 2023, 08:43:22) [GCC 12.3.0]