BUG: Errors caused by DataFrame.all(..., skipna=False, ...) in rows without na values.

[carsten0202]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

I'm familiar with several issues being reported relating to skipna=False, but I have not seen this particular problem elsewhere. In short, the 'skipna=False' setting causes errors in rows not having any missing values.

Code Sample, a copy-pastable example

# Minimal code to reproduce in pandas 1.2.4:
import pandas as pd
df1 = pd.DataFrame({"A": [False,pd.NA,pd.NA], "B": [True,True,False]})
df1
       A      B
0  False   True
1   <NA>   True
2   <NA>  False

df1.all(axis=1, skipna=False)
0     True
1     True
2    False
dtype: bool

# And again with experimental BooleanDtype
df2 = pd.DataFrame({"A": [False,pd.NA,pd.NA], "B": [True,True,False]}, dtype=pd.BooleanDtype())
df2
       A      B
0  False   True
1   <NA>   True
2   <NA>  False

df2.all(axis=1, skipna=False)
0    True
1    True
2    True
dtype: bool

Problem description

With df1 row '0' is clearly wrong; with df2 '0' and '2' are undeniably wrong. The rest are imho also suspicious, but they at least follow the documented behavior of:

If skipna is False, then NA are treated as True, because these are not equal to zero.

...even if I do believe the mathematically correct answers are as given e.g. here:
https://www.ibm.com/docs/en/spss-statistics/SaaS?topic=command-missing-values-logical-operators-if
In other words, row '1' should, imho, evaluate to 'NA'.

Expected Output

Rows '0' and '2' should clearly evaluate to 'False'.
Row '1' should maybe evaluate to 'NA'.

Details

INSTALLED VERSIONS

commit : 2cb9652
python : 3.9.2.final.0
python-bits : 64
OS : Linux
OS-release : 3.10.0-957.12.2.el7.x86_64
Version : #1 SMP Fri Apr 19 21:09:07 UTC 2019
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : C
LANG : C
LOCALE : None.None

pandas : 1.2.4
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 49.6.0.post20210108
Cython : None
pytest : 6.2.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : None
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[mzeitlin11]

Thanks for the report @carsten0202! There's definitely some weird behavior here. For the first case you give, the issue that's occurring is that the call essentially gets dispatched to numpy.all. This will raise a TypeError because of the presence of pd.NA, triggering fallback logic to keep only boolean columns (because of the default bool_only=None). So the end result you see is just

values
Out[7]: 
array([[ True],
       [ True],
       [False]])
np.all(values, axis=1)
Out[8]: array([ True,  True, False])

So that behavior is strange, but somewhat correct, pending the tricky discussion of how to handle pd.NA in object type data (#32931).

For the second example, when using the new nullable boolean type, the goal is to follow Kleene logic (https://pandas.pydata.org/docs/user_guide/boolean.html#kleene-logical-operations)

That's somewhat complicated in the example you give because the aggregation is over axis=1, so even though the columns are BooleanDType, it's not necessarily true that the rows are (in this case it is, but that would require some changes in aggregation logic to know that). The reason this answer comes out different than the first case is that the fallback logic which extracts only boolean data is not recognizing BooleanDType, so the final result just ends up that of all on empty data, which gives True.

The case that should be easier to handle is aggregating directly over columns, eg df2.all(axis=0, skipna=False), but that currently raises.

[mzeitlin11]

The case of df2.all(axis=0, skipna=False) raising is indirectly caused by numpy/numpy#4352. (or #12863). Because of that issue, any/all reductions apply an explicit astype(bool) to the result, which raises when pd.NA is in the result. This issue would be easy to fix if those were resolved since the astype could just be removed, otherwise some workaround might be required.

[carsten0202]

Thanks for your feedback. I think I finally understand what the 'bool_only' option does :-)

And thanks for the link to Kleene logic. I hadn't found that page, but it was very interesting.

Just to update/re-iterate, then if I follow the Kleene logic correctly, the expected outcome for case 2 (df2) should be:

df2.all(axis=1, skipna=False)
0    False
1    <NA>
2    False

[mzeitlin11]

Just to update/re-iterate, then if I follow the Kleene logic correctly, the expected outcome for case 2 (df2) should be:

Yep, exactly!

[ChiQiao]

This bug still exists on version 1.4.0.

Moreover, the docstring is confusing as "bool" and "boolean" are two different types in pandas.

bool_only : bool, default None
Include only boolean columns. If None, will attempt to use everything,
then use only boolean data. Not implemented for Series.

For the examples above, we would expect the same results whether bool_only is True, False, or None, since all the columns have the type "boolean". But in reality, when bool_only is True, all columns are dropped. So the docstring should instead be

bool_only : bool, default None
Include only columns with type "bool". If None, will attempt to use everything,
then use only data with type "bool". Not implemented for Series.

df2.all(axis=1, skipna=True, bool_only=True)

0    True
1    True
2    True
dtype: bool

Expected only when bool_only means excluding "boolean".

df2.all(axis=1, skipna=True, bool_only=False)

0    False
1     True
2    False
dtype: bool

Expected.

df2.all(axis=1, skipna=False, bool_only=False)

raises TypeError: boolean value of NA is ambiguous
But this operation is not ambiguous according to the Kleene logic.

df2.all(axis=1, skipna=False, bool_only=None)

0    True
1    True
2    True
dtype: bool

Expect False, pd.NA, False as discussed above.