Series.update fails with categorical types

[AlexRiina]

cats = pandas.api.types.CategoricalDtype(['a', 'b', 'c', 'd'])
# cats = None
s1 = pandas.Series(['a', 'b', 'c'], index=[1, 2, 3], dtype=cats)
s2 = pandas.Series(['b', 'a'], index=[1, 2], dtype=cats)

s1.update(s2)

With the dtype=None, the s1 series is updated to be ['b', 'a', 'c'] but with the dtype=cats, the update fails with error "ValueError: NumPy boolean array indexing assignment cannot assign 3 input values to the 2 output values where the mask is true"

Details

INSTALLED VERSIONS ------------------ commit: None python: 3.6.7.final.0 python-bits: 64 OS: Linux OS-release: 4.15.0-36-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: 3.3.2
pip: 18.0
setuptools: 40.5.0
Cython: 0.27.3
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: 1.6.6
patsy: 0.5.1
dateutil: 2.7.2
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.4
feather: None
matplotlib: 3.0.1
openpyxl: 2.4.10
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml.etree: 4.1.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[AlexRiina]

Another version of the example code is

s1 = pandas.Series(['a', 'b', 'c'], index=[1, 2, 3]).astype("category")
s2 = pandas.Series(['b', 'a'], index=[1, 2])

s1.update(s2)

which fails just the same.

[mroeschke]

Thanks for the report. Investigation and PRs welcome!

[giuliobeseghi]

This fails with dataframes too:

import pandas as pd

df = pd.DataFrame({"A": ["a", "b", "c"], "B": [1, 2, 3]})
dfs = pd.concat([df] * 3, ignore_index=True)
dfs["C"] = dfs.A

dfs.head()

A B C
0 a 1 a
1 b 2 b
2 c 3 c
3 a 1 a
4 b 2 b

Now imagine I want to have just column 'C' as categorical:

print(dfs.C.astype("category").dtypes) -> category

dfs.update(dfs.C.astype("category"))
print(dfs.dtypes)

[object, int64, object]

[simonjayhawkins]

fixed in master, xref #33984

>>> import pandas as pd
>>>
>>> pd.__version__
'1.1.0.dev0+1482.g18a7e973c'
>>>
>>> cats = pd.api.types.CategoricalDtype(['a', 'b', 'c', 'd'])
>>> # cats = None
>>> s1 = pd.Series(['a', 'b', 'c'], index=[1, 2, 3], dtype=cats)
>>> s2 = pd.Series(['b', 'a'], index=[1, 2], dtype=cats)
>>>
>>> s1.update(s2)
>>>
>>> s1
1    b
2    a
3    c
dtype: category
Categories (4, object): [a, b, c, d]

[erfannariman]

@mroeschke which actions are still open for this issue? I see that a test is added and it is fixed on master/