Assigning values breaks in different ways when duplicate column names

[user347]

Code Sample, a copy-pastable example if possible

Our primary data with two columns with identical name:

df = pd.DataFrame(np.arange(12).reshape(4, 3).T)
df.columns = list('AABC')
print(df)
"""
   A  A  B   C
0  0  3  6   9
1  1  4  7  10
2  2  5  8  11
"""

Issue 1a: Series.replace throws ValueError when assigning:

print(df['B'].replace(6, np.nan))  # will work as expected with int as well
"""
0    NaN
1    7.0
2    8.0
Name: B, dtype: float64
"""
# ValueError: Buffer has wrong number of dimensions (expected 1, got 0):
df['B'] = df['B'].replace(6, np.nan)  # inplace=True does not raise error, but no change
df['B'] = df['B'].replace(6, 5)

Issue 1b: Same ValueError as above thrown when assigning np.nan with loc:

# ValueError: Buffer has wrong number of dimensions (expected 1, got 0):
df.loc[df['B'] == 6, 'B'] = np.nan

# Assigning int with loc will however work: 
df.loc[df['B'] == 6, 'B'] = 5

Issue 2a: assigning np.nan with iloc on column with a duplicate will apply on both columns:

# Assigning np.nan with iloc on column with a duplicate will apply on both columns:
df.iloc[0, 0] = np.nan
print(df)
"""
     A    A  B   C
0  NaN  NaN  5   9
1  1.0  4.0  7  10
2  2.0  5.0  8  11
"""

Issue 2b: assigning int with iloc will work int v0.22.0 but not v0.23.4

df.iloc[0, 0] = 10
print(df)
"""
0.22.0:
    A  A  B   C
0  10  3  5   9
1   1  4  7  10
2   2  5  8  11

0.23.4:
      A     A  B   C
0  10.0  10.0  5   9
1   1.0   4.0  7  10
2   2.0   5.0  8  11
"""
# Assigning with iloc will not break if BOTH columns contain a nan:
x = pd.DataFrame({'a': np.array([np.nan, 1, 2])})
y = pd.DataFrame({'a': np.array([0, np.nan, 2])})

df = pd.concat([x, y], axis=1)

df.iloc[0, 0] = 10
print(df)
"""
      a    a 
0  10.0  0.0
1   1.0  NaN
2   2.0  2.0
"""

Problem description

The main topic for this issue is assigning different values to a DataFrame that contains duplicate column names. List of issues reported:

• Issue 1a: Series.replace throws ValueError when assigning (BUG: iloc.__setitem__ with duplicate columns #32477)
• Issue 1b: Same ValueError as Issue 1a thrown when assigning np.nan with loc (TST: GH24798 df.replace() with duplicate columns #34302)
• Issue 2a: assigning np.nan with iloc on column with a duplicate will apply on both columns (Replacing a column with iloc replaces another column with same name #22036, BUG: .iloc indexing with duplicates #15686 closed by BUG: iloc.__setitem__ with duplicate columns #32477)
• Issue 2b: assigning int with iloc will work in v0.22.0 but not v0.23.4 (Replacing a column with iloc replaces another column with same name #22036, BUG: .iloc indexing with duplicates #15686 closed by BUG: iloc.__setitem__ with duplicate columns #32477)

he issue with iloc and np.nan (above called Issue 2a) was reported and closed as fixed here: #13423 per 0.18.0 but I'm able to recreate that issue with v0.23.4.

Expected Output

Either the same output as we would expect if we had only unique names in our columns or a DuplicateColumnWarning/DuplicateColumnException when DataFrame contains duplicate columns.

Output of pd.show_versions()

Details

pandas: 0.23.4 pytest: None pip: 18.0 setuptools: 39.0.1 Cython: None numpy: 1.16.0 scipy: 1.1.0T pyarrow: None xarray: None IPython: 6.3.1 sphinx: None patsy: None dateutil: 2.7.2 pytz: 2018.3 blosc: None bottleneck: None tables: None numexpr: None feather: None matplotlib: 2.1.0 openpyxl: 2.5.1 xlrd: 1.1.0 xlwt: None xlsxwriter: None lxml: 4.2.1 bs4: 4.6.0 html5lib: 1.0.1 sqlalchemy: 1.2.5 pymysql: None psycopg2: None jinja2: 2.10 s3fs: None fastparquet: None pandas_gbq: None pandas_datareader: None

[WillAyd]

IIUC correctly you have quite a few separate problems here - could you make a checklist of them to aid the discussion? As is now it's a little challenging to dissect the entire sample provided

[user347]

Updated in an attempt to make it a little bit clearer.

[TomAugspurger]

@user347 can you search to see if any of these are duplicate issues? Your 1a sounds familiar.

[user347]

@user347 can you search to see if any of these are duplicate issues? Your 1a sounds familiar.

Maybe I'm blind but I searched before posting and again now, and can't find any open issues that seem the same as either of the issues reported here.

[simonjayhawkins]

Issue 1A (ValueError: Buffer has wrong number of dimensions (expected 1, got 0)) is fixed on master

>>> import numpy as np
>>>
>>> import pandas as pd
>>>
>>> pd.__version__
'1.1.0.dev0+1029.gbdf969cd6'
>>>
>>> df = pd.DataFrame(np.arange(12).reshape(4, 3).T)
>>> df.columns = list("AABC")
>>> print(df)
   A  A  B   C
0  0  3  6   9
1  1  4  7  10
2  2  5  8  11
>>>
>>> print(df["B"].replace(6, np.nan))
0    NaN
1    7.0
2    8.0
Name: B, dtype: float64
>>>
>>> df["B"] = df["B"].replace(6, np.nan)
>>> print(df)
   A  A    B   C
0  0  3  NaN   9
1  1  4  7.0  10
2  2  5  8.0  11
>>>
>>> df = pd.DataFrame(np.arange(12).reshape(4, 3).T)
>>> df.columns = list("AABC")
>>> print(df)
   A  A  B   C
0  0  3  6   9
1  1  4  7  10
2  2  5  8  11
>>>
>>> df["B"] = df["B"].replace(6, 5)
>>> df
   A  A  B   C
0  0  3  5   9
1  1  4  7  10
2  2  5  8  11
>>>

[simonjayhawkins]

Issue 1A (ValueError: Buffer has wrong number of dimensions (expected 1, got 0)) is fixed on master

fixed in #32477

33f67d9 is the first new commit
commit 33f67d9
Author: jbrockmendel jbrockmendel@gmail.com
Date: Tue Mar 10 19:35:21 2020 -0700

BUG: iloc.__setitem__ with duplicate columns (#32477)

This PR also fixed Issue 1b

>>> import numpy as np
>>>
>>> import pandas as pd
>>>
>>> pd.__version__
'1.1.0.dev0+756.g33f67d98a'
>>>
>>> df = pd.DataFrame(np.arange(12).reshape(4, 3).T)
>>> df.columns = list("AABC")
>>> print(df)
   A  A  B   C
0  0  3  6   9
1  1  4  7  10
2  2  5  8  11
>>>
>>> # ValueError: Buffer has wrong number of dimensions (expected 1, got 0):
>>> df.loc[df["B"] == 6, "B"] = np.nan
>>> print(df)
   A  A    B   C
0  0  3  NaN   9
1  1  4  7.0  10
2  2  5  8.0  11
>>>

[simonjayhawkins]

Issue 2a (and Issue 2b) is a duplicate of #22036 and #15686 and also fixed by #32477 (no additional tests needed)