DIfferent result between master and from latest pandas version

[charlesdong1991]

related #28201

I find out the result gotten from pandas master is different from using a specific version recently

Also, minimum reproducible example:

df = pd.DataFrame({"group": ['a', 'a', 'b', 'b'], "A": [0, 1, 2, 3], "B": [5, 6, 7, 8]})
df.columns = pd.MultiIndex.from_tuples([('x', 'group'), ('y', 'A'), ('y', 'B')])

if running the code below on pandas master:

df.groupby(('x', 'group')).agg(a_max=(('y', 'A'), "max"))

But if running the same code in pandas==0.25.3, then will get the error:

ValueError: operands could not be broadcast together with shapes (1,2) (3,) (1,2) 

I didn't find any PRs since 0.25.3 was released related to this issue, so curious why this was caused.

[jbrockmendel]

Can you post the full traceback

[charlesdong1991]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-04b1974e4a36> in <module>
      2 df.columns = pd.MultiIndex.from_tuples([('x', 'group'), ('y', 'A'), ('y', 'B')])
      3 
----> 4 df.groupby(('x', 'group')).agg(a_max=(('y', 'A'), "max"))

~/pyvenv/cw/lib/python3.7/site-packages/pandas/core/groupby/generic.py in aggregate(self, arg, *args, **kwargs)
   1453     @Appender(_shared_docs["aggregate"])
   1454     def aggregate(self, arg=None, *args, **kwargs):
-> 1455         return super().aggregate(arg, *args, **kwargs)
   1456 
   1457     agg = aggregate

~/pyvenv/cw/lib/python3.7/site-packages/pandas/core/groupby/generic.py in aggregate(self, func, *args, **kwargs)
    262 
    263         if relabeling:
--> 264             result = result[order]
    265             result.columns = columns
    266 

~/pyvenv/cw/lib/python3.7/site-packages/pandas/core/frame.py in __getitem__(self, key)
   2999             if is_iterator(key):
   3000                 key = list(key)
-> 3001             indexer = self.loc._convert_to_indexer(key, axis=1, raise_missing=True)
   3002 
   3003         # take() does not accept boolean indexers

~/pyvenv/cw/lib/python3.7/site-packages/pandas/core/indexing.py in _convert_to_indexer(self, obj, axis, is_setter, raise_missing)
   1283                 # When setting, missing keys are not allowed, even with .loc:
   1284                 kwargs = {"raise_missing": True if is_setter else raise_missing}
-> 1285                 return self._get_listlike_indexer(obj, axis, **kwargs)[1]
   1286         else:
   1287             try:

~/pyvenv/cw/lib/python3.7/site-packages/pandas/core/indexing.py in _get_listlike_indexer(self, key, axis, raise_missing)
   1084             if len(ax) or not len(key):
   1085                 key = self._convert_for_reindex(key, axis)
-> 1086             indexer = ax.get_indexer_for(key)
   1087             keyarr = ax.reindex(keyarr)[0]
   1088         else:

~/pyvenv/cw/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_indexer_for(self, target, **kwargs)
   4815         """
   4816         if self.is_unique:
-> 4817             return self.get_indexer(target, **kwargs)
   4818         indexer, _ = self.get_indexer_non_unique(target, **kwargs)
   4819         return indexer

~/pyvenv/cw/lib/python3.7/site-packages/pandas/core/indexes/multi.py in get_indexer(self, target, method, limit, tolerance)
   2447             )
   2448         else:
-> 2449             indexer = self._engine.get_indexer(target)
   2450 
   2451         return ensure_platform_int(indexer)

pandas/_libs/index.pyx in pandas._libs.index.BaseMultiIndexCodesEngine.get_indexer()

pandas/_libs/index.pyx in pandas._libs.index.BaseMultiIndexCodesEngine._extract_level_codes()

~/pyvenv/cw/lib/python3.7/site-packages/pandas/core/indexes/multi.py in _codes_to_ints(self, codes)
     79         # Shift the representation of each level by the pre-calculated number
     80         # of bits:
---> 81         codes <<= self.offsets
     82 
     83         # Now sum and OR are in fact interchangeable. This is a simple

ValueError: operands could not be broadcast together with shapes (1,2) (3,) (1,2)

@jbrockmendel If I rememember correctly after 0.25.2 was released, there was already no error if running this code on pandas master, but running on 0.25.2 could also lead to errors.

[jreback]

@charlesdong1991 there are quite a number of changes in 1.0 vs 0.25.x

so likely this is just fixed in master

if we don’t have a test for it would accept a PR to add one

[charlesdong1991]

@jreback thanks for your reply, will check and add tests for these cases later if there is no test yet!