BUG: Fix replace for different dtype equal value #34878

dsaxton · 2020-06-20T03:03:15Z

Bug noticed when looking at a different issue #34871 (comment)

closes BUG: inconsistent replace #35376
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

jreback · 2020-06-20T12:32:16Z

pandas/core/internals/blocks.py

-                if inplace:
-                    return [self]
-                return [self.copy()]
+                return self.astype(object).replace(


we only want to do this if these are different types, not generally. please show asv's for this.

I think the if not self._can_hold_element(to_replace) check is making sure we have different types.

Not seeing any performance regressions from the change (at least in the replace benchmarks):

(pandas-dev) > asv continuous -f 1.1 upstream/master replace-diff-dtype -b ^replace · Creating environments · Discovering benchmarks ·· Uninstalling from conda-py3.7-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt. ·· Installing 1e014777 <replace-diff-dtype> into conda-py3.7-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.. · Running 12 total benchmarks (2 commits * 1 environments * 6 benchmarks) [ 0.00%] · For pandas commit 9114b4bc <master> (round 1/2): [ 0.00%] ·· Building for conda-py3.7-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt... [ 0.00%] ·· Benchmarking conda-py3.7-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt [ 4.17%] ··· Running (replace.Convert.time_replace--).... [ 20.83%] ··· Running (replace.ReplaceList.time_replace_list--).. [ 25.00%] · For pandas commit 1e014777 <replace-diff-dtype> (round 1/2): [ 25.00%] ·· Building for conda-py3.7-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt... [ 25.00%] ·· Benchmarking conda-py3.7-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt [ 29.17%] ··· Running (replace.Convert.time_replace--).... [ 45.83%] ··· Running (replace.ReplaceList.time_replace_list--).. [ 50.00%] · For pandas commit 1e014777 <replace-diff-dtype> (round 2/2): [ 50.00%] ·· Benchmarking conda-py3.7-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt [ 54.17%] ··· replace.Convert.time_replace ok [ 54.17%] ··· ============= ============ ============ -- replace_data ------------- ------------------------- constructor Timestamp Timedelta ============= ============ ============ DataFrame 98.6±3ms 95.2±0.5ms Series 74.0±0.3ms 74.6±1ms ============= ============ ============ [ 58.33%] ··· replace.FillNa.time_fillna ok [ 58.33%] ··· ========= ============= inplace --------- ------------- True 1.04±0.01ms False 3.01±0.3ms ========= ============= [ 62.50%] ··· replace.FillNa.time_replace ok [ 62.50%] ··· ========= ============= inplace --------- ------------- True 1.10±0.02ms False 4.02±0.07ms ========= ============= [ 66.67%] ··· replace.ReplaceDict.time_replace_series ok [ 66.67%] ··· ========= ============ inplace --------- ------------ True 4.27±0.02s False 4.24±0.04s ========= ============ [ 70.83%] ··· replace.ReplaceList.time_replace_list ok [ 70.83%] ··· ========= ============ inplace --------- ------------ True 51.6±0.6μs False 405±5ms ========= ============ [ 75.00%] ··· replace.ReplaceList.time_replace_list_one_match ok [ 75.00%] ··· ========= ========== inplace --------- ---------- True 89.2±3ms False 467±4ms ========= ========== [ 75.00%] · For pandas commit 9114b4bc <master> (round 2/2): [ 75.00%] ·· Building for conda-py3.7-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt... [ 75.00%] ·· Benchmarking conda-py3.7-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt [ 79.17%] ··· replace.Convert.time_replace ok [ 79.17%] ··· ============= =========== =========== -- replace_data ------------- ----------------------- constructor Timestamp Timedelta ============= =========== =========== DataFrame 102±4ms 103±3ms Series 80.0±4ms 83.1±4ms ============= =========== =========== [ 83.33%] ··· replace.FillNa.time_fillna ok [ 83.33%] ··· ========= ============= inplace --------- ------------- True 1.10±0.04ms False 3.04±0.1ms ========= ============= [ 87.50%] ··· replace.FillNa.time_replace ok [ 87.50%] ··· ========= ============= inplace --------- ------------- True 1.20±0.04ms False 4.26±0.2ms ========= ============= [ 91.67%] ··· replace.ReplaceDict.time_replace_series ok [ 91.67%] ··· ========= ============ inplace --------- ------------ True 4.26±0.01s False 4.24±0.01s ========= ============ [ 95.83%] ··· replace.ReplaceList.time_replace_list ok [ 95.83%] ··· ========= ========== inplace --------- ---------- True 49.6±1μs False 412±6ms ========= ========== [100.00%] ··· replace.ReplaceList.time_replace_list_one_match ok [100.00%] ··· ========= ========== inplace --------- ---------- True 89.3±2ms False 464±30ms ========= ========== BENCHMARKS NOT SIGNIFICANTLY CHANGED.

jreback · 2020-07-17T10:59:58Z

can u merge master and will look

simonjayhawkins · 2020-08-01T13:49:43Z

@dsaxton can you move release note

simonjayhawkins · 2020-09-15T08:38:34Z

marking as stale 😉

jreback · 2020-09-19T02:05:46Z

doc/source/whatsnew/v1.1.1.rst


+**Reshaping**
+
+- Bug where :meth:`DataFrame.replace` and :meth:`Series.replace` would fail when replacement value was of a different dtype but equal to values in the original object (:issue:`34871`).


can you move to 1.2

this looks similar to regression reported in #35376 with open PR (with different fix) #36444

Yeah I think it's the same issue. If another fix avoids astyping to object I would think it's preferred.

dsaxton · 2020-09-19T14:04:20Z

Closing since the other PR looks like a better fix

dsaxton added 3 commits June 19, 2020 21:39

Tests

47c83f9

Replace object block

f45017f

Note

268056c

gfyoung added Bug DataFrame DataFrame data structure Series Series data structure labels Jun 20, 2020

jreback requested changes Jun 20, 2020

View reviewed changes

dsaxton added 4 commits June 20, 2020 14:28

Merge remote-tracking branch 'upstream/master' into replace-diff-dtype

1e01477

Types

60ef5fc

Merge remote-tracking branch 'upstream/master' into replace-diff-dtype

d7f1fa8

Merge remote-tracking branch 'upstream/master' into replace-diff-dtype

bd46f2d

Merge remote-tracking branch 'upstream/master' into replace-diff-dtype

c6eb99b

dsaxton added 3 commits August 1, 2020 16:59

Remove note

d87954e

Merge remote-tracking branch 'upstream/master' into replace-diff-dtype

93d7186

Add note back

3430201

simonjayhawkins mentioned this pull request Sep 15, 2020

CI: Add stale PR action #36336

Merged

simonjayhawkins added the Stale label Sep 15, 2020

dsaxton added 2 commits September 15, 2020 08:34

Merge remote-tracking branch 'upstream/master' into replace-diff-dtype

a428f64

Merge remote-tracking branch 'upstream/master' into replace-diff-dtype

b747b6d

jreback requested changes Sep 19, 2020

View reviewed changes

jreback added this to the 1.2 milestone Sep 19, 2020

jreback removed the Stale label Sep 19, 2020

dsaxton added 2 commits September 18, 2020 21:40

Move

464c81d

Merge remote-tracking branch 'upstream/master' into replace-diff-dtype

5bb9b71

dsaxton closed this Sep 19, 2020

dsaxton deleted the replace-diff-dtype branch September 19, 2020 14:04

QuentinN42 mentioned this pull request Sep 21, 2020

BUG: inconsistent replace #36444

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

BUG: Fix replace for different dtype equal value #34878

BUG: Fix replace for different dtype equal value #34878

Uh oh!

dsaxton commented Jun 20, 2020 •

edited

Loading

Uh oh!

jreback Jun 20, 2020

Uh oh!

dsaxton Jun 21, 2020

Uh oh!

jreback commented Jul 17, 2020

Uh oh!

simonjayhawkins commented Aug 1, 2020

Uh oh!

simonjayhawkins commented Sep 15, 2020

Uh oh!

jreback Sep 19, 2020

Uh oh!

simonjayhawkins Sep 19, 2020

Uh oh!

dsaxton Sep 19, 2020

Uh oh!

dsaxton commented Sep 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		Reshaping

		- Bug where :meth:`DataFrame.replace` and :meth:`Series.replace` would fail when replacement value was of a different dtype but equal to values in the original object (:issue:`34871`).

Uh oh!

BUG: Fix replace for different dtype equal value #34878

BUG: Fix replace for different dtype equal value #34878

Uh oh!

Conversation

dsaxton commented Jun 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jreback Jun 20, 2020

Choose a reason for hiding this comment

Uh oh!

dsaxton Jun 21, 2020

Choose a reason for hiding this comment

Uh oh!

jreback commented Jul 17, 2020

Uh oh!

simonjayhawkins commented Aug 1, 2020

Uh oh!

simonjayhawkins commented Sep 15, 2020

Uh oh!

jreback Sep 19, 2020

Choose a reason for hiding this comment

Uh oh!

simonjayhawkins Sep 19, 2020

Choose a reason for hiding this comment

Uh oh!

dsaxton Sep 19, 2020

Choose a reason for hiding this comment

Uh oh!

dsaxton commented Sep 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dsaxton commented Jun 20, 2020 •

edited

Loading