ENH: Use Kahan summation to calculate groupby.sum() #38903
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
We dispatch to numpy in case of 32bit systems, which returns the wrong result |
arw2019
reviewed
Jan 2, 2021
doc/source/whatsnew/v1.3.0.rst
Outdated
| - :func:`pandas.read_sql_query` now accepts a ``dtype`` argument to cast the columnar data from the SQL database based on user input (:issue:`10285`) | ||
| - Improved integer type mapping from pandas to SQLAlchemy when using :meth:`DataFrame.to_sql` (:issue:`35076`) | ||
| - :func:`to_numeric` now supports downcasting of nullable ``ExtensionDtype`` objects (:issue:`33013`) | ||
| - Improve numerical stability for :meth:`DataFrameGroupBy.sum()` and :meth:`SeriesGroupBy.sum()` through using Kahan summation (:issue:`38778`) |
Contributor
There was a problem hiding this comment.
no brackets after ...GroupBy.sum?
jreback
requested changes
Jan 3, 2021
Contributor
jreback
left a comment
jreback
left a comment
There was a problem hiding this comment.
comments, ping on green.
note that we have been seeing some recent work on combing the aggregation functions for 1d and groupby, this would be a great case here. (slightly complicated because for 1d we actually dispatch to bottleneck if installed), otherwise we use the np.nan* functions so have to see if this is worth it).
doc/source/whatsnew/v1.3.0.rst
Outdated
| - :func:`pandas.read_sql_query` now accepts a ``dtype`` argument to cast the columnar data from the SQL database based on user input (:issue:`10285`) | ||
| - Improved integer type mapping from pandas to SQLAlchemy when using :meth:`DataFrame.to_sql` (:issue:`35076`) | ||
| - :func:`to_numeric` now supports downcasting of nullable ``ExtensionDtype`` objects (:issue:`33013`) | ||
| - Improve numerical stability for :meth:`DataFrameGroupBy.sum` and :meth:`SeriesGroupBy.sum` through using Kahan summation (:issue:`38778`) |
jreback
approved these changes
Jan 3, 2021
Member
Author
|
@jreback green |
Contributor
|
thanks @phofl |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diffThis simplifies the op example.
I think
is dispatched to numpy? The result is wrong too.
100.0Mean doesn't work either. I will look through the functions to determine which need Kahan summation too and open an issue to track these.