Fix three accuracy bugs in zonal stats dask backend (#1090)#1091
Merged
Conversation
1. Dask sum/count/min/max now return NaN (not 0) for zones with all-NaN values, matching the numpy backend. Uses _nanreduce_preserve_allnan wrapper around np.nansum/nanmax/nanmin. 2. Dask std/var replaced the naive one-pass formula with the Chan-Golub-LeVeque parallel merge algorithm, which avoids catastrophic cancellation when the mean is large relative to the variance. 3. _calc_stats and crosstab helpers now skip the nodata_values != comparison when nodata_values is None, avoiding numpy FutureWarning.
- Block-level sum_squares now computes M2 (sum of squared deviations from block mean) instead of raw sum(x²), avoiding float64 precision loss for large values. - Updated test_stats_all_nan_zone and test_stats_nodata_wipes_zone to expect NaN from dask (no longer 0). - Added test_stats_variance_numerical_stability_1090: values near 1e8 with spread of 1, verifying dask matches numpy to 1e-6. - Added test_stats_nodata_none_no_warning_1090: confirms no FutureWarning when nodata_values=None.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1090. Three bugs in
zonal.stats()where the dask backend diverged from numpy:np.nansumconverts all-NaN slices to 0. Added_nanreduce_preserve_allnanwrapper that restores NaN for zones where every block had no valid data.(Σx² - (Σx)²/n) / nwith the Chan-Golub-LeVeque parallel merge algorithm. Block-level computation now produces M2 (sum of squared deviations from block mean) instead of raw sum-of-squares, so precision holds even when values are near 1e8._calc_statscomparedzone_values != Nonewhen nodata_values was None, triggering a numpy FutureWarning. Now skips the comparison entirely.Test plan
test_stats_all_nan_zoneupdated: dask now expects NaN (not 0), passes on all 4 backendstest_stats_nodata_wipes_zoneupdated: same fixtest_stats_variance_numerical_stability_1090: values near 1e8 with spread of 1, verifies dask matches numpy within 1e-6test_stats_nodata_none_no_warning_1090: confirms no FutureWarning with default nodata_values=Nonetest_zonal.pysuite: 119 passedtest_dataset_support.py: 18 passed