Reject oversize bump output rasters before allocation (#1231)#1234
Merged
Conversation
_finish_bump allocated np.zeros((height, width)) with no memory check. The existing bump-count guard only protected the locs/heights arrays, so bump(width=1_000_000, height=1_000_000) passed the guard (count capped at 10M ~ 160 MB) and then tried to allocate an 8 TB float64 raster. Extend the memory budget check to include the output raster bytes (8 * w * h for float64) when the backend will materialize the full array. Dask paths build the output lazily per chunk, so they are excluded from the raster guard and continue to handle large dimensions.
The 20000x20000 default count test allocates a 3.2 GB raster, which exceeds the new memory guard on 2 GB CI runners. The test only verifies count-cap semantics, so bypass the guard via monkeypatch instead of changing the guard or the test dimensions.
Contributor
Author
|
@copilot resolve the merge conflicts in this pull request |
Co-authored-by: brendancol <433221+brendancol@users.noreply.github.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_finish_bumpallocatednp.zeros((height, width))with no memory check. The count guard added in bump: uint16 overflow, unbounded default count, and closure serialization OOM #1206 only covered thelocs/heightsarrays, sobump(width=1_000_000, height=1_000_000)slipped through (count capped at 10M ~ 160 MB) and then tried to allocate ~8 TB for the output raster.raster_bytes = w * h * 8when the backend will materialize the full array. Dask paths build output lazily per-chunk, so they are excluded from the raster guard and keep handling large dimensions.Fixes #1231.
Test plan
pytest xrspatial/tests/test_bump.py: all 18 tests pass locally (15 existing + 3 new)bump(width=1_000_000, height=1_000_000)now raisesMemoryErrorwith a message naming the output raster sizebump(agg=dask_array_100k_x_100k, count=10)still works lazily without tripping the guard