Guard flow_length_dinf() against unbounded memory allocations (#1355)#1358
Merged
Conversation
flow_length_dinf in xrspatial/hydro/ allocates several full-grid working arrays in its eager numpy and cupy backends with no upfront budget check. A 50000x50000 raster asks for about 72 GB of host RAM before anything errors out. Add _check_memory and _check_gpu_memory helpers wired into the numpy and cupy dispatch paths in flow_length_dinf(), mirroring the d8 fix in #1332 and the mfd fix in #1353. Dask path is left alone since per-tile allocations are bounded by chunk size. CPU budget: 29 B/px (in_degree int32 + valid int8 + flow_len float64 + order_r int64 + order_c int64). GPU budget: 32 B/px conservative (input float64 + output float64 + headroom). Tests: oversize numpy raises, normal raster passes, dask backend bypasses the guard, error message includes grid dimensions and the 'dask' hint, cupy gated on cuda_and_cupy_available.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1355.
flow_length_dinfallocates several full-grid working arrays in its eager numpy and cupy backends with no upfront budget check. A 50000x50000 raster asks for ~72 GB of host RAM before anything errors out.Adds
_check_memoryand_check_gpu_memoryhelpers, wired into the numpy and cupy dispatch inflow_length_dinf(). Dask paths are unchanged since per-tile allocations are bounded by chunk size. Same shape as #1332 (d8) and #1353 (mfd).Budget
CPU: 29 B/px
GPU: 32 B/px (input float64 + output float64 + headroom).
Tests
Five new tests in
TestMemoryGuard:test_numpy_huge_raster_raises-- numpy backend raises MemoryError when_available_memory_bytesis patched to 1test_numpy_normal_input_succeeds-- normal-size raster passes the guardtest_dask_path_skips_guard-- dask backend does not trigger the guard at dispatchtest_error_message_mentions_dimensions-- error mentions7x9anddasktest_cupy_huge_raster_raises-- cupy backend raises when GPU memory is patched low (gated on cuda_and_cupy_available)Test plan
pytest xrspatial/hydro/tests/test_flow_length_dinf.py-- 30 passedtest_stream_link_dask_temp_cleanupfailed