From bc784c4d2ddeb869e8bef4e6c5a72cf5bad56abc Mon Sep 17 00:00:00 2001 From: Brendan Collins Date: Tue, 28 Apr 2026 13:32:08 -0700 Subject: [PATCH 1/4] Record security audit results for slope and polygonize slope: clean (no findings) polygonize: MEDIUM only (Cat 1 missing memory guard, Cat 6 missing _validate_raster) sky_view_factor will land separately via PR #1300 (HIGH, Cat 1, fixed). --- .claude/sweep-security-state.csv | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.claude/sweep-security-state.csv b/.claude/sweep-security-state.csv index 20468ae8..9b51069b 100644 --- a/.claude/sweep-security-state.csv +++ b/.claude/sweep-security-state.csv @@ -30,10 +30,12 @@ normalize,2026-04-27,,,,,"Clean. Both rescale and standardize handle the constan pathfinding,2026-04-22,,MEDIUM,1;6,,"No CRITICAL/HIGH findings. Cat 1 already well-guarded: _check_memory(h, w) runs before every numpy/cupy _a_star_search allocation and covers the ~65 bytes/pixel footprint (parent_ys/xs int64, g_cost f64, visited i8, three heap arrays h_keys/h_rows/h_cols sized h*w). Auto-radius falls back when full grid exceeds 50% RAM, HPA* kicks in for long paths. Dask path uses sparse dict/set and on-demand chunk cache so no full-grid numpy materialisation. No CUDA kernels (cupy backend transfers to CPU). No file-path I/O from user input (only /proc/meminfo read). MEDIUM (unfixed, Cat 1): multi_stop_search does not cap len(waypoints); _optimize_waypoint_order builds an O(N^2) dist matrix and runs N^2 A* calls, and _nearest_neighbor_2opt is O(N^3), so pathological waypoint lists can cause extreme CPU consumption (DoS). MEDIUM (unfixed, Cat 6): a_star_search and multi_stop_search do not call _validate_raster(surface) -- only ndim==2 is checked, dtype is not; non-numeric dtypes would fail inside numba with confusing errors rather than a clean TypeError. int64 overflow in height*width (line 214 max_heap) is not reachable given the memory guard (~46340x46340 would already raise MemoryError long before 2**63)." perlin,2026-04-22,1232,HIGH,6,,"HIGH (fixed #1232): perlin() accepted integer-dtyped DataArrays via _validate_raster, but all four backends write float noise into the input buffer in place, then normalize by ptp. With integer storage the float values cast to 0, ptp=0, and the div-by-zero produced NaN/Inf that cast back to INT_MIN on every pixel. Fixed by adding an np.issubdtype(agg.dtype, np.floating) check in perlin() that raises ValueError. MEDIUM (unfixed follow-up): _perlin_numpy/_perlin_cupy/_perlin_dask_numpy/_perlin_dask_cupy all divide by ptp/(max-min) with no zero guard, so degenerate inputs like freq=(0,0) still emit NaN through the normalization step. GPU kernels have bounds guards, shared memory is fixed-size 512 int32 (not user-influenced), cuda.syncthreads() is present after the cooperative load. No file I/O." polygon_clip,2026-04-27,,,,,"Clean. Module is a raster mask-and-clip wrapper -- not a Sutherland-Hodgman polygon-vs-polygon clipper. It resolves a shapely geometry into polygon pairs, optionally crops to bbox, delegates mask construction to xrspatial.rasterize (which has its own memory guards), and applies via xarray.where. No manual line-segment intersection, no recursive clip amplification, no float division on user vertices. Cat 1: list(geometry) materializes the user iterable but the dominant memory cost is the rasterize-built mask which is already bounded by guarded raster size. Cat 2: no integer math. Cat 3: NaN bounds from degenerate geometry are caught by the does-not-overlap ValueError (line 93 _crop_to_bbox); shapely raises GEOSException on malformed input. Cat 4 N/A: no CUDA kernels. Cat 5: dynamic geopandas/shapely.ops imports are import-name strings, not user paths. Cat 6: _validate_raster called with default numeric=True; integer raster + np.nan nodata silently coerces but is a UX nit, not a security issue. Vertex amplification attack surface lives in shapely, not here." +polygonize,2026-04-28,,MEDIUM,1;6,,"Cat 1 MEDIUM: no MemoryError guard like other modules; _calculate_regions allocations (regions uint32, visited uint8, region_lookup that doubles up to 2*uint32_max=~32GB worst case) are all O(N) for input size N, and runtime check at line 328 raises RuntimeError when region count hits uint32 max. Working set is inherent to algorithm, no caller-controlled amplification. Cat 2: flat indices ij=i+j*nx done in numba int64, no overflow possible for realistic dimensions. Region IDs in uint32 with explicit max-region check (line 328). Cat 3: NaN handling correct: numpy backend masks NaN (line 555-560), cupy backend masks NaN (line 605-611), point_in_ring divisions guarded by sign-test, _perpendicular_distance has len_sq==0 guard (line 937). Cat 4: no custom CUDA kernels, uses cupyx.scipy.ndimage.label. Cat 5: no file I/O; _to_geojson returns dict. Cat 6 MEDIUM: polygonize() does not call _validate_raster(), only its own ndim check (line 1623). Missing numeric-dtype check, but generated_jit _is_close handles int/float separately and dtype confusion produces clean errors not silent wrong results. Not fixed (MEDIUM only)." proximity,2026-04-22,,,,,"Clean. Public APIs (proximity/allocation/direction) all call _validate_raster. GPU kernel _proximity_cuda_kernel has bounds guard at lines 359-360. Dask KDTree path has explicit memory guards (lines 897-903 result array, 1297-1312 unbounded distance fallback, 681-682 cache budget). Index math uses np.int64 for pan_near_x/pan_near_y, target_counts, y_offsets/x_offsets -- no int32 overflow risk. Target detection filters NaN via np.isfinite (lines 533, 657). _calc_direction guards x1==x2 & y1==y2 before arctan2. No file I/O. LOW (not flagged): line 1235 pad_y/pad_x omit abs() while line 437 uses it -- minor inconsistency, not exploitable." rasterize,2026-04-21,1223,HIGH,1;2,,HIGH: unbounded out/written allocation in _run_numpy/_run_cupy driven by user-supplied width/height/resolution (no cap). MEDIUM (unfixed): _build_row_csr_numba total=row_ptr[height] is int32 and can wrap for very tall rasters with many long edges. reproject,2026-04-17,,MEDIUM,1;3,, resample,2026-04-28,1295,HIGH,1,,"HIGH (fixed #1295): resample() did not bound output dimensions derived from user-supplied scale_factor / target_resolution. _output_shape returns max(1, round(in_h * scale_y)), max(1, round(in_w * scale_x)) and was passed straight through to the eager numpy / cupy backends, where _run_numpy and _run_cupy / the _AGG_FUNCS numba kernels and _nan_aware_interp_np allocated np.empty / cupy.empty / map_coordinates buffers of that size with no memory check. scale_factor=1e9 on a 4x4 raster requested ~190 EB; target_resolution=1e-9 on a meter-scale raster did the same. Fixed by adding _available_memory_bytes() / _available_gpu_memory_bytes() helpers and _check_resample_memory(out_h, out_w) / _check_resample_gpu_memory(out_h, out_w) guards (12 B/cell budget covering float64 working buffer + float32 output + map_coordinates temporary), wired into resample() before backend dispatch. Eager numpy and cupy paths run the guard; dask paths skip it because per-chunk allocations are bounded by chunk size. Mirrors the kde / line_density (#1287), focal (#1284), geodesic (#1283), cost_distance (#1262), and diffuse (#1267) patterns. No other findings: _validate_raster called at line 698, scale_y > 0 / scale_x > 0 enforced, AGGREGATE_METHODS rejects scale > 1.0, identity fast path bypasses dispatch entirely, all numba kernels guard count > 0 before division, no CUDA kernels (cupy paths use cupy ufuncs + cupyx.scipy.ndimage), no file I/O, all backends cast to float64 before computation and float32 on output." sieve,2026-04-28,1296,HIGH,1,,"HIGH (fixed #1296): sieve() on numpy and cupy backends had no memory guard. _label_connected allocates parent (int32, 4B/px), rank (int32, 4B/px, reused as root_to_id), region_map_flat (int32, 4B/px), plus a float64 result copy (8B/px) ~ 20 B/pixel of working memory before any check. The dask paths (_sieve_dask line 343 and _sieve_dask_cupy line 366) already raised MemoryError via _available_memory_bytes() at 28 B/pixel budget, but the public sieve() API at line 489 dispatched np.ndarray inputs straight into _sieve_numpy with no guard, and _sieve_cupy at line 308 transferred to host via data.get() then called _sieve_numpy, inheriting the gap. A 50000x50000 numpy raster requested ~50 GB silently. Fixed by extracting _check_memory(rows, cols) and _check_gpu_memory(rows, cols) helpers (mirrors cost_distance #1262 / mahalanobis #1288 / multispectral #1291 / kde #1287 pattern) at 28 B/pixel host budget plus 16 B/pixel GPU round-trip budget at 50% of available memory threshold. _check_memory wired into _sieve_numpy at the top before the float64 copy. _check_gpu_memory wired into _sieve_cupy before data.get(); it also calls _check_memory so the host budget still applies. Consolidated _available_memory_bytes definition (was duplicated). All 47 tests pass including 2 new memory-guard tests for the numpy backend (_sieve_numpy direct call + public sieve() API). No other findings: Cat 2 int32 indexing in _label_connected docstring acknowledges <2.1B pixel limit; the new memory guard rejects rasters that large before the int32 issue can trigger so this is a documentation/clarity follow-up rather than an exploitable bug. Cat 3 NaN handled via valid mask; Cat 4 no CUDA kernels; Cat 5 only /proc/meminfo read; Cat 6 _validate_raster called at line 478." +slope,2026-04-28,,,,,"Clean. slope() validates input via _validate_raster (line 383) and _validate_boundary (line 389). Cat 1: planar _cpu/_run_cupy allocate output matching input shape; geodesic paths build (3,H,W) float64 stacked array but are gated by _check_geodesic_memory(rows, cols) at line 410 (already fixed under geodesic audit, PR #1285). Cat 2: no int32 flat-index math; all loops 2D with range(). Cat 3: NaN propagates through arctan in planar kernels; geodesic delegates to _local_frame_project_and_fit which has explicit NaN guards and degenerate det check. Cat 4: _run_gpu (line 146) uses combined bounds+stencil guard 'i-di>=0 and i+di=0 and j+dj ~2B elements in the numpy path. MEDIUM (unfixed): hypsometric_integral() skips _validate_raster on zones/values; _regions_numpy has no memory guard (numpy-only path, bounded by caller-allocated input). MEDIUM (unfixed): _stats_numpy return_type='xarray.DataArray' allocates np.full((n_stats, values.size)) with no guard." From 60044c5366c23c1756efb932c1e30dec9efadcf6 Mon Sep 17 00:00:00 2001 From: Brendan Collins Date: Tue, 28 Apr 2026 17:42:35 -0700 Subject: [PATCH 2/4] Record terrain security audit (clean, MEDIUM-only Cat 1/3 noted) --- .claude/sweep-security-state.csv | 1 + 1 file changed, 1 insertion(+) diff --git a/.claude/sweep-security-state.csv b/.claude/sweep-security-state.csv index 9b51069b..75623701 100644 --- a/.claude/sweep-security-state.csv +++ b/.claude/sweep-security-state.csv @@ -37,5 +37,6 @@ reproject,2026-04-17,,MEDIUM,1;3,, resample,2026-04-28,1295,HIGH,1,,"HIGH (fixed #1295): resample() did not bound output dimensions derived from user-supplied scale_factor / target_resolution. _output_shape returns max(1, round(in_h * scale_y)), max(1, round(in_w * scale_x)) and was passed straight through to the eager numpy / cupy backends, where _run_numpy and _run_cupy / the _AGG_FUNCS numba kernels and _nan_aware_interp_np allocated np.empty / cupy.empty / map_coordinates buffers of that size with no memory check. scale_factor=1e9 on a 4x4 raster requested ~190 EB; target_resolution=1e-9 on a meter-scale raster did the same. Fixed by adding _available_memory_bytes() / _available_gpu_memory_bytes() helpers and _check_resample_memory(out_h, out_w) / _check_resample_gpu_memory(out_h, out_w) guards (12 B/cell budget covering float64 working buffer + float32 output + map_coordinates temporary), wired into resample() before backend dispatch. Eager numpy and cupy paths run the guard; dask paths skip it because per-chunk allocations are bounded by chunk size. Mirrors the kde / line_density (#1287), focal (#1284), geodesic (#1283), cost_distance (#1262), and diffuse (#1267) patterns. No other findings: _validate_raster called at line 698, scale_y > 0 / scale_x > 0 enforced, AGGREGATE_METHODS rejects scale > 1.0, identity fast path bypasses dispatch entirely, all numba kernels guard count > 0 before division, no CUDA kernels (cupy paths use cupy ufuncs + cupyx.scipy.ndimage), no file I/O, all backends cast to float64 before computation and float32 on output." sieve,2026-04-28,1296,HIGH,1,,"HIGH (fixed #1296): sieve() on numpy and cupy backends had no memory guard. _label_connected allocates parent (int32, 4B/px), rank (int32, 4B/px, reused as root_to_id), region_map_flat (int32, 4B/px), plus a float64 result copy (8B/px) ~ 20 B/pixel of working memory before any check. The dask paths (_sieve_dask line 343 and _sieve_dask_cupy line 366) already raised MemoryError via _available_memory_bytes() at 28 B/pixel budget, but the public sieve() API at line 489 dispatched np.ndarray inputs straight into _sieve_numpy with no guard, and _sieve_cupy at line 308 transferred to host via data.get() then called _sieve_numpy, inheriting the gap. A 50000x50000 numpy raster requested ~50 GB silently. Fixed by extracting _check_memory(rows, cols) and _check_gpu_memory(rows, cols) helpers (mirrors cost_distance #1262 / mahalanobis #1288 / multispectral #1291 / kde #1287 pattern) at 28 B/pixel host budget plus 16 B/pixel GPU round-trip budget at 50% of available memory threshold. _check_memory wired into _sieve_numpy at the top before the float64 copy. _check_gpu_memory wired into _sieve_cupy before data.get(); it also calls _check_memory so the host budget still applies. Consolidated _available_memory_bytes definition (was duplicated). All 47 tests pass including 2 new memory-guard tests for the numpy backend (_sieve_numpy direct call + public sieve() API). No other findings: Cat 2 int32 indexing in _label_connected docstring acknowledges <2.1B pixel limit; the new memory guard rejects rasters that large before the int32 issue can trigger so this is a documentation/clarity follow-up rather than an exploitable bug. Cat 3 NaN handled via valid mask; Cat 4 no CUDA kernels; Cat 5 only /proc/meminfo read; Cat 6 _validate_raster called at line 478." slope,2026-04-28,,,,,"Clean. slope() validates input via _validate_raster (line 383) and _validate_boundary (line 389). Cat 1: planar _cpu/_run_cupy allocate output matching input shape; geodesic paths build (3,H,W) float64 stacked array but are gated by _check_geodesic_memory(rows, cols) at line 410 (already fixed under geodesic audit, PR #1285). Cat 2: no int32 flat-index math; all loops 2D with range(). Cat 3: NaN propagates through arctan in planar kernels; geodesic delegates to _local_frame_project_and_fit which has explicit NaN guards and degenerate det check. Cat 4: _run_gpu (line 146) uses combined bounds+stencil guard 'i-di>=0 and i+di=0 and j+dj0, weight in ridged mode, w_noise in worley mode, plus a tmp buffer on GPU); for a 30000x30000 caller-allocated template (~7 GB float64) the marginal scratch footprint is ~14-28 GB without a memory check. The dask + worley path (line 197 _terrain_dask_numpy, line 466 _terrain_dask_cupy) calls dask.persist(raw_worley) which materialises the entire dask array in worker memory before computing min/max for the worley_norm_range pre-pass, defeating chunked processing when worley_blend>0. Cat 2 N/A: no flat-index math in this module (perlin/worley kernels live elsewhere). Cat 3 MEDIUM (unfixed): octaves is bounded >= 1 (line 594) but lacunarity and persistence are not validated; lacunarity**i with user-supplied octaves could overflow to inf for large octaves -- exploitable only via explicit large octaves which is also bounded by user. Division by w_ptp guarded at line 110 (numpy) and line 364 (gpu). Division by warp_norm at line 67 / 184 / 289 / 451 is safe because persistence**0=1 always contributes >= 1 to the sum and warp_octaves >=1 in practice (no validation but caller-controlled). Cat 4 N/A: no CUDA kernels owned by this module; perlin/worley GPU kernels are imported from sibling modules. Cat 5 N/A: no file I/O. Cat 6: _validate_raster runs on agg; noise_mode validated against {'fbm','ridged'}; numeric scalars (seed, zfactor, lacunarity, persistence, warp_strength, warp_octaves, worley_blend, worley_seed) accepted without _validate_scalar but mostly cosmetic since arithmetic naturally rejects bad types. MEDIUM not fixed per task instructions." viewshed,2026-04-22,1229,HIGH,1,,"HIGH (fixed #1229): _viewshed_cpu allocated ~500 bytes/pixel of working memory (event_list 3*H*W*7*8 bytes + status_values/status_struct/idle + visibility_grid + lexsort temporary) with no guard. A 20000x20000 raster tried to allocate ~200 GB. Fixed by adding peak-memory guard mirroring the _viewshed_dask pattern (_available_memory_bytes() check, raises MemoryError with max_distance= hint). No other HIGH findings: dask path already guarded, _validate_raster is called, distance-sweep uses dtype=float64, _calc_dist_n_grad guards zero distance." zonal,2026-04-22,1227,HIGH,1;2;6,,"HIGH (fixed #1227): _stats_cupy used `if nodata_values:` (truthy) so nodata_values=0 silently skipped the filter on the cupy backend, producing wrong stats vs every other backend. MEDIUM (unfixed): _strides uses np.int32 for stride indices — can wrap for arrays > ~2B elements in the numpy path. MEDIUM (unfixed): hypsometric_integral() skips _validate_raster on zones/values; _regions_numpy has no memory guard (numpy-only path, bounded by caller-allocated input). MEDIUM (unfixed): _stats_numpy return_type='xarray.DataArray' allocates np.full((n_stats, values.size)) with no guard." From 8941161f1f9b2f187781f1b6fb4c961ce66ca92d Mon Sep 17 00:00:00 2001 From: Brendan Collins Date: Tue, 28 Apr 2026 17:46:05 -0700 Subject: [PATCH 3/4] Guard surface_distance() against unbounded eager allocations (#1303) The numpy and cupy backends allocated the Dijkstra heap and four output arrays sized directly from input dimensions with no memory check. _surface_distance_numpy called _init_arrays (4x H*W float64 + int64 = 32 B/pixel) plus _dijkstra/_dijkstra_geodesic, which allocated h_keys/h_rows/h_cols of size H*W (24 B/pixel) and a visited mask (1 B/pixel), for ~80 B/pixel total. _surface_distance_cupy did the same on the GPU at ~72 B/pixel. A 50000x50000 raster requested ~200 GB before erroring out and got OOM-killed. Add _available_memory_bytes / _available_gpu_memory_bytes and _check_memory / _check_gpu_memory helpers (80 B/pixel host, 72 B/pixel GPU, 50% threshold), wired into _surface_distance_numpy and _surface_distance_cupy before any allocation. Dask paths inherit the guard via per-chunk numpy calls, so very-large rasters with bounded chunk sizes still work. Same memory-guard pattern as sky_view_factor (#1299), sieve (#1296), kde (#1287), resample (#1295), focal (#1284), geodesic (#1283), mahalanobis (#1288), true_color (#1291), diffuse (#1267), erode (#1275), emerging_hotspots (#1274), and dasymetric (#1261). TestMemoryGuard covers the huge-raster MemoryError path on all three public functions, normal input still succeeding, ValueError still winning over MemoryError on invalid args, the dask path being bounded per-chunk rather than per-full-shape, and the error message mentioning the grid size and the dask alternative. --- xrspatial/surface_distance.py | 94 ++++++++++++++++++ xrspatial/tests/test_surface_distance.py | 119 +++++++++++++++++++++++ 2 files changed, 213 insertions(+) diff --git a/xrspatial/surface_distance.py b/xrspatial/surface_distance.py index 572181cc..c3cfcda9 100644 --- a/xrspatial/surface_distance.py +++ b/xrspatial/surface_distance.py @@ -64,6 +64,98 @@ class cupy: # type: ignore[no-redef] ALLOCATION = 1 DIRECTION = 2 + +# --------------------------------------------------------------------------- +# Memory guards +# --------------------------------------------------------------------------- +# Peak working set per pixel for the eager numpy backend: +# dist (float64) 8 +# alloc (float64) 8 +# src_row (int64) 8 +# src_col (int64) 8 +# visited (int8) 1 +# h_keys (float64) 8 +# h_rows (int64) 8 +# h_cols (int64) 8 +# output (float32) 4 +# direction-mode temps ~16 +# Total ~80 bytes/pixel. A 50000x50000 raster needs ~200 GB. +_BYTES_PER_PIXEL = 80 + +# CuPy backend skips the explicit binary heap (parallel relaxation instead) +# but still allocates dist, alloc, srow, scol, src cast, elev cast, mask, +# row_idx/col_idx, output. ~72 bytes/pixel. +_GPU_BYTES_PER_PIXEL = 72 + + +def _available_memory_bytes(): + """Best-effort estimate of available host memory in bytes.""" + try: + with open("/proc/meminfo", "r") as f: + for line in f: + if line.startswith("MemAvailable:"): + return int(line.split()[1]) * 1024 + except (OSError, ValueError, IndexError): + pass + try: + import psutil + + return psutil.virtual_memory().available + except (ImportError, AttributeError): + pass + return 2 * 1024**3 + + +def _available_gpu_memory_bytes(): + """Best-effort estimate of free GPU memory in bytes. + + Returns 0 when CuPy / CUDA is unavailable or the query fails. + Callers treat that as "no GPU info, skip the guard". + """ + try: + import cupy as _cp + + free, _total = _cp.cuda.runtime.memGetInfo() + return int(free) + except Exception: + return 0 + + +def _check_memory(rows, cols): + """Raise MemoryError if the eager numpy pass would exceed 50% of RAM.""" + required = int(rows) * int(cols) * _BYTES_PER_PIXEL + available = _available_memory_bytes() + if required > 0.5 * available: + raise MemoryError( + f"surface_distance() on a {rows}x{cols} raster needs " + f"~{required / 1e9:.1f} GB of working memory but only " + f"~{available / 1e9:.1f} GB is available. Set a finite " + f"`max_distance=` to bound the search, or use a dask-backed " + f"DataArray for out-of-core processing." + ) + + +def _check_gpu_memory(rows, cols): + """Raise MemoryError when the cupy allocation would not fit. + + Checks host memory first because the input may already be staged on + the host before transfer. Skips silently when the GPU query fails. + """ + _check_memory(rows, cols) + available = _available_gpu_memory_bytes() + if available <= 0: + return + required = int(rows) * int(cols) * _GPU_BYTES_PER_PIXEL + if required > 0.5 * available: + raise MemoryError( + f"surface_distance() on a {rows}x{cols} cupy raster needs " + f"~{required / 1e9:.1f} GB of GPU memory but only " + f"~{available / 1e9:.1f} GB is free on the active device. " + f"Set a finite `max_distance=` to bound the search, or use a " + f"dask+cupy DataArray for out-of-core processing." + ) + + # --------------------------------------------------------------------------- # Numba kernels # --------------------------------------------------------------------------- @@ -360,6 +452,7 @@ def _surface_distance_numpy(source_data, elev_data, cellsize_x, cellsize_y, dd_grid, use_geodesic, mode): """NumPy backend: run Dijkstra and extract requested output.""" H, W = source_data.shape + _check_memory(H, W) dist, alloc, src_row, src_col = _init_arrays(H, W) _seed_sources(source_data, elev_data, target_values, @@ -442,6 +535,7 @@ def _surface_distance_cupy(source_data, elev_data, cellsize_x, cellsize_y, import cupy as cp H, W = source_data.shape + _check_gpu_memory(H, W) src = source_data.astype(cp.float64) elev = elev_data.astype(cp.float64) diff --git a/xrspatial/tests/test_surface_distance.py b/xrspatial/tests/test_surface_distance.py index b8f6a8b9..74ae62fb 100644 --- a/xrspatial/tests/test_surface_distance.py +++ b/xrspatial/tests/test_surface_distance.py @@ -673,3 +673,122 @@ def test_geodesic_basic(): # Cardinal neighbours should be ~111 km (1 degree at equator) for pos in [(0, 1), (2, 1), (1, 0), (1, 2)]: assert 100000 < sd[pos] < 130000 # roughly 100-130 km + + +# --------------------------------------------------------------------------- +# Memory guard +# --------------------------------------------------------------------------- + + +class TestMemoryGuard: + """Memory guard on the eager numpy / cupy backends.""" + + def test_numpy_huge_raster_raises(self): + """Numpy backend raises MemoryError when projected RAM exceeds budget.""" + from unittest.mock import patch + + source = np.zeros((4, 4), dtype=np.float64) + source[1, 1] = 1.0 + elev = np.zeros((4, 4), dtype=np.float64) + raster = _make_raster(source) + elevation = _make_raster(elev) + + # Mock available memory to 1 byte so even a 4x4 raster trips it. + with patch( + "xrspatial.surface_distance._available_memory_bytes", + return_value=1, + ): + with pytest.raises(MemoryError, match="working memory"): + surface_distance(raster, elevation) + with pytest.raises(MemoryError, match="working memory"): + surface_allocation(raster, elevation) + with pytest.raises(MemoryError, match="working memory"): + surface_direction(raster, elevation) + + def test_numpy_normal_input_succeeds(self): + """Normal-size raster passes the guard with real memory.""" + source = np.zeros((10, 10), dtype=np.float64) + source[5, 5] = 1.0 + elev = np.zeros((10, 10), dtype=np.float64) + raster = _make_raster(source) + elevation = _make_raster(elev) + # Should not raise -- 10x10 needs ~8 KB. + result = surface_distance(raster, elevation) + assert result.shape == (10, 10) + + def test_validation_error_takes_precedence(self): + """Invalid args raise ValueError before the memory guard runs.""" + from unittest.mock import patch + + source = np.zeros((4, 4), dtype=np.float64) + elev_wrong = np.zeros((5, 5), dtype=np.float64) + raster = _make_raster(source) + elevation = xr.DataArray( + elev_wrong, + dims=['y', 'x'], + coords={'y': np.arange(5, dtype=np.float64), + 'x': np.arange(5, dtype=np.float64)}, + attrs={'res': (1.0, 1.0)}, + ) + + with patch( + "xrspatial.surface_distance._available_memory_bytes", + return_value=1, + ): + # Mismatched shapes raise ValueError before any allocation. + with pytest.raises(ValueError, match="same shape"): + surface_distance(raster, elevation) + + # Invalid connectivity raises ValueError too. + elev_ok = _make_raster(np.zeros((4, 4), dtype=np.float64)) + with pytest.raises(ValueError, match="connectivity"): + surface_distance(raster, elev_ok, connectivity=5) + + def test_dask_path_bounded_per_chunk(self): + """Dask backend inherits the guard per-chunk (not on the full shape). + + A dask raster whose total footprint would trip the guard but whose + per-chunk footprint fits comfortably should compute successfully. + """ + if da is None: + pytest.skip("dask not installed") + + from unittest.mock import patch + + # 200x200 total (~6.4 MB at 80 B/pixel) chunked at 20x20 + # (~32 KB per chunk). Mock available memory to 1 MB: the full + # array would exceed 50% of that, but each 20x20 chunk needs + # only ~32 KB so per-chunk allocation passes. + source = np.zeros((200, 200), dtype=np.float64) + source[100, 100] = 1.0 + elev = np.zeros((200, 200), dtype=np.float64) + raster = _make_raster(source, backend='dask+numpy', chunks=(20, 20)) + elevation = _make_raster(elev, backend='dask+numpy', chunks=(20, 20)) + + with patch( + "xrspatial.surface_distance._available_memory_bytes", + return_value=1024 * 1024, # 1 MB + ): + # max_distance=5 keeps map_overlap depth small (< chunk size). + result = surface_distance(raster, elevation, max_distance=5.0) + # Force a small compute window to prove per-chunk passes. + _ = result.data[:4, :4].compute() + + def test_error_message_mentions_grid_size(self): + """The error message names the grid dimensions and the dask alternative.""" + from unittest.mock import patch + + source = np.zeros((7, 11), dtype=np.float64) + source[3, 5] = 1.0 + elev = np.zeros((7, 11), dtype=np.float64) + raster = _make_raster(source) + elevation = _make_raster(elev) + + with patch( + "xrspatial.surface_distance._available_memory_bytes", + return_value=1, + ): + with pytest.raises(MemoryError, match="7x11"): + surface_distance(raster, elevation) + with pytest.raises(MemoryError, match="dask"): + surface_distance(raster, elevation) From e5b1054ad15b0634ca589c11b197a7be8150b6e8 Mon Sep 17 00:00:00 2001 From: Brendan Collins Date: Tue, 28 Apr 2026 17:47:00 -0700 Subject: [PATCH 4/4] Record surface_distance security audit (PR #1305) Append row to .claude/sweep-security-state.csv noting the HIGH-severity unbounded allocation in _surface_distance_numpy / _surface_distance_cupy fixed in PR #1305. Other categories clean. --- .claude/sweep-security-state.csv | 1 + 1 file changed, 1 insertion(+) diff --git a/.claude/sweep-security-state.csv b/.claude/sweep-security-state.csv index 75623701..5dc725af 100644 --- a/.claude/sweep-security-state.csv +++ b/.claude/sweep-security-state.csv @@ -40,3 +40,4 @@ slope,2026-04-28,,,,,"Clean. slope() validates input via _validate_raster (line terrain,2026-04-28,,MEDIUM,1;3,,"No CRITICAL/HIGH findings. generate_terrain accepts a template DataArray (agg) so the OUTPUT raster is already allocated by the caller before dispatch -- this provides a natural memory bound that distinguishes terrain from purely-generative modules like bump.py. _validate_raster is called at line 580. Cat 1 MEDIUM (unfixed): _gen_terrain (numpy line 41) and _terrain_gpu (line 237) allocate 5-7 same-shape float32 scratch arrays per call (x/y meshgrid, warp_x/warp_y when warp_strength>0, weight in ridged mode, w_noise in worley mode, plus a tmp buffer on GPU); for a 30000x30000 caller-allocated template (~7 GB float64) the marginal scratch footprint is ~14-28 GB without a memory check. The dask + worley path (line 197 _terrain_dask_numpy, line 466 _terrain_dask_cupy) calls dask.persist(raw_worley) which materialises the entire dask array in worker memory before computing min/max for the worley_norm_range pre-pass, defeating chunked processing when worley_blend>0. Cat 2 N/A: no flat-index math in this module (perlin/worley kernels live elsewhere). Cat 3 MEDIUM (unfixed): octaves is bounded >= 1 (line 594) but lacunarity and persistence are not validated; lacunarity**i with user-supplied octaves could overflow to inf for large octaves -- exploitable only via explicit large octaves which is also bounded by user. Division by w_ptp guarded at line 110 (numpy) and line 364 (gpu). Division by warp_norm at line 67 / 184 / 289 / 451 is safe because persistence**0=1 always contributes >= 1 to the sum and warp_octaves >=1 in practice (no validation but caller-controlled). Cat 4 N/A: no CUDA kernels owned by this module; perlin/worley GPU kernels are imported from sibling modules. Cat 5 N/A: no file I/O. Cat 6: _validate_raster runs on agg; noise_mode validated against {'fbm','ridged'}; numeric scalars (seed, zfactor, lacunarity, persistence, warp_strength, warp_octaves, worley_blend, worley_seed) accepted without _validate_scalar but mostly cosmetic since arithmetic naturally rejects bad types. MEDIUM not fixed per task instructions." viewshed,2026-04-22,1229,HIGH,1,,"HIGH (fixed #1229): _viewshed_cpu allocated ~500 bytes/pixel of working memory (event_list 3*H*W*7*8 bytes + status_values/status_struct/idle + visibility_grid + lexsort temporary) with no guard. A 20000x20000 raster tried to allocate ~200 GB. Fixed by adding peak-memory guard mirroring the _viewshed_dask pattern (_available_memory_bytes() check, raises MemoryError with max_distance= hint). No other HIGH findings: dask path already guarded, _validate_raster is called, distance-sweep uses dtype=float64, _calc_dist_n_grad guards zero distance." zonal,2026-04-22,1227,HIGH,1;2;6,,"HIGH (fixed #1227): _stats_cupy used `if nodata_values:` (truthy) so nodata_values=0 silently skipped the filter on the cupy backend, producing wrong stats vs every other backend. MEDIUM (unfixed): _strides uses np.int32 for stride indices — can wrap for arrays > ~2B elements in the numpy path. MEDIUM (unfixed): hypsometric_integral() skips _validate_raster on zones/values; _regions_numpy has no memory guard (numpy-only path, bounded by caller-allocated input). MEDIUM (unfixed): _stats_numpy return_type='xarray.DataArray' allocates np.full((n_stats, values.size)) with no guard." +surface_distance,2026-04-28,1303,HIGH,1,,Fixed in PR #1305: added _check_memory and _check_gpu_memory guards to _surface_distance_numpy (line ~233) and _surface_distance_cupy (line ~448) before O(H*W) heap+output allocations. Dask paths inherit via per-chunk numpy call. Other categories clean.