Skip to content

surface_distance geodesic dd_grid allocates (8, H, W) float64 without memory guard #1128

@brendancol

Description

@brendancol

Describe the bug

_precompute_dd_grid at line 308 allocates np.zeros((n_neighbors, H, W), dtype=float64) where n_neighbors is 8 for 8-connectivity. This is 64 bytes per pixel. For a 30TB raster (~3.75 billion pixels) this needs 240TB.

Called from surface_distance() when use_geodesic=True (line 1228).

Expected behavior

Add a memory guard before the allocation. For dask inputs, the dd_grid could potentially be computed per-chunk, but the current numpy-only implementation needs the full grid.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingoomOut-of-memory risk with large datasets

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions