Memory-safe rechunk, preview chunk budget, plot improvements#1075
Merged
Conversation
Add column/rasterize_kw params, fix accessor namespace to .xrs, clarify nodata semantics, specify float64 output dtype, add list-of-pairs zones support, note dask chunk alignment strategy.
rechunk_no_shuffle now accepts Datasets and re-opens unmodified zarr stores with larger chunks instead of layering a rechunk-merge graph. preview() derives a per-task memory budget from the active dask cluster and re-opens zarr sources when chunks exceed it. Accessor gains Dataset.xrs.plot() for subplot grids and DataArray plot auto-computes dask arrays, sets equal aspect, and avoids kwargs mutation. Removes fused_overlap/multi_overlap from the Dataset accessor (the underlying functions only accept DataArrays).
Keep our fixes: kwargs mutation fix in DataArray.xrs.plot(), removal of fused_overlap/multi_overlap from Dataset accessor.
The zarr tests called xr.open_zarr / to_zarr which fails on CI environments without zarr (e.g. macOS 3.14). The failure triggered fail-fast cancellation of all other matrix jobs.
Drop _is_unmodified_zarr, _reopen_preview_chunks, and _preview_chunk_budget. Dataset rechunk now uses ds.chunk() instead of re-opening the zarr store under the hood.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
rechunk_no_shuffleacceptsxr.Datasetand re-opens unmodified zarr stores with larger chunks instead of adding a rechunk-merge graph layer. For file-backed data the task graph is much smaller.preview()derives a per-task memory budget from the active dask cluster (worker_memory * 0.7 / nthreads) and re-opens zarr sources when chunks exceed it, avoiding OOM on multi-threaded workers.DataArray.xrs.plot()auto-computes dask arrays, sets equal aspect ratio, and creates a figure with reasonable defaults. NewDataset.xrs.plot()renders 2D variables as a subplot grid with GeoTIFF colormap support.fused_overlap/multi_overlapfrom the Dataset accessor (the underlying functions only accept DataArrays, so these would always raise TypeError).Test plan
Dataset.xrs.plot()with a multi-variable Dataset