-
-
Notifications
You must be signed in to change notification settings - Fork 371
Open
Labels
V2Affects the v2 branchAffects the v2 branchbugPotential issues with the zarr-python libraryPotential issues with the zarr-python library
Description
Zarr version
2.16.1
Numcodecs version
0.11.0
Python Version
3.10.12
Operating System
Linux
Installation
using conda
Description
The zarr.storage.FSStore chunk store currently ignores an important exception that the underlying fsspec.mapping.FSMap is able to provide when calling getitems. The FSMap detects when the reference file system is not reachable (because of network problems or missing/invalid credentials) and returns an exception. But because the FSStore.getitems method sets the on_error argument to "omit", the FSStore treats the values as missing. They're not missing, there was an error that the user needs to know about.
Steps to reproduce
import zarr
from fsspec.implementations.reference import ReferenceFileSystem
fo = {
"version": 1,
"refs": {
".zgroup": '{"zarr_format":2}',
"time/.zarray": '{"chunks":[512],"compressor":null,"dtype":"<f8","fill_value":null,"filters":null,"order":"C","shape":[1],"zarr_format":2}',
"time/0": [
"s3://gesdisc-cumulus-prod-protected/GLDAS/GLDAS_NOAH025_3H.2.1/2018/152/GLDAS_NOAH025_3H.A20180601.0000.021.nc4",
22998,
4096,
],
},
}
fs = ReferenceFileSystem(fo, remote_protocol="s3", remote_options={"key": "foo", "secret": "bar", "token": "baz"})
group = zarr.open_group(fs.get_mapper(""))
group["time"][:]
The returned value is bogus if the credentials are invalid, as above, but no error is raised. The correct value for "time" is array([9685260.]).
Additional output
No response
Metadata
Metadata
Assignees
Labels
V2Affects the v2 branchAffects the v2 branchbugPotential issues with the zarr-python libraryPotential issues with the zarr-python library