cannot upload zarr files

[gmazzamuto]

Hi all,

using the latest version I can't upload zarr data because an IsADirectoryError exception is raised. See below.

2023-06-30T17:19:50+0200 [INFO    ] dandi 3077715:140494520747840 dandi v0.55.1, hdmf v3.6.1, pynwb v2.3.3, h5py v3.9.0
2023-06-30T17:19:50+0200 [INFO    ] dandi 3077715:140494520747840 sys.argv = ['/opt/bin/dandi', 'upload', '--allow-any-path', '--validation', 'ignore', 'sub-I59/ses-SPIM/micr/sub-I59_ses-SPIM_sample-BrocaAreaS01_stain-Calretinin_SPIM.ome.zarr']
2023-06-30T17:19:50+0200 [INFO    ] dandi 3077715:140494520747840 os.getcwd() = /mnt/NIH/DANDI/000026
2023-06-30T17:19:50+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 Starting new HTTPS connection (1): rig.mit.edu:443
2023-06-30T17:19:51+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 https://rig.mit.edu:443 "GET /et/projects/dandi/dandi-cli HTTP/1.1" 200 579
2023-06-30T17:19:51+0200 [DEBUG   ] dandi 3077715:140494520747840 No newer (than 0.55.1) version of dandi/dandi-cli found available
...
2023-06-30T17:19:53+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 Starting new HTTPS connection (1): dandiarchive.org:443
2023-06-30T17:19:53+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 https://dandiarchive.org:443 "GET /server-info HTTP/1.1" 200 354
2023-06-30T17:19:53+0200 [DEBUG   ] dandi 3077715:140494520747840 GET https://api.dandiarchive.org/api/info/
2023-06-30T17:19:53+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 Starting new HTTPS connection (1): api.dandiarchive.org:443
2023-06-30T17:19:53+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 https://api.dandiarchive.org:443 "GET /api/info/ HTTP/1.1" 200 354
2023-06-30T17:19:53+0200 [DEBUG   ] dandi 3077715:140494520747840 Response: 200
...
2023-06-30T17:19:58+0200 [DEBUG   ] dandi 3077715:140494520747840 Using API key from keyring
2023-06-30T17:19:58+0200 [DEBUG   ] dandi 3077715:140494520747840 GET https://api.dandiarchive.org/api/auth/token
2023-06-30T17:19:58+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 https://api.dandiarchive.org:443 "GET /api/auth/token HTTP/1.1" 301 0
2023-06-30T17:19:58+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 https://api.dandiarchive.org:443 "GET /api/auth/token/ HTTP/1.1" 200 42
2023-06-30T17:19:58+0200 [DEBUG   ] dandi 3077715:140494520747840 Response: 200
2023-06-30T17:19:58+0200 [DEBUG   ] dandi 3077715:140494520747840 Found identifier DANDI:000026 in top level 'identifier'
2023-06-30T17:20:17+0200 [INFO    ] dandi 3077715:140494520747840 Found 2 files to consider
2023-06-30T17:20:19+0200 [WARNING ] bids-schema 3077715:140493548414720 No BIDS reference root provided.
2023-06-30T17:20:19+0200 [INFO    ] bids-schema 3077715:140493548414720 No schema path specified, defaulting to the bundled schema, `/opt/lib/python3.8/site-packages/bidsschematools/data/schema`.
...
2023-06-30T17:21:58+0200 [DEBUG   ] dandi 3077715:140493548414720 GET https://api.dandiarchive.org/api/dandisets/000026/versions/draft/assets/
2023-06-30T17:21:58+0200 [DEBUG   ] urllib3.connectionpool 3077715:140493548414720 Resetting dropped connection: api.dandiarchive.org
2023-06-30T17:21:59+0200 [DEBUG   ] urllib3.connectionpool 3077715:140493548414720 https://api.dandiarchive.org:443 "GET /api/dandisets/000026/versions/draft/assets/?path=sub-I59%2Fses-SPIM%2Fmicr%2Fsub-I59_ses-SPIM_sample-BrocaAreaS01_stain-Calretinin_SPIM.ome.zarr HTTP/1.1" 200 52
2023-06-30T17:21:59+0200 [DEBUG   ] dandi 3077715:140493548414720 Response: 200
2023-06-30T17:22:01+0200 [DEBUG   ] fscacher.cache 3077715:140493548414720 Calling memoized version of <function get_dandietag at 0x7fc69e831ca0> for /mnt/NIH/DANDI/000026/sub-I59/ses-SPIM/micr/sub-I59_ses-SPIM_sample-BrocaAreaS01_stain-Calretinin_SPIM.ome.zarr
2023-06-30T17:22:01+0200 [DEBUG   ] fscacher.cache 3077715:140493548414720 Running original <function get_dandietag at 0x7fc69e831ca0> on PosixPath('/mnt/NIH/DANDI/000026/sub-I59/ses-SPIM/micr/sub-I59_ses-SPIM_sample-BrocaAreaS01_stain-Calretinin_SPIM.ome.zarr')
2023-06-30T17:22:01+0200 [ERROR   ] dandi 3077715:140493548414720 Error uploading /mnt/NIH/DANDI/000026/sub-I59/ses-SPIM/micr/sub-I59_ses-SPIM_sample-BrocaAreaS01_stain-Calretinin_SPIM.ome.zarr:
Traceback (most recent call last):
  File "/opt/lib/python3.8/site-packages/dandi/upload.py", line 263, in process_path
    for r in dfile.iter_upload(
  File "/opt/lib/python3.8/site-packages/dandi/files/bases.py", line 339, in iter_upload
    etagger = get_dandietag(self.filepath)
  File "/opt/lib/python3.8/site-packages/fscacher/cache.py", line 157, in fingerprinter
    ret = fingerprinted(*args, **kwargs_)
  File "/opt/lib/python3.8/site-packages/joblib/memory.py", line 655, in __call__
    return self._cached_call(args, kwargs)[0]
  File "/opt/lib/python3.8/site-packages/joblib/memory.py", line 598, in _cached_call
    out, metadata = self.call(*args, **kwargs)
  File "/opt/lib/python3.8/site-packages/joblib/memory.py", line 856, in call
    output = self.func(*args, **kwargs)
  File "/opt/lib/python3.8/site-packages/fscacher/cache.py", line 97, in fingerprinted
    return f(path, *args, **kwargs)
  File "/opt/lib/python3.8/site-packages/dandi/support/digests.py", line 100, in get_dandietag
    return DandiETag.from_file(filepath)
  File "/opt/lib/python3.8/site-packages/dandischema/digests/dandietag.py", line 147, in from_file
    with open(path, "rb") as f:
IsADirectoryError: [Errno 21] Is a directory: '/mnt/NIH/DANDI/000026/sub-I59/ses-SPIM/micr/sub-I59_ses-SPIM_sample-BrocaAreaS01_stain-Calretinin_SPIM.ome.zarr'
2023-06-30T17:22:01+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 Starting new HTTPS connection (1): rig.mit.edu:443
2023-06-30T17:22:01+0200 [DEBUG   ] urllib3.connectionpool 3077715:140494520747840 https://rig.mit.edu:443 "GET /et/projects/dandi/dandi-cli HTTP/1.1" 200 579
2023-06-30T17:22:01+0200 [DEBUG   ] dandi 3077715:140494520747840 Caught exception [Errno 21] Is a directory: '/mnt/NIH/DANDI/000026/sub-I59/ses-SPIM/micr/sub-I59_ses-SPIM_sample-BrocaAreaS01_stain-Calretinin_SPIM.ome.zarr'

[jwodder]

@gmazzamuto

• Is the dataset on an NFS file system?
• Is sub-I59_ses-SPIM_sample-BrocaAreaS01_stain-Calretinin_SPIM.ome.zarr a symlink or the target of a symlink?
• Is sub-I59_ses-SPIM_sample-BrocaAreaS01_stain-Calretinin_SPIM.ome.zarr empty?

[gmazzamuto]

Yes, the dataset is on an NFS file system. The directory is not a symlink and is not empty.

[gmazzamuto]

Hi all, any news on this one? I have a bunch of new data to upload. Otherwise I'll have to upload in OME-TIFF rather than zarr, to meet the project deadlines

[yarikoptic]

I thought we provided guidance, sorry -- slipped through, probably mixed up with #1321. Could you please

• reconfirm that issue persists and traceback is similarly implicating fscacher
• try upload while specifying DANDI_CACHE=clear, and if that doesn't help:
• try upload while specifying DANDI_CACHE=ignore
• report back on either 👍 or 👎

edit note: I have added clear in addition to original ignore only

[yarikoptic]

1 more question: which dandiset it is? it is BIDS one right?

[yarikoptic]

one more... if you get a chance before trying above, sorry for the flood, it would help

DANDI_DEVEL=1 dandi --pdb upload  --devel-debug ...the rest of options like you did

then whenever it errors out and gets into debugger, could you please show me what is that object is . Do command up in pdb a number of times until you get to the level of File "/opt/lib/python3.8/site-packages/dandi/files/bases.py", line 339, in iter_upload. There just do p self and show what that self is ;-)

PS edit: above is for myself to see if it is due to the fact that ZarrBIDSAsset(BIDSAsset, ZarrAsset): and BIDSAsset(LocalFileAsset): so I think that iter_upload gets from LocalFileAsset not from ZarrAsset here. We would need to overload to ensure that correct one is used, or may be just place ZarrAsset first in multiple inheritance?... need to check if we have any test which uploads zarr from a bids dataset.

[github-actions]

🚀 Issue was released in 0.56.2 🚀