Do not traverse .ome.zarr and .ngff folders while looking for BIDS datasets

[yarikoptic]

Fixes #1069

TODO

• check on real case if that helps enough

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.40%. Comparing base (045e9ff) to head (716453b).
Report is 1013 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1071      +/-   ##
==========================================
+ Coverage   88.38%   88.40%   +0.02%     
==========================================
  Files          72       72              
  Lines        9261     9272      +11     
==========================================
+ Hits         8185     8197      +12     
+ Misses       1076     1075       -1     

Flag	Coverage Δ
unittests	88.40% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yarikoptic]

ok -- with this I fail (incorrect token) on our target use case within 3 seconds, so we manage to traverse enough fast enough, so effectively addresses the case. but may be we would tune bids detection more - see #1073

[satra]

@yarikoptic - does the check still run when --allow-any-path is used? as in is there a necessity of any bids-specific traversal other than dandiset.yaml detection?

and secondarily, if we don't use that flag and do --validation skip then does it care for anything else other than the specific directory contents and the presence of a dataset_description.json. i.e. it should not care about files in the other subject's directory.

(and the one ngff file in MITU01 is being removed).

[yarikoptic]

@yarikoptic - does the check still run when --allow-any-path is used?

the patch is only for _bids_discover_and_validate( for discovery of dataset_description.json, no other operation should be affected AFAIK.

and secondarily, if we don't use that flag and do --validation skip then does it care for anything else other than the specific directory contents and the presence of a dataset_description.json. i.e. it should not care about files in the other subject's directory.

I don't think this patch should affect that behavior either, IMHO should not . If that behavior is wrong, we need a separate issue issue for that particular case - I am just trying to fix up #1069 here.

[yarikoptic]

filed #1078 for a failed test, filed #1077 for a separate aspect identified by @jwodder , got blessing from @jwodder. So Let's proceed with this but I will delay release -- may be more PRs would get merged in short time.