S3 anonymous Zarr data examples...

[DennisHeimbigner]

I am in the process of constructing th initial netcdf-c library handler
for the Zarr format. As part of this, I need to verify my assumptions
about the mapping of the storage to S3.
Are there any anonymously accessible zarr datasets that I can access
(read-only)?

[jhamman]

@DennisHeimbigner - this dataset is on GCS but may work for you: https://storage.googleapis.com/pangeo-data/ecco/eccov4r3/

[DennisHeimbigner]

This prefix is readable
https://storage.googleapis.com/pangeo-data
so it may work. Thanks.

[jakirkham]

There's also a fixture directory in this repo, which has some dummy data for testing/validating the format/spec is still met during testing. This may be useful for seeding your own S3 bucket. Though it is quite small.

[DennisHeimbigner]

Unfortunately, these do not appear to reside on S3 itself.

[jhamman]

Right. This data is in GCS. Perhaps @jacobtomlinson knows of a public s3 zarr out there?

[jacobtomlinson]

No, but I can make one if you like?

[DennisHeimbigner]

That would be helpful if you did. It does not have to be complex, I am just
trying to get the basic access correct.

[alimanfoo]

The S3 example in the zarr tutorial uses a very small toy dataset that is publicly accessible. Bucket is here: http://zarr-demo.s3-eu-west-2.amazonaws.com/

[joshmoore]

Would there be any interest in having a https://www.minio.io/ -based setup using docker within travis so that s3 tests could be run? This would carry a s3fs requirement at least at the testing scope.

Edit: Looks like gh-293 may either make this unnecessary or be a good template for adding this for a AWS clone.

[alimanfoo]

Would there be any interest in having a https://www.minio.io/ -based setup using docker within travis so that s3 tests could be run? This would carry a s3fs requirement at least at the testing scope.

Edit: Looks like gh-293 may either make this unnecessary or be a good template for adding this for a AWS clone.

Sorry for slow follow up here. I think this would be excellent. I had been concerned that the cloud storage class implementations that are not within the zarr code base were not getting put through the test suite, but this would solve that very nicely. I think #293 provides a template, but it would need a new PR to add test coverage for AWS S3 via s3fs.S3Map.

Also I noticed recently that GCS has support now for local emulation, so it should be possible to get something for GCS too via gcsfs.GCSMap. That could be done separately from the open PR to implement a GCS storage class via the official Python SDK (#252), which would be nice to finish but is a parallel piece of work.

[martindurant]

GCS has support now for local emulation

how? where? I'd love to see it. I think I saw this mentioned elsewhere.

To @joshmoore , you don't need minio, you can more easily use moto, which is what the s3fs tests use.

[alimanfoo]

Re emulation, sorry I think I got confused, I had seen this page about emulation for Google Cloud Datastore but of course that's something completely different from Google Cloud Storage.

[joshmoore]

you don't need minio, you can more easily use moto, which is what the s3fs tests use.

Thanks, @martindurant. I hadn't seen moto before. Happy to have the tests use whatever's appropriate in this repo, especially if mocking is preferred to integration tests. For me, the minio setup is also useful for more production testing. Would you also suggest using moto in server mode for that?

[martindurant]

I don't see why not. Moto lacks some rather specific features such as file versioning, but is pretty complete. minio also isn't exactly S3...