feature(store): make list_* methods async generators #110
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ature/store-list-methods-async-generators
jhamman
commented
Apr 19, 2024
6 tasks
d-v-b
reviewed
Apr 19, 2024
d-v-b
reviewed
Apr 19, 2024
- simplify code comments - use `removeprefix` instead of `strip`
d-v-b
approved these changes
Apr 19, 2024
d-v-b
added a commit
that referenced
this pull request
May 21, 2024
* feat: functional .children method for groups * changes necessary for correctly generating list of children * add stand-alone test for group.children * give type hints a glow-up * test: use separate assert statements to avoid platform-dependent ordering issues * test: put fixtures in conftest, add MemoryStore fixture * docs: release notes * test: remove prematurely-added mock s3 fixture * chore: move v3 tests into v3 folder * chore: type hints * test: add schema for group method tests * chore: add type for zarr_formats * chore: remove localstore for now * test: add __init__.py to support imports from top-level conftest.py, and add some docstrings, and remove redundant def * fix: return valid JSON from GroupMetadata.to_bytes for v2 metadata * fix: don't use a type as a value * test: add getitem test * fix: replace reference to nonexistent method in with , which does exist * test: declare v3ness via directory structure, not test file name * add a docstring to _get, and pass auto_mkdir to _put * fix: add docstring to LocalStore.get_partial_values; adjust body of LocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get. * test: add tests for localstore init, set, get, get_partial * fix: Rename children to members; AsyncGroup.members yields tuples of (name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict. * fix: make Group.members return a tuple of str, Array | Group pairs * fix: revert changes to synchronization code; this is churn that we need to deal with * chore: move v3 tests into v3 folder * chore: type hints * test: add schema for group method tests * chore: add type for zarr_formats * chore: remove localstore for now * test: add __init__.py to support imports from top-level conftest.py, and add some docstrings, and remove redundant def * fix: return valid JSON from GroupMetadata.to_bytes for v2 metadata * fix: don't use a type as a value * test: add getitem test * fix: replace reference to nonexistent method in with , which does exist * test: declare v3ness via directory structure, not test file name * add a docstring to _get, and pass auto_mkdir to _put * fix: add docstring to LocalStore.get_partial_values; adjust body of LocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get. * test: add tests for localstore init, set, get, get_partial * fix: remove pre-emptive fetching from group.open * fix: use removeprefix (removes a substring) instead of strip (removes any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem * xfail v2 tests that are sure to fail; add delitem tests; partition xfailing tests into subtests * fix: handle byte_range[0] being None * fix: adjust test for localstore.get to check that get on nonexistent keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial * fix: add zarr_format parameter to array creation routines (which raises if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup. * test: add group init test * feature(store): make list_* methods async generators (#110) * feature(store): make list_* methods async generators * Update src/zarr/v3/store/memory.py * Apply suggestions from code review - simplify code comments - use `removeprefix` instead of `strip` --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * fix: define utility for converting asyncarray to array, and similar for group, largely to appease mypy * chore: remove checks that only existed because of implicit groups * chore: clean up docstring and modernize some type hints * chore: move imports to top-level * remove fixture files * remove commented imports * remove explicit asyncio marks; use __eq__ method of LocalStore for test * rename test_storage to test_store * modern type hints --------- Co-authored-by: Joe Hamman <jhamman1@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This goes on top of zarr-developers#1743
TODO: