feat: iter_posts and iter_comments auto-paginating generators#23
Merged
jackparnell merged 1 commit intomainfrom Apr 9, 2026
Merged
feat: iter_posts and iter_comments auto-paginating generators#23jackparnell merged 1 commit intomainfrom
jackparnell merged 1 commit intomainfrom
Conversation
Adds generator methods that walk paginated endpoints and yield one item
at a time, transparently fetching new pages as needed:
ColonyClient.iter_posts(colony, sort, post_type, tag, search,
page_size, max_results) -> Iterator[dict]
ColonyClient.iter_comments(post_id, max_results) -> Iterator[dict]
AsyncColonyClient.iter_posts(...) -> AsyncIterator[dict]
AsyncColonyClient.iter_comments(...) -> AsyncIterator[dict]
Why: today users have to track offsets manually or buffer everything
into a list via get_all_comments. Iterators give two wins:
1. Memory: walk a 5000-comment thread without holding it all at once
2. Early-exit: `for post in iter_posts(...): if cond: break` stops
fetching as soon as you've found what you need, instead of fetching
a fixed `limit=100` and discarding the rest
Both endpoints supported:
- /posts uses offset+limit pagination — iter_posts increments offset
by page_size each round trip and stops on a partial page
- /posts/{id}/comments uses page+fixed-20 pagination — iter_comments
increments page and stops on a partial page
Refactor: get_all_comments() (sync and async) now delegates to
iter_comments(), eliminating the duplicated pagination loop. Behaviour
is unchanged — old callers still get a list[dict].
Notifications and search are NOT included in this PR — the server
endpoints don't accept offset/page params, so iteration would just be
re-fetching the same first page.
Tests: 8 sync + 7 async iterator tests covering single-page, multi-page
with partial last page, max_results stop-early (mid-page and across
page boundaries), filter propagation, custom page_size, empty response,
malformed response shape, and verifying get_all_comments still works.
Coverage stays at 100% (514/514 statements).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
ColonistOne
added a commit
that referenced
this pull request
Apr 9, 2026
Two changes that ship together so v1.5.0 can be the first release cut
via the new automation:
1. Release workflow at .github/workflows/release.yml — triggered on
`v*` tag push. Stages:
- test: runs ruff, mypy, pytest before anything else
- build: builds wheel + sdist, refuses to proceed if
the tag version doesn't match pyproject.toml
- publish: uploads to PyPI via OIDC trusted publishing
(no API token stored anywhere — short-lived
token minted by PyPI from the GitHub Actions
OIDC identity at publish time)
- github-release: extracts the matching CHANGELOG section and
creates a GitHub Release with the wheel + sdist
attached
2. Version bump 1.4.0 → 1.5.0 in pyproject.toml and __init__.py.
3. CHANGELOG: consolidated the 1.5.0 section into a clean, ordered
summary covering everything that's landed since 1.4.0:
- AsyncColonyClient (PR #18)
- Typed error hierarchy (PR #19)
- RetryConfig + 5xx default retry (PR #20)
- py.typed + verify_webhook + Dependabot (PR #21)
- Pagination iterators (PR #23)
- Coverage + Codecov (PR #17)
- This release automation
Coverage at 100% (514/514 statements). 215 tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6 tasks
ColonistOne
added a commit
that referenced
this pull request
Apr 9, 2026
Two changes that ship together so v1.5.0 can be the first release cut
via the new automation:
1. Release workflow at .github/workflows/release.yml — triggered on
`v*` tag push. Stages:
- test: runs ruff, mypy, pytest before anything else
- build: builds wheel + sdist, refuses to proceed if
the tag version doesn't match pyproject.toml
- publish: uploads to PyPI via OIDC trusted publishing
(no API token stored anywhere — short-lived
token minted by PyPI from the GitHub Actions
OIDC identity at publish time)
- github-release: extracts the matching CHANGELOG section and
creates a GitHub Release with the wheel + sdist
attached
2. Version bump 1.4.0 → 1.5.0 in pyproject.toml and __init__.py.
3. CHANGELOG: consolidated the 1.5.0 section into a clean, ordered
summary covering everything that's landed since 1.4.0:
- AsyncColonyClient (PR #18)
- Typed error hierarchy (PR #19)
- RetryConfig + 5xx default retry (PR #20)
- py.typed + verify_webhook + Dependabot (PR #21)
- Pagination iterators (PR #23)
- Coverage + Codecov (PR #17)
- This release automation
Coverage at 100% (514/514 statements). 215 tests passing.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds generator methods that walk paginated endpoints and yield one item at a time, transparently fetching new pages as needed:
Why
Today users have to track offsets manually or buffer everything into a list via
get_all_comments. Iterators give two wins:for post in iter_posts(...): if cond: breakstops fetching as soon as you've found what you need, instead of fetching a fixedlimit=100and discarding the restEndpoints covered
/posts—offset + limitpagination →iter_postsincrementsoffsetbypage_sizeeach round trip and stops on a partial page/posts/{id}/comments—page + fixed-20pagination →iter_commentsincrementspageand stops on a partial page/notifications,/search, and/coloniesare not included in this PR — the server endpoints don't acceptoffset/pageparams, so iteration would just re-fetch the same first page. (We could add iterators once the server supports it; out of scope here.)Refactor
get_all_comments()(sync and async) now delegates toiter_comments(), eliminating the duplicated pagination loop. Behaviour is unchanged — old callers still get alist[dict]back.Examples
iter_postsacceptspage_size=(default 20, max 100) to tune the per-request size.iter_commentsis fixed at 20 per page (server-enforced). Both acceptmax_results=to stop early.Test plan
TestIterPosts+TestIterComments: single-page, multi-page with partial last page,max_resultsstop-early (mid-page and across page boundaries), filter propagation, custompage_size, empty response, malformed shapeTestAsyncIterPosts+TestAsyncIterCommentsmirroring the same scenariosget_all_commentsstill works (sync + async) after the refactorruff check/ruff format --check/mypy src/all clean