Skip to content

feat: iter_posts and iter_comments auto-paginating generators#23

Merged
jackparnell merged 1 commit intomainfrom
feature/pagination-iterators
Apr 9, 2026
Merged

feat: iter_posts and iter_comments auto-paginating generators#23
jackparnell merged 1 commit intomainfrom
feature/pagination-iterators

Conversation

@ColonistOne
Copy link
Copy Markdown
Collaborator

Summary

Adds generator methods that walk paginated endpoints and yield one item at a time, transparently fetching new pages as needed:

ColonyClient.iter_posts(...)    -> Iterator[dict]
ColonyClient.iter_comments(...) -> Iterator[dict]

AsyncColonyClient.iter_posts(...)    -> AsyncIterator[dict]
AsyncColonyClient.iter_comments(...) -> AsyncIterator[dict]

Why

Today users have to track offsets manually or buffer everything into a list via get_all_comments. Iterators give two wins:

  1. Memory — walk a 5000-comment thread without holding it all at once
  2. Early exitfor post in iter_posts(...): if cond: break stops fetching as soon as you've found what you need, instead of fetching a fixed limit=100 and discarding the rest

Endpoints covered

  • /postsoffset + limit pagination → iter_posts increments offset by page_size each round trip and stops on a partial page
  • /posts/{id}/commentspage + fixed-20 pagination → iter_comments increments page and stops on a partial page

/notifications, /search, and /colonies are not included in this PR — the server endpoints don't accept offset/page params, so iteration would just re-fetch the same first page. (We could add iterators once the server supports it; out of scope here.)

Refactor

get_all_comments() (sync and async) now delegates to iter_comments(), eliminating the duplicated pagination loop. Behaviour is unchanged — old callers still get a list[dict] back.

Examples

# Sync
for post in client.iter_posts(colony="general", sort="top", max_results=50):
    print(post["title"])

for comment in client.iter_comments(post_id):
    if comment["author"] == "alice":
        print(comment["body"])

# Async
async for post in client.iter_posts(colony="general", max_results=100):
    print(post["title"])

iter_posts accepts page_size= (default 20, max 100) to tune the per-request size. iter_comments is fixed at 20 per page (server-enforced). Both accept max_results= to stop early.

Test plan

  • 8 new sync tests in TestIterPosts + TestIterComments: single-page, multi-page with partial last page, max_results stop-early (mid-page and across page boundaries), filter propagation, custom page_size, empty response, malformed shape
  • 7 new async tests in TestAsyncIterPosts + TestAsyncIterComments mirroring the same scenarios
  • Verified get_all_comments still works (sync + async) after the refactor
  • Coverage stays at 100% (514/514 statements)
  • ruff check / ruff format --check / mypy src/ all clean
  • CI green on Python 3.10 / 3.11 / 3.12 / 3.13

Adds generator methods that walk paginated endpoints and yield one item
at a time, transparently fetching new pages as needed:

  ColonyClient.iter_posts(colony, sort, post_type, tag, search,
                          page_size, max_results) -> Iterator[dict]
  ColonyClient.iter_comments(post_id, max_results) -> Iterator[dict]

  AsyncColonyClient.iter_posts(...) -> AsyncIterator[dict]
  AsyncColonyClient.iter_comments(...) -> AsyncIterator[dict]

Why: today users have to track offsets manually or buffer everything
into a list via get_all_comments. Iterators give two wins:

  1. Memory: walk a 5000-comment thread without holding it all at once
  2. Early-exit: `for post in iter_posts(...): if cond: break` stops
     fetching as soon as you've found what you need, instead of fetching
     a fixed `limit=100` and discarding the rest

Both endpoints supported:
  - /posts uses offset+limit pagination — iter_posts increments offset
    by page_size each round trip and stops on a partial page
  - /posts/{id}/comments uses page+fixed-20 pagination — iter_comments
    increments page and stops on a partial page

Refactor: get_all_comments() (sync and async) now delegates to
iter_comments(), eliminating the duplicated pagination loop. Behaviour
is unchanged — old callers still get a list[dict].

Notifications and search are NOT included in this PR — the server
endpoints don't accept offset/page params, so iteration would just be
re-fetching the same first page.

Tests: 8 sync + 7 async iterator tests covering single-page, multi-page
with partial last page, max_results stop-early (mid-page and across
page boundaries), filter propagation, custom page_size, empty response,
malformed response shape, and verifying get_all_comments still works.
Coverage stays at 100% (514/514 statements).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 9, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@jackparnell jackparnell merged commit 8d3c06c into main Apr 9, 2026
7 checks passed
@jackparnell jackparnell deleted the feature/pagination-iterators branch April 9, 2026 08:49
ColonistOne added a commit that referenced this pull request Apr 9, 2026
Two changes that ship together so v1.5.0 can be the first release cut
via the new automation:

1. Release workflow at .github/workflows/release.yml — triggered on
   `v*` tag push. Stages:
     - test:           runs ruff, mypy, pytest before anything else
     - build:          builds wheel + sdist, refuses to proceed if
                       the tag version doesn't match pyproject.toml
     - publish:        uploads to PyPI via OIDC trusted publishing
                       (no API token stored anywhere — short-lived
                       token minted by PyPI from the GitHub Actions
                       OIDC identity at publish time)
     - github-release: extracts the matching CHANGELOG section and
                       creates a GitHub Release with the wheel + sdist
                       attached

2. Version bump 1.4.0 → 1.5.0 in pyproject.toml and __init__.py.

3. CHANGELOG: consolidated the 1.5.0 section into a clean, ordered
   summary covering everything that's landed since 1.4.0:
     - AsyncColonyClient (PR #18)
     - Typed error hierarchy (PR #19)
     - RetryConfig + 5xx default retry (PR #20)
     - py.typed + verify_webhook + Dependabot (PR #21)
     - Pagination iterators (PR #23)
     - Coverage + Codecov (PR #17)
     - This release automation

Coverage at 100% (514/514 statements). 215 tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ColonistOne added a commit that referenced this pull request Apr 9, 2026
Two changes that ship together so v1.5.0 can be the first release cut
via the new automation:

1. Release workflow at .github/workflows/release.yml — triggered on
   `v*` tag push. Stages:
     - test:           runs ruff, mypy, pytest before anything else
     - build:          builds wheel + sdist, refuses to proceed if
                       the tag version doesn't match pyproject.toml
     - publish:        uploads to PyPI via OIDC trusted publishing
                       (no API token stored anywhere — short-lived
                       token minted by PyPI from the GitHub Actions
                       OIDC identity at publish time)
     - github-release: extracts the matching CHANGELOG section and
                       creates a GitHub Release with the wheel + sdist
                       attached

2. Version bump 1.4.0 → 1.5.0 in pyproject.toml and __init__.py.

3. CHANGELOG: consolidated the 1.5.0 section into a clean, ordered
   summary covering everything that's landed since 1.4.0:
     - AsyncColonyClient (PR #18)
     - Typed error hierarchy (PR #19)
     - RetryConfig + 5xx default retry (PR #20)
     - py.typed + verify_webhook + Dependabot (PR #21)
     - Pagination iterators (PR #23)
     - Coverage + Codecov (PR #17)
     - This release automation

Coverage at 100% (514/514 statements). 215 tests passing.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants