feat: paginate sem get pipelines to return all results within age window#247
Merged
loadez merged 10 commits intosemaphoreci:masterfrom Apr 24, 2026
Merged
Conversation
`sem get pipelines` previously returned only the first page of results (~30 pipelines) with no way to fetch more. This was a problem when projects had many pipelines queued — they were invisible to the CLI. Add an --all flag that fetches every page using the API's Link header pagination with page_size=200, matching the pattern used by `sem get projects`. Includes retry logic with exponential backoff for resilience against transient failures. Default behavior (no --all) is unchanged — returns a single page.
Always fetch all pages for the requested age range rather than requiring a separate --all flag. The --age flag already expresses user intent for how much history they want — silently truncating to one page was the actual bug.
cchristous
changed the title
Match the projects pagination pattern by retrying deserialization alongside the HTTP request, so truncated or corrupted responses are retried rather than failing immediately.
Cap at 500 pages (100,000 pipelines) to prevent infinite loops if the server has a bug where it always returns a Link rel="next" header.
- hasNextPage: nil headers, no Link header, with/without rel=next, multiple header values - PipelinesListV1Alpha: marshal, unmarshal, round-trip, empty list, invalid JSON
- Reset page/headers at top of retry closure to prevent stale data from a failed attempt causing runaway pagination - Add NonRetryableError to retry package so 4xx errors and deserialization failures are not retried (avoids ~6s wasted waits) - Wrap exhausted-retry errors with attempt count for debuggability - Log each retry attempt with failure reason and backoff duration - Return an error when maxPages safety limit is hit instead of silently truncating results - Remove redundant custom MarshalJSON/UnmarshalJSON on PipelinesListV1Alpha (standard encoding/json handles slices) - Initialize allPipelines as empty slice to produce [] not null - Use %w for error wrapping to support errors.Is/errors.As - Add 9 integration tests for ListPplWithOptions covering single/multi-page, error paths, retry behavior, and stale headers
- Truncate large response bodies in error messages (cap at 200 chars) - Add pagination context to mid-pagination errors (page number, accumulated count, completed pages) - Wrap json.Marshal error with descriptive context - Add [retry] prefix to retry log messages for filterability - Improve doc comments for RetryWithMaxFailures and NonRetryableError - Make maxPages a package-level var for test overriding - Add tests for: maxPages safety limit, 429 retry behavior, retry exhaustion on middle page, large error body truncation
The core logic is already covered by unit tests for hasNextPage, retry, and model serialization. No other client in this package has HTTP-level tests — this was over-testing.
…NonRetryable - Expand non-retryable HTTP range to >= 300 so redirects fail immediately instead of retrying uselessly - Add nil guard before hasNextPage to fail loudly if response headers are unexpectedly missing - Guard NonRetryable against nil error argument
The pagination changes add page and page_size query parameters to pipeline list requests. Update httpmock URL matchers to account for the new parameters.
loadez
approved these changes
Mar 17, 2026
4 tasks
cchristous
added a commit
to cchristous/semaphore-cli
that referenced
this pull request
Apr 25, 2026
ListTasks previously issued a single GET /api/v1alpha/tasks and returned whatever the first page contained (~30 tasks), silently discarding the rest. Projects with >30 tasks were invisible beyond the first page. Rewrite ListTasks to loop with page_size=200, following the API's Link: rel="next" header until exhausted, and aggregating all pages. Mirrors the pattern established for pipelines in semaphoreci#247, reusing retry.RetryWithMaxFailures, retry.NonRetryable, and hasNextPage. Includes a safety cap of 500 pages (100k tasks) and nil-header guard. No CLI surface changes -- sem get tasks -p <project> just works now. Existing httpmock responders that matched task URLs exactly are switched to regexp matchers to accommodate the new page/page_size query params. A new Test__ListTasks__MultiPage verifies both pages are fetched via Link: rel=next and aggregated into the output. Reported by Kumar Utkarsh against semaphoreci#246.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sem get pipelinesnow paginates through all results within the requested--agewindowpage_size=200and the API'sLinkheader (rel="next") to fetch all pagesWhy
sem get pipelines --age 48hpreviously returned only the first page (~30 results), silently discarding the rest. The--ageflag already expresses how much history the user wants — truncating to a single page was the actual bug. For example, a project with 60+ queued pipelines would only show the 30 most recent, hiding the backlog entirely.The pagination approach follows the pattern used by
sem get projects(page-based with retry), adapted for the pipelines API which usesLinkheaders instead ofx-has-more.Changes
api/client/pipelines_v1_alpha.go: Add pagination loop withLinkheader parsing, integrated with retry logic. Error responses include pagination context (page number, accumulated count) and truncate large response bodies to prevent noisy output. 3xx and 4xx (except 429) are treated as non-retryable; 5xx and 429 are retried.api/retry/with_max_failures.go: IntroduceRetryWithMaxFailureswith exponential backoff and aNonRetryableErrortype that short-circuits retries for permanent failures. The wrapper is unwrapped before returning so callers see the original error.api/client/pipelines_v1_alpha_test.go: Unit tests forhasNextPageLink header parsing (nil headers, comma-separated values, multiple header entries).api/models/pipeline_list_v1_alpha_test.go: Tests forPipelinesListV1AlphaJSON serialization round-trip, empty list, and invalid input.api/retry/with_max_failures_test.go: Tests for retry logic including backoff timing, non-retryable short-circuit, attempt counting, and mixed retryable/non-retryable sequences.Test plan
sem get pipelines -p <project> --age 1hreturns all pipelines from the last hour (not truncated to 30)sem get pipelines -p <project> --age 48hreturns all pipelines from the last 48 hours across multiple pagessem get pipelines <id>(describe single pipeline) still works