Extend action-pin tooling: list mode, branch-pin validation, upgrade reporting

[lex57ukr]

Goal

Expand the action-pin tooling in ci-tools beyond strict tag-pin verification, so it can:

1. Inventory pinned actions without hitting the API (fast audits).
2. Validate branch-pinned actions (e.g. @main) and report drift from branch HEAD.
3. Surface available upgrades for tag- and branch-pinned actions, so we can review pins periodically without waiting on dependabot.

Context: homebrew-tap just SHA-pinned Homebrew/actions/setup-homebrew@main (upstream has no tagged releases). Dependabot cannot track branch-SHA pins, so we need tool-supported visibility into drift.

Scope

Extend validate-action-pins:

• --list flag: print every uses: <owner>/<repo>@<ref> in the provided files with no API calls. Works on SHA pins and tag/branch pins alike. Useful for audits and piping into other tools.
• Branch-ref support: accept a branch hint in the comment (e.g. # main) and verify via /git/ref/heads/<branch>. Use /repos/{owner}/{repo}/compare/{pinned-sha}...{head-sha} to report "N commits behind HEAD" so stale branch pins are visible without being a hard failure.

New sibling tool (proposed name: check-action-updates):

• Scans the same workflow files and, for each pinned action, reports the most recent semver tag (newest tag matching the current major, plus absolute latest) and the branch HEAD for branch pins.
• Output is a compact diff-style report — suitable for a scheduled workflow that opens an issue when upgrades are available.

Outcome

• Branch pins become maintainable: drift from HEAD is observable without manual gh api calls.
• Periodic upgrade audits can be scripted (e.g. a cron workflow that files an issue per stale pin).
• --list gives a fast, API-free inventory for ad-hoc review.

Notes

Reasoning for splitting upgrade reporting into a separate tool: "verify pins are consistent" and "report available upgrades" are different jobs. Bundling them behind flags bloats the flag surface and muddies exit-code semantics (upgrade-available is informational; pin-mismatch is a failure). Separate binaries keep each tool focused.

Driving use case: knight-owl-dev/homebrew-tap#50 — SHA-pinning in that repo surfaced the branch-pin staleness gap.

[lex57ukr]

Approach summary

PR delivers the three goals from the issue body but diverges from two structural choices in the original spec. Branch is 122-extend-action-pin-tooling, 15 commits, 55 BATS tests.

What landed

Single binary, subcommand dispatch on validate-action-pins:

• check — existing SHA-vs-tag verification plus branch-pin support. The Dependabot-style # <ref> comment may be a tag OR a branch; branch drift is WARN (informational), tag mismatch stays FAIL.
• list — offline inventory of every uses: pin. No API, no auth, no connectivity probe.
• updates — upgrade inventory per unique pin.

Bare validate-action-pins FILE... still routes to check, so Makefile, scripts/ci-tools/verify.sh, and scripts/verify-deb-install.sh needed no changes. --format=plain|tsv on list and updates for scriptable pipelines.

Deviations from the issue body

1. No separate check-action-updates binary — subcommands on the existing binary instead.

Subcommand dispatch keeps exit-code semantics clean per-command (updates exits 0 regardless, check exits 1 on mismatch) without splitting packaging, man pages, version handling, or the scripts/ci-tools/resolve.sh tool list. The concern in the issue ("upgrade-available is informational; pin-mismatch is a failure") is satisfied at the subcommand boundary just as well as at the binary boundary, with less infrastructure cost.

2. --list landed as a subcommand, not a flag.

With check + list + updates all being full-scale modes, flag-on-flag would be asymmetric. Subcommand style matches the git / gh composition idiom and lets each subcommand own its own flags.

3. updates reports every newer tag, not "newest same-major + overall latest".

Initial implementation did the latter, but real-world action tag conventions are too varied (v6, v6.0.2, 0.35.0 without v-prefix, major-only aliases) to confidently pick "the" next version. Dependabot already doesn't open major-version PRs by default, so the value of this command is an inventory the reviewer can skim — not a policy decision the tool makes on their behalf. Output lists every tag strictly newer than the current ref, sorted ascending. Tag classification is tolerant: v?X[.Y[.Z]], so @v6 and @v6.0.0 both route through the tag path and compare equal for the newer-than check.

Scope delivered by side effect

• Testability refactor: validate-action-pins is now sourceable (BASH_SOURCE guard, GITHUB_API_BASE env override, optional connectivity-probe skip). Byte-identical behaviour when env vars are unset.
• BATS harness under tests/bats/ — new test-location convention mirroring images/<image>/ under the tests root. 55 cases covering parse / resolve / check / list / updates, with file:// fixtures standing in for the GitHub API.
• DOCKER_TTY Makefile variable so make test-bats, make verify, and make scan render interactively (coloured output) while keeping CI logs clean.

Deferred

• Pagination of /tags — first page (30 entries) suffices for typical action repos; documented as a known limit in the man page.
• Cron workflow that opens issues per stale pin — the tooling is in place; wiring it up isn't in this PR.

[lex57ukr]

Conscious punt list

In-session DevOps + architect reviews surfaced four items we deliberately didn't address in this PR. Listing them here for the record, not as a promise to come back.

Parallelize API calls in check / updates
Sequential pin resolution works fine at our workflow sizes (~15 pins per repo, ~3s total). Bash parallelism is fragile — output-order non-determinism, no locking on the cache associative arrays, secondary rate-limit risk at high fan-out. The clean solution is a rewrite in Go or Python, which is a different project. Revisit if a consumer with 100+ pins complains about runtime.

Unify TSV schema across subcommands
list --format=tsv and updates --format=tsv both emit 5 columns, but columns 4–5 carry different taxonomies (pin format vs classified kind; comment vs newer-list). Per-subcommand schemas are a valid design choice — list and updates do different jobs — and unifying would break every downstream consumer we've already documented in the man page and in the comment above. Man page explains each subcommand's columns; fix the day a real consumer tries to merge TSV streams and gets bitten.

Permissive flag placement (kubectl / gh style)
Today the subcommand must be $1, so validate-action-pins --format=tsv list workflows/*.yml doesn't route correctly — flag-then-subcommand isn't accepted. The fix is a two-pass arg parser plus a handful of permutation tests. No user has hit it; nice-to-have polish.

Nightly smoke against the real GitHub API
All 71 bats cases use file:// fixtures, so a GitHub API schema drift wouldn't trip them. make lint-actions already runs check against this repo's own workflows on every PR, so there's an implicit integration test on the happy path. Proper version would be a scheduled workflow hitting a known-stable ref (e.g. actions/checkout@v6) with a clear escalation path — different review surface (CI infra + secrets) than this PR.

All four are real improvements with real trade-offs. None block shipping the behavior changes in this branch; each gets revisited if/when evidence arrives.

[lex57ukr]

Addendum (post-review iterations)

Two reviewer-hat rounds after the approach-summary and punt-list comments above, a few items shifted:

No longer deferred: "Permissive flag placement (kubectl / gh style)" from the punt list shipped in bc9e798. A reviewer noted that "public contract" isn't a real constraint yet (nothing's shipped), so the argument for deferring dissolved. validate-action-pins --format=tsv list foo.yml now routes correctly, and the subcommand word can appear anywhere in the arg list; -- disambiguates the edge case of a workflow literally named check/list/updates.

Updated numbers: branch is 37 commits and 83 bats tests at the time of this note. Tests are split into six files by abstraction layer (cli / preflight / helpers / subcommand-*) under tests/bats/images/ci-tools/validate-action-pins/, with fixtures/ co-located so each future ci-tools binary gets its own namespace.

Added since the approach summary: curl retries + --max-time, distinct WARN paths for auth (401/403) / secondary rate limit (429) / transport failure / unexpected HTTP, rate-limit-budget awareness (soft WARN at <20 remaining, hard skip at 0), a CI bats job, RETURN-trap tempfile cleanup in the preflight, sourced-test coverage for the preflight status classifier, and a BATS_RUNNER Makefile knob for the container-vs-host split.

Still deferred: parallelize API calls, TSV-schema unification across subcommands, nightly real-API integration test. Same reasoning as the original punt list.