Skip to content

install.md: add CI-readiness guidance so autoloop's CI-gated acceptance loop can actually make progress #54

@mrjf

Description

@mrjf

Summary

install.md walks users through installing autoloop but is silent on a critical prerequisite for the autoloop to actually work: the consuming repo's CI must be in a state where autoloop's CI-gated acceptance loop (#37) can make forward progress. Without explicit guidance here, users install autoloop into repos whose CI fails every PR for reasons unrelated to the iteration, and every autoloop-authored PR lands red. The fix-retry loop from #37 stalls; the program pauses with ci-fix-exhausted; users conclude autoloop is broken when the actual problem is CI hygiene.

Add a CI-readiness section to install.md that walks users through:

  1. What needs to be true about CI for autoloop to work.
  2. How to check whether their repo's current CI meets the bar.
  3. Concrete remediation for the common failure modes.

Concrete failure modes we've seen

These are all "autoloop is trying, CI is blocking, none of it is the agent's fault":

  1. Legacy lint violations on main. Linter was upgraded (or rules tightened) and main has pre-existing violations that the linter now flags. Every PR inherits them; the agent can't fix them because they're outside the program's Target list. Symptom: lint step red on every PR, with violations in files the PR never touched.

  2. Action-approval gates. GitHub's "Require approval for all outside collaborators" setting (or the stricter variants) requires a maintainer to click "Approve and run" on every workflow run from github-actions[bot]. Until approved, CI has no conclusion — so gh pr checks --watch either times out or sees pending indefinitely. Fix-retry pushes speculative fixes on top of already-pending runs, stacking up red commits before any real result lands.

  3. CI runs lint/type-check against the full codebase, not the diff. Legacy violations in files unrelated to the PR block the PR. Common in long-lived repos where lint rules have tightened over time.

  4. Tests fail on main. Flaky tests, environment-dependent tests, or tests that depend on services the CI runner can't reach. Every PR inherits the red state.

  5. No CI at all. Rare but: some repos installing autoloop don't have a CI workflow that runs on PRs. Fix-retry has nothing to gate on.

Proposed addition to install.md

Insert as Step 2.5 (between gh-aw init and workflow copy) or as a new "CI prerequisites" section right before Step 5 (commit + PR):

## CI Prerequisites

Autoloop's per-iteration acceptance gate waits for CI to turn green on each pushed commit. For autoloop to actually make forward progress, CI must be in a state where the agent's iterations have a realistic chance of passing. Before enabling autoloop on your repo:

### 1. Confirm CI runs on PRs from bot actors

In repo Settings → Actions → General → **Fork pull request workflows from outside collaborators**, set this to `Require approval for first-time contributors who are new to GitHub` or looser. The stricter settings (`Require approval for all outside collaborators`, `Require approval for first-time contributors`) block every autoloop-authored workflow run behind a maintainer click, making the CI-gated acceptance loop unable to get a real conclusion.

Similarly, if you use environment protections or CODEOWNERS rules that gate CI on maintainer review, scope them so `github-actions[bot]`-authored PRs can still exercise CI.

### 2. Confirm main is currently green

Run your full CI command (`npm test`, `cargo test`, `pytest`, …) against a fresh clone of main. If it fails, fix main before enabling autoloop. Every PR inherits main's state; an iteration that introduces a clean change can still land red if main was already red.

This includes lint: many linters evolve their rules between releases. A linter upgrade can silently introduce violations on main that now fail every PR.

### 3. Consider scoping CI lint to changed files

Long-lived codebases frequently accumulate lint violations as tooling evolves. Running lint against the whole codebase on every PR means legacy violations block new work forever. For autoloop-authored PRs this is especially painful: the agent is bound to a program's Target list and cannot fix violations outside it.

Scope lint to changed files:

\`\`\`yaml
- name: Lint changed files only
  run: |
    set -euo pipefail
    base=\${{ github.event.pull_request.base.sha || 'origin/main' }}
    files=\$(git diff --name-only --diff-filter=ACMR "\$base" HEAD \
            | grep -E '\\.(ts|tsx|js|jsx|py|rs|go)\$' || true)
    [ -z "\$files" ] && { echo "No lint-eligible files changed."; exit 0; }
    echo "\$files" | xargs <your-lint-command>
\`\`\`

Pair with a scheduled job on main that runs the linter against the whole codebase and files a tracking issue when new violations appear. That way legacy violations neither block in-flight PRs nor silently accumulate forever.

### 4. Pin linter/formatter versions

Auto-upgrades to linters (especially ones with experimental/nursery rule groups) cause the symptom in (2). Pin to a known-good version in your package manifest and update deliberately in a dedicated PR that fixes the new rules.

### 5. Verify with a no-op PR

Before enabling autoloop on a schedule, open a no-op PR yourself (touch a single file, like adding a trailing newline to `README.md`). Confirm CI runs, passes, and the PR can be merged without intervention. If that fails, autoloop will fail the same way.

Plus a short troubleshooting addition near the end of install.md:

### Autoloop opens PRs, but they all fail CI

If every autoloop-authored PR is red with violations in files the iteration didn't touch, your main is likely red against the current CI ruleset. See **CI Prerequisites** above and run the steps there against main. Fix main, then autoloop iterations will start landing green.

If only some iterations fail CI and the agent's fix-retry loop isn't converging, the iteration is genuinely wrong — that's expected autoloop behaviour, and the program will pause with a `ci-fix-exhausted` reason once the retry budget is exhausted. Read the program's state file on `memory/autoloop` and the latest iteration comment on the program issue for details.

Why this belongs in install.md

The alternative is a separate CI_PREREQUISITES.md document. install.md is better because:

  • The user reading install.md hasn't decided yet whether autoloop fits their repo. A dedicated prerequisites document is skippable; an install step isn't.
  • The steps are actionable during install — "try a no-op PR" and "check that lint on main passes" are things the user can do before they even commit the autoloop workflow files.
  • Failures caught here are failures avoided later. Once autoloop is running on a schedule and filing issues hourly, root-causing "why don't PRs merge" is much more painful than a 5-minute prerequisites check up front.

Acceptance

  • install.md has a ## CI Prerequisites section between the gh-aw init and the PR-creation steps.
  • The section covers: approval gates, main-green check, scoped lint, pinned tool versions, no-op PR verification.
  • install.md has a troubleshooting note for the "every PR fails" symptom pointing back to the prerequisites section.
  • Existing installations aren't blocked by the addition (it's advisory; nothing is gated on it).

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions