feat: extend compliance audit with CI/automation health survey#13
Merged
feat: extend compliance audit with CI/automation health survey#13
Conversation
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 11 minutes and 27 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdded a new weekly GitHub Actions workflow that runs a repo compliance audit, collects cross-repo health telemetry, and invokes an AI analysis step to synthesize findings and optionally create routed issues; the prior single-file compliance workflow was removed. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Scheduler as "GitHub Actions\nScheduler / Manual Dispatch"
participant GH as "GitHub Actions\nRunner"
participant AuditScript as "audit script\n(scripts/compliance-audit.sh)"
participant API as "GitHub API / gh"
participant ArtifactStore as "Artifacts"
participant Claude as "Claude Code Action\n(anthropics/claude-code-action)"
participant RepoIssues as "Target Repo\n(issues)"
Scheduler->>GH: trigger workflow (cron / manual)
GH->>AuditScript: run `audit` job (checkout .github, run script)
AuditScript->>ArtifactStore: upload `compliance-report` artifact
GH->>API: run `health-survey` job (collect telemetry per repo)
API-->>GH: return health JSON report
GH->>ArtifactStore: download audit artifact (if present) in `analyze`
GH->>Claude: invoke analysis with audit + health JSON, dry_run input
Claude-->>GH: return analysis & issue actions
alt dry_run = false
GH->>RepoIssues: create/update routed issues per analysis
else dry_run = true
GH-->>GH: produce analysis output only
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an organization-level GitHub Actions workflow to periodically discover actionable PRs/issues across petry-projects and invoke a Claude Code agent to prioritize and work on a small set of items.
Changes:
- Introduces a scheduled + manually-dispatchable workflow (every 4 hours) with concurrency locking.
- Implements a discovery job that scans repos for open PRs/issues and produces a JSON “work manifest”.
- Adds an improvement job that runs
anthropics/claude-code-actionwith a detailed operating prompt using the manifest.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/continuous-improvement.yml:
- Line 240: The workflow hard-codes the organization name ("petry-projects") in
git clone commands and similar places (e.g., the line containing "gh repo clone
petry-projects/<repo> /tmp/<repo>"); replace those hard-coded org strings with
the GitHub Actions context variable for the repository owner (use ${{
github.repository_owner }}) and apply the same replacement to all other
occurrences (lines referenced in the comment such as the other clone/URL usages)
so the workflow works for any org without manual edits.
- Line 398: The current dry-run relies on the agent following natural-language
instructions; to enforce it programmatically, split the existing improve job
into two jobs (e.g., improve-dry-run and improve-live) and gate them by
inputs.dry_run, ensure improve-dry-run sets permissions.contents: read only and
does only analysis steps (no Claude Code write steps), ensure improve-live has
write permissions (contents/pull-requests/issues/id-token) and contains the
Claude Code action and any push/PR/comment steps, and alternatively wrap the
Claude Code action and any write steps with an explicit conditional that checks
inputs.dry_run (skip when true) so no write operations are executed when dry_run
is enabled.
- Line 180: The workflow's job timeout is excessive and risks long blocking;
change the timeout-minutes value from 120 to 60 and update the agent's CI-wait
behavior so it does not block for long-running CI: implement a threshold (e.g.,
10 minutes) in the logic that handles the --watch/CI polling loop and, if
exceeded, have the agent stop waiting and leave a comment on the PR instead of
continuing to wait; update any relevant steps referring to the Claude Code
action and the --watch loop to document and enforce this new timeout/threshold
behavior.
- Around line 298-303: Replace the blocking `gh pr checks <number> --watch
--fail-level all` usage with a non-blocking poll: run `sleep 30` then use `gh pr
checks <number> --repo petry-projects/<repo> --json name,state` in a loop that
re-checks every 30s up to a timeout (e.g., 10 minutes); if checks are still
pending after the timeout, stop waiting, post a PR comment with the current CI
states, and move on; if a failure is observed, retry at most once for the item
before posting results and continuing.
- Around line 345-365: The workflow currently instructs the agent to run "gh pr
create" automatically, which can create unsolicited PRs; change the CI job/step
that invokes the gh pr create command so it defaults to a dry-run mode and
requires explicit human confirmation before creating a PR (e.g., add a
conditional that only runs the "gh pr create" step when a workflow input or
repository secret like CONFIRM_CREATE_PR is set, and otherwise posts a comment
on the corresponding issue describing the proposed changes), and update the step
name that performs PR creation to clearly reflect "dry-run" vs "create" modes so
maintainers can validate output before any PR is opened.
- Line 394: Update the "Respect existing work." rule that currently checks for
author activity within the last 24h: change the inactivity window to a shorter
span (6–12 hours) and add an additional recent-update guard that skips automated
pushes if the PR was updated within the last 4 hours; implement these checks in
the discovery phase or at the start of Phase 2 so the agent first evaluates both
last-commit timestamp and last-PR-update timestamp before deciding to push
changes, and update the human-facing rule text to reflect the new windows.
- Around line 236-266: The workflow currently instructs the agent to directly
commit and push fixes in the "If the PR has unresolved comments" and "If the PR
has failing CI" checklist items (the blocks starting with "If the PR has
unresolved comments" and "If the PR has failing CI"); change these to prevent
unconsented pushes by: replace the direct "Make the requested changes... Push
the changes to the PR branch" and "Fix the root cause... Commit and push the
fix" language with steps that (1) always leave detailed review comments and
suggested patches, (2) only perform commits/pushes when the PR author explicitly
requests fixes (e.g., a trigger like "@agent please fix"), and (3) if making
changes, prefer creating a new branch and opening a separate PR against the
original branch (or verify write access to the fork before attempting any push);
also add a check to verify branch ownership/fork write permissions before any
automated push.
- Around line 89-99: The GraphQL query assigned to UNRESOLVED uses
reviewThreads(first: 50) which can miss threads when a PR has >50 threads;
update the logic in the workflow to paginate reviewThreads using
pageInfo/endCursor (or increase to a safe max like first: 100 if acceptable) and
loop requests until pageInfo.hasNextPage is false, aggregating nodes and
computing the unresolved count; look for the UNRESOLVED assignment and the
reviewThreads(first: 50) fragment to modify the query and the surrounding shell
logic to collect all pages before computing the final length.
- Line 121: The jq label-filter currently uses
any(test("help|bug|enhancement|good first issue|scorecard|compliance")) which
does substring matching and can match unintended labels; replace that part of
the expression (the select(... .labels | map(.name) | any(test(...)) ...)) with
either anchored regexes like any(test("^(help|bug|enhancement|good first
issue|scorecard|compliance)$")) or, preferably, explicit equality checks e.g.
any(. == "help" or . == "bug" or . == "enhancement" or . == "good first issue"
or . == "scorecard" or . == "compliance") so only exact label names are matched.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8dcd0949-44d1-4980-9e5a-871605e8df3b
📒 Files selected for processing (1)
.github/workflows/continuous-improvement.yml
don-petry
changed the title
don-petry
force-pushed
the
feat/continuous-improvement-agent
branch
from
e245144 to
04c31c0
Compare
don-petry
changed the title
don-petry
force-pushed
the
feat/continuous-improvement-agent
branch
3 times, most recently
from
c175547 to
f0831cf
Compare
don-petry
force-pushed
the
feat/continuous-improvement-agent
branch
from
f0831cf to
8d035da
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/compliance-audit-and-improvement.yml:
- Around line 213-216: The current steps build REPORT from REPORT_FILE and write
it into GITHUB_OUTPUT via REPORT and REPORT_EOF which risks hitting output size
limits; instead write the normalized JSON into a dedicated file (e.g.,
health-survey-report.json) and use the actions/upload-artifact action (or
equivalent artifact upload) to publish that file as the health survey report
artifact; update the workflow to remove the echo to GITHUB_OUTPUT
(REPORT/REPORT_EOF) and add an upload-artifact step that references the
generated file so large reports are handled robustly.
- Line 126: Replace the subshell sed invocation that sets REPOS from TARGET_REPO
with bash parameter expansion to remove the "$ORG/" prefix; update the
assignment that references REPOS and TARGET_REPO (and ORG) to use parameter
expansion (strip the "$ORG/" prefix from TARGET_REPO) instead of echo|sed so the
linter SC2001 is satisfied and no external command is invoked.
- Around line 185-202: The current inline Python script embeds shell variables
directly (e.g., $REPO, $FULL_REPO, $BRANCH_PROTECTION, $WORKFLOWS) which is
fragile and can break on quotes; change the step to export those values as
environment variables and have the Python snippet read them from os.environ (or
write JSON payloads like BRANCH_PROTECTION/WORKFLOWS to temp files and read them
in Python) before loading REPORT_FILE and appending the repo entry; update the
code that builds the JSON to use json.loads(os.environ['BRANCH_PROTECTION']) and
json.loads(os.environ['WORKFLOWS']) (or file reads) and keep REPORT_FILE as the
path passed into the script so no direct shell-to-Python string interpolation
occurs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 3e2bcc0f-290f-46a5-a7f1-cd560c903cdd
📒 Files selected for processing (1)
.github/workflows/compliance-audit-and-improvement.yml
don-petry
force-pushed
the
feat/continuous-improvement-agent
branch
2 times, most recently
from
a8f1b77 to
2df7591
Compare
don-petry
force-pushed
the
feat/continuous-improvement-agent
branch
from
2df7591 to
e3460b1
Compare
Replaces compliance-audit.yml with compliance-audit-and-improvement.yml,
extending the existing weekly compliance audit with runtime health
telemetry and a forward-looking best practices research phase.
Architecture (3 jobs):
Job 1 — Compliance Audit (unchanged)
Deterministic shell script checking all repos against org standards.
Creates/updates/closes compliance issues per finding.
Job 2 — Health Survey (new)
Collects runtime telemetry across all org repos:
CI run failures (7d), security alerts (Dependabot/secret/code scanning),
PR staleness, branch protection status, workflow inventory.
Job 3 — Analyze & Create Issues (Claude, rewritten)
Six-phase analysis combining both datasets:
1. Load compliance + health data and org standards
2. Correlate and categorize findings by severity
3. Research root causes and automation opportunities
4. Evaluate against industry best practices and emerging capabilities
(agentic guardrails, supply chain integrity, reliability SLOs, etc.)
— outputs only standards proposals, not implementation issues
5. Create issues: repo-specific go in that repo, org-wide in .github,
every issue gets the claude label for agent pickup
6. Summary report to step summary
Issue rules:
- Every issue must have the `claude` label
- Repo-specific issues are created in that repo
- Org-wide and standards proposals go in .github
- Deduplicates against existing open issues
- Max 3 standards-improvement + 3 best-practices proposals per run
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
don-petry
force-pushed
the
feat/continuous-improvement-agent
branch
from
e3460b1 to
afd3af1
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces
compliance-audit.ymlwithcompliance-audit-and-improvement.yml, extending the existing weekly compliance audit with runtime health telemetry and a forward-looking best practices research phase.Architecture
What's New
Job 2 — Health Survey
Collects live runtime telemetry across all org repos:
gh run listJob 3 — Combined Analysis (6 Phases)
claudelabel, deduplicationIssue Rules
claudelabel for agent pickup.githubStandards:prefixed proposals with proposed standard, rationale, current gap, and implementation pathWhat's Unchanged
scripts/compliance-audit.shdry_runandtarget_repoinputsTest Plan
dry_run: true— verify both jobs collect data, no issues createdtarget_repoinput)claudelabel.githubwithclaudelabelStandards:proposals in.githubInitial Survey Issues (2026-04-05)
All issues have the
claudelabel.Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
Summary by CodeRabbit