Skip to content

feat: add weekly compliance audit workflow#12

Merged
don-petry merged 4 commits intomainfrom
feat/weekly-compliance-audit
Apr 5, 2026
Merged

feat: add weekly compliance audit workflow#12
don-petry merged 4 commits intomainfrom
feat/weekly-compliance-audit

Conversation

@don-petry
Copy link
Copy Markdown
Contributor

@don-petry don-petry commented Apr 5, 2026

Summary

  • Adds a weekly GitHub Actions workflow that audits all petry-projects repositories against org standards
  • Deterministic shell script checks: required CI workflows, Dependabot config, repo settings, labels, rulesets, action SHA pinning, CODEOWNERS, permissions
  • Claude Code Action job researches standards improvements and posts summary notifications
  • Creates/updates/closes GitHub Issues for each finding, assigns them to Claude for remediation

Architecture

Job 1: Compliance Audit (deterministic scripts/compliance-audit.sh)

  • Iterates all non-archived repos in the org (excluding .github)
  • Checks 9 categories: required workflows, action pinning, dependabot config, repo settings, discussions, labels, rulesets, CODEOWNERS, workflow permissions
  • Outputs JSON findings + markdown summary
  • Creates issues in each repo for gaps; closes issues when resolved
  • Uploads report as artifact (90-day retention)

Job 2: Standards Review (Claude Code Action)

  • Downloads audit findings from Job 1
  • Reviews org standards for gaps/improvements
  • Creates up to 3 improvement issues in .github per run
  • Posts weekly summary notification issue for org owners

Standards Covered

Standard Checks
CI Standards 6 required workflows, SHA pinning, least-privilege permissions
Dependabot Policy Config exists, github-actions ecosystem, limit:0 for apps, labels
GitHub Settings Auto-merge, delete branches, wiki, discussions, labels, rulesets, CODEOWNERS

Test Plan

  • Dry-run tested locally — successfully audited all 6 repos, found 61 findings
  • Trigger via workflow_dispatch with dry_run: true to validate in CI
  • Trigger via workflow_dispatch with dry_run: false to validate issue creation
  • Verify idempotent behavior on second run (updates, not duplicates)
  • Verify Claude Code Action standards review job completes

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores

    • Added automated weekly compliance audits running Mondays at 08:00 UTC
    • Generates detailed compliance reports with summaries and metrics, uploaded as retained artifacts
    • Automatically files, updates, and closes issues for detected compliance items and maintains a dated weekly summary
  • New Features

    • Supports manual on-demand runs and a dry-run mode for safe testing
    • Uses an automated review step to propose labeled enhancement issues and publish an executive weekly summary

Adds automated weekly audit that checks all petry-projects repos
against org standards (CI, Dependabot, settings, labels, rulesets)
and creates/updates/closes issues for each finding.

- Deterministic shell script for reliable, repeatable checks
- Claude Code Action job for standards improvement research
- Issues auto-assigned to Claude for remediation
- Summary notification for org owners
- Idempotent: updates existing issues, closes resolved ones

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 5, 2026 17:16
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 5, 2026

📝 Walkthrough

Walkthrough

Adds a new scheduled and manual GitHub Actions workflow .github/workflows/compliance-audit.yml and a new executable audit script scripts/compliance-audit.sh that enumerates org repositories, generates machine-readable findings and a human summary, uploads artifacts, and optionally creates/updates/closes per-repo compliance issues; also runs a standards-review step using an LLM action.

Changes

Cohort / File(s) Summary
Workflow
.github/workflows/compliance-audit.yml
New "Weekly Compliance Audit" workflow: weekly cron + workflow_dispatch with dry_run boolean; two jobs (audit, standards-review); global permissions: {}, non-canceling concurrency group, artifact upload, job outputs, and step summaries.
Audit Script
scripts/compliance-audit.sh
New executable Bash script performing org-wide checks (list repos, skip archives and .github, validate workflow use pinning, Dependabot checks, repo settings vs standards, labels, rulesets, CODEOWNERS, SonarCloud heuristics, workflow permissions). Emits findings.json and summary.md, manages per-repo "Compliance: " issues (create/comment/close) when enabled, supports DRY_RUN and CREATE_ISSUES.

Sequence Diagram

sequenceDiagram
    participant GH as GitHub Actions
    participant AuditJob as Audit Job
    participant Script as compliance-audit.sh
    participant GHAPI as GitHub API
    participant Artifact as Artifact Storage
    participant ReviewJob as Standards-Review Job
    participant Claude as Claude Action

    GH->>AuditJob: trigger (cron / manual)
    AuditJob->>Script: run compliance-audit.sh (REPORT_DIR, DRY_RUN, CREATE_ISSUES)
    Script->>GHAPI: list org repos, fetch files, inspect workflows & settings
    Script->>Script: build findings.json and summary.md
    Script->>GHAPI: create/update/close per-repo compliance issues (if enabled)
    AuditJob->>Artifact: upload compliance-report artifact
    GH->>ReviewJob: needs: audit (success)
    ReviewJob->>Artifact: download compliance-report
    ReviewJob->>Claude: submit direct_prompt with findings + standards + workflow URL
    Claude->>GHAPI: create/deduplicate enhancement issues and file weekly summary issue
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: adding a weekly compliance audit workflow to the repository.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/weekly-compliance-audit

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/compliance-audit.yml:
- Around line 50-59: The current REPO_COUNT/jq query ('[.[].repo] | unique |
length') counts unique repositories present in findings, not total audited
repos; rename the variable and output key to something explicit (e.g.,
REPOS_WITH_FINDINGS_COUNT and repos_with_findings_count), update the echo that
writes to GITHUB_OUTPUT to use the new key, and adjust the workflow output
definition and the user-facing prompt text that referenced repo_count to match
the new name so it clearly reads "Repos with findings"; keep the same jq
expression (REPO_COUNT/REPOS_WITH_FINDINGS_COUNT) and update any references to
repo_count elsewhere in the file to the new identifier.

In `@scripts/compliance-audit.sh`:
- Around line 602-603: The current repos assignment uses gh repo list with a
fixed --limit 100 which can silently omit repos; update the
scripts/compliance-audit.sh logic that sets the repos variable so it fetches all
non-archived repositories (e.g., switch to a paginated fetch or use gh repo list
--paginate if available, or increase/remove the --limit and loop until all pages
are returned) — modify the line that runs gh repo list "$ORG" --no-archived
--json name -q '.[].name' --limit 100 to a paginated/looped version that
accumulates results into the repos variable to ensure no repositories are
skipped.
- Line 35: The array REQUIRED_WORKFLOWS in scripts/compliance-audit.sh currently
lists dependabot-automerge.yml and dependency-audit.yml which are not documented
as mandatory; update the script so REQUIRED_WORKFLOWS only contains the four
universally required workflows (ci.yml, codeql.yml, sonarcloud.yml, claude.yml)
to avoid false positives, or if you intend to require the Dependabot workflows
instead, update standards/ci-standards.md to declare dependabot-automerge.yml
and dependency-audit.yml as mandatory; locate and edit the REQUIRED_WORKFLOWS
definition to implement one of these two options.
- Around line 472-475: The script attempts to assign issues to "app/claude"
using the gh CLI in the block that checks new_issue and calls gh issue edit
--add-assignee; GitHub Apps cannot be assigned this way and the current call
fails silently. Update the gh issue edit invocation in the new_issue assignment
block to use a valid assignee (replace "app/claude" with the actual bot username
"claude" if that account exists) or remove the --add-assignee flag entirely and
skip assignment when no valid user exists; ensure the change is applied to the
gh issue edit call that references "$ORG/$repo" and --add-assignee so
assignments succeed or are omitted.
- Around line 363-388: The check_workflow_permissions function currently flags
any workflow missing a top-level permissions: key; update it to allow single-job
workflows that define permissions at the job level by: after decoding into
decoded, first test for a top-level permissions: (as now), but if missing then
parse the jobs block to determine if there is exactly one job and whether that
job contains a job-level permissions: declaration—only call add_finding when
neither a top-level permissions: exists nor (jobs count == 1 and that single job
has permissions:). Use the existing variables (decoded, wf, repo) and adjust the
conditional logic in check_workflow_permissions to inspect the jobs section in
decoded before invoking add_finding.
- Around line 73-85: The gh_api retry wrapper currently discards all stderr
(2>/dev/null) which hides useful error messages; update the gh_api function so
that during the first retries you continue to suppress stderr but on the final
failed attempt you capture and emit the stderr output for debugging. Concretely,
keep the existing loop and suppressed calls for i < retries, but on the final
iteration run gh api "$@" while capturing stderr (e.g., into a variable) and
then print that stderr to stderr or via logger before returning failure;
reference the gh_api function, the retries variable and the loop checking if [
"$i" -lt "$retries" ] to locate where to add this behavior.
- Around line 196-230: The current string checks assume double-quoted YAML;
update the grep/awk patterns to accept single-quoted, double-quoted, or unquoted
values. Replace occurrences that look for package-ecosystem:.*"X" (used in the
github-actions check and the for-loop eco detection/awk) with a regex like
package-ecosystem:\s*['"]?X['"]? (use grep -E or awk regex) and change the label
checks that grep for `"security"` and `"dependencies"` to grep -E -q
"['\"]?(security|dependencies)['\"]?" (or equivalent regex) so values in lists,
single quotes, double quotes, or unquoted forms are all matched; keep using the
same variables (decoded, eco) and add_finding calls.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 34661d9e-941b-4f17-a1da-243477f4a176

📥 Commits

Reviewing files that changed from the base of the PR and between b7f6e7d and 9394b07.

📒 Files selected for processing (2)
  • .github/workflows/compliance-audit.yml
  • scripts/compliance-audit.sh

Comment thread .github/workflows/compliance-audit.yml Outdated
Comment thread scripts/compliance-audit.sh
Comment thread scripts/compliance-audit.sh
Comment thread scripts/compliance-audit.sh
Comment thread scripts/compliance-audit.sh
Comment thread scripts/compliance-audit.sh
Comment thread scripts/compliance-audit.sh Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an org-wide, scheduled compliance auditing capability for the petry-projects organization, aligning repositories with the documented standards under standards/ and automating remediation tracking via GitHub Issues.

Changes:

  • Adds scripts/compliance-audit.sh to audit repositories for required workflows, action SHA pinning, Dependabot policy, settings, labels, rulesets, CODEOWNERS, SonarCloud properties, and workflow permissions.
  • Adds a scheduled GitHub Actions workflow to run the audit weekly (with workflow_dispatch support), upload audit artifacts, and run a Claude Code “standards review” follow-up job.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.

File Description
scripts/compliance-audit.sh Implements the compliance checks, report generation, and issue create/update/close automation across org repos.
.github/workflows/compliance-audit.yml Schedules and runs the audit script weekly, uploads the report artifact, and triggers a Claude Code standards review job.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread scripts/compliance-audit.sh Outdated
Comment thread scripts/compliance-audit.sh
Comment thread scripts/compliance-audit.sh
Comment thread scripts/compliance-audit.sh
Comment thread scripts/compliance-audit.sh
Comment thread scripts/compliance-audit.sh Outdated
Comment thread .github/workflows/compliance-audit.yml Outdated
Comment thread scripts/compliance-audit.sh
- Add retry error logging to gh_api helper
- Fix pnpm detection when package.json absent
- Fix empty ecosystem array display
- Replace heredoc with direct assignment for issue body
- Add jq error safety in close_resolved_issues
- Increase repo list limit to 500 with empty check
- Use process substitution instead of pipe subshell
- Add concurrency group and timeout to workflow
- Add timeout-minutes to audit job

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/compliance-audit.yml:
- Around line 155-161: The current step that closes previous "Weekly Compliance
Audit Summary" issues is brittle because it relies only on title matching;
update the workflow to require a specific auxiliary label (e.g., add and check
for the label weekly-summary) when searching/closing prior issues and ensure
created issues are tagged with that label in addition to compliance-audit;
specifically modify the logic that closes previous issues to filter GitHub
issues by label "weekly-summary" and title pattern "Weekly Compliance Audit
Summary" before closing, and ensure the Create Issue step applies the
"weekly-summary" label when opening the new "Weekly Compliance Audit Summary —
YYYY-MM-DD" issue.
- Around line 109-114: The workflow step "Run Claude Code for standards review"
currently pins the action via an intermediate commit SHA
(anthropics/claude-code-action@bee87b3258c251f9279e5371b0cc3660f37f3f77) which
is unstable; update the uses reference to a proper release tag (for example
anthropics/claude-code-action@v1.0.88 or a later stable tag) so the action is
pinned to a release that includes the fix for the Grep EACCES bug, leaving the
rest of the step (env GH_TOKEN and input claude_code_oauth_token) unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 263cd924-e496-466f-aa60-8350d88d3c4a

📥 Commits

Reviewing files that changed from the base of the PR and between 9394b07 and 88101eb.

📒 Files selected for processing (2)
  • .github/workflows/compliance-audit.yml
  • scripts/compliance-audit.sh

Comment thread .github/workflows/compliance-audit.yml
Comment thread .github/workflows/compliance-audit.yml Outdated
- Handle single-job workflows with job-level permissions
- Add has_issues to required settings checks
- Soften CODEOWNERS wording (SHOULD not MUST per standards)
- Remove misleading issues:write from audit job permissions
- Rename repo_count to repos_with_findings for clarity

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@don-petry
Copy link
Copy Markdown
Contributor Author

Review Response

All actionable review comments have been addressed in commits 88101eb and 7f659ec. Here's the summary:

Fixed

  • REPO_COUNT naming → renamed to repos_with_findings for clarity
  • gh_api retry logging → added error logging on final retry failure
  • Permissions false positives → now handles single-job workflows with job-level permissions:
  • --limit 100 → increased to --limit 500 with empty-result validation
  • Missing has_issues check → added to REQUIRED_SETTINGS_BOOL
  • CODEOWNERS wording → softened to "recommended" (SHOULD not MUST per standards)
  • issues: write permission → removed from audit job (cross-repo access uses ORG_SCORECARD_TOKEN)

Reviewed, No Change Needed

  • REQUIRED_WORKFLOWS includes Dependabot workflowsci-standards.md explicitly lists all 6 as "Every repository MUST have these workflows" (sections 5 and 6)
  • YAML quote style fragility → all org dependabot.yml files consistently use double-quoted values; will track as future improvement
  • app/claude assignment|| true makes this best-effort; issue is still created and labeled if assignment fails
  • O(n²) add_finding → acceptable for current scale (~60 findings/~6 repos); will optimize if needed
  • detect_ecosystems API call → provides context for Claude Code AI analysis step and future ecosystem-aware checks
  • Missing open-pull-requests-limit key → when absent, Dependabot defaults vary; other checks catch missing/malformed ecosystem entries

Per feedback, only humans should close summary/notification
issues. Changed Claude prompt to explicitly not close them.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

♻️ Duplicate comments (1)
scripts/compliance-audit.sh (1)

74-88: 🧹 Nitpick | 🔵 Trivial

Final retry failure should use warn for better visibility.

Line 84 uses info for the failure message, but failures warrant warn (which emits ::warning:: to the Actions log). This was noted in a past review comment about logging failures.

Proposed fix
     else
-      info "gh api $1 failed after $retries retries" >&2
+      warn "gh api $1 failed after $retries retries"
     fi
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/compliance-audit.sh` around lines 74 - 88, The gh_api retry wrapper
uses info for the final failure message; change it to call warn instead so
GitHub Actions emits a visible warning. In the gh_api function replace the info
"gh api $1 failed after $retries retries" >&2 call with a warn invocation (e.g.,
warn "gh api $* failed after $retries retries") to include the full command and
ensure it goes through the warn logging helper; keep the existing retry logic
and exit behavior intact.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/compliance-audit.sh`:
- Around line 387-402: The job-counting logic using grep on the decoded YAML is
fragile; update the check in the block that computes job_count and has_job_perms
(variables: job_count, has_job_perms, operating on decoded) to use a real YAML
parser when available (prefer yq) to read .jobs and per-job .permissions (e.g.,
use yq to compute job_count as length of .jobs and to test each job's
permissions) and fall back to the existing grep heuristic only if yq is not
installed; ensure you still call add_finding(repo, "ci-workflows",
"missing-permissions-$wf", ...) when no top-level permissions and either more
than one job or no job-level permissions are present, and add a short comment
documenting the yq fallback assumption.
- Around line 211-225: The current limit extraction uses "limit=$(echo "$block"
| grep 'open-pull-requests-limit:' | head -1 | grep -oE '[0-9]+' || echo "")"
which can pick up unrelated digits; instead, first locate the specific
"open-pull-requests-limit:" line in the captured block and then parse only the
value following that key (accounting for optional spaces and possible YAML
formats like "open-pull-requests-limit: 0" or "open-pull-requests-limit: '0'"),
replacing the fragile two-grep approach that sets the variable limit; update the
logic around the variables block and limit (used in the for-loop over eco and
the add_finding call) to extract the numeric value from the matched line only
and fall back to empty if not present.
- Around line 104-111: The current replacement
ECOSYSTEMS=("${ECOSYSTEMS[@]/npm/pnpm}") is fragile because it replaces the
substring "npm" inside any element; instead, update the branch that handles pnpm
lockfiles to perform an exact-element replacement: iterate over the ECOSYSTEMS
array, if an element equals "npm" replace that element with "pnpm" (preserving
array order and indices), and if no exact match is found append "pnpm" to
ECOSYSTEMS; use the existing conditional that checks the pnpm lockfile and the
ECOSYSTEMS variable to locate where to implement this change.
- Around line 689-691: The two stdout echo lines (echo "findings=$FINDINGS_FILE"
and echo "summary=$SUMMARY_FILE") are not captured by the workflow; either
remove them if they were only debug prints or redirect them into the GitHub
Actions outputs file by appending the keys and values to $GITHUB_OUTPUT (use
FINDINGS_FILE and SUMMARY_FILE variables to produce "findings=..." and
"summary=..." entries). Update the script where those echoes appear so the
intended output is written to $GITHUB_OUTPUT when meant as workflow outputs,
otherwise delete the lines to avoid noisy unused stdout.

---

Duplicate comments:
In `@scripts/compliance-audit.sh`:
- Around line 74-88: The gh_api retry wrapper uses info for the final failure
message; change it to call warn instead so GitHub Actions emits a visible
warning. In the gh_api function replace the info "gh api $1 failed after
$retries retries" >&2 call with a warn invocation (e.g., warn "gh api $* failed
after $retries retries") to include the full command and ensure it goes through
the warn logging helper; keep the existing retry logic and exit behavior intact.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: a4acc60d-c840-47ee-91ab-320c7d924018

📥 Commits

Reviewing files that changed from the base of the PR and between 88101eb and 734db45.

📒 Files selected for processing (2)
  • .github/workflows/compliance-audit.yml
  • scripts/compliance-audit.sh

Comment on lines +104 to +111
if echo "$tree" | grep -qE '(^|/)pnpm-lock\.yaml$'; then
# Override npm with pnpm if lock file present, or add pnpm directly
if [[ " ${ECOSYSTEMS[*]} " == *" npm "* ]]; then
ECOSYSTEMS=("${ECOSYSTEMS[@]/npm/pnpm}")
else
ECOSYSTEMS+=("pnpm")
fi
fi
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Fragile array element replacement for pnpm override.

The substitution "${ECOSYSTEMS[@]/npm/pnpm}" replaces the substring npm in all array elements, not just exact matches. This works currently because no other ecosystem name contains "npm" as a substring, but it's fragile.

More robust approach
   if echo "$tree" | grep -qE '(^|/)pnpm-lock\.yaml$'; then
-    # Override npm with pnpm if lock file present, or add pnpm directly
-    if [[ " ${ECOSYSTEMS[*]} " == *" npm "* ]]; then
-      ECOSYSTEMS=("${ECOSYSTEMS[@]/npm/pnpm}")
-    else
+    # Override npm with pnpm if lock file present
+    local new_ecos=()
+    local replaced=false
+    for eco in "${ECOSYSTEMS[@]}"; do
+      if [ "$eco" = "npm" ]; then
+        new_ecos+=("pnpm")
+        replaced=true
+      else
+        new_ecos+=("$eco")
+      fi
+    done
+    ECOSYSTEMS=("${new_ecos[@]}")
+    if [ "$replaced" = false ]; then
       ECOSYSTEMS+=("pnpm")
     fi
   fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if echo "$tree" | grep -qE '(^|/)pnpm-lock\.yaml$'; then
# Override npm with pnpm if lock file present, or add pnpm directly
if [[ " ${ECOSYSTEMS[*]} " == *" npm "* ]]; then
ECOSYSTEMS=("${ECOSYSTEMS[@]/npm/pnpm}")
else
ECOSYSTEMS+=("pnpm")
fi
fi
if echo "$tree" | grep -qE '(^|/)pnpm-lock\.yaml$'; then
# Override npm with pnpm if lock file present
local new_ecos=()
local replaced=false
for eco in "${ECOSYSTEMS[@]}"; do
if [ "$eco" = "npm" ]; then
new_ecos+=("pnpm")
replaced=true
else
new_ecos+=("$eco")
fi
done
ECOSYSTEMS=("${new_ecos[@]}")
if [ "$replaced" = false ]; then
ECOSYSTEMS+=("pnpm")
fi
fi
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/compliance-audit.sh` around lines 104 - 111, The current replacement
ECOSYSTEMS=("${ECOSYSTEMS[@]/npm/pnpm}") is fragile because it replaces the
substring "npm" inside any element; instead, update the branch that handles pnpm
lockfiles to perform an exact-element replacement: iterate over the ECOSYSTEMS
array, if an element equals "npm" replace that element with "pnpm" (preserving
array order and indices), and if no exact match is found append "pnpm" to
ECOSYSTEMS; use the existing conditional that checks the pnpm lockfile and the
ECOSYSTEMS variable to locate where to implement this change.

Comment on lines +211 to +225
for eco in npm pip gomod cargo terraform; do
if echo "$decoded" | grep -q "package-ecosystem:.*\"$eco\""; then
# Check if this ecosystem has limit: 0
# Simple heuristic: find the ecosystem line and look for limit in the next ~10 lines
local block
block=$(echo "$decoded" | awk "/package-ecosystem:.*\"$eco\"/{found=1} found{print; if(/package-ecosystem:/ && NR>1 && !/\"$eco\"/) exit}" | head -15)
local limit
limit=$(echo "$block" | grep 'open-pull-requests-limit:' | head -1 | grep -oE '[0-9]+' || echo "")
if [ -n "$limit" ] && [ "$limit" != "0" ]; then
add_finding "$repo" "dependabot" "wrong-limit-$eco" "warning" \
"Dependabot \`$eco\` ecosystem has \`open-pull-requests-limit: $limit\` (should be \`0\` for security-only policy)" \
"standards/dependabot-policy.md#policy"
fi
fi
done
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Limit extraction may capture wrong numbers in edge cases.

Line 218 uses grep -oE '[0-9]+' which extracts the first number found in the block. If the ecosystem block contains other numbers (e.g., in comments or other settings) before open-pull-requests-limit, the wrong value could be extracted.

Proposed fix for more precise extraction
       local limit
-      limit=$(echo "$block" | grep 'open-pull-requests-limit:' | head -1 | grep -oE '[0-9]+' || echo "")
+      limit=$(echo "$block" | grep 'open-pull-requests-limit:' | head -1 | sed -n 's/.*open-pull-requests-limit:[[:space:]]*\([0-9]*\).*/\1/p' || echo "")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/compliance-audit.sh` around lines 211 - 225, The current limit
extraction uses "limit=$(echo "$block" | grep 'open-pull-requests-limit:' | head
-1 | grep -oE '[0-9]+' || echo "")" which can pick up unrelated digits; instead,
first locate the specific "open-pull-requests-limit:" line in the captured block
and then parse only the value following that key (accounting for optional spaces
and possible YAML formats like "open-pull-requests-limit: 0" or
"open-pull-requests-limit: '0'"), replacing the fragile two-grep approach that
sets the variable limit; update the logic around the variables block and limit
(used in the for-loop over eco and the add_finding call) to extract the numeric
value from the matched line only and fall back to empty if not present.

Comment on lines +387 to +402
# Check if the workflow has a top-level permissions key
# Single-job workflows may define permissions at job level instead
if ! echo "$decoded" | grep -qE '^permissions:'; then
# Count jobs and check if the single job has job-level permissions
local job_count
job_count=$(echo "$decoded" | grep -cE '^ [a-zA-Z_-]+:$' || echo "0")
local has_job_perms
has_job_perms=$(echo "$decoded" | grep -cE '^ permissions:' || echo "0")
if [ "$job_count" -gt 1 ] || [ "$has_job_perms" -eq 0 ]; then
add_finding "$repo" "ci-workflows" "missing-permissions-$wf" "warning" \
"Workflow \`$wf\` missing top-level \`permissions:\` declaration (least-privilege policy)" \
"standards/ci-standards.md#permissions-policy"
fi
fi
done
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Job counting heuristic is fragile for complex workflows.

The regex '^ [a-zA-Z_-]+:$' at line 392 counts any 2-space indented key as a job, which could include non-job keys if workflow structure varies. Similarly, line 394's check for job-level permissions assumes exactly 4-space indentation.

This works for standard workflows but may produce false positives/negatives for unconventional formatting.

Consider using yq for more robust YAML parsing

If yq is available, parsing the actual jobs section would be more reliable:

job_count=$(echo "$decoded" | yq '.jobs | keys | length' 2>/dev/null || echo "0")

Alternatively, document the assumption about standard YAML formatting in the script comments.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/compliance-audit.sh` around lines 387 - 402, The job-counting logic
using grep on the decoded YAML is fragile; update the check in the block that
computes job_count and has_job_perms (variables: job_count, has_job_perms,
operating on decoded) to use a real YAML parser when available (prefer yq) to
read .jobs and per-job .permissions (e.g., use yq to compute job_count as length
of .jobs and to test each job's permissions) and fall back to the existing grep
heuristic only if yq is not installed; ensure you still call add_finding(repo,
"ci-workflows", "missing-permissions-$wf", ...) when no top-level permissions
and either more than one job or no job-level permissions are present, and add a
short comment documenting the yq fallback assumption.

Comment on lines +689 to +691
# Output report paths
echo "findings=$FINDINGS_FILE"
echo "summary=$SUMMARY_FILE"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Stdout outputs appear unused.

Lines 690-691 echo findings=... and summary=... to stdout, but the workflow doesn't capture these outputs. Consider removing them or redirecting to $GITHUB_OUTPUT if they're intended to be workflow outputs.

If intended as debug info, keep as-is. If intended for workflow, redirect:
-  echo "findings=$FINDINGS_FILE"
-  echo "summary=$SUMMARY_FILE"
+  # These paths are already known to the workflow via REPORT_DIR
+  info "Findings: $FINDINGS_FILE"
+  info "Summary: $SUMMARY_FILE"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@scripts/compliance-audit.sh` around lines 689 - 691, The two stdout echo
lines (echo "findings=$FINDINGS_FILE" and echo "summary=$SUMMARY_FILE") are not
captured by the workflow; either remove them if they were only debug prints or
redirect them into the GitHub Actions outputs file by appending the keys and
values to $GITHUB_OUTPUT (use FINDINGS_FILE and SUMMARY_FILE variables to
produce "findings=..." and "summary=..." entries). Update the script where those
echoes appear so the intended output is written to $GITHUB_OUTPUT when meant as
workflow outputs, otherwise delete the lines to avoid noisy unused stdout.

@don-petry
Copy link
Copy Markdown
Contributor Author

Functional Test Results

Run 1: Full Live Test (all repos)

  • 61 issues created across 6 repositories
  • Issue counts match findings exactly:
    • bmad-bgreat-suite: 19 issues
    • google-app-scripts: 11 issues
    • ContentTwin: 10 issues
    • TalkTerm: 8 issues
    • markets: 7 issues
    • broodly: 6 issues
  • All issues labeled compliance-audit
  • Issue bodies contain severity, detail, and linked standard reference

Run 2: Idempotency Test (broodly)

  • Re-ran against broodly which already had 6 open issues
  • 0 duplicates created — all 6 existing issues received a comment update instead
  • Comment format: "Weekly Compliance Audit (date) — This finding is still open."

Verified

  • Issue creation works across all org repos
  • Idempotent: updates existing issues, doesn't duplicate
  • Labels correctly applied
  • Issue body format correct with standard references
  • CI passing (CodeQL, CodeRabbit)
  • All 14 review comments resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants