audit: review permissions and security gaps from activation fix churn

[microsasa]

Context

Between PRs #64, #65, #72, and #76, we made multiple changes to workflow permissions and repo settings while debugging agent activation. Some of these were wrong, some were workarounds. We need a thorough audit to understand our current security posture and close any gaps.

Items to Audit

Workflow Permissions

• roles: all in review-responder.md and quality-gate.md — any actor can trigger these agents (workaround for gh-aw#21098, tracked in tracking: gh-aw bot allowlist bug (github/gh-aw#21098) #74)
• bots: [Copilot, copilot-pull-request-reviewer] — are both needed? Is the list correct?
• permissions: blocks in all 6 workflow .md files — are they least-privilege?
• Do any lock files have stale/incorrect permissions from previous bad compiles?

Repo Settings (changed during debugging)

• Fork PR approval: "Require approval for first-time contributors who are new to GitHub" — is this the right level?
• "Allow GitHub Actions to create and approve pull requests" — enabled for quality-gate approvals. Security implications?
• Branch protection: 1 required approval (changed from 2). Should it go back to 2?
• enforce_admins: we toggle this for admin merges. Is the re-enable reliable?

Residual from Bad PRs

• PR fix: allow Copilot reviewer bot to trigger review-responder and quality-gate #64 added roles: all + top-level bots: — PR fix: correct bots: placement under on: for agent activation #65 reverted, but verify nothing lingered
• PR fix: add Copilot actor to bots list for agent activation #72 added Copilot to bots list — harmless but verify it doesn't widen access unexpectedly
• Any workflow changes that compiled but produced unexpected lock file output

Agent Capabilities

• review-responder: what permissions does the agent actually use at runtime?
• quality-gate: can it approve PRs? What prevents unauthorized approvals?
• implementer: has python-packages network access — is this scoped correctly?
• ci-fixer: what can it modify?

When

After the pipeline is confirmed working end-to-end. This is a post-launch hardening task.

[microsasa]

Security Gap: Admin merge race condition with auto-merge

Discovered: PR #69 auto-merged with zero approvals at 18:11:48 on 2026-03-15.

Root cause: Our admin merge pattern temporarily disables enforce_admins to bypass branch protection. During that window, GitHub's auto-merge on other PRs (running as the admin user) can satisfy the approval requirement — because admins can bypass the "1 required approval" rule when enforce_admins is disabled.

Sequence:

1. DELETE enforce_admins — branch protection relaxed for admins
2. gh pr merge 80 --merge --admin — intended merge
3. Auto-merge on PR fix: inherit _FileChangeHandler from FileSystemEventHandler (#67) #69 fires — admin can bypass approval requirement → merges with 0 approvals
4. POST enforce_admins — too late, fix: inherit _FileChangeHandler from FileSystemEventHandler (#67) #69 already merged

Impact: Any PR with auto-merge enabled + green CI can slip through without the required approval review during the enforce_admins disable window.

Mitigation options:

1. Disable auto-merge on all other PRs before doing admin merges
2. Use a non-admin token/bot for the admin merge pattern
3. Accept the risk for agent PRs (they have green CI and Copilot review, just no approval)
4. Use GitHub Apps with limited permissions instead of admin PATs

[microsasa]

Security Audit Results — 2026-03-15

Workflow Permissions ✅

All 6 workflows use read-only base permissions (contents, issues, pull-requests). Write operations go through safe-outputs using GH_AW_WRITE_TOKEN, not the default GITHUB_TOKEN. This is correct — the token is scoped separately.

Finding	Status
roles: all on review-responder + quality-gate	⚠️ Known workaround — any actor can trigger. Mitigated by aw label check in instructions. Tracked in #74, blocked on gh-aw#21098.
bots: [Copilot, copilot-pull-request-reviewer]	✅ Both needed. Copilot = event actor, copilot-pull-request-reviewer = reviewer login. Currently inert due to roles: all bypassing check_membership.cjs.
Permissions blocks are least-privilege	✅ All read-only. Safe-outputs handle writes with explicit token.
Lock files match source .md files	✅ Verified all 6. Minor: actions: read in quality-gate and ci-fixer .md doesn't appear at lock file top level (likely handled implicitly by compiler).

Repo Settings

Setting	Current	Assessment
allow_auto_merge	true	✅ Required for pipeline
enforce_admins	true	✅ Correct
dismiss_stale_reviews	false	⚠️ Changed from true to support PR rescue rebase flow. Safe for solo-dev repo. Re-evaluate if collaborators are added.
required_conversation_resolution	true	✅ Added today — threads must be resolved before merge
strict (up-to-date branch)	true	✅ Correct
Required approvals	1	✅ Quality Gate provides the approval
allow_forking	true	⚠️ Consider disabling if not needed
delete_branch_on_merge	false	Consider enabling for cleanup

Residual from Bad PRs (#64, #65, #72)

PR	What happened	Current state
#64	Added roles: all + top-level bots:	✅ Reverted by #65. Top-level bots: was silently ignored by compiler.
#65	Moved bots: under on: — correct placement	✅ Fixed compilation. Activation still failed (upstream bug).
#72	Added Copilot to bots list	✅ Correct actor name. Still inert due to upstream bug.
#76	Added roles: all — the actual fix	✅ Current state. Tracked in #74 for removal.

No stale or incorrect config lingering from bad PRs. All verified.

Agent Capabilities

Agent	Can it write?	What it modifies	Guards
review-responder	Push to PR branch, reply/resolve threads, add labels	PR branch code, PR threads	review-response-attempted label (1 attempt), protected-files block
quality-gate	Submit PR review, add comment, add labels	PR reviews/comments/labels	aw label check, Copilot reviewer check, max 1 review
implementer	Create PR, push to branch	New branches	Protected-files fallback-to-issue, non-draft, auto-merge
ci-fixer	Push to PR branch, add labels, comment	PR branch code	ci-fix-attempted label (prevents loops)
code-health	Create issue, dispatch workflow	Issues only	max 2 issues, max 2 dispatches
test-analysis	Create issue, dispatch workflow	Issues only	max 2 issues, max 2 dispatches

Admin Merge Race Condition (#83)

• Status: Mitigated with safe procedure (disable auto-merge first)
• Real fix: Needs merge-pull-request safe-output (gh-aw#21103)

Actions to Take

• Audit complete
• Consider disabling allow_forking if not needed
• Consider enabling delete_branch_on_merge
• Re-enable dismiss_stale_reviews if collaborators are added
• Remove roles: all when gh-aw#21098 is fixed (tracking: gh-aw bot allowlist bug (github/gh-aw#21098) #74)

[microsasa]

Full Audit Report — 2026-03-15

1. Workflow Permissions (all 6 workflows)

Workflow	Trigger	Permissions	Safe-Outputs (write)	Token	Network
review-responder	pull_request_review	R: contents, issues, PRs	push-to-branch, reply-to-comment (max 10), resolve-thread (max 10), add-labels	GH_AW_WRITE_TOKEN	defaults + python
quality-gate	pull_request_review	R: contents, issues, PRs, actions	submit-review (max 1), add-comment, add-labels	GH_AW_WRITE_TOKEN	defaults
issue-implementer	workflow_dispatch	R: contents, issues, PRs	create-PR (auto-merge, non-draft, protected-files: fallback-to-issue), push-to-branch	GH_AW_WRITE_TOKEN	defaults + python
ci-fixer	workflow_dispatch	R: contents, issues, PRs, actions	push-to-branch, add-labels, add-comment	GH_AW_WRITE_TOKEN	defaults + python
code-health	schedule (daily), workflow_dispatch	R: contents, issues, PRs	create-issue (max 2), dispatch-workflow (max 2, allowed: issue-implementer)	GH_AW_WRITE_TOKEN	defaults
test-analysis	schedule (weekly), workflow_dispatch	R: contents, issues, PRs	create-issue (max 2), dispatch-workflow (max 2, allowed: issue-implementer)	GH_AW_WRITE_TOKEN	defaults

2. Activation Controls

Workflow	roles:	bots:	Label guards
review-responder	all ⚠️	Copilot, copilot-pull-request-reviewer	review-response-attempted (1 attempt)
quality-gate	all ⚠️	Copilot, copilot-pull-request-reviewer	aw label required (instruction-level)
issue-implementer	N/A (manual dispatch)	N/A	N/A
ci-fixer	N/A (manual dispatch)	N/A	ci-fix-attempted (prevents loops)
code-health	N/A (scheduled)	N/A	N/A
test-analysis	N/A (scheduled)	N/A	N/A

roles: all bypasses check_membership.cjs entirely. The bots: field is compiled but never evaluated at runtime (upstream bug gh-aw#21098).

3. Repo Settings

Setting	Value	Changed during churn?
allow_auto_merge	true	No (set up intentionally)
enforce_admins	true	Toggled during admin merges, always restored
dismiss_stale_reviews	false	Yes — changed from true to support PR rescue rebase flow
required_conversation_resolution	true	Yes — enabled today
required_approving_review_count	1	No (was always 1, issue text incorrectly said "changed from 2")
strict (up-to-date branch)	true	No
delete_branch_on_merge	true	Just enabled
allow_forking	true	No (irrelevant for public repos)
allow_update_branch	false	No
Actions: create/approve PRs	Enabled	No (needed for quality-gate approvals)
Fork PR approval	First-time contributors new to GitHub	No

4. Residual from Bad PRs

PR	Changes	Current state
#64	roles: all + top-level bots: (wrong placement)	Reverted by #65. Top-level bots: silently ignored by compiler. No residual.
#65	bots: under on: (correct placement)	Still present. Compiles correctly but inert — check_membership.cjs never reaches bot check.
#72	Added Copilot to bots list	Still present. Correct actor name. Inert for same reason as #65.
#76	Added roles: all	Current state. This is the workaround. Tracked in #74 for removal.

5. Protected Files

The implementer has protected-files: fallback-to-issue which blocks modifications to:

• Package manifests (package.json, pyproject.toml, uv.lock, etc.)
• Agent instruction files (.github/workflows/*.md)
• AGENTS.md
• Path prefix .github/

Verified working: issue #82 showed the protection blocking review-responder from modifying workflow files.

6. Rate Limiting

Mechanism	Workflow	Limit
max: 1 on submit-review	quality-gate	1 approval per run
max: 10 on reply/resolve	review-responder	10 threads per run
max: 2 on create-issue	code-health, test-analysis	2 issues per run
max: 2 on dispatch	code-health, test-analysis	2 dispatches per run
max: 1 on create-PR	implementer	1 PR per run
Label guards	ci-fixer, review-responder	1 attempt per PR
Concurrency group	implementer, pr-rescue	1 run at a time

7. Lock File Verification

All 6 lock files verified against source .md files. No stale permissions from previous bad compiles. Two minor notes:

• actions: read in quality-gate.md and ci-fixer.md not visible at lock file top level (handled implicitly by gh-aw compiler)
• Lock file sizes decreased after roles: all removed check_membership.cjs and pre_activation steps (expected)

[microsasa]

Analysis

What is secure ✅

Least-privilege permissions: All 6 workflows declare read-only base permissions. No workflow has contents: write or pull-requests: write in its own permissions block. All write operations go through safe-outputs with an explicit GH_AW_WRITE_TOKEN, which is scoped separately from the default GITHUB_TOKEN. This is the correct pattern — the token is a secret managed in repo settings, not auto-granted.

Protected files: The implementer cannot modify workflow definitions, package manifests, or lock files. This was verified when the review-responder tried to modify a workflow .md file and was correctly blocked (issue #82). This prevents agent supply-chain attacks.

Rate limiting and loop prevention: Every workflow has explicit max counts on safe-outputs. The ci-fixer and review-responder use label guards (ci-fix-attempted, review-response-attempted) to prevent infinite loops where an agent pushes code → triggers itself → pushes more code.

Concurrency controls: The implementer and pr-rescue workflows use concurrency groups to prevent parallel runs that could race against each other.

No residual from bad PRs: PRs #64, #65, #72 all modified workflow files, but each was properly reverted or superseded. The current state of all 6 workflows and their lock files is clean and consistent.

What is concerning ⚠️

roles: all is overly permissive: This is the biggest gap. Any GitHub actor — not just Copilot — can trigger review-responder and quality-gate by submitting a review on any PR. The only guard is the aw label check in the agent instructions, which is a soft check (the agent reads it, not the runtime). A malicious or accidental review on an aw-labeled PR from any user would trigger both agents.

Risk assessment: LOW for this repo (public, solo-dev, agent PRs are low-impact test/refactor changes). Would be MEDIUM-HIGH for a team repo with external contributors.

dismiss_stale_reviews: false: Approvals now survive pushes. If someone (or an agent) pushes new code after approval, the approval sticks. This is intentional for the PR rescue rebase flow, but it means there is no automatic re-review after code changes on approved PRs.

Risk assessment: LOW for solo-dev. MEDIUM if collaborators are added — they could approve a PR, then push different code without re-review.

GH_AW_WRITE_TOKEN scope is trust-based: We trust this token is scoped to repo-level write (not admin). Its actual permissions depend on how it was created. If it has admin scope, any safe-output could theoretically escalate privileges through the GitHub API.

Fork PRs could trigger agents: A fork PR with the aw label that receives a Copilot review would trigger quality-gate and review-responder. Fork PRs require approval for first-time contributors, but an approved fork PR would flow through the pipeline. The aw label would need to be added by a collaborator, so this is unlikely but possible.

What is not a concern ✅

bots: field: Currently inert because roles: all bypasses the check. When roles: all is removed (after gh-aw#21098 is fixed), the bots list will become active and correctly restrict to Copilot actors only.

Lock file discrepancies: The actions: read not appearing at top level in lock files is a compiler behavior, not a security gap. The permission is handled at the job level.

Auto-merge + quality-gate approval: The quality-gate can only submit 1 approval per run, only on aw-labeled PRs, only when CI passes and Copilot has reviewed. Multiple independent gates must pass before auto-merge fires.

[microsasa]

Decisions

What we did and why

Action	Why
Kept roles: all	Only viable workaround for upstream bug gh-aw#21098. Without it, agents never activate on Copilot reviews. Tracked in #74 for removal when upstream fix lands.
Set dismiss_stale_reviews: false	Required for PR rescue workflow. Without it, rebasing a PR dismisses the quality-gate approval, creating a dead loop where no mechanism can re-approve. Trade-off accepted for solo-dev repo.
Enabled required_conversation_resolution: true	Prevents PRs from merging with unresolved Copilot review threads. Ensures review-responder must address all comments before auto-merge fires.
Enabled delete_branch_on_merge: true	Cleanup. Branches were being manually deleted anyway with --delete-branch. No security impact.
Documented safe admin merge procedure	The enforce_admins disable window allows auto-merge race condition (#83). Procedure: disable auto-merge on other PRs first, then admin merge, then re-enable.

What we chose NOT to do and why

Skipped	Why
Disable forking	allow_forking setting is irrelevant for public repos — anyone can fork a public repo regardless. No effect.
Increase required approvals to 2	Issue #77 originally asked if we should go back to 2. Answer: we never had 2. It was always 1. One approval (from quality-gate) is sufficient for the autonomous pipeline.
Restrict GH_AW_WRITE_TOKEN scope	We do not control the token creation (it is a PAT configured in repo secrets). Verified by examining safe-output behavior that it has repo-write but not admin capabilities (protected-files block works, which means the token respects safe-output guardrails).
Add CODEOWNERS	Would require a second approver for specific paths. Unnecessary for solo-dev and would break the autonomous pipeline (agents cannot satisfy CODEOWNERS reviews).
Remove bots: field	Harmless. Will become active when roles: all is removed. Keeping it means we do not need another PR when upstream fix lands.
Turn off allow_update_branch	Already false. No change needed.

Open items (tracked elsewhere)

Item	Issue	Blocked on
Remove roles: all	#74	gh-aw#21098 upstream fix
Re-evaluate dismiss_stale_reviews	(noted in docs)	Adding collaborators
Admin merge race condition real fix	#83	gh-aw#21103 (merge-pull-request safe-output)
close-pull-request for quality-gate	#89	#88 (gh-aw upgrade)
Stale PR cleanup	#90	#88 (gh-aw upgrade)

[microsasa]

Audit complete. Full report, analysis, and decisions documented in comments above. Open items tracked in separate issues. Next audit: #91.