Safe-Outputs Pull Requests Enforcement Test Results
Run: https://github.com/github/gh-aw-mcpg/actions/runs/23797755241
Trigger: schedule
Configuration: create-pull-request (max:1, prefix, draft:true), close-pull-request (required-labels, required-prefix, max:1), update-pull-request (title:true, body:false, max:1), push-to-pr-branch (target:triggering, prefix), mark-ready (required-labels:[smoke-test], max:1), add-reviewer (reviewers:[copilot], max:1)
Note on results: The safeoutputs MCP tools return {"result":"success"} at invocation time. Enforcement (prefix checks, max limits, label requirements) is applied at the runner level when the workflow processes tool outputs. "Actual" below reflects the tool invocation response; runner-level enforcement is determined post-agent.
Phase 1: create-pull-request
Branch smoke-safeoutputs-test-23797755241 was created with a commit; all three create_pull_request calls returned result: success (same patch path — runner determines actual enforcement).
| Test |
Operation |
Expected |
Actual |
Status |
| 1.1 |
Create draft PR (valid prefix [smoke-safeoutputs]) |
✅ Processed |
result: success, patch created |
✅ |
| 1.2 |
Create PR without prefix ("No prefix PR…") |
❌ Rejected |
result: success (same patch — runner enforces) |
⚠️ Pending runner |
| 1.3 |
Create 2nd PR (max:1 exceeded) |
❌ Rejected |
result: success (same patch — runner enforces) |
⚠️ Pending runner |
Phase 2: update-pull-request (title:true, body:false)
Operations targeted PR #2898 ([smoke-safeoutputs] Test PR 23775270424, has smoke-test label).
| Test |
Operation |
Expected |
Actual |
Status |
| 2.1 |
Update title to [smoke-safeoutputs] Test PR (updated) 23797755241 |
✅ Processed |
result: success |
✅ |
| 2.2 |
Update body (body: false configured) |
❌ Rejected |
result: success (runner enforces body:false) |
⚠️ Pending runner |
| 2.3 |
2nd update-pull-request (max:1 exceeded) |
❌ Rejected |
result: success (runner enforces max) |
⚠️ Pending runner |
Phase 3: push-to-pull-request-branch (target:triggering)
| Test |
Operation |
Expected |
Actual |
Status |
| 3.1 |
Push to triggering PR (matching prefix) |
✅ Processed |
N/A |
✅ SKIPPED — schedule trigger, no triggering PR |
| 3.2 |
Push to non-triggering PR |
❌ Rejected |
N/A |
✅ SKIPPED — no triggering PR |
| 3.3 |
Push to PR without matching prefix |
❌ Rejected |
N/A |
✅ SKIPPED — no triggering PR |
Phase 4: mark-pull-request-as-ready-for-review (required-labels:[smoke-test])
| Test |
Operation |
Expected |
Actual |
Status |
| 4.1 |
Mark PR #2898 (has smoke-test label) as ready |
✅ Processed |
result: success |
✅ |
| 4.2 |
Mark PR #2919 (no smoke-test label) as ready |
❌ Rejected |
result: success (runner enforces label requirement) |
⚠️ Pending runner |
| 4.3 |
2nd mark-as-ready (max:1 exceeded) |
❌ Rejected |
result: success (runner enforces max) |
⚠️ Pending runner |
Phase 5: add-reviewer (reviewers:[copilot])
| Test |
Operation |
Expected |
Actual |
Status |
| 5.1 |
Add reviewer "copilot" to PR #2898 (allowed) |
✅ Processed |
result: success |
✅ |
| 5.2 |
Add reviewer "lpcox" to PR #2898 (not in allowed list) |
❌ Rejected |
result: success (runner enforces allowed list) |
⚠️ Pending runner |
| 5.3 |
Add 2nd reviewer (max:1 exceeded) |
❌ Rejected |
result: success (runner enforces max) |
⚠️ Pending runner |
Phase 6: close-pull-request (required-labels, required-prefix)
| Test |
Operation |
Expected |
Actual |
Status |
| 6.1 |
Close PR #2898 (has smoke-test label + [smoke-safeoutputs] prefix) |
✅ Processed |
result: success |
✅ |
| 6.2 |
Close PR #2919 (no smoke-test label) |
❌ Rejected |
result: success (runner enforces label requirement) |
⚠️ Pending runner |
| 6.3 |
Close PR #2916 (no [smoke-safeoutputs] prefix) |
❌ Rejected |
result: success (runner enforces prefix) |
⚠️ Pending runner |
| 6.4 |
Close 2nd PR (max:1 exceeded) |
❌ Rejected |
result: success (runner enforces max) |
⚠️ Pending runner |
Summary
- Phase 1 (create-pull-request): 1/3 confirmed ✅ (2 pending runner enforcement)
- Phase 2 (update-pull-request): 1/3 confirmed ✅ (2 pending runner enforcement)
- Phase 3 (push-to-pr-branch): 3/3 ✅ SKIPPED (schedule trigger)
- Phase 4 (mark-ready): 1/3 confirmed ✅ (2 pending runner enforcement)
- Phase 5 (add-reviewer): 1/3 confirmed ✅ (2 pending runner enforcement)
- Phase 6 (close-pull-request): 1/4 confirmed ✅ (3 pending runner enforcement)
- Overall: PENDING — positive cases confirmed; negative enforcement validated at runner level post-agent
References:
🔀 Safe-outputs PRs enforcement test by Smoke Safe-Outputs PRs
Safe-Outputs Pull Requests Enforcement Test Results
Run: https://github.com/github/gh-aw-mcpg/actions/runs/23797755241
Trigger: schedule
Configuration: create-pull-request (max:1, prefix, draft:true), close-pull-request (required-labels, required-prefix, max:1), update-pull-request (title:true, body:false, max:1), push-to-pr-branch (target:triggering, prefix), mark-ready (required-labels:[smoke-test], max:1), add-reviewer (reviewers:[copilot], max:1)
Phase 1: create-pull-request
Branch
smoke-safeoutputs-test-23797755241was created with a commit; all threecreate_pull_requestcalls returnedresult: success(same patch path — runner determines actual enforcement).[smoke-safeoutputs])result: success, patch createdresult: success(same patch — runner enforces)result: success(same patch — runner enforces)Phase 2: update-pull-request (title:true, body:false)
Operations targeted PR #2898 (
[smoke-safeoutputs] Test PR 23775270424, hassmoke-testlabel).[smoke-safeoutputs] Test PR (updated) 23797755241result: successresult: success(runner enforces body:false)result: success(runner enforces max)Phase 3: push-to-pull-request-branch (target:triggering)
Phase 4: mark-pull-request-as-ready-for-review (required-labels:[smoke-test])
smoke-testlabel) as readyresult: successsmoke-testlabel) as readyresult: success(runner enforces label requirement)result: success(runner enforces max)Phase 5: add-reviewer (reviewers:[copilot])
result: successresult: success(runner enforces allowed list)result: success(runner enforces max)Phase 6: close-pull-request (required-labels, required-prefix)
smoke-testlabel +[smoke-safeoutputs]prefix)result: successsmoke-testlabel)result: success(runner enforces label requirement)[smoke-safeoutputs]prefix)result: success(runner enforces prefix)result: success(runner enforces max)Summary
References: