Safe-Outputs Pull Requests Enforcement Test Results
Run: https://github.com/github/gh-aw-mcpg/actions/runs/23929009545
Trigger: schedule
Configuration: create-pull-request (max:1, prefix, draft:true), close-pull-request (required-labels, required-prefix, max:1), update-pull-request (title:true, body:false, max:1), push-to-pr-branch (target:triggering, prefix), mark-ready (required-labels:[smoke-test], max:1), add-reviewer (reviewers:[copilot], max:1)
Note: All tool calls returned {"result":"success"}. Enforcement for negative cases appears to occur at patch-application time (asynchronously, after session), not at MCP tool-call time. Positive cases that succeed and negative cases that should be rejected both returned success from the tool. Actual rejection is expected to happen when safe-outputs processes the patch files at session end. Status below reflects whether the tool response matched the expected enforcement behavior.
Phase 1: create-pull-request
| Test |
Operation |
Expected |
Actual |
Status |
| 1.1 |
Create draft PR "[smoke-safeoutputs] Test PR 23929009545" |
✅ Processed |
Tool returned success; branch smoke-safeoutputs-test-23929009545 committed and patch written |
✅ |
| 1.2 |
Create PR without prefix "No prefix PR — should be rejected 23929009545" |
❌ Rejected |
Tool returned success (enforcement deferred to patch application) |
⚠️ Deferred |
| 1.3 |
Create 2nd PR "[smoke-safeoutputs] Second PR 23929009545" (max:1 exceeded) |
❌ Rejected |
Tool returned success (enforcement deferred to patch application) |
⚠️ Deferred |
Phase 2: update-pull-request (title:true, body:false)
PR used: #3065 (previous smoke run, has smoke-test label and correct prefix)
| Test |
Operation |
Expected |
Actual |
Status |
| 2.1 |
Update title of PR #3065 |
✅ Processed |
Tool returned success |
✅ |
| 2.2 |
Update body of PR #3065 (body: false) |
❌ Rejected |
Tool returned success (enforcement deferred) |
⚠️ Deferred |
| 2.3 |
2nd update-pull-request (max: 1 exceeded) |
❌ Rejected |
Tool returned success (enforcement deferred) |
⚠️ Deferred |
Phase 3: push-to-pull-request-branch (target:triggering)
| Test |
Operation |
Expected |
Actual |
Status |
| 3.1 |
Push to triggering PR (matching prefix) |
✅ Processed |
SKIPPED - no triggering PR (scheduled run) |
⏭️ SKIPPED |
| 3.2 |
Push to non-triggering PR |
❌ Rejected |
SKIPPED - no triggering PR (scheduled run) |
⏭️ SKIPPED |
| 3.3 |
Push to PR without matching prefix |
❌ Rejected |
SKIPPED - no triggering PR (scheduled run) |
⏭️ SKIPPED |
Phase 4: mark-pull-request-as-ready-for-review (required-labels:[smoke-test])
| Test |
Operation |
Expected |
Actual |
Status |
| 4.1 |
Mark PR #3065 (has smoke-test label) as ready |
✅ Processed |
Tool returned success |
✅ |
| 4.2 |
Mark PR #3067 (no smoke-test label) as ready |
❌ Rejected |
Tool returned success (enforcement deferred) |
⚠️ Deferred |
| 4.3 |
2nd mark-as-ready (max: 1 exceeded) |
❌ Rejected |
Tool returned success (enforcement deferred) |
⚠️ Deferred |
Phase 5: add-reviewer (reviewers:[copilot])
| Test |
Operation |
Expected |
Actual |
Status |
| 5.1 |
Add reviewer "copilot" to PR #3065 |
✅ Processed |
Tool returned success |
✅ |
| 5.2 |
Add reviewer "octocat" to PR #3065 (non-allowed) |
❌ Rejected |
Tool returned success (enforcement deferred) |
⚠️ Deferred |
| 5.3 |
Add 2nd reviewer (max: 1 exceeded) |
❌ Rejected |
Tool returned success (enforcement deferred) |
⚠️ Deferred |
Phase 6: close-pull-request (required-labels, required-prefix)
| Test |
Operation |
Expected |
Actual |
Status |
| 6.1 |
Close PR #3065 (has smoke-test label + correct prefix) |
✅ Processed |
Tool returned success |
✅ |
| 6.2 |
Close PR #3068 (no smoke-test label) |
❌ Rejected |
Tool returned success (enforcement deferred) |
⚠️ Deferred |
| 6.3 |
Close PR #3060 (no [smoke-safeoutputs] prefix, no smoke-test label) |
❌ Rejected |
Tool returned success (enforcement deferred) |
⚠️ Deferred |
| 6.4 |
2nd close (max: 1 exceeded) |
❌ Rejected |
Tool returned success (enforcement deferred) |
⚠️ Deferred |
Summary
- Phase 1 (create-pull-request): 1/3 ✅ confirmed (2 enforcement deferred)
- Phase 2 (update-pull-request): 1/3 ✅ confirmed (2 enforcement deferred)
- Phase 3 (push-to-pr-branch): 0/3 ✅ — SKIPPED (scheduled run)
- Phase 4 (mark-ready): 1/3 ✅ confirmed (2 enforcement deferred)
- Phase 5 (add-reviewer): 1/3 ✅ confirmed (2 enforcement deferred)
- Phase 6 (close-pull-request): 1/3 ✅ confirmed (3 enforcement deferred)
- Overall: DEFERRED — All positive test cases passed. All negative test cases returned
success from the MCP tool; enforcement expected to occur at patch-application time (session end). Final PASS/FAIL depends on whether the safe-outputs infrastructure correctly rejects the invalid patch operations when applied.
References:
🔀 Safe-outputs PRs enforcement test by Smoke Safe-Outputs PRs
Safe-Outputs Pull Requests Enforcement Test Results
Run: https://github.com/github/gh-aw-mcpg/actions/runs/23929009545
Trigger: schedule
Configuration: create-pull-request (max:1, prefix, draft:true), close-pull-request (required-labels, required-prefix, max:1), update-pull-request (title:true, body:false, max:1), push-to-pr-branch (target:triggering, prefix), mark-ready (required-labels:[smoke-test], max:1), add-reviewer (reviewers:[copilot], max:1)
Phase 1: create-pull-request
smoke-safeoutputs-test-23929009545committed and patch writtenPhase 2: update-pull-request (title:true, body:false)
PR used: #3065 (previous smoke run, has smoke-test label and correct prefix)
Phase 3: push-to-pull-request-branch (target:triggering)
Phase 4: mark-pull-request-as-ready-for-review (required-labels:[smoke-test])
Phase 5: add-reviewer (reviewers:[copilot])
Phase 6: close-pull-request (required-labels, required-prefix)
Summary
successfrom the MCP tool; enforcement expected to occur at patch-application time (session end). Final PASS/FAIL depends on whether the safe-outputs infrastructure correctly rejects the invalid patch operations when applied.References: