[prompt-analysis] Copilot PR Prompt Analysis - 2026-03-20

[github-actions[bot]]

Analysis of 1,000 Copilot-generated PRs from the last 30 days (github/gh-aw), focusing on which prompt characteristics lead to successful merges.

Summary

Metric	Count	Percentage
Total PRs	1,000	—
✅ Merged	821	82.1%
❌ Closed	174	17.4%
🔄 Open	5	0.5%

Overall success rate: 82.1% — a high baseline reflecting well-scoped, actionable prompts across this repo.

---

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
🐛 Bug Fix	732	614	83.9%
🔧 Refactor	12	10	83.3%
📝 Documentation	75	61	81.3%
✨ Feature	16	13	81.2%
🧪 Testing	153	119	77.8%
❓ Other (unclassified)	7	4	57.1%

Bug fix prompts dominate (73% of all PRs) and perform best. Testing PRs lag slightly, possibly because test additions are more likely to surface integration complexity.

---

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt body: ~910 words
• Most common title keywords: fix, add, feat, workflow, safe, outputs, mcp, cli
• 46% use conventional commit format (fix:, feat:, refactor:) in the title
• Only 2.2% are labeled WIP

Body length vs success rate:

Prompt Length	Merged	Closed	Success Rate
Empty	4	1	80%
< 50 words	3	3	50%
50–200 words	177	45	80%
200–500 words	209	43	83%
500–1000 words	134	22	86% ← peak
> 1000 words	294	60	83%

Key finding: Prompts in the 500–1000 word range achieve the highest success rate (86%). Very short prompts (< 50 words) perform worst at 50%.

Example Merged PR Bodies (top 3)

#21906 — fix(docs): redirect /gh-aw (no trailing slash) to /gh-aw/

GET /gh-aw returned 404 instead of redirecting to /gh-aw/, and the resulting 404 page rendered a doubled canonical URL...

#21902 — Bump absolute maximum for update_issue and update_discussion safe outputs

The max field for update_issue and update_discussion safe output types was capped at 100 in the JSON schema, preventing legitimate batch workflows...

#21899 — feat: update gh-aw-metadata payload to v3 with agent id/mode

Summary — Updates the gh-aw-metadata comment embedded in compiled lock files from schema v2 to v3, adding four new fields...

---

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt body: ~880 words (slightly shorter than merged)
• 21.8% are labeled WIP (vs only 2.2% of merged PRs) — strongest predictor of closure
• Only 35.1% use conventional commit format (vs 46% of merged PRs)
• Common title keywords unique to closed: wip, remove, playwright, mode, default

Example Closed PR Bodies (top 3)

#21833 — fix: make pre-activation app token generation resilient to missing app installation

The Issue Monster workflow has been failing for hundreds of consecutive runs... (likely closed due to superseding fix)

#21755 — fix: raise safe output max cap from 100 to 1000

The max field on all safe output types was capped at 100 in the JSON schema, blocking legitimate batch workflows... (superseded by #21902)

#21734 — feat(spec): add bots field to GitHub MCP Access Control Specification

The GitHub MCP Server Access Control Specification lacked a way to grant bot accounts team-member-level access...

---

Key Insights

• WIP in the title is the single strongest predictor of closure — PRs with "wip" in the title are closed at 10× the rate of merged PRs (21.8% vs 2.2%). Avoid submitting work-in-progress Copilot PRs.
• Conventional commit format correlates strongly with merges — Titles like fix:, feat:, refactor: appear in 46% of merged PRs but only 35% of closed ones. Structured titles help reviewers quickly assess scope.
• Medium-length prompts (500–1000 words) are the sweet spot — Providing enough context for Copilot to act confidently without overwhelming it. Very short prompts (< 50 words) halve the success rate.
• Bug fix prompts dominate and succeed — 73% of all PRs target bugs, likely because bug fix prompts naturally include specific reproduction steps, error messages, and scope constraints.

---

Recommendations

1. DO: Use conventional commit format in PR titles (fix:, feat:, docs:, refactor:) — it signals clear scope and correlates with 11% higher merge rate.
2. DO: Write prompts in the 500–1000 word range — include the problem statement, affected files/functions, expected behavior, and any constraints.
3. DO: Reference specific error messages, file paths, or function names — bug fix prompts succeed because they anchor Copilot to concrete evidence.
4. AVOID: Submitting PRs with "WIP" in the title — these are closed at 10× the rate. If the work isn't ready, don't submit.
5. AVOID: Vague one-liner prompts (< 50 words) — these have only a 50% success rate, the lowest of any length bucket.

---

Historical Trend (last 7 days)

Date	Success Rate
2026-03-14	79.0%
2026-03-15	79.7%
2026-03-16	80.3%
2026-03-17	81.2%
2026-03-18	82.0%
2026-03-19	81.9%
2026-03-20	82.5% ↑

Trend: Gradual improvement (+3.5 percentage points over 7 days). The overall success rate has been steadily climbing, suggesting prompts in this repo are getting more targeted and specific over time.

---

References: §23329758301

AI generated by Copilot PR Prompt Pattern Analysis · history

• expires on Mar 21, 2026, 5:07 AM UTC

[github-actions[bot]]

🤖 Smoke test agent was here! Just popped in from the agentic vortex to say hello 👋 All systems nominal. beep boop Carry on, humans! 🚀

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• pr:Skip write permissions for staged safe output handlers #21903 (pull_request_read: Resource 'pr:Skip write permissions for staged safe output handlers #21903' has lower integrity than agent requires. Agent would need to drop integrity tags [unapproved:all approved:all] to trust this resource.)

📰 BREAKING: Report filed by Smoke Copilot · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #22089.