[prompt-analysis] Copilot PR Prompt Analysis - 2026-04-04

[github-actions[bot]]

Summary

Analysis Period: Last 30 days | Total PRs: 1,000 | Merged: 829 (83.1%) | Closed: 169 (16.9%) | Open: 2

Overall, Copilot PRs in this repository have a strong 83.1% merge rate, consistent with previous analysis runs (83.5% on Apr 1, 82.5% on Apr 2).

---

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Bug Fix	896	746	150	83.3%
Feature Addition	80	66	14	82.5%
Refactoring/Upgrade	14	11	3	78.6%
Documentation	5	4	1	80.0%
Tests	1	1	0	100.0%
Other	2	1	1	50.0%

Note: The dominant category is Bug Fix (89.6% of all PRs), largely driven by the automated workflow prompt template that includes "fix", "error", "resolve" keywords.

---

Prompt Pattern Analysis

✅ Successful Prompt Patterns

Characteristics of merged PRs:

• Conventional commit titles: 58.7% of merged PRs use fix:, feat:, chore:, refactor: prefixes vs only 39.1% of closed PRs — a 19.6 percentage point advantage
• Inline code references: 99.6% of merged PRs contain backtick code references vs 94.1% of closed PRs
• Average body length: ~822 words (slightly more detailed than closed PRs at 794 words)
• Low checklist usage: Only 2.3% of merged PRs contain task checklists (vs 18.3% of closed PRs)

Example merged PRs:

1. PR fix: format pkg/cli/remote_workflow_test.go with go fmt #23450: fix: format pkg/cli/remote_workflow_test.go with go fmt → Merged — Specific file, clear action
2. PR chore: Bump AWF firewall version to v0.25.11 #24133: chore: Bump AWF firewall version to v0.25.11 → Merged — Version bump with clear context
3. PR ci-doctor: remove workflow_run, stop-after, and if #22615: ci-doctor: remove workflow_run, stop-after, and if → Merged — Targeted changes enumerated

❌ Unsuccessful Prompt Patterns

Characteristics of closed PRs:

• [WIP] prefix: Multiple closed PRs use [WIP] in the title indicating incomplete work
• Vague/generic titles: "Completing task", "Install Autoloop" without specifics
• Task checklists dominate: 18.3% of closed PRs have - [ ] checklists (8x more than merged), suggesting the agent left tracking-style content in PR bodies instead of focused descriptions
• "No-op" PRs: Tracking-only PRs (version updates with no code changes) tend to be closed

Example closed PRs:

1. PR Completing task #21143: Completing task → Closed — Generic, no actionable context
2. PR No-op: CLI version tracking — Claude Code 2.1.92, Copilot CLI 1.0.18 #24440: No-op: CLI version tracking — Claude Code 2.1.92, Copilot CLI 1.0.18 → Closed — No actual code change
3. PR [WIP] Rename 'Checkout code' to 'Checkout repository' in daily-copilot-token-report #24167: [WIP] Rename 'Checkout code' to 'Checkout repository' → Closed — Work-in-progress state

---

Key Insights

• Conventional commit format strongly correlates with success: PRs using fix:, feat:, chore: etc. achieve 58.7% adoption among merged PRs vs 39.1% in closed PRs. When the workflow prompt explicitly requests conventional commit style, merge rates improve noticeably.
• Checklists signal incomplete or agent-confusion PRs: A task checklist (- [ ]) in the PR body is an 8x stronger signal of a closed PR than a merged one — these appear in PRs where the agent tracked its own tasks rather than describing the actual change.
• Specificity matters more than length: Merged and closed PRs have similar body lengths (~821 vs ~794 words), but merged PRs nearly universally contain inline code references (99.6%) — indicating that referencing specific symbols, files, and packages is key.

---

Recommendations

1. DO: Always use conventional commit format in PR titles (fix:, feat:, chore:, refactor:) — this is the single strongest predictor of success
2. DO: Include specific file paths and symbol names with backtick formatting — 99.6% of merged PRs do this
3. DO: Write focused, declarative PR descriptions (what changed and why) rather than task checklists
4. AVOID: [WIP] prefix — commit work-in-progress only when ready for review
5. AVOID: "No-op" or tracking-only PRs without actual code changes — these are consistently closed
6. AVOID: Task-list style PR bodies (- [ ] step 1, - [ ] step 2) — these indicate agent confusion rather than clear change descriptions

---

Historical Trends

Date	Total PRs	Merged	Success Rate
2026-04-01	1,000	832	83.5%
2026-04-02	1,000	823	82.5%
2026-04-04	1,000	829	83.1%

Trend: The 30-day merge rate has stabilized in the 82–84% range. The slight uptick today (+0.6pp from Apr 2) suggests no significant regression. The ratio of conventional-commit-titled PRs continues to differentiate merged from closed consistently across all analysis runs.

View Full Category Breakdown

Category	Merged	Closed	Rate	Notes
bug_fix	746	150	83.3%	Largest category; driven by automated workflow prompts
feature	66	14	82.5%	Includes add, implement, create keywords
refactor	11	3	78.6%	Includes upgrade, bump, migrate
docs	4	1	80.0%	Small sample; includes readme, document
tests	1	0	100%	Very small sample
other	1	1	50.0%	Uncategorized

View Methodology Notes

Data Source: Pre-fetched copilot/* branch PRs from the last 30 days (1,000 PRs total).

Categorization: Keyword-based classification of PR titles + bodies. Due to the automated prompt template used in this repository, most PRs contain broad keywords (fix, resolve, error) which biases toward the bug_fix category.

Conventional commit detection: Regex match on ^(fix|feat|chore|refactor|docs|test|ci|build|perf|style|revert)(\(.+\))?: in PR title.

Checklist detection: Presence of - [ pattern in PR body.

References:

• §23971603751

Generated by Copilot PR Prompt Pattern Analysis · ● 739.6K · ◷

• expires on Apr 5, 2026, 4:51 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #24672.