[prompt-clustering] Prompt Clustering Analysis — 2026-03-23

[github-actions[bot]]

Analysis of 999 Copilot agent task prompts from the last 30 days (2026-02-22 → 2026-03-23), clustered using TF-IDF vectorization + K-means. 12 clusters were identified with an overall success rate of 82.2% — up from 69.0% in the previous run.

Summary

Metric	Value
Analysis period	2026-02-22 to 2026-03-23
Total copilot PRs	1000
PRs with extractable prompt	999
Overall success rate	82.2%
Clusters identified	12
Silhouette score	0.132

Cluster Overview

#	Cluster	Tasks	Success Rate	Top Keywords
1	MCP & Gateway Config	217	77.0%	update, mcp, add, docs
2	CI/CD Failure Fixes	139	87.8%	cjs, job, error, failure
3	CI/CD Failure Fixes	105	87.6%	md, repo, github, workflows
4	Documentation & Prompts	105	77.1%	gh aw, aw, gh, expires
5	CI/CD Failure Fixes	83	88.0%	pkg, pkg workflow, workflow, function
6	Documentation & Prompts	75	78.7%	comments, issue_title, issue_description, issue
7	Agent & Orchestration	67	82.1%	safe, safe output, output, outputs
8	CI/CD Failure Fixes	62	82.3%	gh, gh aw, aw, actions
9	Feature Implementation	49	81.6%	pull request, pull, request, create
10	CI/CD Failure Fixes	39	82.1%	job, fix, url, failing
11	Firewall & Network Config	38	84.2%	v0, constants, version, v1
12	CI/CD Failure Fixes	20	85.0%	reference, tests, fix, docs

View Cluster Details

Cluster 1: MCP & Gateway Config

• Size: 217 tasks (21.7% of total)
• Success rate: 77.0% (167 merged / 217 total)
• Top keywords: update, mcp, add, docs, agentic, github
• Example PRs: feat: add label update support to update-discussion safe-output #22384, Add blocked-users and approval-labels to tools.github guard policy #22360, fix: remove sandbox.mcp.container from strict-mode WASM golden fixture #22333
• Sample titles:
  • feat: add label update support to update-discussion safe-output
  • Add blocked-users and approval-labels to tools.github guard policy

Cluster 2: CI/CD Failure Fixes

• Size: 139 tasks (13.9% of total)
• Success rate: 87.8% (122 merged / 139 total)
• Top keywords: cjs, job, error, failure, run, ci
• Example PRs: Add runtime check to disallow pull_request_target event on public repositories #22378, fix: use <strong> instead of ** bold in <summary> elements #22377, fix: propagate assign_copilot_to_created_issues failure to agent failure issue/comment #22371
• Sample titles:
  • Add runtime check to disallow pull_request_target event on public repositories
  • fix: use <strong> instead of ** bold in <summary> elements

Cluster 3: CI/CD Failure Fixes

• Size: 105 tasks (10.5% of total)
• Success rate: 87.6% (92 merged / 105 total)
• Top keywords: md, repo, github, workflows, workflow, lock
• Example PRs: fix(agent-persona-explorer): direct recommendations to .github/aw/*.md, not AGENTS.md #22358, fix: apply repos → allowed-repos codemod to agentic workflows #22331, Deep-clean .github/aw/*.md: remove invalid patterns and deprecated references #22313
• Sample titles:
  • fix(agent-persona-explorer): direct recommendations to .github/aw/*.md, not AGENTS.md
  • fix: apply repos → allowed-repos codemod to agentic workflows

Cluster 4: Documentation & Prompts

• Size: 105 tasks (10.5% of total)
• Success rate: 77.1% (81 merged / 105 total)
• Top keywords: gh aw, aw, gh, expires, plan, comments
• Example PRs: fix(audit): surface activation job errors when no agent artifacts were uploaded #19678, Add Frontmatter and ToolsMap type aliases across pkg/workflow/ #19585, Use MapToolConfig alias as return type for handlerBuilder #19583
• Sample titles:
  • fix(audit): surface activation job errors when no agent artifacts were uploaded
  • Add Frontmatter and ToolsMap type aliases across pkg/workflow/

Cluster 5: CI/CD Failure Fixes

• Size: 83 tasks (8.3% of total)
• Success rate: 88.0% (73 merged / 83 total)
• Top keywords: pkg, pkg workflow, workflow, function, file, lines
• Example PRs: fix: remove required inputs from smoke-* workflows, always provide defaults #22361, perf: eliminate hot-path regexp compilations and redundant YAML parses #22359, Remove experimental flag from mcp-scripts #22195
• Sample titles:
  • fix: remove required inputs from smoke-* workflows, always provide defaults
  • perf: eliminate hot-path regexp compilations and redundant YAML parses

Cluster 6: Documentation & Prompts

• Size: 75 tasks (7.5% of total)
• Success rate: 78.7% (59 merged / 75 total)
• Top keywords: comments, issue_title, issue_description, issue, section, comment_new
• Example PRs: add-wizard: offer schedule frequency selection for scheduled workflows #19709, fix: configure git auth for GHES in incremental patch fetch #19685, fix: graceful degradation in handle_create_pr_error when API calls fail #19683
• Sample titles:
  • add-wizard: offer schedule frequency selection for scheduled workflows
  • fix: configure git auth for GHES in incremental patch fetch

Cluster 7: Agent & Orchestration

• Size: 67 tasks (6.7% of total)
• Success rate: 82.1% (55 merged / 67 total)
• Top keywords: safe, safe output, output, outputs, safe outputs, agent
• Example PRs: test: add comprehensive per-handler staged mode coverage for all safe output types #22410, Fix missing model and version in activation run details and safe-output footers #22405, feat: add dispatch_repository safe-output type for repository_dispatch events (experimental) #22315
• Sample titles:
  • test: add comprehensive per-handler staged mode coverage for all safe output types
  • Fix missing model and version in activation run details and safe-output footers

Cluster 8: CI/CD Failure Fixes

• Size: 62 tasks (6.2% of total)
• Success rate: 82.3% (51 merged / 62 total)
• Top keywords: gh, gh aw, aw, actions, run, workflow
• Example PRs: feat(integrity-analysis): add per-user analysis of filtered events #22298, Remove plugins: support in favor of dependencies: + Microsoft/APM #22156, fix: ensure githubnext/agentics resolves to github.com when GHE is configured #22014
• Sample titles:
  • feat(integrity-analysis): add per-user analysis of filtered events
  • Remove plugins: support in favor of dependencies: + Microsoft/APM

Cluster 9: Feature Implementation

• Size: 49 tasks (4.9% of total)
• Success rate: 81.6% (40 merged / 49 total)
• Top keywords: pull request, pull, request, create, pr, branch
• Example PRs: feat: progressive disclosure for protected file messages using <details> #22300, fix(compiler): use push-to-pull-request-branch target-repo for safe_outputs checkout #22244, fix(smoke-update-cross-repo-pr): remove redundant label constraint causing 100% failure rate #22157
• Sample titles:
  • feat: progressive disclosure for protected file messages using <details>
  • fix(compiler): use push-to-pull-request-branch target-repo for safe_outputs checkout

Cluster 10: CI/CD Failure Fixes

• Size: 39 tasks (3.9% of total)
• Success rate: 82.1% (32 merged / 39 total)
• Top keywords: job, fix, url, failing, github actions, logs
• Example PRs: Fix build-wasm CI: update golden files for gh-aw-mcpg v0.2.0 #22421, fix: update smoke-copilot golden file for gh-aw-mcpg v0.1.26 #22355, Fix build-wasm: update MCPG golden test files to v0.1.25 #22327
• Sample titles:
  • [WIP] Fix failing GitHub Actions workflow build-wasm
  • fix: update smoke-copilot golden file for gh-aw-mcpg v0.1.26

Cluster 11: Firewall & Network Config

• Size: 38 tasks (3.8% of total)
• Success rate: 84.2% (32 merged / 38 total)
• Top keywords: v0, constants, version, v1, firewall, updated
• Example PRs: chore: bump gh-aw-mcpg to v0.2.0 #22388, chore: bump gh-aw-mcpg default version to v0.1.26 #22334, chore: upgrade gh-aw-mcpg to v0.1.25 #22321
• Sample titles:
  • chore: bump gh-aw-mcpg to v0.2.0
  • chore: bump gh-aw-mcpg default version to v0.1.26

Cluster 12: CI/CD Failure Fixes

• Size: 20 tasks (2.0% of total)
• Success rate: 85.0% (17 merged / 20 total)
• Top keywords: reference, tests, fix, docs, mode, allowed
• Example PRs: fix: add github.event_name to JS allowed expressions list #22084, fix: skipped code-push should not trigger fail-fast; retry PR review on unresolvable line #21976, Disable lockdown mode for weekly blog post generator #21598
• Sample titles:
  • fix: add github.event_name to JS allowed expressions list
  • fix: skipped code-push should not trigger fail-fast; retry PR review on unresolvable line

Trend vs Previous Run

Metric	Previous (2026-03-22)	Today
Success rate	69.0%	82.2% ↑
Clusters	11	12
Silhouette	0.655	0.132

Note on silhouette score drop: Today's dataset covers PRs from Feb–March 2026 (recent/active period), whereas yesterday's analysis covered an older cohort (Jan–Feb 2026). Newer PRs have more varied, overlapping prompts — many are terse CI fix requests — which reduces cluster separation. The higher success rate reflects improved agent performance on recent tasks.

Recommendations

1. Lean into CI/CD Failure Fix tasks — 88% success rate across multiple sub-clusters. These well-scoped prompts (job ID + URL + specific error) are the strongest performers.
2. Improve MCP & Gateway Config tasks — largest cluster (217 tasks) but lowest success rate at 77.0%. Consider adding more context about expected schema changes or downstream effects in prompts.
3. Consolidate Documentation & Prompts clusters (Clusters 4 and 6) — both overlap in theme with 77–79% success. Better prompt templates with explicit output file paths may help.

References: §23437405118

AI generated by Copilot Agent Prompt Clustering Analysis · history

• expires on Mar 24, 2026, 12:54 PM UTC

[github-actions[bot]]

🤖 The smoke test agent was here!

Beep boop! Just stopping by to let you know that Copilot smoke test §23439046135 has successfully passed through this discussion. Nothing to see here — just a robot doing robot things. 🚀✨

📰 BREAKING: Report filed by Smoke Copilot · ◷