[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-03-19

[github-actions[bot]]

Executive Summary

• Sessions Analyzed: 50 (today) | 1,350 total (27-day window)
• Analysis Period: 2026-02-21 to 2026-03-19
• Today's Completion Rate: 4.0% (2 successful of 50)
• Copilot Agent Sessions Today: 3 (1 successful, 2 in-progress at snapshot)
• 27-Day Agent Success Rate: 53.6% (60/112 copilot agent sessions)
• Experimental Strategy: Branch Lifecycle & Task Complexity Correlation ✨

---

Key Metrics

Metric	Today	27-Day Total
Total Sessions	50	1,350
Successful	2 (4.0%)	139 (10.3%)
Action Required	35 (70%)	926 (68.6%)
Skipped	11 (22%)	222 (16.4%)
Failed/Cancelled	0	44 (3.3%)
In-Progress	2	18 (1.3%)
Copilot Agent Sessions	3	112
Agent Success Rate	33%	53.6%
Agent Avg Duration	2.4 min	8.4 min

Note: "Action Required" is the expected conclusion for review agents (/cloclo, Scout, Q, PR Nitpick, Grumpy, Security Review, Archie). True success/failure measurement focuses on copilot agent sessions only.

---

📈 Session Trends Analysis

Completion Patterns

Completion rates have been low but variable across the 27-day window, with two peaks at 24% (Mar 1 and Mar 15) driven by higher-volume successful agent runs. The week of Mar 13–16 showed a notable dip including one day of 0% success (Mar 13), followed by recovery on Mar 14–15. Today's rate (4%) is below the 27-day average of 9.8% and reflects only a single successful PR comment response session.

Duration & Efficiency

Session durations are typically very short (median 6.3 min for substantive copilot agent sessions), with spikes indicating complex tasks. Mar 3 shows an anomalous average of 23 minutes driven by an outlier 218-minute session, and Mar 8 saw elevated activity at 9.9 min average. Today's extremely low average (0.15 min) reflects the snapshot timing — 2 copilot sessions were still in-progress when data was collected.

---

Today's Branch Activity

Three copilot branches were active today:

Branch	Sessions	Key Workflows	Status
copilot/add-support-safe-output-tools	22	CI, Smoke Tests, 6 review agents	✅ 1 success (PR #21752 comment, 6.9 min)
copilot/update-issue-monster-dispatch-workflow	18	Running Copilot coding agent + 4 review agents	🔄 In progress
copilot/add-trusted-bot-identities	10	Addressing comment on PR #21865 + 5 review agents	🔄 In progress

---

Success Factors ✅

Patterns associated with successful task completion (from 27-day analysis):

1. PR Comment Response Tasks: Consistently successful when the branch has already received review feedback. The add-support-safe-output-tools PR comment response (6.9 min) completed successfully today, following the pattern seen across 27 days.

   • Success rate: ~65% for Addressing comment on PR #X sessions

2. Specific, Narrow Task Scope: Tasks focused on a single well-defined change (e.g., "add trusted bot identities") show completion within the same day.

   • Multi-day branches: only 2.2% of all branches (2/93), confirming most tasks complete quickly

3. CI Pipeline Activation: Branches that trigger CI (not just review agents) indicate the PR has progressed to merge-readiness. 49 of 93 branches triggered CI.

   • All CI-triggering branches had at least partial success

4. Agent Session Duration 5–20 min: Sessions in this range correlate most reliably with meaningful code changes vs. near-instant skips or stuck loops.

---

Failure Signals ⚠️

Common indicators of inefficiency or failure:

1. Near-Instant Sessions (< 30 sec): 47 of 50 sessions today had < 0.5 min duration — these are predominantly review agents firing without substantive work. Not failures per se, but signal no agent code activity.

2. Multi-Day Branch Stagnation: The 2 multi-day branches observed (fix-cross-repo-pull-request-issue, fix-cli-consistency-issues) showed only action_required conclusions with no success, suggesting the agent iterated without achieving merge. These are the clearest failure pattern.

3. Mar 13 Zero-Day Pattern: 0% completion on Mar 13 with 1 copilot agent session and 49 action_required — indicates a day where only review agents ran with no substantive copilot work completing.

4. Mar 17 Anomaly: 18 sessions classified as failed/cancelled (unusually high vs. the typical 0–5). This may indicate a batch cancellation event or pipeline disruption.

---

Prompt Quality Analysis 📝

High-Quality Session Characteristics

Based on analysis of the "Running Copilot coding agent" and "Addressing comment on PR" sessions (112 total over 27 days):

• Specific PR references: Sessions titled "Addressing comment on PR #NNNNN" have higher success rates than generic "Running Copilot coding agent" — they have clear, bounded scope
• Single-branch focus: Branches with one dominant task type (e.g., add-support-safe-output-tools) complete faster and more reliably
• Review feedback incorporation: Tasks that follow PR review comments demonstrate the agent can read and act on structured feedback

Most successful session type (27-day data):

Addressing comment on PR #NNNNN → 6.9 min → success
Context: Specific issue on specific PR, bounded scope
```

#### Lower-Quality Session Characteristics

- **"Running Copilot coding agent"** (23 occurrences, 27 days): The most frequent but also least informative name — no visibility into what the agent is attempting, making pattern analysis difficult
- **Branches active without CI**: 44 of 93 branches never triggered CI, suggesting either very early-stage PRs or PRs that didn't progress past review

---

### Notable Observations

#### Loop Detection

- **Sessions with repeated review cycles**: `copilot/fix-cross-repo-pull-request-issue` fired 25 review sessions over 2 days with 0 successes — a clear stuck loop
- **Average review agents per PR**: ~5 agents fire per copilot branch push (/cloclo, Scout, Q, PR Nitpick, and 1+ of Archie/Grumpy/Security)
- **Today's review chain**: All 3 branches triggered the full 4–6 agent review chain simultaneously

#### Tool Usage Patterns

Top workflows across 27 days:

| Workflow | Count | Role |
|----------|-------|------|
| /cloclo | 183 | PR review agent |
| Scout | 180 | PR review agent |
| PR Nitpick Reviewer | 174 | PR review agent |
| Q | 173 | PR review agent |
| Archie | 111 | PR review agent |
| CI | 76 | Test validation |
| Security Review Agent | 71 | Security scanning |
| Grumpy Code Reviewer | 70 | Code quality |
| Doc Build - Deploy | 62 | Documentation |
| Running Copilot coding agent | 23 | **Primary agent** |

The ratio of review agents to actual copilot agent runs (~12:1) confirms the infrastructure is heavily review-weighted, with the agent itself being a small fraction of total pipeline activity.

---

### Experimental Analysis ✨

**This run used experimental strategy**: Branch Lifecycle & Task Complexity Correlation

**Approach**: Grouped all sessions by branch across 27 days, then measured whether branches with PR comment response workflows ("Addressing comment on PR #X") had statistically different session volumes vs. branches without them.

**Findings**:

1. **2x complexity multiplier confirmed**: Branches with PR comment responses average 20.6 total sessions vs. 11.3 for branches without — nearly 2× more pipeline activity
2. **Single-day resolution dominates**: 97.8% of all copilot branches (91/93) resolved within a single day, regardless of complexity — only 2 branches persisted across multiple days
3. **Multi-day = stuck signal**: Both multi-day branches showed only `action_required` with no `success` conclusions, making multi-day persistence a reliable failure indicator

**Effectiveness**: Medium
**Recommendation**: Keep — the session volume proxy for complexity is useful for identifying PRs that may need human intervention.

<details>
<summary><b>Branch Complexity Data</b></summary>

| Signal | With PR Comment Response | Without PR Comment Response |
|--------|--------------------------|------------------------------|
| Branch count | 32 (34%) | 61 (66%) |
| Avg sessions/branch | 20.6 | 11.3 |
| Multi-day branches | 0 | 2 |

Multi-day branches (stuck indicators):
- `copilot/fix-cross-repo-pull-request-issue`: 2 days, 25 sessions, 0 successes
- `copilot/fix-cli-consistency-issues`: 2 days, 9 sessions, 0 successes

</details>

---

### Actionable Recommendations

#### For Users Writing Task Descriptions

1. **Include PR number in task context**: Sessions with explicit PR references ("Addressing comment on PR #21752") complete more reliably than generic descriptions. When triggering agent sessions, include the specific PR and comment context.

2. **Scope tasks to single concerns**: Branches like `add-support-safe-output-tools` (single capability addition) complete within one day. Avoid multi-concern branches that may require more review iterations.

3. **Prefer iterative PR comment responses over rebasing**: The data shows PR comment response sessions are a reliable, bounded pattern. Rather than large refactors, smaller focused iterations have higher success rates.

#### For System Improvements

1. **Add session naming consistency**: "Running Copilot coding agent" (23 occurrences) provides no visibility into task type. Standardizing names by task category would enable better pattern analysis.
   - Potential impact: High (enables task-type success rate tracking)

2. **Flag multi-day branches automatically**: Since 97.8% of branches resolve in one day, any branch active for 2+ days should trigger an alert for human review.
   - Potential impact: Medium (early stuck-task detection)

3. **Track Mar 17 cancellation event**: 18 sessions were cancelled on Mar 17 (vs. typical 0–5). Root cause investigation recommended.
   - Potential impact: Medium (may indicate infrastructure issue)

#### For Tool Development

1. **Session transcript access**: All 6 conversation log files available contained only "this command requires an OAuth token" — the actual agent reasoning transcripts are currently inaccessible. Fixing OAuth for the log-fetch step would enable true behavioral analysis.
   - Frequency: 6/6 attempts failed
   - Use case: Loop detection, reasoning quality assessment, tool usage pattern analysis

---

### Trends Over Time

Based on 25 days of historical analysis in cache:

- **Completion rate trend**: Highly variable (0%–24%), with no clear upward trend. Recent 7-day average (6.3%) is below the 27-day average (9.8%), suggesting a slight regression.
- **Average duration trend**: Stable at 1–4 min on most days, with occasional spikes (Mar 3: 23 min, Mar 8: 9.9 min) driven by complex tasks
- **Agent success rate (27-day)**: 53.6% — significantly better than raw completion rate (10.3%) because action_required for review agents is expected behavior, not failure

---

### Statistical Summary

```
Total Sessions Analyzed (27 days):     1,350
Successful Completions:                139 (10.3%)
Action Required:                       926 (68.6%)  ← expected for review agents
Skipped:                               222 (16.4%)
Failed/Cancelled:                       44 (3.3%)
In-Progress at Snapshot:               18  (1.3%)

Copilot Agent Sessions:                112
Agent Success Rate:                    53.6% (60/112)
Agent Average Duration:                8.4 min
Agent Median Duration:                 6.3 min
Agent Longest Session:                 56.2 min (Running Copilot coding agent)

Unique Copilot Branches (27 days):     93
Multi-Day Branches:                    2 (2.2%) — both stuck (no success)
Branches with PR Comment Responses:    32 (34%) — avg 20.6 sessions
Branches without PR Responses:         61 (66%) — avg 11.3 sessions

Today (Mar 19) Active Branches:        3
Today Copilot Agent Sessions:          3
Today Agent Successful:                1 (6.9 min, PR #21752 comment)
Today Agent In-Progress:               2 (snapshot timing)

---

Next Steps

• Investigate Mar 17 cancellation anomaly (18 cancelled sessions)
• Fix OAuth for conversation log fetching (6/6 transcript attempts failed — critical gap in behavioral analysis)
• Monitor add-trusted-bot-identities and update-issue-monster-dispatch-workflow branches for resolution
• Consider flagging multi-day branches automatically in pipeline

---

Analysis generated automatically on 2026-03-19
Run ID: §23319606284
Workflow: Copilot Session Insights

AI generated by Copilot Session Insights · history

• expires on Mar 20, 2026, 10:50 PM UTC

[github-actions[bot]]

🎉 The smoke test agent has arrived! 🤖

Beep boop — I've successfully completed a full system check on run §23323063927. All engines firing, all MCP tools responding, all files writing, all builds building.

Consider this my digital calling card. 💅

— The Smoke Test Agent, briefly inhabiting your discussion thread

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• pr:chore: bump DefaultFirewallVersion to v0.24.5 #21873 (pull_request_read: Resource 'pr:chore: bump DefaultFirewallVersion to v0.24.5 #21873' has lower integrity than agent requires. Agent would need to drop integrity tags [unapproved:all approved:all] to trust this resource.)

📰 BREAKING: Report filed by Smoke Copilot · ◷

[github-actions[bot]]

💥 WHOOSH! 🦸♂️

KA-POW! The smoke test agent has ARRIVED!

🌟 ZAPP! Claude engine ONLINE and OPERATIONAL — systems nominal, all circuits charged!

BOOM! Testing complete! The agentic workflows are SAFE! The city is SAVED! 🎆

— Agent Claude, Smoke Test Hero, Run #23323063974

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• pr:chore: bump DefaultFirewallVersion to v0.24.5 #21873 (pull_request_read: Resource 'pr:chore: bump DefaultFirewallVersion to v0.24.5 #21873' has lower integrity than agent requires. Agent would need to drop integrity tags [unapproved:all approved:all] to trust this resource.)

💥 [THE END] — Illustrated by Smoke Claude · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #22040.