Enable conversation transcript access for Copilot Session Insights

[Copilot]

Copilot Session Insights workflow was analyzing GitHub Actions infrastructure logs, which lack agent conversation content needed for behavioral pattern analysis.

Changes

shared/copilot-session-data-fetch.md

• Fetch conversation transcripts via gh agent-task view --log instead of workflow run logs
• Extract session IDs from branch names (copilot/issue-123 → 123)
• Save as {session_number}-conversation.txt with agent internal monologue, reasoning, and tool usage
• Fallback to GitHub Actions logs if gh agent-task unavailable (requires gh CLI v2.80.0+)

copilot-session-insights.md

• Guide analysis of conversation logs: reasoning patterns, tool effectiveness, error recovery, prompt understanding
• Add conversation log access patterns and behavioral insights to extract

Example

# Before: Downloaded GitHub Actions workflow logs (infrastructure only)
gh api "repos/$REPO/actions/runs/${run_id}/logs" > logs.zip

# After: Fetch agent conversation transcript with reasoning
gh agent-task view --repo $REPO 123 --log > logs/123-conversation.txt

Conversation logs contain:

• Agent's step-by-step reasoning and planning
• Tool calls and results
• Code changes and validation attempts
• Error handling and recovery strategies

Enables true behavioral analysis instead of infrastructure metrics.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[plan] Add Conversation Transcript Access for Copilot Session Insights</issue_title>
<issue_description>## Objective

Enable Copilot Session Insights workflow to access Copilot agent conversation transcripts for behavioral pattern analysis, or redesign workflow to work with available data.

Context

From Discussion github/gh-aw#14345, the Copilot Session Insights workflow §21779587315 reported missing data:

Missing: Copilot agent session conversation transcripts (user prompts, agent responses, tool calls)

Impact: Cannot perform:

• Behavioral pattern analysis
• Prompt quality assessment
• Success factor identification

Currently only GitHub Actions infrastructure logs are available, which lack the actual conversation content needed for insights.

Approach

1. Research available data sources for Copilot agent conversations:
   • GitHub API endpoints for agent session data
   • Copilot telemetry/analytics APIs
   • Alternative data collection methods
2. Evaluate feasibility of accessing conversation transcripts:
   • Required permissions/scopes
   • API availability and stability
   • Data format and completeness
3. If conversation data is accessible:
   • Implement data collection in workflow
   • Add necessary authentication/permissions
   • Parse and analyze conversation patterns
4. If conversation data is NOT accessible:
   • Redesign workflow to work with available data (GitHub Actions logs, workflow metrics)
   • Document limitations and adjust workflow goals
   • Consider alternative approaches for gaining insights

Files to Review

• .github/workflows/copilot-session-insights.md - Workflow definition
• Copilot API documentation for available data sources
• Related data collection utilities

Acceptance Criteria

• Research completed on available Copilot conversation data sources
• Feasibility assessment documented
• If data accessible: implement data collection and analysis
• If data not accessible: redesign workflow with realistic scope
• Test workflow completes successfully with chosen approach
• Update workflow documentation with data source details and limitations

AI generated by Plan Command for discussion #14345

• expires on Feb 9, 2026, 2:05 PM UTC

Comments on the Issue (you are @copilot in this section)

• Fixes [plan] Add Conversation Transcript Access for Copilot Session Insights #14352

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[Copilot]

Pull request overview

Updates the Copilot Session Insights workflow to fetch and analyze Copilot agent conversation transcripts (via gh agent-task view --log) instead of relying on GitHub Actions run logs, enabling behavior-focused analysis (reasoning, tool usage, recovery patterns).

Changes:

• Extend the shared session data fetch module to download *-conversation.txt transcripts using gh agent-task view --log, with an Actions-logs fallback.
• Update the Session Insights workflow guidance to incorporate conversation transcript analysis.
• Regenerate the compiled workflow lockfile to reflect the new fetch behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
.github/workflows/shared/copilot-session-data-fetch.md	Switches log acquisition to gh agent-task view --log, adds fallback behavior, and updates documentation/output expectations.
.github/workflows/copilot-session-insights.md	Updates analysis instructions to leverage conversation transcript files and outlines what behavioral signals to extract.
.github/workflows/copilot-session-insights.lock.yml	Recompiled workflow output reflecting the updated shared fetch module logic.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The gh agent-task view ... > ".../${session_number}-conversation.txt" 2>&1 redirection will create/overwrite the conversation file even when the command fails, leaving an error message in place of a transcript. Since the fallback path doesn’t remove or distinguish that file, downstream analysis may treat an error output as a valid transcript. Write to a temp file and only move it into place on success (or remove the output file on failure).

[Copilot]

The new tooling permissions are very broad ("gh agent-task *" and especially "cat *"). Since this workflow is already using an allowlist approach, consider tightening these patterns to only the subcommands/paths actually needed (e.g., gh agent-task view ... and cat /tmp/gh-aw/session-data/logs/*) to reduce the blast radius if the agent is compromised or mis-prompted.

[Copilot]

Because the cache key stays the same (copilot-session-data) while the cached log contents/format changed (now transcripts instead of extracted zip directories), workflows may reuse an existing same-day cache that contains only the old Actions log data and never attempt to fetch transcripts. Consider bumping the cache key/version or adding a cache-hit validation step that fetches transcripts when *-conversation.txt files are missing.

[Copilot]

The workflow instructions suggest running find ... | wc -l and cat /tmp/gh-aw/session-data/logs/123-conversation.txt, but the tools.bash allowlist currently doesn’t include wc and may not allow these more specific find/cat invocations (e.g., it only allows find /tmp -type f and cat /tmp/*). Update the allowlist to include the exact commands/patterns needed to read and count conversation transcripts so the agent can follow the documented process.

Suggested change

	# List available conversation logs
	find /tmp/gh-aw/session-data/logs -type f -name "*-conversation.txt"
	# Read a specific conversation log
	cat /tmp/gh-aw/session-data/logs/123-conversation.txt
	# Count conversation logs
	find /tmp/gh-aw/session-data/logs -type f -name "*-conversation.txt" | wc -l
	# List available files under /tmp (includes conversation logs under /tmp/gh-aw/session-data/logs)
	find /tmp -type f
	# Read a specific conversation log
	# Replace PATH_WITH_CONVERSATION_LOG with an actual path from the find output
	cat /tmp/*
	# Count conversation logs
	# Run find and count the matching conversation log paths from its output (without using wc)
	find /tmp -type f