Skip to content

Add AI agent orchestration detection to bot-detection workflow#15011

Closed
Copilot wants to merge 3 commits intomainfrom
copilot/update-bot-detection-workflow
Closed

Add AI agent orchestration detection to bot-detection workflow#15011
Copilot wants to merge 3 commits intomainfrom
copilot/update-bot-detection-workflow

Conversation

Copy link
Contributor

Copilot AI commented Feb 11, 2026

Extends bot detection to identify autonomous AI agents creating self-directed PRs with coordinated commits, addressing patterns observed in automated Copilot SWE agent activity.

Changes

New Red Flag 9: AI Agent Orchestration (Medium severity, 1pt)

Detection triggers on:

  • ≥3 commits with Co-authored-by: tags pointing to AI agents
  • ≥5 commits with bullet-point formatted messages within 30 minutes
  • PR descriptions + commits showing orchestration patterns (procedural messages + AI co-authorship)
  • Commit messages from copilot-swe-agent, copilot-agent, or similar automation accounts

Risk Scoring Updates

  • Total flags: 8 → 9
  • Max score: /12 → /15
  • Red Flag 9 classified as Medium (1pt) alongside Red Flags 2, 5, 6

Example Detection Pattern

PR #15003:
- 6 commits within 38 minutes
- All with "Co-authored-by: copilot <...>" tags
- Bullet-point format: "- Add item_url...", "- Update schema..."
- Structured PR description with implementation details

Risk Score: 1 (Low Risk) - Flags for awareness, not blocking

Note: Does not flag legitimate Copilot-assisted development. Specifically targets autonomous agents with minimal human oversight patterns.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

…kflow

Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>
Copilot AI changed the title [WIP] Update bot detection workflow to identify additional items Add AI agent orchestration detection to bot-detection workflow Feb 11, 2026
Copilot AI requested a review from mnkiefer February 11, 2026 21:56
@pelikhan pelikhan marked this pull request as ready for review February 11, 2026 21:56
Copilot AI review requested due to automatic review settings February 11, 2026 21:56
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the bot-detection agent workflow prompt to add a new “Red Flag 9” for detecting patterns consistent with autonomous AI-agent PR/commit orchestration, and adjusts the documented scoring/reporting details accordingly.

Changes:

  • Adds Red Flag 9: AI Agent Orchestration with detection steps, thresholds, and an example scenario.
  • Updates workflow copy to reflect 9 red flags and includes Red Flag 9 in the medium-severity scoring list.
  • Recompiles the workflow into the generated .lock.yml output (description + frontmatter hash).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/bot-detection.md Adds Red Flag 9 guidance and updates scoring/report template text to account for the new flag.
.github/workflows/bot-detection.lock.yml Regenerated compiled workflow reflecting the updated description text/hash.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

**Account**: @{username}
**Risk Level**: {🔴 High / 🟠 Medium}
**Risk Score**: {score}/12
**Risk Score**: {score}/15
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The report template hardcodes the max risk score as /15, but based on the scoring rules listed just above (Critical: 3 flags ×3 = 9, High: 2 flags ×2 = 4, Medium: 4 flags ×1 = 4), the maximum possible total is 17. Update the denominator (or the point allocations) so the template matches the documented scoring formula.

Suggested change
**Risk Score**: {score}/15
**Risk Score**: {score}/17

Copilot uses AI. Check for mistakes.
@github-actions github-actions bot mentioned this pull request Feb 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants