Skip to content

Replace per-skill AI disclosure with a PreToolUse hook#125889

Open
lewing wants to merge 20 commits intodotnet:mainfrom
lewing:hooks/ai-disclosure
Open

Replace per-skill AI disclosure with a PreToolUse hook#125889
lewing wants to merge 20 commits intodotnet:mainfrom
lewing:hooks/ai-disclosure

Conversation

@lewing
Copy link
Copy Markdown
Member

@lewing lewing commented Mar 21, 2026 •
edited
Loading

Replace per-skill AI disclosure with a single PreToolUse hook

Replaces the per-skill duplication from #125842 with a single PreToolUse hook that reminds about AI-generated content disclosure at the point of action.

What changed

  • Added .github/hooks/hooks.json — a single PreToolUse hook (Copilot CLI format) with command (Node.js), bash, and powershell variants that fires on write-verb tool calls (Bash|PowerShell|create|comment|update|edit|review|push|merge) and checks for gh pr/issue commands or GitHub MCP tool usage
  • Added .claude/settings.json — equivalent Claude Code hook configuration
  • Kept the full "AI-Generated Content Disclosure" section in copilot-instructions.md (unchanged from Require AI-generated content disclosure in copilot instructions and skills #125842) — this is the source of truth
  • Removed the duplicated disclosure text from 6 skill files:
    • api-proposal/SKILL.md
    • ci-analysis/SKILL.md
    • code-review/SKILL.md
    • issue-triage/SKILL.md
    • performance-benchmark/SKILL.md
    • vmr-codeflow-status/SKILL.md

Design

  • The hook is a nudge, not a gate — it writes to stderr and always exits 0, so it never blocks tool execution
  • A single consolidated hook covers both gh CLI commands and GitHub MCP tools, avoiding doubled context overhead from running two hooks on every tool call
  • The short message (~15 tokens) avoids accumulating tokens in long sessions, pointing to the full rule in copilot-instructions.md
  • Cross-tool compatible: Copilot CLI (v1.0.6+), VS Code, and Claude Code

Note

This PR was drafted with the help of an AI agent.

cc @stephentoub

@lewing
Replace per-skill AI disclosure with a PreToolUse hook
148082d 
Move the AI-generated content disclosure from duplicated text in
copilot-instructions.md and 6 SKILL.md files into a single
.github/hooks/hooks.json PreToolUse hook that fires when gh CLI
commands post to GitHub (pr/issue create/comment/edit/review).

This eliminates token waste from repeated instructions and ensures
the disclosure applies uniformly to all GitHub-posting actions,
including any future skills that use gh.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 21, 2026 15:20
@github-actions github-actions Bot added the area-skills Agent Skills label Mar 21, 2026
@lewing lewing marked this pull request as draft March 21, 2026 15:22
Copilot AI reviewed Mar 21, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR centralizes the repository’s “AI-generated content disclosure” reminder by removing duplicated disclosure paragraphs from Copilot instructions and multiple skill prompts, and introducing a single lifecycle hook intended to trigger only when gh is used to post to GitHub.

Changes:

  • Added a .github/hooks/hooks.json PreToolUse hook to detect gh pr/issue {create,comment,edit,review,merge} and emit a disclosure reminder.
  • Removed the duplicated disclosure text from .github/copilot-instructions.md and several .github/skills/*/SKILL.md files.
  • Kept skill workflows otherwise unchanged, relying on the hook as the new single source of truth.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
.github/hooks/hooks.json Adds a PreToolUse hook for Bash that detects certain gh commands and prints a disclosure reminder.
.github/copilot-instructions.md Removes the “AI-Generated Content Disclosure” section from the main Copilot instructions.
.github/skills/api-proposal/SKILL.md Removes the per-skill disclosure paragraph from the publish phase guidance.
.github/skills/ci-analysis/SKILL.md Removes the per-skill disclosure paragraph near the top-level workflow guidance.
.github/skills/code-review/SKILL.md Removes the per-skill disclosure paragraph from the review output format guidance.
.github/skills/issue-triage/SKILL.md Removes the per-skill disclosure paragraph from the anti-patterns section.
.github/skills/performance-benchmark/SKILL.md Removes the per-skill disclosure paragraph near EgorBot posting instructions.
.github/skills/vmr-codeflow-status/SKILL.md Removes the per-skill disclosure paragraph near the top-level workflow guidance.

Comment thread .github/hooks/hooks.json Outdated
Comment thread .github/hooks/hooks.json Outdated
Comment thread .github/copilot-instructions.md
@lewing
Address review: pure shell hook, broader matcher, keep fallback pointer
691492f 
- Replace node -e with grep+shell (no external runtime dependency)
- Broaden matcher to Bash|PowerShell for cross-platform coverage
- Add one-line disclosure pointer back to copilot-instructions.md
  for discoverability in environments without hook support

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lewing
lewing commented Mar 21, 2026
View reviewed changes
Copy link
Copy Markdown
Member Author

@lewing lewing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed all three review comments in 691492f:

  1. Node dependency → Replaced node -e with pure shell (grep -qE ... && echo >&2; :). No external runtime needed, always exits 0.
  2. Matcher scope → Broadened to "Bash|PowerShell" for cross-platform coverage.
  3. Discoverability → Added a one-line pointer back in copilot-instructions.md linking to .github/hooks/hooks.json so the rule is visible even in environments without hook support.

Note

This comment was generated with the help of Copilot.

@lewing lewing requested a review from Copilot March 21, 2026 15:29
@lewing
Use official hook format with bash/powershell keys
d740af9 
Use the version 1 hooks schema with separate bash and powershell
keys for cross-platform support. Pure shell builtins on both
platforms — no dependency on grep, node, or other external tools.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI reviewed Mar 21, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Comment thread .github/hooks/hooks.json Outdated
Comment thread .github/hooks/hooks.json Outdated
Comment thread .github/copilot-instructions.md Outdated
lewing and others added 2 commits March 21, 2026 10:34
@lewing
Soften disclosure wording: reminder, not enforcement
081b325 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lewing
Broaden hook to cover all GitHub interactions, not just gh CLI
f750f38 
- Shell hook: match any command mentioning 'github' or 'gh' (covers
  gh CLI, curl to api.github.com, etc.)
- GitHub MCP tools hook: unconditional reminder for any tool with
  'github' in the name
- copilot-instructions.md: remove gh-specific wording; the rule
  applies to all GitHub posting methods

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 21, 2026 15:39
Copilot AI reviewed Mar 21, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comment thread .github/hooks/hooks.json Outdated
lewing and others added 2 commits March 21, 2026 10:49
@lewing
Drop MCP tool hook — keep it simple
8605e50 
The copilot-instructions.md rule is the source of truth. The shell
hook is a best-effort nudge when gh/github commands run, not a gate.
Remove the overly broad GitHub MCP tool matcher that would spam on
every read-only query.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lewing
Add write-only MCP tool hook alongside shell hook
44b9d43 
Two hooks:
1. Shell (Bash|PowerShell): fires when command mentions github/gh
2. MCP tools: matcher catches write verbs (create|comment|update|
   edit|review|push|merge), then body checks for 'github' in the
   tool args to avoid false positives from non-GitHub tools

No metadata on MCP tools to distinguish read vs write, so we match
on tool name patterns containing write-like verbs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 21, 2026 15:52
Copilot AI reviewed Mar 21, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comment thread .github/hooks/hooks.json Outdated
lewing and others added 2 commits March 21, 2026 11:00
@lewing
Tighten hook: write-only gh subcommands + short message
1c10b95 
Shell hook now only fires on gh pr/issue create/comment/review/edit/
merge/close — not on reads like 'gh pr view' or 'gh issue list'.
Message shortened to ~15 tokens referencing copilot-instructions.md
for the full rule, avoiding context bloat in long sessions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lewing
Restore full disclosure section in copilot-instructions.md
5a6fd89 
The hook message is a short pointer ('per copilot-instructions.md'),
so the full rule needs to live here. Restores the original section
from dotnet#125842. The per-skill duplicates remain removed — the hook
provides the point-of-action reminder, this section is the source
of truth.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 21, 2026 16:03
Copilot AI reviewed Mar 21, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

@lewing
Copy link
Copy Markdown
Member Author

lewing commented Mar 22, 2026

opened microsoft/vscode#303928

Copilot AI review requested due to automatic review settings March 28, 2026 19:34
Copilot AI reviewed Mar 28, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

@build-analysis build-analysis Bot mentioned this pull request Mar 28, 2026
Open
@lewing lewing marked this pull request as ready for review April 1, 2026 21:59
danmoseley
danmoseley approved these changes Apr 2, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@danmoseley danmoseley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK but I'm a bit unclear how this is an improvement?

@lewing
Copy link
Copy Markdown
Member Author

lewing commented Apr 2, 2026

OK but I'm a bit unclear how this is an improvement

it avoids sprinkling the same text in a bunch of places that aren't related?

@build-analysis build-analysis Bot mentioned this pull request Apr 12, 2026
Open
Copilot AI review requested due to automatic review settings April 16, 2026 03:45
Copilot AI reviewed Apr 16, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@build-analysis build-analysis Bot mentioned this pull request Apr 16, 2026
Open
This was referenced Apr 16, 2026
Open
Closed
Copilot AI review requested due to automatic review settings April 20, 2026 20:11
Copilot AI reviewed Apr 20, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comment thread .github/skills/issue-triage/SKILL.md
Comment on lines 451 to +455
When posting any content to GitHub under a user's credentials (not a dedicated bot account), you **MUST** include a concise, visible note (e.g. a `> [!NOTE]` alert) indicating the content was AI/Copilot-generated. Skip this if the user explicitly asks you to omit it.

- **NEVER** use `gh issue close`, `gh issue edit`, `gh issue comment`, or `gh pr review --approve`/`--request-changes` **unless the user explicitly asks you to** after picking an outcome.

- **Security concerns are out of scope.** Do not assess, discuss, or make recommendations about potential security implications. Security assessment is handled through separate processes.
- **NEVER** use `gh issue close`, `gh issue edit`, `gh issue comment`, or
`gh pr review --approve`/`--request-changes` **unless the user explicitly
asks you to** after picking an outcome.
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says the duplicated AI disclosure text was removed from issue-triage/SKILL.md, but this file still contains the per-skill disclosure guidance (line 451). Either remove this line to fully rely on the new PreToolUse hook + .github/copilot-instructions.md, or update the PR description to reflect that issue-triage intentionally keeps the explicit disclosure text.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member Author

@lewing lewing Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot apply changes based on this feedback

Copy link
Copy Markdown
Member Author

@lewing lewing Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot please remove the ai disclosure text here too

@build-analysis build-analysis Bot mentioned this pull request Apr 21, 2026
Open
lewing and others added 2 commits April 24, 2026 13:32
@lewing
Merge upstream/main: resolve conflicts
82d68d6 
- .claude/settings.json: merged hooks (from PR) with marketplace config (from upstream)
- ci-analysis/SKILL.md: accept upstream deletion (skill moved to dotnet/arcade-skills)
- vmr-codeflow-status/SKILL.md: accept upstream deletion (skill moved to dotnet/arcade-skills)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lewing
Merge branch 'main' into hooks/ai-disclosure
b4c5f18
Copilot AI review requested due to automatic review settings April 27, 2026 22:50
lewing

This comment was marked as outdated.

Sign in to view

@lewing

This comment was marked as low quality.

Sign in to view
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@danmoseley danmoseley danmoseley approved these changes

@stephentoub stephentoub Awaiting requested review from stephentoub

Copilot code review Copilot Awaiting requested review from Copilot

Assignees

@lewing lewing

Labels

area-skills Agent Skills

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lewing @danmoseley