Add validation script and nested AGENTS.md to enforce agent skill quality#53721
Add validation script and nested AGENTS.md to enforce agent skill quality#53721lbussell merged 5 commits intodotnet:release/10.0.3xxfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds repository-local guidance and validation to improve Agent Skill quality/consistency, and migrates the incremental test skill to the spec-compliant .github/skills/ layout.
Changes:
- Add
.github/skills/AGENTS.mdwith structure + checklist for creating skills. - Add a
ValidateSkill.csfile-based app to validate basicSKILL.mdfrontmatter/size requirements. - Move/replace the
incremental-testskill content into.github/skills/incremental-test/and add a supporting copy script.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/ValidateSkill.cs | New validator script for SKILL.md frontmatter and size rules. |
| .github/skills/incremental-test/SKILL.md | New spec-style skill document for incremental dotnet.Tests runs. |
| .github/skills/incremental-test/scripts/Copy-ToRedist.ps1 | New helper script to copy built outputs into the redist SDK layout. |
| .github/skills/AGENTS.md | New contributor guidance/checklist for skills under .github/skills/. |
| .github/copilot/skills/incremental-test.md | Removes old skill doc from the previous location. |
| .github/copilot-instructions.md | Updates docs to reference the incremental-test skill instead of a specific file path. |
| IDeserializer deserializer = new DeserializerBuilder().Build(); | ||
| Dictionary<string, object> frontmatter = deserializer.Deserialize<Dictionary<string, object>>(yaml); | ||
|
|
There was a problem hiding this comment.
Deserialize<Dictionary<string, object>> can throw on invalid YAML, which will surface as an unhandled exception/stack trace rather than a clear validation failure. Catch YAML parsing exceptions and return exit code 1 with a concise error message.
| int lineCount = text.Split('\n').Length; | ||
| if (lineCount > 500) | ||
| { | ||
| Console.Error.WriteLine($"SKILL.md is {lineCount} lines (max 500). See \"Progressive Disclosure\" at https://agentskills.io/specification.md"); | ||
| return 1; |
There was a problem hiding this comment.
text.Split('\n').Length can overcount by 1 when the file ends with a trailing newline (common in repo files), potentially rejecting a 500-line file as 501. Use File.ReadLines(...).Count() or a split that removes empty trailing entries / handles CRLF consistently.
Where is this specified? Copilot CLI and Copilot in VSCode found and used my skill without a problem. |
|
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
/ba-g |
|
you have to provide a reason, like this: |
|
/ba-g docs-only change |
|
Thanks @jjonescz! |
#53479 attempted to introduce an agent skill but it was not properly formatted or located in the correct directory. It also did not follow some of the Agent Skills best practices.
This PR adds
.github/skills/AGENTS.mdwhich is intended to improve skill quality. It also adds a script that validates basic SKILL.md formatting requirements based on the specification.Then in a new session, I had Copilot fix the incremental-test skill based on the spec + best practices. /cc @jjonescz
This should also be backported to
main.