From c8f47733688a00861db2d3343a595dbf211c70b3 Mon Sep 17 00:00:00 2001 From: Shane Neuville Date: Fri, 24 Apr 2026 15:48:32 -0500 Subject: [PATCH] fix: drift agent now exhaustively covers all release features MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes to ensure complete drift coverage: 1. Capped untracked_closed_issues to 20 in JSON output (was 7200+ at 1.2MB — the agent wasted turns parsing noise instead of reading release notes) 2. Restructured prompt with explicit 6-step workflow: - Extract release notes with python3 (not raw cat of 2.6MB) - Build exhaustive checklist of ALL changes before editing - Classify P0/P1/P2/P3 — implement ALL P0-P2, only skip P3 - Where-to-add guide for each change type - Verify completeness before committing - PR body must list every item implemented AND skipped 3. Removed 'note P2 in description' instruction that caused the agent to skip new features instead of implementing them Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../scripts/Check-Staleness.ps1 | 9 +- .github/workflows/instruction-drift.agent.md | 84 +++++++++++++++---- 2 files changed, 75 insertions(+), 18 deletions(-) diff --git a/.claude/skills/instruction-drift/scripts/Check-Staleness.ps1 b/.claude/skills/instruction-drift/scripts/Check-Staleness.ps1 index efb2993c0..3b643e9a1 100644 --- a/.claude/skills/instruction-drift/scripts/Check-Staleness.ps1 +++ b/.claude/skills/instruction-drift/scripts/Check-Staleness.ps1 @@ -683,11 +683,16 @@ foreach ($manifestPath in $manifests) { $untrackedCount = ($closedResult.untracked | Measure-Object).Count if ($untrackedCount -gt 0) { Write-Host " ⚠️ Found $untrackedCount recently closed issue(s) not in manifest:" -ForegroundColor Yellow - foreach ($issue in $closedResult.untracked) { + foreach ($issue in ($closedResult.untracked | Select-Object -First 10)) { $labels = if ($issue.labels) { " [$($issue.labels -join ', ')]" } else { '' } Write-Host " #$($issue.number): $($issue.title)$labels" -ForegroundColor Yellow } - $results[-1].untracked_closed_issues = @($closedResult.untracked) + if ($untrackedCount -gt 10) { + Write-Host " ... and $($untrackedCount - 10) more" -ForegroundColor Yellow + } + # Cap at 20 in JSON to avoid bloating the report + $results[-1].untracked_closed_issues = @($closedResult.untracked | Select-Object -First 20) + $results[-1].untracked_closed_issues_total = $untrackedCount } else { Write-Host " ✅ No new closed issues (checked $($closedResult.total_checked) in last 90 days)" -ForegroundColor Green diff --git a/.github/workflows/instruction-drift.agent.md b/.github/workflows/instruction-drift.agent.md index 395b3dea7..c49c20130 100644 --- a/.github/workflows/instruction-drift.agent.md +++ b/.github/workflows/instruction-drift.agent.md @@ -81,14 +81,25 @@ timeout-minutes: 30 The staleness check ran in `steps:` (before you started) with authenticated `gh` CLI. **Do NOT run Check-Staleness.ps1 yourself** — `gh` CLI is not authenticated inside this container. -Read the results files: +Read the results. The file is large — extract only the releases and key signals: ```bash -cat /tmp/drift-results/staleness.json +# Check if changes were detected +python3 -c "import json; d=json.load(open('/tmp/drift-results/staleness.json')); print('changes_detected:', d.get('changes_detected')); [print(f' {s[\"type\"]}: {s.get(\"last_reviewed_release\",\"\")} -> releases={len(s.get(\"result\",{}).get(\"releases\",[]))}') for m in d['manifests'] for s in m['sources'] if s['type']=='releases']" ``` -If the file contains `"changes_detected": true`, also read: +If changes detected, extract the release notes: ```bash -cat /tmp/drift-results/upstream.json +python3 -c " +import json +d=json.load(open('/tmp/drift-results/staleness.json')) +for m in d['manifests']: + for s in m['sources']: + if s['type']=='releases' and s.get('result',{}).get('releases'): + for r in s['result']['releases']: + print(f'=== {r[\"tag\"]} ({r[\"published_at\"]}) ===') + print(r.get('release_notes','')[:8000]) + print() +" ``` If there were errors, check: @@ -112,23 +123,59 @@ Do not create issues, PRs, or comments. Stop after calling noop. ## Analyze and Update -Read the staleness report above. For each signal, read the affected skill files and make updates: -- `.claude/skills/gh-aw-guide/SKILL.md` -- `.claude/skills/gh-aw-guide/references/architecture.md` -- `.github/instructions/gh-aw-workflows.instructions.md` -- `.github/instructions/gh-aw-workflows.sync.yaml` +### Step 2: Build a checklist of ALL changes from release notes + +Before editing any files, read EVERY release note above and build a complete list of changes. For each item, classify: +- **P0** = factually wrong in our docs +- **P1** = security-relevant change +- **P2** = new feature, new safe output, new config option, behavior change, bug fix that affects workflow authors +- **P3** = internal/cosmetic only (OTLP traces, CI pipeline changes, internal refactors) + +**You MUST implement P0, P1, AND P2 items. Only skip P3.** + +### Step 3: Read current skill files + +Read ALL of these to understand what we currently cover: +```bash +cat .claude/skills/gh-aw-guide/SKILL.md +cat .claude/skills/gh-aw-guide/references/architecture.md +cat .github/instructions/gh-aw-workflows.instructions.md +cat .github/instructions/gh-aw-workflows.sync.yaml +``` + +### Step 4: Implement ALL changes + +For EACH P0/P1/P2 item from your checklist: +1. Find the right section in the skill files +2. Add or update the content +3. Check off the item + +**Where to add each type of change:** +- New safe output → anti-patterns table in SKILL.md + Safe Outputs section +- New frontmatter option → Frontmatter Features section in SKILL.md +- New trigger behavior → Trigger Selection Guide in SKILL.md +- Security change → Security-Critical Patterns in SKILL.md +- Protected files change → architecture.md Protected Files section +- Bug fix affecting workflow authors → relevant section + Known Issues if applicable + +### Step 5: Update the sync manifest + +In `gh-aw-workflows.sync.yaml`, update `last_reviewed_release` to the latest tag. + +### Step 6: Verify completeness + +Before committing, re-read your checklist. Every P0/P1/P2 item must be addressed. If any are missing, go back and add them. ### Rules 1. **Respect `divergence` sections** — NEVER remove: "Security Boundaries", "Safe Pattern: Checkout + Restore", "Common Patterns" -2. **Classify P0-P3:** P0=factually wrong, P1=security, P2=new features, P3=nice-to-have -3. **Auto-fix P0, P1, AND P2.** Only skip P3 (cosmetic/nice-to-have). -4. **For new releases:** Read ALL release notes in the report. For each new feature, check if our skill covers it. Add new features to the anti-patterns table, trigger guide, or security patterns as appropriate. Update the `last_reviewed_release` field in `gh-aw-workflows.sync.yaml` to the latest tag after processing. -5. **Update sync manifest** — `resolution_expected`, `last_reviewed_release`, new issues +2. **Be exhaustive** — every author-facing feature, behavior change, and bug fix must be documented +3. **Include YAML examples** for new config options +4. **List what was skipped as P3** in the PR description with reasoning -Commit each change: +Commit all changes together: ```bash -git add -git commit -m "docs: update +git add .claude/skills/gh-aw-guide/SKILL.md .claude/skills/gh-aw-guide/references/architecture.md .github/instructions/gh-aw-workflows.instructions.md .github/instructions/gh-aw-workflows.sync.yaml +git commit -m "docs: update gh-aw skill for Co-authored-by: copilot-agentic-workflow[bot] <224017+copilot-agentic-workflow[bot]@users.noreply.github.com>" ``` @@ -136,3 +183,8 @@ Co-authored-by: copilot-agentic-workflow[bot] <224017+copilot-agentic-workflow[b ## Create PR The `create-pull-request` safe output packages changes into a draft PR. + +In the PR body, include: +1. **Complete list of P0/P1/P2 items** with which file/section was updated +2. **Complete list of P3 items skipped** with reasoning +3. The release versions covered