feat(ci): automate apm-triage-panel on new and reopened issues#954
feat(ci): automate apm-triage-panel on new and reopened issues#954danielmeppiel merged 4 commits intomainfrom
Conversation
Adds .github/workflows/triage-panel.md (gh-aw substrate) that auto-invokes
the existing apm-triage-panel skill on issue events:
- issues opened
- issues reopened
- issues labeled with status/needs-triage (filtered via on.steps gate)
- workflow_dispatch (manual fallback with issue_number input)
Mirrors the proven pattern from pr-review-panel.md:
- Pre-activation label-name guard kills the pipeline at the cheapest
point for out-of-scope label events (no runner cold-start, no agent
spin-up).
- APM imports pinned to microsoft/apm@main so the triage panel always
runs against the trusted, already-reviewed skill.
- Read-only agent permissions; writes go through gh-aw safe-outputs
(add-comment max:2, update-issue target:*).
- roles: [admin, maintainer, write] matches the trust model of
applying / opening triageable issues.
Clean separation of concerns:
- This workflow decides labels and milestone via the panel skill.
- The existing project-sync.yml workflow then picks up theme/* label
changes and syncs the issue to the PGS board column.
- apm-triage-panel skill content (personas, rubric, output template)
is untouched -- this PR only adds the trigger surface.
Validation:
- gh aw compile triage-panel: 0 errors, 0 warnings.
- actionlint on the generated lock: 2 artifact_prefix warnings that
also appear (4x) in the existing pr-review-panel.lock.yml -- a
known gh-aw compiler artifact, not from this change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three lock files (cli-consistency-checker, daily-doc-updater, daily-test-improver)
plus the auto-generated agentics-maintenance.yml were still pinned to gh-aw
compiler v0.52.1. pr-review-panel and the new triage-panel are already at
v0.68.3, so this aligns the entire repo on a single compiler version.
Ran from a clean checkout:
gh aw compile --approve-updates
Approved baselines:
- CREATE_PR_PAT secret -- already declared in daily-doc-updater.md and
daily-test-improver.md sources; the v0.52.1 metadata format simply
didn't track it. No new external surface.
actionlint findings on the regenerated files are all pre-existing gh-aw
codegen patterns (artifact_prefix property warnings, empty-string choice
option in agentics-maintenance.yml). None are introduced by the upgrade.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Added second commit recompiling all gh-aw workflows on v0.68.3. Scope of the upgrade: 3 stale lock files ( Security baseline approved: actionlint: only pre-existing codegen patterns flagged (artifact_prefix, empty-string choice option in agentics-maintenance) — neither introduced by the upgrade. |
There was a problem hiding this comment.
Pull request overview
Adds a new Agentic Workflows (gh-aw) automation to run the existing apm-triage-panel skill on issue activity, post a synthesized triage comment, and apply the panel-decided labels/milestone via Safe Outputs.
Changes:
- Add a new gh-aw source workflow to trigger triage on
issuesevents (opened/reopened/labeled) andworkflow_dispatch. - Add the compiled lock workflow generated by
gh aw compile.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/triage-panel.md | New gh-aw workflow definition to orchestrate apm-triage-panel on issue events and use Safe Outputs for comments + issue updates. |
| .github/workflows/triage-panel.lock.yml | Compiled GitHub Actions workflow generated from the gh-aw source, including Safe Outputs tool schema and execution plumbing. |
Copilot's findings
Comments suppressed due to low confidence (1)
.github/workflows/triage-panel.lock.yml:623
- Safe Outputs
update_issuetool schema requirescustomValidation: requiresOneOf:status,title,body. If this validator is enforced, label/milestone-only updates (the stated intent of this workflow) will fail unless the agent also provides a no-opstatus/title/body. Either adjust the workflow/prompt to meet this constraint or configure Safe Outputs validation to allow label/milestone-only updates.
"update_issue": {
"defaultMax": 1,
"fields": {
"assignees": {
"type": "array",
"itemType": "string",
"itemSanitize": true,
"itemMaxLength": 39
},
"body": {
"type": "string",
"sanitize": true,
"maxLength": 65000
},
"issue_number": {
"issueOrPRNumber": true
},
"labels": {
"type": "array",
"itemType": "string",
"itemSanitize": true,
"itemMaxLength": 128
},
"milestone": {
"optionalPositiveInteger": true
},
"operation": {
"type": "string",
"enum": [
"replace",
"append",
"prepend",
"replace-island"
]
},
"repo": {
"type": "string",
"maxLength": 256
},
"status": {
"type": "string",
"enum": [
"open",
"closed"
]
},
"title": {
"type": "string",
"sanitize": true,
"maxLength": 128
}
},
"customValidation": "requiresOneOf:status,title,body"
}
- Files reviewed: 5/6 changed files
- Comments generated: 5
| if [ "$ACTION" = "labeled" ] && [ "$LABEL_NAME" = "status/needs-triage" ]; then | ||
| echo "Triggering label is 'status/needs-triage' -- proceeding." | ||
| exit 0 | ||
| fi | ||
| echo "Event '$ACTION' / label '$LABEL_NAME' is out of scope; skipping." | ||
| exit 1 |
There was a problem hiding this comment.
The pre-activation label gate exits with status 1 for out-of-scope issues:labeled events. That will mark the workflow run as failed for every non-status/needs-triage label applied to an issue (and potentially for label changes performed by this workflow), which can create significant Actions noise. Consider removing labeled from the trigger (keeping only opened/reopened + workflow_dispatch), or restructuring the gate so out-of-scope label events end in a neutral/green outcome while still preventing agent activation.
| cat > "${RUNNER_TEMP}/gh-aw/safeoutputs/config.json" << 'GH_AW_SAFE_OUTPUTS_CONFIG_efa05e814fb58cf7_EOF' | ||
| {"add_comment":{"max":2},"create_report_incomplete_issue":{},"missing_data":{},"missing_tool":{},"noop":{"max":1,"report-as-issue":"true"},"report_incomplete":{},"update_issue":{"allow_body":true,"max":1,"target":"*"}} | ||
| GH_AW_SAFE_OUTPUTS_CONFIG_efa05e814fb58cf7_EOF |
There was a problem hiding this comment.
Safe Outputs config enables update_issue.allow_body: true, which allows the agent to edit issue bodies via the write-sink even though the workflow intent is label/milestone-only updates. If the Safe Outputs framework supports disabling body/title edits for update_issue, it would be safer to do so here to reduce prompt-injection impact (issue bodies are untrusted input).
This issue also appears on line 570 of the same file.
| fi | ||
| echo "Event '$ACTION' / label '$LABEL_NAME' is out of scope; skipping." | ||
| exit 1 | ||
| roles: [admin, maintainer, write] |
There was a problem hiding this comment.
roles: [admin, maintainer, write] means the workflow only activates when the triggering actor has write access. That prevents auto-triage for issues opened/reopened by external contributors, which is the primary use case described in this PR. Drop the roles gate (or change the triggers to only run on maintainer-applied labels) so newly opened issues from non-members are actually processed.
| roles: [admin, maintainer, write] |
| # 1. issues: opened / reopened / labeled. We listen on plain `issues` | ||
| # (not `issues_target`) because issues -- unlike PRs -- don't have a | ||
| # fork-head untrusted code surface; the only untrusted input is the |
There was a problem hiding this comment.
The header comment says "issues_target" is an alternative event, but GitHub Actions does not have an issues_target trigger (unlike pull_request_target). This is likely to confuse future maintainers; please reword to just explain why plain issues is sufficient here.
| # 1. issues: opened / reopened / labeled. We listen on plain `issues` | |
| # (not `issues_target`) because issues -- unlike PRs -- don't have a | |
| # fork-head untrusted code surface; the only untrusted input is the | |
| # 1. issues: opened / reopened / labeled. Plain `issues` is sufficient | |
| # here because issues -- unlike PRs -- do not introduce a fork-head | |
| # untrusted code execution surface; the only untrusted input is the |
| `safe-outputs.update-issue`. The `theme/*` label, if assigned, | ||
| will automatically trigger the existing `project-sync.yml` | ||
| workflow to add the issue to the appropriate PGS board column -- | ||
| no extra action needed here. | ||
|
|
||
| Do not perform any other writes. Do not edit the issue body or title. | ||
| Do not close, reopen, lock, or assign the issue. |
There was a problem hiding this comment.
The prompt instructs the agent to apply only labels/milestone via safe-outputs.update-issue, but the compiled workflow's Safe Outputs tool schema requires at least one of status, title, or body (customValidation requiresOneOf:status,title,body). With the current instructions, label-only updates are likely to be rejected by the Safe Outputs validator and the workflow won't apply the panel's decisions. Update the instructions (or Safe Outputs config, if possible) so the agent includes a no-op status value (e.g., open) when updating labels/milestone, or otherwise satisfies the validator without changing title/body.
| `safe-outputs.update-issue`. The `theme/*` label, if assigned, | |
| will automatically trigger the existing `project-sync.yml` | |
| workflow to add the issue to the appropriate PGS board column -- | |
| no extra action needed here. | |
| Do not perform any other writes. Do not edit the issue body or title. | |
| Do not close, reopen, lock, or assign the issue. | |
| `safe-outputs.update-issue`, and include `status: open` in that | |
| update payload as a validator no-op. This satisfies the Safe | |
| Outputs schema requirement that at least one of `status`, `title`, | |
| or `body` be present, while preserving the existing issue state and | |
| avoiding any title/body edits. The `theme/*` label, if assigned, | |
| will automatically trigger the existing `project-sync.yml` | |
| workflow to add the issue to the appropriate PGS board column -- | |
| no extra action needed here. | |
| Do not perform any other writes. Do not edit the issue body or title. | |
| Do not close, reopen, lock, or assign the issue. When calling | |
| `safe-outputs.update-issue` for labels/milestone, use `status: | |
| open` only as the required no-op field for validator compliance. |
…ication framing
Reshapes the triage workflow trigger model to bound LLM cost without
sacrificing 'no genuine issue left untriaged'. Grounded in actual repo
data: 200 issues / 30 days mean 6.7/day, peak 17/day, 19 bot-authored,
and only 5% currently get a theme/* label -- so unbounded per-issue
runs would mean ~10M tokens/month with no ceiling.
Trigger model:
1. SCHEDULED_SWEEP (gh-aw fuzzy daily, ~12:49 UTC): finds open
human-authored issues lacking 'status/triaged', oldest first,
processes up to 10 per run. Hard ceiling: 10 * 30 = 300 panel
runs / month. Drains the typical queue (~140 eligible/month) with
headroom for spike days.
2. OPT_IN_RETRIAGE (issues: labeled, status/needs-triage): explicit
fast-path for maintainers who need immediate triage, e.g. on a
security report. Fires within minutes. Front-gate filters bots,
locked, closed at zero cost.
3. MANUAL_DISPATCH (workflow_dispatch with optional issue_number):
replay / debugging.
Lifecycle invariants now explicitly designed and documented in the
agent prompt:
- Bots are never triaged (front-gate + sweep filter).
- Every human-authored open issue is triaged within 24h, or sooner
if a maintainer applies status/needs-triage.
- Maintainer label edits ALWAYS WIN: agent never reverts human label
changes; the panel may surface disagreement in the verdict comment
but never silently overrides.
- status/triaged is the 'do not re-sweep me' signal. Removing it
re-enrolls the issue in the next sweep. Applying status/needs-
triage bypasses the sweep cap entirely (immediate fast-path).
- status/needs-triage is consumed (removed) by the workflow after
re-triage so the trigger doesn't loop.
- Existing milestone is never overwritten -- panel only sets a
milestone if none exists today.
Verdict comment now appends a verbatim 'agentic proposal pending human
ratification, silence is approval' footer that documents the override
mechanics inline with every triage comment, so contributors and
maintainers see the human-in-the-loop framing on the issue itself.
Concurrency group prevents double-triage on the same issue when fast
path and sweep race.
Compile: 0 errors, 0 warnings.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Four targeted hardenings address realistic abuse vectors that could inflate panel cost beyond the per-month ceiling. 1. Body size cap (64 KB / 65536 chars) -- protects against a single adversarially-large issue body inflating per-run token cost from ~50k to >500k. Generous margin: typical APM PRD/design issues are <16 KB. Truncated bodies are flagged in the verdict so a maintainer can read the original. 2. Per-author quota in scheduled sweep (max 2 issues per distinct author per sweep) -- prevents a sock-puppet account from monopolizing daily triage capacity (10 slots/day). Excess issues from the same author roll to subsequent sweeps; they still get triaged eventually, just spread across days. 3. Output safety rails / batch allow-list -- explicit prompt-level rail that the agent computes its write allow-list (BATCH_ALLOW_LIST) from the workflow-controlled candidate selection BEFORE reading any issue body, so prompt-injection in a body cannot redirect writes to unrelated issues. Compensates for safe-outputs.update-issue.target: "*" being required for sweep mode (gh-aw does not support per-mode targets). Every workflow run logs every safe-output call for audit. 4. Spam-shape filter in scheduled sweep -- crude pre-panel filter skips issues whose bodies are >50 consecutive identical chars, >80% URLs, dominated by a single 3-char repeat, or have <20 chars of real content. Skipped issues stay in queue for manual review; maintainers can force a panel run via status/needs-triage. Threat model NOT addressed by this PR: open-close-reopen loop (workflow does not subscribe to those events; cannot fire), label manipulation by external users (gated by repo write permission), fabricated bot login (we use user.type == Bot which is unspoofable from user side). Lock file unchanged because gh-aw embeds the prompt via runtime-import, so .md edits propagate without recompile changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Hardening commit (f0e7abe) addresses the open-close-loop concern + adjacent abuse vectors. What I checked firstThe specific attack named ("open-close issue indefinitely to burn tokens") doesn't actually fire any triggers in this PR -- workflow subscribes to Real surfaces I found and hardened
Why
|
Live validation outcomeRan two end-to-end scenarios against the merged workflow on Scenarios run
Both issues received: full panel verdict comment, complete label set (8 labels each, including Scenarios 2 and 3 (spam-shape filter, empty-body filter) only exercise the Bugs caught during validation (all fixed on
|
Triage Panel — architecture referenceNote This is a self-contained reference for the system landed by this PR. If you maintain microsoft/apm and want issues to get triaged — or want to understand exactly what gets written to your repo and why — start here. TL;DRA daily scheduled sweep plus a one-click opt-in fast-path that runs the Problem (WHY)
Approach (WHAT)
Implementation (HOW)
Three execution modes share Step 2 (run the panel) and Step 3 (apply decisions); they differ only in Step 1 (which issues to triage):
Defensive layers, in order:
DiagramsTrigger fan-in into the shared three-step pipeline: flowchart LR
A1[issues: labeled] -->|status/needs-triage only| G[Front-gate shell step]
A2[schedule: daily] --> M{Mode}
A3[workflow_dispatch] --> M
G --> M
M -->|labeled event| R1[OPT_IN_RETRIAGE]
M -->|cron OR dispatch w/o issue| R2[SCHEDULED_SWEEP]
M -->|dispatch w/ issue_number| R3[MANUAL_DISPATCH]
R1 --> S1[Step 1: gather candidates]
R2 --> S1
R3 --> S1
S1 --> S2[Step 2: run apm-triage-panel skill]
S2 --> S3[Step 3: emit verdict + apply labels/milestone]
Per-issue Step 3 emission, showing the safe-output surface and the human-ratification contract: sequenceDiagram
autonumber
participant Agent
participant SO as safe-outputs
participant Issue as GitHub issue
participant Maint as Maintainer
Agent->>SO: add-comment (verdict + ratification footer)
SO->>Issue: post comment
Agent->>SO: add-labels (theme/area/type/priority + status/triaged)
SO->>Issue: apply labels (allow-list strict-equality)
Agent->>SO: remove-labels (status/needs-triage if present)
SO->>Issue: consume trigger label
Agent->>SO: assign-milestone (only if issue had none)
SO->>Issue: set milestone
Note over Issue,Maint: Silence is approval. Any human edit is authoritative and not reverted.
Maint->>Issue: optional override (edit label / milestone)
Trade-offs
Benefits
ValidationLive-validation evidence is in the prior comment on this PR. Three bugs in Compile check (run locally to verify the lock is in sync)How to use this — three workflows for maintainers
To re-enroll an issue into the next daily sweep, remove the To extend the taxonomy, add the label in GitHub, then add it to Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com |
* chore(release): cut 0.9.4 CHANGELOG entry for 0.9.4 covers all 7 PRs merged since v0.9.3: - #974 SKILL_BUNDLE day-0 install parity (Added) - #954 automate apm-triage-panel workflow (Added) - #970 python-architect mermaid classDiagram trap (Changed) - #911 REQUESTS_CA_BUNDLE TLS validation (Fixed) - #971 triage-panel project-sync dispatch (Fixed) - #910 CLI consistency cleanup (Fixed) - #958 issue templates label taxonomy (Fixed) - #953 docs auto-deploy after bot-cut releases (Fixed) Open milestone 0.9.4 issues (41) reassigned to 0.9.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(changelog): tighten 0.9.4 entries (so-what for developers) Refactor per Keep-a-Changelog spirit: lead with developer impact, trim agent-internals prose, group maintainer-only changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(changelog): add #660 install.sh air-gapped entry to 0.9.4 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TL;DR
Adds an automated triage workflow that:
apm-triage-panelskill on a daily fuzzy schedule (gh-aw "daily" → ~12:49 UTC), processing up to 10 oldest-first untriaged open issues per sweep — hard cost ceiling.status/needs-triageto trigger immediate re-triage on any specific issue (security reports, urgent items).Problem (WHY)
The
apm-triage-panelskill exists at.apm/skills/apm-triage-panel/SKILL.mdbut a maintainer has to invoke it by hand on every issue. Real data from this repo, last 30 days:theme/*label eventuallyManual triage is severely under-resourced. Automation is the right call. But naive
on: issues.openedtriggers would mean ~200 panel runs/month with no ceiling — at ~30-50k tokens per panel run that's ~10M LLM tokens/month with no upper bound, and every spike day burns through quota.Approach (WHAT)
Two cooperative paths plus manual dispatch:
Lifecycle invariants (designed and documented in the agent prompt)
status/triagedstatus/needs-triagestatus/needs-triage; keepsstatus/triagedtheme/*)status/triagedstatus/triagedis removed; maintainer appliesstatus/needs-triageto force re-triageImplementation (HOW)
Files
.github/workflows/triage-panel.md— gh-aw source (new).github/workflows/triage-panel.lock.yml— generated (new).github/workflows/{cli-consistency-checker,daily-doc-updater,daily-test-improver}.lock.yml— recompiled v0.52.1 → v0.68.3.github/workflows/agentics-maintenance.yml— auto-regenerated by gh-aw v0.68.3Verdict-comment footer (appended verbatim by the agent)
This puts the human-in-the-loop contract on every issue, visible to contributor and maintainer alike.
Substrate choices
on: issues.labeled(filtered tostatus/needs-triageviaon.steps:)pr-review-panel.mdpattern. Pre-activation gate kills out-of-scope label events at zero cost.schedule: cron 'daily'49 12 * * *). Avoids deterministic top-of-the-hour load spikes.issues.opened/issues.reopenedsubscriptionsstatus/needs-triageis the immediate-path escape hatch. Unbounded per-issue triggers would have no cost ceiling.concurrency.groupon issue number@mainpr-review-panel.md.safe-outputs.update-issue(not directgh issue edit)Cost shape
Trade-offs
status/needs-triagelabel is the always-immediate escape hatch for anything urgent.apm-triage-panelskill is single-issue by contract). A future "lighter classifier first, escalate to panel" two-tier model could cut tokens further; out of scope here.Validation
Security review (gh-aw safe-update baselines approved)
All flagged items are already in active use elsewhere in the repo:
microsoft/apm-action@v1.4.2— same pin aspr-review-panel.lock.yml,daily-doc-updater.lock.yml,cli-consistency-checker.lock.yml.GH_AW_PLUGINS_TOKEN— already configured at the repo level for the same imports.CREATE_PR_PAT— already declared indaily-doc-updater.mdanddaily-test-improver.mdsources; the v0.52.1 metadata schema simply didn't track it. No new external surface.Pre-merge checklist
status/triagedlabel:How to test (post-merge)
Fast path smoke test — apply
status/needs-triageto any open untriaged issue. Within ~1-2 min the panel should run, post a verdict, apply labels, addstatus/triaged, and removestatus/needs-triage. Confirmproject-sync.ymlthen picks up thetheme/*label change and adds the issue to the project board.Sweep dry run —
gh aw run triage-panel(no input) before the first scheduled fire. Confirm the agent finds untriaged issues, processes ≤10 oldest, and skips bot-authored / locked / already-status/triagedones.Out-of-scope label test — apply a
theme/authlabel directly to a fresh issue. Theon.stepsgate should drop the event before the agent runs (visible as a quick "skipping" step in the Actions log, no agent job).Human-edit-wins regression — after a sweep triages an issue, manually swap one of its
theme/*labels for a different one. Re-trigger by applyingstatus/needs-triage. Verify the new comment surfaces any disagreement as a recommendation but does not revert your edit.Out of scope
apm-triage-panel/SKILL.md(already shippable; last refactored in50d24b0a).project-sync.yml(clean handoff via labels already works).