-
Notifications
You must be signed in to change notification settings - Fork 298
Description
Analysis of the last 24 hours of workflow runs identified 1 confirmed error where an agent incorrectly used the add_comment safe output tool without an explicit target in a workflow_dispatch context. The same ambiguity affects add_labels and add_reviewer. The workflow prompt is not the root cause — the tool descriptions are misleading about when auto-targeting works vs. silently fails.
Error Analysis
Error Pattern: Omitting item_number in Non-PR/Issue/Discussion Trigger Contexts
Occurrences: 1 confirmed error (run §22229256501, Smoke Copilot, workflow_dispatch)
What the agent did wrong:
Called add_comment without item_number when the workflow was triggered by workflow_dispatch — a context with no triggering issue, PR, or discussion to auto-resolve.
Example from workflow Smoke Copilot (Run §22229256501):
{
"body": "### Smoke Test Results...",
"type": "add_comment"
}Expected behavior: Either include item_number explicitly, or skip the add_comment call entirely when there is no triggering PR.
Why this happened: The item_number description for add_comment says:
"If omitted, the tool will attempt to resolve the target from the current workflow context (triggering issue, PR, or discussion)."
The agent interpreted "attempt to resolve" as "will succeed" and omitted item_number. For workflow_dispatch triggers there is no triggering issue/PR/discussion, so the resolution silently fails — the server returns {"result":"success"} but no comment is posted. The agent receives no feedback that the operation was a no-op.
Contrast with smoke-claude: The equivalent Claude smoke workflow (smoke-claude.md) works around this correctly because it contains an explicit conditional instruction:
"Only if this workflow was triggered by a pull_request event: Use the
add_commenttool ... (omit theitem_numberparameter to auto-target the triggering PR)"
The Copilot smoke workflow lacks this conditional, and the tool description's vague language about auto-targeting gave the agent no reason to add it. This pattern is likely to recur in other workflows written by less-careful prompt authors.
Current Tool Descriptions
Relevant excerpts from pkg/workflow/js/safe_outputs_tools.json
add_comment → item_number:
{
"type": "number",
"description": "The issue, pull request, or discussion number to comment on. This is the numeric ID from the GitHub URL (e.g., 123 in github.com/owner/repo/issues/123). If omitted, the tool will attempt to resolve the target from the current workflow context (triggering issue, PR, or discussion)."
}add_labels → item_number:
{
"type": "number",
"description": "Issue or PR number to add labels to. This is the numeric ID from the GitHub URL (e.g., 456 in github.com/owner/repo/issues/456). If omitted, adds labels to the item that triggered this workflow."
}add_reviewer → pull_request_number:
{
"type": ["number", "string"],
"description": "Pull request number to add reviewers to. This is the numeric ID from the GitHub URL (e.g., 876 in github.com/owner/repo/pull/876). If omitted, adds reviewers to the PR that triggered this workflow."
}Root Cause Analysis
Three related tool description problems:
-
Auto-targeting claim is unconditional: "If omitted, the tool will attempt to resolve..." / "If omitted, adds to the item that triggered this workflow." These phrases imply the field is always safely optional. They don't mention the condition for success.
-
Silent failure is undocumented: When auto-targeting fails (no triggering context), the server returns
{"result":"success"}and does nothing. Agents have no signal that the operation failed. -
Trigger type restriction is unspecified: Auto-targeting only works for
issues,pull_request,discussion,issue_comment, and similar event triggers. It does not work forworkflow_dispatch,schedule,push,workflow_run, or similar non-item triggers. This constraint is completely absent from the descriptions.
Recommended Improvements
Update Field Descriptions in pkg/workflow/js/safe_outputs_tools.json
For add_comment → item_number:
- Current: "If omitted, the tool will attempt to resolve the target from the current workflow context (triggering issue, PR, or discussion)."
- Suggested: "If omitted, the tool auto-targets the issue, PR, or discussion that triggered this workflow. Auto-targeting only works for issue, pull_request, discussion, and comment event triggers — it does NOT work for schedule, workflow_dispatch, push, or workflow_run triggers. For those trigger types, always provide item_number explicitly, or the comment will be silently discarded."
For add_labels → item_number:
- Current: "If omitted, adds labels to the item that triggered this workflow."
- Suggested: "If omitted, adds labels to the issue or PR that triggered this workflow. Only works for issue or pull_request event triggers. For schedule, workflow_dispatch, or other triggers, item_number is required — omitting it will silently skip the label operation."
For add_reviewer → pull_request_number:
- Current: "If omitted, adds reviewers to the PR that triggered this workflow."
- Suggested: "If omitted, adds reviewers to the PR that triggered this workflow. Only works for pull_request event triggers. For workflow_dispatch, schedule, or other triggers, pull_request_number is required — omitting it will silently skip the reviewer assignment."
Also update the smoke-copilot.md prompt
The Smoke Copilot workflow's OUTPUT section should be updated to mirror the conditional logic in smoke-claude.md:
- Add "Only if this workflow was triggered by a pull_request event:" before the
add_commentinstruction for the PR summary - Add "Only if triggered by a pull_request event:" before the
add_labels/remove_labelsinstructions
Affected Workflows
The following workflows had errors with this pattern:
Smoke Copilot— 1 error (run §22229256501, workflow_dispatch)
Testing Plan
After updating the tool descriptions:
- Trigger
smoke-copilotviaworkflow_dispatchand verify the agent either skips the PR comment or provides an explicititem_number - Trigger
smoke-copilotvia a PR labeled event and verify auto-targeting still works (existing behavior preserved) - Monitor logs for 2-3 days to verify the pattern does not recur
- Check other workflows using
add_comment,add_labels,add_reviewerwithout explicit targets to identify any other affected prompts
Implementation Checklist
- Update
item_numberdescription foradd_commentinpkg/workflow/js/safe_outputs_tools.json - Update
item_numberdescription foradd_labelsinpkg/workflow/js/safe_outputs_tools.json - Update
pull_request_numberdescription foradd_reviewerinpkg/workflow/js/safe_outputs_tools.json - Update
smoke-copilot.mdOutput section to add trigger-type conditional guards - Run
make buildto rebuild binary - Run
make recompileto update all workflows - Run
make testto ensure no regressions - Deploy and monitor error rates
References
- Tool schema:
pkg/workflow/js/safe_outputs_tools.json - MCP server loader:
actions/setup/js/safe_outputs_tools_loader.cjs - Validator:
actions/setup/js/safe_output_validator.cjs - Smoke Claude prompt (good example):
.github/workflows/smoke-claude.mdlines 186–194 - Smoke Copilot prompt (needs fix):
.github/workflows/smoke-copilot.mdlines 148–160
Run IDs with errors: §22229256501
Generated by Daily Safe Output Tool Optimizer
- expires on Feb 22, 2026, 4:01 PM UTC