Improve agent projects safe output tools support

[mnkiefer]

• When using an agentic workflow with both issue and projects safe outputs, but minimal instructions without direct tool call mentions (just natural language), the following issue occurred as described:

The agent creates issues via create_issue with temporary_ids, but doesn't know it needs to call update_project using those same temporary IDs to add them to the project board. The issues get created fine but never get added to the project. The agent assumed the safe_outputs handler would automatically add created issues to the configured project, but it seems like update_project calls are required explicitly.

[mnkiefer]

/scout

[github-actions]

🔭 Recon complete! Scout has charted the territory. Map ready! 🗺️

[github-actions]

🔍 Scout Research Report

Triggered by @mnkiefer

Executive Summary

Research focused on AI agent tool discovery and multi-step workflow patterns reveals that agents struggle with implicit tool chaining—they don't naturally understand that creating an issue with temporary_ids should trigger a subsequent update_project call. This is a known pattern in agent design where explicit instructions and tool descriptions are critical for multi-step operations.

Click to expand detailed findings

Key Findings

1. Tool Chaining Requires Explicit Guidance

Problem: Agents don't automatically infer multi-step workflows involving temporary identifiers.

Research Evidence:

• Anthropic's "Building Effective Agents" recommends prompt chaining for multi-step tasks where each step processes the previous output
• Tool chaining requires passing intermediate results between tools—agents need explicit instructions on this flow
• Temporal identifier patterns (like temporary_ids) need special handling in workflow design

Sources:

• Building Effective AI Agents - Anthropic
• [AI Agent Tools Tutorial - Patronus AI]((www.patronus.ai/redacted)
• [How to Build Tool Chaining - OneUptime]((oneuptime.com/redacted)

2. Tool Descriptions Must Be Explicit About Relationships

Problem: Tool descriptions don't communicate dependencies or related operations.

Research Evidence:

• Anthropic's engineering guide: "Make implicit context explicit" - describe relationships between tools
• Tool descriptions should explain when to use them (not just what they do)
• Parameters should be unambiguous (e.g., temporary_issue_id vs generic id)

Concrete Example from Research:

"When writing tool descriptions, think of how you would describe your tool to a new hire. Consider context you might implicitly bring and make it explicit. Instead of user, use user_id."

Sources:

• Writing Effective Tools for AI Agents - Anthropic
• [AI Agents: The Art of Explicit Instructions]((nebulaconsulting.co.uk/redacted)

3. GitHub Projects API Patterns

Finding: GitHub's Projects v2 API requires explicit two-step operations:

1. Create issue (returns node_id)
2. Call addProjectV2ItemById with both projectId and contentId (issue's node_id)

Research shows:

• This is not automatic - no built-in GitHub automation links new issues to projects without explicit API calls
• The actions/add-to-project GitHub Action exists specifically to handle this workflow
• Classic Projects vs Projects v2 have different APIs (potential confusion point)

Sources:

• Using the API to manage Projects - GitHub Docs
• actions/add-to-project - GitHub Actions

4. Best Practices for Multi-Tool Agents

Pattern: Sequential Execution with Explicit State

• Don't rely on agent context to carry state
• Use explicit artifacts (structured outputs) that persist between tool calls
• Build validation steps: "verify X succeeded before calling Y"

Pattern: Tool Discovery at Scale

• When agents have 10+ tools, description quality matters more
• Use semantic tool discovery (RAG) for 20+ tools
• Group related tools in documentation

Sources:

• [Multi-Step Task Breakdown Analysis]((dev.to/redacted)
• [Semantic Tool Discovery - Rocket Connect]((www.rconnect.tech/redacted)

Recommendations

1. Enhance Tool Descriptions with Workflow Context (High Priority)

Current State: create_issue description doesn't mention project workflow.

Recommended Change: Update tool descriptions to include:

create_issue:
  description: "Creates a GitHub issue. Returns temporary_id that can be 
                used with update_project to add the issue to project boards."
  
update_project:
  description: "Adds items to a GitHub project board. Use temporary_id 
                from create_issue to link newly created issues."
```

### 2. **Add Explicit Workflow Examples to System Prompt** (High Priority)

Include example workflows in agent instructions:
```
Example multi-step workflow:
1. Call create_issue with temporary_id="issue-1"
2. Wait for issue creation response
3. Call update_project with temporary_id="issue-1" to add to project

3. Implement Tool Chaining Hints (Medium Priority)

Add a related_tools field to tool metadata:

{
  "tool": "create_issue",
  "related_tools": {
    "follow_up": ["update_project"],
    "description": "After creating issues, use update_project to add them to boards"
  }
}

4. Consider Prompt Chaining Pattern (Medium Priority)

Instead of relying on the agent to chain tools, implement explicit workflow steps:

• Step 1: "Create all issues"
• Step 2: "For each created issue, add to project using temporary_ids"

This makes the multi-step nature explicit in the orchestration layer.

5. Add Validation Tool (Low Priority)

Create a verify_project_items tool that checks if issues were added to projects:

• Helps agents detect when they've forgotten the second step
• Provides clear error messages for recovery

Implementation Priority

1. Quick Win: Update tool descriptions (1-2 hours)
2. High Impact: Add workflow examples to system prompts (2-4 hours)
3. Long Term: Implement tool relationship metadata system (1-2 days)

Key Sources Referenced

• Anthropic: Building Effective Agents
• Anthropic: Writing Tools for AI Agents
• GitHub Projects API Documentation
• [Tool Chaining Patterns - OneUptime]((oneuptime.com/redacted)
• [AI Agent Tool Calling Guide - Patronus AI]((www.patronus.ai/redacted)

Next Steps

1. Review current create_issue and update_project tool descriptions
2. Add explicit workflow documentation to tool descriptions
3. Update agent system prompts with multi-step workflow examples
4. Consider implementing tool relationship metadata for future scalability

AI generated by Scout

[mnkiefer]

• Resolved with prompt improvements here: chore: improve project prompt instructions #14984