perf: optimize secret-digger-copilot token usage#1887
Merged
Conversation
Address recommendations from #1879 (token optimization report): 1. Reduce timeout-minutes from 30 to 15 - Failure runs were spending 31 turns over ~7 min then timing out at 30 min - Halves the max cost ceiling for runaway failure runs - Note: Copilot engine does not support max-turns; timeout is the available control 2. Remove duplicate context from user message - Repository, Run ID, Workflow, Engine lines were already injected by gh-aw framework into <system> context - Removes 4 redundant lines that slightly inflate the per-run unique prompt portion 3. Trim shared/secret-audit.md prompt - Condensed Investigation Workflow steps 1-4 into 3 concise lines (~450 chars saved per turn) - Condensed Background Knowledge Tracking section - Removed Security Research Guidelines section (covered by MISSION) - Added explicit turn budget: "Complete in 6-8 tool calls" - Fixed step numbering after condensing These changes also affect secret-digger-claude and secret-digger-codex (which import shared/secret-audit.md) but their lock files are unchanged since the prompt changes are embedded at compile time. Closes #1879 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
Contributor
There was a problem hiding this comment.
Pull request overview
Optimizes the Secret Digger Copilot workflow to reduce token/cost in failure runs by shortening job timeouts and trimming repeated/verbose prompt content used by the agent.
Changes:
- Reduced Copilot workflow
timeout-minutesfrom 30 to 15. - Removed the “Current Run Context” block from the Copilot workflow prompt.
- Condensed
shared/secret-audit.md(background/workflow text) and added an explicit investigation budget guideline.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/shared/secret-audit.md | Trims shared prompt content and adds a turn/tool-call budget to steer shorter investigations. |
| .github/workflows/secret-digger-copilot.md | Lowers timeout and removes redundant per-run context from the user message section. |
| .github/workflows/secret-digger-copilot.lock.yml | Regenerates the compiled workflow to reflect the new timeout/prompt inputs. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (1)
.github/workflows/shared/secret-audit.md:83
- The new Turn budget guidance mixes concepts (“6–8 tool calls” but then “Load cache state (1 turn) …”), which can be interpreted inconsistently by the agent. Consider using one unit consistently (either “tool calls” or “turns”) and aligning the breakdown accordingly so the budget is unambiguous.
**Turn budget**: Complete your investigation in **6–8 tool calls**. Load cache state (1 turn), investigate deeply (4–5 turns), update cache and report (1–2 turns). Do not exhaustively check all 10 investigation areas in a single run — pick ONE area and go deep.
- Files reviewed: 3/3 changed files
- Comments generated: 3
Comment on lines
80
to
+84
| ## Emergency Exit Rule | ||
|
|
||
| If you have nothing substantive to report after completing your investigation, call `noop` with a summary message instead of producing empty or minimal output. Never produce output shorter than one complete sentence. | ||
|
|
||
| ## Security Research Guidelines | ||
| **Turn budget**: Complete your investigation in **6–8 tool calls**. Load cache state (1 turn), investigate deeply (4–5 turns), update cache and report (1–2 turns). Do not exhaustively check all 10 investigation areas in a single run — pick ONE area and go deep. | ||
|
|
||
| - Be thorough and systematic | ||
| - Try unusual and creative approaches | ||
| - Don't repeat the same techniques every run | ||
| - Document everything in cache-memory | ||
| - Focus on finding real security boundary violations | ||
| - This security testing is conducted within a sandboxed environment specifically designed for this purpose | ||
| If you have nothing substantive to report after completing your investigation, call `noop` with a summary message instead of producing empty or minimal output. Never produce output shorter than one complete sentence. |
There was a problem hiding this comment.
The Turn budget paragraph is currently placed under the "## Emergency Exit Rule" heading, even though it’s not part of the emergency exit behavior. Consider moving it to its own heading (e.g., "## Turn Budget") or placing it near the Investigation Workflow so the document structure matches the content.
This issue also appears on line 82 of the same file.
See below for a potential fix:
## Turn Budget
**Turn budget**: Complete your investigation in **6–8 tool calls**. Load cache state (1 turn), investigate deeply (4–5 turns), update cache and report (1–2 turns). Do not exhaustively check all 10 investigation areas in a single run — pick ONE area and go deep.
## Emergency Exit Rule
| timeout-minutes: 30 | ||
| timeout-minutes: 15 | ||
| --- | ||
|
|
There was a problem hiding this comment.
The "Current Run Context" block removal also drops the "Runner: Check your environment carefully" instruction, which is not redundant with the injected GitHub context. If that guidance is still desired for agent behavior/safety, consider keeping it (e.g., as a single line above the main instruction) while removing only the truly duplicate context lines.
Suggested change
| Runner: Check your environment carefully. |
Comment on lines
149
to
153
| GH_AW_GITHUB_EVENT_ISSUE_NUMBER: ${{ github.event.issue.number }} | ||
| GH_AW_GITHUB_EVENT_PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }} | ||
| GH_AW_GITHUB_REPOSITORY: ${{ github.repository }} | ||
| GH_AW_GITHUB_RUN_ID: ${{ github.run_id }} | ||
| GH_AW_GITHUB_WORKFLOW: ${{ github.workflow }} | ||
| GH_AW_GITHUB_WORKSPACE: ${{ github.workspace }} |
There was a problem hiding this comment.
In this lock file, GH_AW_GITHUB_WORKFLOW is no longer passed into prompt rendering/substitution, which means the workflow name won’t appear in the <github-context> block (unlike the Claude/Codex Secret Digger workflows). If keeping workflow name in the system context is still intended, re-introduce GH_AW_GITHUB_WORKFLOW in these env/substitution lists so it can be injected.
This comment has been minimized.
This comment has been minimized.
Closed
Contributor
|
Smoke test results:
|
Contributor
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
|
Contributor
Smoke Test Results
Overall: PASS
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses the token optimization recommendations from #1879.
Changes
1. Reduced
timeout-minutesfrom 30 → 15max-turns(only Claude does);timeout-minutesis the available control lever2. Removed duplicate context from user message
<system>context3. Trimmed
shared/secret-audit.mdpromptImpact
These prompt changes also affect
secret-digger-claudeandsecret-digger-codex(which importshared/secret-audit.md), but their lock files are unchanged since they were already compiled at this gh-aw version.Why not
max-turns?The Copilot engine sets
supportsMaxTurns: false— it usesmax-continuationsinstead, which controls autopilot sequential runs (not per-run turn count). The turn budget is communicated via prompt instructions as the next best approach.Closes #1879