Skip to content

Optimize security-guard token usage with pre-run relevance gating, lower turn cap, and leaner prompt context#2113

Merged
lpcox merged 3 commits intomainfrom
copilot/optimize-claude-token-costs
Apr 20, 2026
Merged

Optimize security-guard token usage with pre-run relevance gating, lower turn cap, and leaner prompt context#2113
lpcox merged 3 commits intomainfrom
copilot/optimize-claude-token-costs

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 20, 2026

security-guard was spending full Claude runs on PRs with no security-relevant changes, and carrying unnecessary prompt/context token overhead. This PR reduces avoidable token cost by skipping irrelevant runs, capping turns, and compressing high-volume guidance.

  • Workflow-level relevance gating (skip non-security PRs)

    • Added check_security_relevance job in security-guard.md.
    • Gated agent execution with:
      • if: needs.check_security_relevance.outputs.security_files_changed != '0'
    • Mirrored this in security-guard.lock.yml (agent.needs, agent.if, and downstream needs updates).
  • Lower Claude turn budget

    • Changed engine.max-turns from 10 to 6 in source workflow.
    • Updated compiled lock runtime settings (--max-turns and GH_AW_MAX_TURNS) to match.
  • Prompt footprint reduction

    • Condensed the Security Checks section in security-guard.md from multiple verbose subsections into a compact checklist of critical risk classes.
    • Preserved security-review intent while reducing repeated token payload.
  • System-context trimming for security workflow efficiency

    • Trimmed non-essential sections from CLAUDE.md / AGENTS.md that were identified as irrelevant to security-review execution context.
  • Pattern precision hardening

    • Tightened relevance matcher from containers/ to (^|/)containers/ to avoid accidental path overmatching.
# security-guard.md
if: needs.check_security_relevance.outputs.security_files_changed != '0'

jobs:
  check_security_relevance:
    outputs:
      security_files_changed: ${{ steps.check.outputs.count }}

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/repos/actions/github-script/git/ref/tags/v9
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq .object.sha 0 -j ACCEPT (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq .object.sha (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv /tmp/go-build2519296211/b315/symabis -c=4 -nolocalimports -importcfg /tmp/go-build2519296211/b315/importcfg -pack -asmhdr (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v0.68.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.68.3 --jq .object.sha 0 -j ACCEPT (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.68.3 --jq .object.sha (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI linked an issue Apr 20, 2026 that may be closed by this pull request
9 tasks
Copilot AI changed the title [WIP] Optimize Claude Token usage in security-guard workflow Optimize security-guard token usage with pre-run relevance gating, lower turn cap, and leaner prompt context Apr 20, 2026
Copilot AI requested a review from lpcox April 20, 2026 13:58
@lpcox lpcox marked this pull request as ready for review April 20, 2026 14:33
@lpcox lpcox requested a review from Mossaka as a code owner April 20, 2026 14:34
Copilot AI review requested due to automatic review settings April 20, 2026 14:34
@github-actions
Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation build failed for this PR. View logs.

Built from commit d8df57d

@github-actions
Copy link
Copy Markdown
Contributor

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 84.31% 84.39% 📈 +0.08%
Statements 83.55% 83.63% 📈 +0.08%
Functions 87.46% 87.46% ➡️ +0.00%
Branches 74.72% 74.76% 📈 +0.04%
📁 Per-file Coverage Changes (1 files)
File Lines (Before → After) Statements (Before → After)
src/docker-manager.ts 86.8% → 87.1% (+0.30%) 86.4% → 86.7% (+0.29%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test Results:

Status: PASS

💥 [THE END] — Illustrated by Smoke Claude

@github-actions
Copy link
Copy Markdown
Contributor

🔥 Smoke Test: Copilot BYOK (Offline) Mode

Test Result
GitHub MCP (list merged PRs) #2102 Allow chatgpt.com in Codex smoke workflow AWF domain allowlist
GitHub.com connectivity (HTTP 200)
File write/read (smoke-test-copilot-byok-24670564605.txt) BYOK smoke test passed at Mon Apr 20 14:34:37 UTC 2026
BYOK inference (this response)

Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com

Overall: PASS — PR by @app/copilot-swe-agent, assigned to @lpcox and @Copilot

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions
Copy link
Copy Markdown
Contributor

🔬 Smoke Test Results

Test Status
GitHub PR list (MCP) ✅ Latest merged: "Allow chatgpt.com in Codex smoke workflow AWF domain allowlist" (#2102)
GitHub.com connectivity ❌ Pre-step data unavailable (template vars not substituted)
File write/read ❌ Pre-step data unavailable (template vars not substituted)

Overall: FAIL (pre-step outputs missing — template substitution did not occur)

PR by @app/copilot-swe-agent · Assignees: @lpcox, @Copilot

📰 BREAKING: Report filed by Smoke Copilot

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test Results

Check Status Notes
Redis PING (host.docker.internal:6379) ❌ FAIL redis-cli not installed; apt unavailable in this environment
PostgreSQL pg_isready (host.docker.internal:5432) ✅ PASS Accepting connections
PostgreSQL SELECT 1 (db: smoketest, user: postgres) ✅ PASS Returned 1

2/3 checks passed. Redis connectivity could not be verified due to missing tooling.

🔌 Service connectivity validated by Smoke Services

@github-actions
Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

⚠️ ALL CLONES FAILED — Repository clones could not complete due to environment-level failures (thread resource limits or proxy access denied).

Ecosystem Project Build/Install Tests Status
Bun elysia N/A ❌ CLONE_FAILED
Bun hono N/A ❌ CLONE_FAILED
C++ fmt N/A ❌ CLONE_FAILED
C++ json N/A ❌ CLONE_FAILED
Deno oak N/A ❌ CLONE_FAILED
Deno std N/A ❌ CLONE_FAILED
.NET hello-world N/A ❌ CLONE_FAILED
.NET json-parse N/A ❌ CLONE_FAILED
Go color N/A ❌ CLONE_FAILED
Go env N/A ❌ CLONE_FAILED
Go uuid N/A ❌ CLONE_FAILED
Java gson N/A ❌ CLONE_FAILED
Java caffeine N/A ❌ CLONE_FAILED
Node.js clsx N/A ❌ CLONE_FAILED
Node.js execa N/A ❌ CLONE_FAILED
Node.js p-limit N/A ❌ CLONE_FAILED
Rust fd N/A ❌ CLONE_FAILED
Rust zoxide N/A ❌ CLONE_FAILED

Overall: 0/8 ecosystems passed — ❌ FAIL

Error Details

Bun: failed to run git: fork/exec /usr/bin/git: resource temporarily unavailable

C++: remote: access denied: unrecognized endpoint — HTTP 403 from proxy ((localhost/redacted))

Deno: runtime/cgo: pthread_create failed: Resource temporarily unavailable — SIGABRT

.NET: /tmp/awf-lib/gh: exit: Illegal number: EAGAIN — thread resource limit

Go: runtime/cgo: pthread_create failed: Resource temporarily unavailable — SIGABRT

Java: runtime/cgo: pthread_create failed: Resource temporarily unavailable — SIGABRT

Node.js: runtime/cgo: pthread_create failed: Resource temporarily unavailable — SIGABRT

Rust: remote: access denied: unrecognized endpoint — HTTP 403 from proxy

The sandbox environment hit OS thread limits (pthread_create: EAGAIN) and/or the gh CLI proxy rejected outbound git clone requests. No test repositories could be cloned.

Generated by Build Test Suite for issue #2113 · ● 156.5K ·

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Optimizes the security-guard agent workflow to reduce unnecessary Claude token spend by skipping runs on PRs without security-relevant changes, lowering the max turn budget, and shrinking prompt/context payload.

Changes:

  • Added a PR file-diff–based relevance gate (check_security_relevance) and used it to skip the agent job when no security-critical files changed.
  • Reduced Claude max turns from 10 → 6 and updated the compiled lock workflow to match.
  • Condensed security guidance in the prompt and trimmed high-volume sections from CLAUDE.md.
Show a summary per file
File Description
CLAUDE.md Removes large, non-security-focused guidance sections to reduce system context size for the security workflow.
.github/workflows/security-guard.md Adds security relevance gating job + workflow-level gate, lowers max-turns, and compresses the “Security Checks” prompt content.
.github/workflows/security-guard.lock.yml Updates compiled workflow to include the new gating job/needs/if logic and max-turns=6 runtime settings.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

.github/workflows/security-guard.md:89

  • SECURITY_RE (and the PR-files API call) is duplicated here and in the new check_security_relevance job. This increases maintenance risk (regex drift) and adds an extra API call per run. Consider reusing needs.check_security_relevance.outputs.security_files_changed for reporting in the prompt/summary, or centralizing the regex in one place so both checks stay consistent.
    run: |
      SECURITY_RE="host-iptables|setup-iptables|squid-config|docker-manager|seccomp-profile|domain-patterns|entrypoint\.sh|Dockerfile|(^|/)containers/"
      COUNT=$(gh api "repos/${GH_REPO}/pulls/${PR_NUMBER}/files" \
        --paginate --jq '.[].filename' \
        | grep -cE "$SECURITY_RE" || true)
      echo "security_files_changed=$COUNT" >> "$GITHUB_OUTPUT"
  • Files reviewed: 3/3 changed files
  • Comments generated: 1

Comment on lines +39 to +43
SECURITY_RE="host-iptables|setup-iptables|squid-config|docker-manager|seccomp-profile|domain-patterns|entrypoint\.sh|Dockerfile|(^|/)containers/"
COUNT=$(gh api "repos/${GH_REPO}/pulls/${PR_NUMBER}/files" \
--paginate --jq '.[].filename' \
| grep -cE "$SECURITY_RE" || true)
echo "count=$COUNT" >> "$GITHUB_OUTPUT"
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The relevance gate currently treats any gh api failure as COUNT=0 (because the pipeline yields no lines and grep -c returns 0), which would skip the agent job and potentially miss a required security review. Please detect GitHub API errors explicitly (e.g., capture filenames first and check exit status / enable set -o pipefail) and default to a safe value that still runs the agent when the file list cannot be retrieved.

Copilot uses AI. Check for mistakes.
@lpcox
Copy link
Copy Markdown
Collaborator

lpcox commented Apr 20, 2026

@copilot update the PR with this feedback #2113 (review)

@lpcox lpcox merged commit 690912a into main Apr 20, 2026
61 of 68 checks passed
@lpcox lpcox deleted the copilot/optimize-claude-token-costs branch April 20, 2026 14:56
Copilot AI added a commit that referenced this pull request Apr 20, 2026
…lower turn cap, and leaner prompt context (#2113)

* Initial plan

* feat: optimize security-guard token usage and gating

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/fa62b406-4f98-4c20-9901-392bfda30bfa

* fix: refine security relevance gating regex and non-pr fallback

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/fa62b406-4f98-4c20-9901-392bfda30bfa

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copilot AI added a commit that referenced this pull request Apr 20, 2026
…lower turn cap, and leaner prompt context (#2113)

* Initial plan

* feat: optimize security-guard token usage and gating

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/fa62b406-4f98-4c20-9901-392bfda30bfa

* fix: refine security relevance gating regex and non-pr fallback

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/fa62b406-4f98-4c20-9901-392bfda30bfa

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

⚡ Claude Token Optimization2026-04-19 — security-guard

3 participants