From 5c5475c41b87da4a6142da656f882a6e23ccf93f Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 11 Apr 2026 21:33:38 +0000 Subject: [PATCH 1/4] Initial plan From 43ec13a34bc161440e5c3f1739c290ed978ea842 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 11 Apr 2026 21:39:31 +0000 Subject: [PATCH 2/4] fix: add token usage artifact locations to debug agent prompt - Add 'Token Usage Data Locations' section to debug-agentic-workflow.md showing exact artifact names, file paths, download commands, and field descriptions for token-usage.jsonl vs agent_usage.json - Add terminology note distinguishing firewall network logs from the firewall-audit-logs artifact - Add cross-reference link to artifacts.md from cost-management.md - Add cross-reference link to artifacts.md from audit.md Agent-Logs-Url: https://github.com/github/gh-aw/sessions/4018c614-db61-4f66-bf8a-fb7837325e0e Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com> --- .github/aw/debug-agentic-workflow.md | 32 +++++++++++++++++-- docs/src/content/docs/reference/audit.md | 1 + .../content/docs/reference/cost-management.md | 1 + 3 files changed, 31 insertions(+), 3 deletions(-) diff --git a/.github/aw/debug-agentic-workflow.md b/.github/aw/debug-agentic-workflow.md index f584661b53..f42b2571f7 100644 --- a/.github/aw/debug-agentic-workflow.md +++ b/.github/aw/debug-agentic-workflow.md @@ -272,7 +272,33 @@ When the user chooses to analyze existing logs: - Provides JSON output with metrics, errors, and summaries - Includes token usage, cost estimates, and execution time -2. **Analyze the Results** +2. **Token Usage Data Locations** + + Token usage data is split across two separate artifacts. Always check the right one: + + | Data | Artifact | Path | Content | + |------|----------|------|---------| + | Per-request detail | `firewall-audit-logs` | `api-proxy-logs/token-usage.jsonl` | JSONL with per-API-call token counts, model, effective tokens | + | Aggregated summary | `agent` | `agent_usage.json` | Total input/output/cache tokens, effective tokens | + | Step summary | Job log | "Parse token usage" step | Markdown table in step summary | + + > **Important:** Per-request token detail is in the `firewall-audit-logs` artifact, **not** the `agent` artifact. `agent_usage.json` contains aggregated totals only. + + ```bash + # Download per-request token usage data + gh run download -n firewall-audit-logs + cat firewall-audit-logs/api-proxy-logs/token-usage.jsonl + ``` + + The `token-usage.jsonl` file contains one JSON object per API request with fields: + - `model`: Model name used (string, e.g. `"gpt-4.1"`) + - `input_tokens`, `output_tokens`: Raw token counts (integers) + - `cache_read_tokens`, `cache_write_tokens`: Cache-related tokens (integers; zero when caching is not used) + - `effective_tokens`: Cost-normalized token count (integer) + + > **Note on terminology:** "Firewall network logs" (domain allow/deny decisions, parsed by `parse_firewall_logs.cjs`) are different from the "firewall-audit-logs artifact" (which contains `token-usage.jsonl` and API proxy logs). + +3. **Analyze the Results** Review the JSON output and identify: - **Errors and Warnings**: Look for error patterns in logs @@ -281,7 +307,7 @@ When the user chooses to analyze existing logs: - **Execution Time**: Identify slow steps or timeouts - **Success/Failure Patterns**: Analyze workflow conclusions -3. **Provide Insights** +4. **Provide Insights** Based on the analysis, provide: - Clear explanation of what went wrong (if failures exist) @@ -289,7 +315,7 @@ When the user chooses to analyze existing logs: - Suggested workflow changes (frontmatter or prompt modifications) - Command to apply fixes: `gh aw compile ` -4. **Iterative Refinement** +5. **Iterative Refinement** If changes are made: - Help user edit the workflow file diff --git a/docs/src/content/docs/reference/audit.md b/docs/src/content/docs/reference/audit.md index 222af046ab..2ac7e1dadc 100644 --- a/docs/src/content/docs/reference/audit.md +++ b/docs/src/content/docs/reference/audit.md @@ -131,6 +131,7 @@ gh aw logs --format markdown --repo owner/repo --count 10 ## Related Documentation - [Cost Management](/gh-aw/reference/cost-management/) — Track token usage and inference spend +- [Artifacts](/gh-aw/reference/artifacts/) — Artifact names, directory structures, and token usage file locations (`token-usage.jsonl` in `firewall-audit-logs`) - [Effective Tokens Specification](/gh-aw/reference/effective-tokens-specification/) — How effective tokens are computed - [Network](/gh-aw/reference/network/) — Firewall and domain allow/deny configuration - [MCP Gateway](/gh-aw/reference/mcp-gateway/) — MCP server health and debugging diff --git a/docs/src/content/docs/reference/cost-management.md b/docs/src/content/docs/reference/cost-management.md index 93372063cc..28c239986b 100644 --- a/docs/src/content/docs/reference/cost-management.md +++ b/docs/src/content/docs/reference/cost-management.md @@ -225,6 +225,7 @@ These are rough estimates to help with budgeting. Actual costs vary by prompt si ## Related Documentation - [Audit Commands](/gh-aw/reference/audit/) - Single-run analysis, diff, and cross-run reporting +- [Artifacts](/gh-aw/reference/artifacts/) - Artifact names, directory structures, and token usage file locations - [Effective Tokens Specification](/gh-aw/reference/effective-tokens-specification/) - How effective token counts are computed - [Triggers](/gh-aw/reference/triggers/) - Configuring workflow triggers and skip conditions - [Rate Limiting Controls](/gh-aw/reference/rate-limiting-controls/) - Preventing runaway workflows From db92324cfe7787af5d780e7fe512dc4773251cb7 Mon Sep 17 00:00:00 2001 From: Landon Cox Date: Sat, 11 Apr 2026 14:48:04 -0700 Subject: [PATCH 3/4] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .github/aw/debug-agentic-workflow.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/aw/debug-agentic-workflow.md b/.github/aw/debug-agentic-workflow.md index f42b2571f7..95662e4032 100644 --- a/.github/aw/debug-agentic-workflow.md +++ b/.github/aw/debug-agentic-workflow.md @@ -278,7 +278,7 @@ When the user chooses to analyze existing logs: | Data | Artifact | Path | Content | |------|----------|------|---------| - | Per-request detail | `firewall-audit-logs` | `api-proxy-logs/token-usage.jsonl` | JSONL with per-API-call token counts, model, effective tokens | + | Per-request detail | `firewall-audit-logs` | `api-proxy-logs/token-usage.jsonl` | JSONL with per-API-call model and raw token counts | | Aggregated summary | `agent` | `agent_usage.json` | Total input/output/cache tokens, effective tokens | | Step summary | Job log | "Parse token usage" step | Markdown table in step summary | @@ -294,7 +294,7 @@ When the user chooses to analyze existing logs: - `model`: Model name used (string, e.g. `"gpt-4.1"`) - `input_tokens`, `output_tokens`: Raw token counts (integers) - `cache_read_tokens`, `cache_write_tokens`: Cache-related tokens (integers; zero when caching is not used) - - `effective_tokens`: Cost-normalized token count (integer) + - Effective tokens are **not** stored per request in `token-usage.jsonl`; they are computed during aggregation and reported in `agent_usage.json` / the step summary > **Note on terminology:** "Firewall network logs" (domain allow/deny decisions, parsed by `parse_firewall_logs.cjs`) are different from the "firewall-audit-logs artifact" (which contains `token-usage.jsonl` and API proxy logs). From 92c1cb12a8f4622e91d7b8adb38f6152e67eec11 Mon Sep 17 00:00:00 2001 From: Landon Cox Date: Sat, 11 Apr 2026 14:48:19 -0700 Subject: [PATCH 4/4] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .github/aw/debug-agentic-workflow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/aw/debug-agentic-workflow.md b/.github/aw/debug-agentic-workflow.md index 95662e4032..8c301248ce 100644 --- a/.github/aw/debug-agentic-workflow.md +++ b/.github/aw/debug-agentic-workflow.md @@ -280,7 +280,7 @@ When the user chooses to analyze existing logs: |------|----------|------|---------| | Per-request detail | `firewall-audit-logs` | `api-proxy-logs/token-usage.jsonl` | JSONL with per-API-call model and raw token counts | | Aggregated summary | `agent` | `agent_usage.json` | Total input/output/cache tokens, effective tokens | - | Step summary | Job log | "Parse token usage" step | Markdown table in step summary | + | Step summary | Job log | "Parse token usage for step summary" step | Markdown table in step summary | > **Important:** Per-request token detail is in the `firewall-audit-logs` artifact, **not** the `agent` artifact. `agent_usage.json` contains aggregated totals only.