Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 29 additions & 3 deletions .github/aw/debug-agentic-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,33 @@ When the user chooses to analyze existing logs:
- Provides JSON output with metrics, errors, and summaries
- Includes token usage, cost estimates, and execution time

2. **Analyze the Results**
2. **Token Usage Data Locations**

Token usage data is split across two separate artifacts. Always check the right one:

| Data | Artifact | Path | Content |
|------|----------|------|---------|
| Per-request detail | `firewall-audit-logs` | `api-proxy-logs/token-usage.jsonl` | JSONL with per-API-call model and raw token counts |
| Aggregated summary | `agent` | `agent_usage.json` | Total input/output/cache tokens, effective tokens |
Comment thread
lpcox marked this conversation as resolved.
| Step summary | Job log | "Parse token usage for step summary" step | Markdown table in step summary |

> **Important:** Per-request token detail is in the `firewall-audit-logs` artifact, **not** the `agent` artifact. `agent_usage.json` contains aggregated totals only.

```bash
# Download per-request token usage data
gh run download <run-id> -n firewall-audit-logs
cat firewall-audit-logs/api-proxy-logs/token-usage.jsonl
```

The `token-usage.jsonl` file contains one JSON object per API request with fields:
- `model`: Model name used (string, e.g. `"gpt-4.1"`)
- `input_tokens`, `output_tokens`: Raw token counts (integers)
- `cache_read_tokens`, `cache_write_tokens`: Cache-related tokens (integers; zero when caching is not used)
- Effective tokens are **not** stored per request in `token-usage.jsonl`; they are computed during aggregation and reported in `agent_usage.json` / the step summary

> **Note on terminology:** "Firewall network logs" (domain allow/deny decisions, parsed by `parse_firewall_logs.cjs`) are different from the "firewall-audit-logs artifact" (which contains `token-usage.jsonl` and API proxy logs).

3. **Analyze the Results**

Review the JSON output and identify:
- **Errors and Warnings**: Look for error patterns in logs
Expand All @@ -281,15 +307,15 @@ When the user chooses to analyze existing logs:
- **Execution Time**: Identify slow steps or timeouts
- **Success/Failure Patterns**: Analyze workflow conclusions

3. **Provide Insights**
4. **Provide Insights**

Based on the analysis, provide:
- Clear explanation of what went wrong (if failures exist)
- Specific recommendations for improvement
- Suggested workflow changes (frontmatter or prompt modifications)
- Command to apply fixes: `gh aw compile <workflow-name>`

4. **Iterative Refinement**
5. **Iterative Refinement**

If changes are made:
- Help user edit the workflow file
Expand Down
1 change: 1 addition & 0 deletions docs/src/content/docs/reference/audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ gh aw logs --format markdown --repo owner/repo --count 10
## Related Documentation

- [Cost Management](/gh-aw/reference/cost-management/) — Track token usage and inference spend
- [Artifacts](/gh-aw/reference/artifacts/) — Artifact names, directory structures, and token usage file locations (`token-usage.jsonl` in `firewall-audit-logs`)
- [Effective Tokens Specification](/gh-aw/reference/effective-tokens-specification/) — How effective tokens are computed
- [Network](/gh-aw/reference/network/) — Firewall and domain allow/deny configuration
- [MCP Gateway](/gh-aw/reference/mcp-gateway/) — MCP server health and debugging
Expand Down
1 change: 1 addition & 0 deletions docs/src/content/docs/reference/cost-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,7 @@ These are rough estimates to help with budgeting. Actual costs vary by prompt si
## Related Documentation

- [Audit Commands](/gh-aw/reference/audit/) - Single-run analysis, diff, and cross-run reporting
- [Artifacts](/gh-aw/reference/artifacts/) - Artifact names, directory structures, and token usage file locations
- [Effective Tokens Specification](/gh-aw/reference/effective-tokens-specification/) - How effective token counts are computed
- [Triggers](/gh-aw/reference/triggers/) - Configuring workflow triggers and skip conditions
- [Rate Limiting Controls](/gh-aw/reference/rate-limiting-controls/) - Preventing runaway workflows
Expand Down
Loading