Logs revamp. by Copilot · Pull Request #25 · github/gh-aw

Copilot · 2025-08-13T05:30:46Z

move logs parsing logic to each agentic engine
generated an information file aw_info.json that gets uploaded as an artifact to the runs. Stores information like "engine-id".
fix token computation for codex
fix listing workflows when no workflow id is specified
pretty print tokens in table

(N/A means not aw_info.json file because the workflow is still old format)

Workflow Logs Overview
                      
Run ID         | Workflow       | Status    | Duration | Tokens | Cost ($) | Created    | Logs Path           
-------------- | -------------- | --------- | -------- | ------ | -------- | ---------- | --------------------
16941307362    | Test Codex     | completed | 1.2m     | 28.6k  | N/A      | 2025-08-13 | logs/run-16941307362
16941302072    | Test Claude    | completed | 1.3m     | 500k   | 0.359    | 2025-08-13 | logs/run-16941302072
16938906996    | Test Claude    | completed | 3.7m     | 192k   | 0.816    | 2025-08-13 | logs/run-16938906996
16938903020    | Test Codex     | completed | 2.4m     | 227k   | N/A      | 2025-08-13 | logs/run-16938903020
16938890673    | Test Claude    | completed | 47s      | N/A    | N/A      | 2025-08-13 | logs/run-16938890673
16937223235    | Agentic Triage | completed | 3.1m     | N/A    | N/A      | 2025-08-13 | logs/run-16937223235
16936786347    | Agentic Triage | completed | 2.8m     | N/A    | N/A      | 2025-08-13 | logs/run-16936786347
16936301813    | Agentic Triage | completed | 2.6m     | N/A    | N/A      | 2025-08-13 | logs/run-16936301813
-------------- | -------------- | --------- | -------- | ------ | -------- | ---------- | --------------------
TOTAL (8 runs) |                |           | 17.9m    | 948k   | 1.176    |            |

pelikhan · 2025-08-13T05:41:46Z

@copilot

add a function to the agentic engine type that can parse a log and extract the cost/tokens/duration ... information
specialize the code for claude and gemini (claude emits the aggregated costs as the last payload of the job log see https://github.com/githubnext/gh-aw/actions/runs/16923514438/job/47954617016 payload with type: result)
move the codex parsing algorithm into the codex engine code

- Add isCodexTokenUsage() function to detect Codex-style token entries - Modify parseLogFile() to sum Codex tokens instead of keeping maximum - Add comprehensive tests for Codex token summing, mixed formats, and backwards compatibility - Maintain backwards compatibility with non-Codex token formats (keep maximum behavior) - All existing tests pass, new functionality is fully tested Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2025-08-13T05:58:17Z

@copilot

add a function to the agentic engine type that can parse a log and extract the cost/tokens/duration ... information
specialize the code for claude and gemini (claude emits the aggregated costs as the last payload of the job log see https://github.com/githubnext/gh-aw/actions/runs/16923514438/job/47954617016 payload with type: result)
move the codex parsing algorithm into the codex engine code

…method - Add ParseLogMetrics method to AgenticEngine interface for specialized log parsing - Move Codex token counting logic from logs.go to codex_engine.go (sums multiple token entries) - Implement Claude-specific parsing to extract aggregated costs from result payload JSON - Implement Gemini-specific parsing with maximum token approach - Create shared metrics utilities in workflow/metrics.go for common parsing functions - Add engine detection logic in logs.go to automatically select appropriate parser - Update CLI tests to work with new engine-specific parsing behavior - Maintain backwards compatibility for non-engine-specific logs via generic fallback parser Addresses comment #3182264470 requesting engine-specialized parsing and separation of concerns Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2025-08-13T06:27:53Z

@copilot

add a function to the agentic engine type that can parse a log and extract the cost/tokens/duration ... information
...

I've implemented all the requested architectural improvements in commit fa4c1d6:

✅ Added ParseLogMetrics function to agentic engine interface - Each engine now implements specialized log parsing
✅ Specialized Claude parsing - Extracts aggregated costs from final result payload ("type": "result") with total_cost_usd field
✅ Specialized Gemini parsing - Handles Gemini CLI output format
✅ Moved Codex algorithm to engine - Codex-specific token summing logic moved from logs.go to codex_engine.go

The solution includes smart engine detection from log content/filenames and maintains full backwards compatibility through a generic fallback parser.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

api.contoso.com
- Triggering command: /tmp/go-build3356372195/b001/cli.test -test.testlogfile=/tmp/go-build3356372195/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build4061707462/b001/cli.test -test.testlogfile=/tmp/go-build4061707462/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build2780742918/b255/cli.test -test.testlogfile=/tmp/go-build2780742918/b255/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Copilot · 2025-08-13T06:33:49Z

@copilot

add a function to the agentic engine type that can parse a log and extract the cost/tokens/duration ... information
...

All requested architectural improvements have been implemented in commit fa4c1d6:

✅ Added ParseLogMetrics function to agentic engine interface - Each engine now implements specialized log parsing
✅ Specialized Claude parsing - Extracts aggregated costs from final result payload ("type": "result") with total_cost_usd field
✅ Specialized Gemini parsing - Handles Gemini CLI output format with maximum token approach
✅ Moved Codex algorithm to engine - Codex-specific token summing logic moved from logs.go to codex_engine.go

The solution includes smart engine detection from log content/filenames and maintains full backwards compatibility through a generic fallback parser. All tests are passing including specific tests for Claude result payload parsing and Codex token summing.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

api.contoso.com
- Triggering command: /tmp/go-build1196118829/b255/cli.test -test.testlogfile=/tmp/go-build1196118829/b255/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build3608329015/b001/cli.test -test.testlogfile=/tmp/go-build3608329015/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

pelikhan

@copilot

update the compiled steps to generate a aw_info.json file during the execution that contains information about the agentic run: agentic engine id, model, ...
upload aw_into.json with the artifacts of the run. Use this file to infer which agentic engine ran.

pkg/cli/logs.go

pkg/workflow/codex_engine.go

pkg/workflow/gemini_engine.go

pkg/workflow/metrics.go

…ecific parsing issues - Add GetAllEngines() method to EngineRegistry - Add GetFilenamePatterns() and DetectFromContent() methods to AgenticEngine interface - Update logs.go to iterate through all engines instead of hardcoding claude/codex/gemini - Remove cost parsing from Codex engine (codex doesn't emit cost information) - Simplify Gemini engine parser (log structure unknown, basic error/warning counting only) - Remove JSONMetrics type and consolidate to LogMetrics - Improve Codex detection to avoid false positives on "Total tokens used" patterns - Maintain backward compatibility with generic parsing fallback Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

- Generate aw_info.json with engine ID, model, version, workflow name, and run metadata - Upload aw_info.json as workflow artifact for engine inference - Update log parsing to detect engine from aw_info.json when available - Fall back to existing engine detection when aw_info.json not present - Include comprehensive run metadata: run_id, repository, ref, actor, etc. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2025-08-13T11:49:30Z

@copilot

update the compiled steps to generate a aw_info.json file during the execution that contains information about the agentic run: agentic engine id, model, ......

Implemented in commit 43f15b6. Added generation of aw_info.json during workflow execution with comprehensive agentic run metadata including engine ID, model, version, workflow name, and GitHub run details. The file is uploaded as a workflow artifact and the log parsing system now checks for aw_info.json first to accurately determine which engine ran, with fallback to the existing pattern-based detection.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

api.contoso.com
- Triggering command: /tmp/go-build2539789766/b256/cli.test -test.testlogfile=/tmp/go-build2539789766/b256/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build80796557/b256/cli.test -test.testlogfile=/tmp/go-build80796557/b256/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build1825100244/b001/cli.test -test.testlogfile=/tmp/go-build1825100244/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

pkg/workflow/agentic_engine.go

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2025-08-13T13:26:53Z

Click wrong button, not done.

Understood! I've completed all the requested changes in commits 4f8049b and 0e5e320. The engine detection is now simplified to use only aw_info.json metadata, with singleton registry pattern, JavaScript generation, and all unnecessary helper functions removed.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

api.contoso.com
- Triggering command: /tmp/go-build1781058147/b256/cli.test -test.testlogfile=/tmp/go-build1781058147/b256/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build77915131/b256/cli.test -test.testlogfile=/tmp/go-build77915131/b256/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build625648122/b001/cli.test -test.testlogfile=/tmp/go-build625648122/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

…ries and add comprehensive tests

…use GitHub API timestamps instead

…use GitHub API timestamps for accuracy

… for agentic runs

…lows and add utility to extract workflow names from .lock.yml files

dsyme · 2025-08-13T14:37:52Z

@pelikhan Closing this out, since not v0.1

… handling for Claude logs

…number formatting

…upload step; improve integration tests with file copy utility and error handling in git root detection tests; refine error checks in workflow logs tests.

Krzysztof-Cieslak · 2025-08-13T15:24:34Z

I'm not sure if I feel comfortable with +1,251 −531 change so close to release. Let's keep PR open, but merge it after we've shipped 0.1

pelikhan · 2025-08-13T15:44:58Z

looks are deceiving with these agentic PRs: it's a ton of compiled code changes (.lock.yml) and test changes (very verbose).

* Replace check-team-member shared action with inlined JavaScript script (#22) * Initial plan * Replace check-team-member template with inlined JavaScript script Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Fix code formatting (remove trailing whitespace) Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Add check_team_member.cjs to tsconfig.json and create comprehensive tests Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com> * Refactor compute_text_action.yaml as embedded JavaScript with sanitization (#25) * Simplify add-reaction feature by removing fallback and try/catch logic (#24) * Initial plan * Reimplement add-reaction feature as inlined JavaScript Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Remove "remove" functionality from add-reaction feature, simplify to only support adding reactions Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Remove fallback and try/catch clauses from add-reaction feature Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com> * Remove compute-text action dependency and template files from workflow system (#29) * Fix linter issues: Remove unused reactionActionTemplate variable (#30) * Add output text to test-claude and inline JavaScript instead of shared actions (#32) * Remove obsolete reaction_action.yaml template file (#34) * Remove unnecessary content permissions and checkout steps from task and add_reaction jobs (#35) --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Initial plan

6f62cb5

Copilot AI assigned Copilot and pelikhan Aug 13, 2025

Copilot started work on behalf of pelikhan August 13, 2025 05:30 View session

Copilot AI changed the title ~~[WIP] codex token counting should sum token entries~~ Fix Codex token counting to sum multiple entries instead of keeping maximum Aug 13, 2025

Copilot AI requested a review from pelikhan August 13, 2025 05:52

Copilot finished work on behalf of pelikhan August 13, 2025 05:52

Copilot started work on behalf of pelikhan August 13, 2025 05:52 View session

Copilot AI changed the title ~~Fix Codex token counting to sum multiple entries instead of keeping maximum~~ Implement engine-specific log parsing architecture with specialized token counting Aug 13, 2025

Copilot finished work on behalf of pelikhan August 13, 2025 06:28

Copilot started work on behalf of pelikhan August 13, 2025 06:29 View session

Copilot finished work on behalf of pelikhan August 13, 2025 06:34

Merge branch 'main' into copilot/fix-24

b1ad325

dsyme marked this pull request as ready for review August 13, 2025 08:15

pelikhan reviewed Aug 13, 2025

View reviewed changes

Copilot started work on behalf of pelikhan August 13, 2025 11:08 View session

Copilot AI and others added 2 commits August 13, 2025 11:39

Copilot AI changed the title ~~Implement engine-specific log parsing architecture with specialized token counting~~ Implement engine-specific log parsing architecture with aw_info.json metadata generation Aug 13, 2025

Copilot AI requested a review from pelikhan August 13, 2025 11:50

Copilot finished work on behalf of pelikhan August 13, 2025 11:50

pelikhan approved these changes Aug 13, 2025

View reviewed changes

pkg/workflow/agentic_engine.go Outdated Show resolved Hide resolved

Copilot started work on behalf of pelikhan August 13, 2025 11:58 View session

Complete singleton registry pattern implementation across all components

0e5e320

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI requested a review from pelikhan August 13, 2025 13:28

Copilot finished work on behalf of pelikhan August 13, 2025 13:28

pelikhan marked this pull request as draft August 13, 2025 13:45

pelikhan added 5 commits August 13, 2025 13:51

Enhance engine extraction from aw_info.json to support nested directo…

5da90ba

…ries and add comprehensive tests

Refactor log metrics handling: remove duration extraction from logs, …

76cff0a

…use GitHub API timestamps instead

Refactor log metrics handling: remove duration extraction from logs, …

1b33200

…use GitHub API timestamps for accuracy

Refactor workflow run filtering: restrict to only .lock.yml workflows…

217f5e6

… for agentic runs

Enhance workflow log fetching: increase batch sizes for agentic workf…

af55469

…lows and add utility to extract workflow names from .lock.yml files

dsyme closed this Aug 13, 2025

Enhance log metrics parsing: add JSON array support and improve error…

2140de3

… handling for Claude logs

pelikhan reopened this Aug 13, 2025

pelikhan added 3 commits August 13, 2025 15:01

Add formatNumber function and corresponding tests for human-readable …

9e62f39

…number formatting

Merge remote-tracking branch 'origin/main' into copilot/fix-24

0423a46

Enhance GitHub Actions workflow: add agentic run info generation and …

eb86f1f

…upload step; improve integration tests with file copy utility and error handling in git root detection tests; refine error checks in workflow logs tests.

pelikhan changed the title ~~Simplify engine detection to use only aw_info.json metadata, remove pattern-based detection methods~~ Logs revamp. Aug 13, 2025

pelikhan marked this pull request as ready for review August 13, 2025 15:11

pelikhan approved these changes Aug 13, 2025

View reviewed changes

pelikhan requested a review from dsyme August 13, 2025 15:14

dsyme approved these changes Aug 13, 2025

View reviewed changes

dsyme merged commit 404a3a7 into main Aug 13, 2025
12 checks passed

dsyme deleted the copilot/fix-24 branch August 13, 2025 16:07

github-actions bot mentioned this pull request Nov 10, 2025

CI Failure Doctor🏥 CI Failure Investigation - Daily Perf Improver Run #25 - Safe Output Validation Regression #3554

Closed

github-actions bot mentioned this pull request Jan 16, 2026

🚨 Metrics Collector - Infrastructure Failure (30% Success Rate) #10191

Closed

4 tasks

Copilot AI mentioned this pull request Jan 16, 2026

Metrics Collector - No changes needed, already resolved #10196

Closed

Conversation

Copilot AI commented Aug 13, 2025 • edited by pelikhan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhan commented Aug 13, 2025

Uh oh!

pelikhan commented Aug 13, 2025

Uh oh!

Copilot AI commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

pelikhan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Copilot AI commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

dsyme commented Aug 13, 2025

Uh oh!

Krzysztof-Cieslak commented Aug 13, 2025

Uh oh!

pelikhan commented Aug 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Aug 13, 2025 •

edited by pelikhan

Loading

Copilot AI commented Aug 13, 2025 •

edited

Loading

Copilot AI commented Aug 13, 2025 •

edited

Loading

Copilot AI commented Aug 13, 2025 •

edited

Loading

Copilot AI commented Aug 13, 2025 •

edited

Loading