[observability] Observability Coverage Report - 2026-04-24

[github-actions[bot]]

Executive Summary

Analyzed 36 unique workflow runs from the last 7 days after deduplicating one duplicate entry in the MCP logs index. Coverage metrics below are calculated from 32 completed runs with run_summary.json, while 4 in-progress runs are tracked separately because their final artifacts have not been uploaded yet.

AWF Firewall coverage was 96.4% across 28 completed firewall-enabled runs, with 1 confirmed missing access.log artifact. MCP telemetry coverage was 93.8% across 32 completed agent runs, with 2 runs missing all MCP telemetry. No gateway.jsonl artifacts were observed; healthy MCP coverage in this window came entirely from the canonical fallback mcp-logs/rpc-messages.jsonl.

Overall completed-run observability coverage was 93.8% (30/32 runs meeting all applicable artifact checks). Additional telemetry was present in 30/32 completed runs for agent-stdio.log and 26/32 completed runs for safeoutputs.jsonl.

Key Alerts and Anomalies

🔴 Critical Issues:

• Smoke CI §24913094891 is firewall-enabled but missing sandbox/firewall/logs/access.log.
• Smoke CI §24914380736 is missing both gateway.jsonl and rpc-messages.jsonl.
• Smoke CI §24913094891 is missing both gateway.jsonl and rpc-messages.jsonl.

⚠️ Warnings:

• Smoke CI §24914380736 has no aw_info.json; firewall enablement could not be confirmed from artifacts.
• 4 run(s) are still in progress and were excluded from coverage percentages to avoid undercounting due to partial uploads.
• Raw JSONL and firewall file contents were only directly readable for a small sample because many downloaded artifact directories are mode-restricted; per-run status below is based on artifact manifests plus run_summary.json.

Coverage Summary

Component	Runs Analyzed	Logs Present	Coverage	Status
AWF Firewall (access.log)	28 (firewall_enabled_workflows)	27	96.4%	⚠️
MCP Gateway (gateway.jsonl or rpc-messages.jsonl)	32 (mcp_enabled_workflows)	30	93.8%	🔴

Detailed Run Analysis

Firewall-Enabled Runs

Workflow	Run ID	access.log	Entries	Allowed	Blocked	Status
Package Specification Librarian	§24916469921	✅	60	60	0	✅
Daily Project Performance Summary Generator (Using MCP Scripts)	§24916432030	✅	35	15	20	✅
Daily Skill Optimizer Improvements	§24916318508	✅	5	5	0	✅
Daily Safe Output Integrator	§24915731926	✅	26	14	12	✅
Smoke CI	§24915461608	✅	0	0	0	⚠️
Daily Secrets Analysis Agent	§24914522853	✅	15	7	8	✅
Smoke CI	§24914398618	✅	0	0	0	⚠️
Test Quality Sentinel	§24914376215	✅	13	13	0	✅
Design Decision Gate 🏗️	§24914376196	✅	9	9	0	✅
Agent Container Smoke Test	§24913976870	✅	6	3	3	✅
Smoke Copilot	§24913976862	✅	16	16	0	✅
Smoke Codex	§24913976859	✅	19	15	4	✅
Changeset Generator	§24913976852	✅	128	120	8	✅
Smoke Claude	§24913976844	✅	17	17	0	✅
Draft PR Cleanup	§24913958764	✅	24	24	0	✅
Daily Safe Outputs Conformance Checker	§24913950248	✅	8	8	0	✅
Release	§24913567782	✅	18	18	0	✅
Smoke CI	§24913380498	✅	0	0	0	⚠️
Design Decision Gate 🏗️	§24913376378	✅	9	9	0	✅
Test Quality Sentinel	§24913376376	✅	4	4	0	✅
Design Decision Gate 🏗️	§24913143084	✅	9	9	0	✅
Test Quality Sentinel	§24913143079	✅	10	10	0	✅
Smoke CI	§24913126991	✅	0	0	0	⚠️
Daily Compiler Quality Check	§24913114026	✅	79	46	33	✅
Smoke CI	§24913094891	❌	N/A	N/A	N/A	🔴
Test Quality Sentinel	§24913089154	✅	16	16	0	✅
Design Decision Gate 🏗️	§24913089153	✅	21	21	0	✅
Daily Code Metrics and Trend Tracking Agent	§24912924613	✅	633	633	0	✅

Missing Firewall Logs (access.log)

Workflow	Run ID	Date	Link
Smoke CI	24913094891	2026-04-24	§24913094891

MCP-Enabled Runs

Workflow	Run ID	Telemetry Source	Entries	Servers	Tool Calls	Errors	Status
Package Specification Librarian	§24916469921	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Daily Project Performance Summary Generator (Using MCP Scripts)	§24916432030	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Daily Skill Optimizer Improvements	§24916318508	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Daily Safe Output Integrator	§24915731926	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke CI	§24915461608	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Daily Secrets Analysis Agent	§24914522853	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke CI	§24914398618	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke CI	§24914380736	❌ None	N/A	N/A	N/A	N/A	🔴
Test Quality Sentinel	§24914376215	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Design Decision Gate 🏗️	§24914376196	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Agent Container Smoke Test	§24913976870	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke Copilot	§24913976862	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke Codex	§24913976859	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Changeset Generator	§24913976852	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke Gemini	§24913976851	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke Claude	§24913976844	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke OpenCode	§24913976843	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke Crush	§24913976826	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Draft PR Cleanup	§24913958764	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Daily Safe Outputs Conformance Checker	§24913950248	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Release	§24913567782	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke CI	§24913380498	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Design Decision Gate 🏗️	§24913376378	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Test Quality Sentinel	§24913376376	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Design Decision Gate 🏗️	§24913143084	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Test Quality Sentinel	§24913143079	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke CI	§24913126991	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Daily Compiler Quality Check	§24913114026	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Smoke CI	§24913094891	❌ None	N/A	N/A	N/A	N/A	🔴
Test Quality Sentinel	§24913089154	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Design Decision Gate 🏗️	§24913089153	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅
Daily Code Metrics and Trend Tracking Agent	§24912924613	rpc-messages.jsonl	N/A	N/A	N/A	N/A	✅

Missing MCP Telemetry (no gateway.jsonl or rpc-messages.jsonl)

Workflow	Run ID	Date	Link
Smoke CI	24914380736	2026-04-24	§24914380736
Smoke CI	24913094891	2026-04-24	§24913094891

In-Progress Runs

• Daily Safe Output Tool Optimizer §24916938567 is still in progress; final artifacts have not uploaded yet.
• Daily Observability Report for AWF Firewall and MCP Gateway §24916931881 is still in progress; final artifacts have not uploaded yet.
• Daily Semgrep Scan §24916897881 is still in progress; final artifacts have not uploaded yet.
• GitHub API Consumption Report Agent §24916593308 is still in progress; final artifacts have not uploaded yet.

Telemetry Quality Analysis

Firewall Log Quality

• Total access.log requests summarized: 1225
• Allowed requests: 1135
• Blocked requests: 90 (7.3%)
• Distinct domains observed: 20
• Most accessed domains: github.com:443 (604), api.githubcopilot.com:443 (241), api.anthropic.com:443 (104)
• Highest block pressure workflow: Daily Project Performance Summary Generator (Using MCP Scripts) with 20/35 blocked requests (57.1%)

MCP Log Quality

• Telemetry source observed in healthy runs: rpc-messages.jsonl only
• gateway.jsonl runs observed: 0
• rpc-messages.jsonl runs observed: 30
• Readable JSONL sample validation: 3 runs, all with valid JSONL and required top-level fields (timestamp, direction, type, server_id, payload)
• Sample readable runs:
  • 24913567782: 70 events, 33 outgoing tools/call requests, servers github, safeoutputs
  • 24913976844: 80 events, 33 outgoing tools/call requests, servers agenticworkflows, github, mcpscripts, playwright, safeoutputs, serena, tavily
  • 24913976862: 74 events, 31 outgoing tools/call requests, servers agenticworkflows, github, mcpscripts, playwright, safeoutputs, serena
• Aggregate MCP server usage across the fetched window: 7 servers with visible activity (agenticworkflows, github, mcpscripts, playwright, safeoutputs, serena, tavily)
• Aggregate tool call counts by server from the cached logs index: safeoutputs 62, github 55, serena 20, mcpscripts 7, playwright 5, agenticworkflows 1, tavily 1

Healthy Runs Summary

• 30 completed runs had full applicable observability coverage
• Healthy MCP coverage currently depends on the rpc-messages.jsonl fallback rather than gateway.jsonl
• Readable firewall samples contained both successful CONNECT 200 TCP_TUNNEL entries and proxy noise such as NONE_NONE:HIER_NONE error:transaction-end-before-headers; the summarized firewall counters remain the more reliable source for allow/block totals

Recommended Actions

1. Investigate why cancelled Smoke CI runs §24913094891 and §24914380736 exited after the agent job started but before MCP telemetry was uploaded.
2. Move or duplicate critical telemetry uploads earlier in the agent lifecycle so cancelled runs still retain access.log and MCP JSONL artifacts.
3. Add an explicit audit assertion for rpc-messages.jsonl presence on every completed agent run, and keep gateway.jsonl optional until that artifact is actually emitted in this repository.
4. Review the repeated block pressure in Daily Project Performance Summary Generator (Using MCP Scripts) and Daily Safe Output Integrator to confirm whether the blocked traffic is expected or indicates configuration drift.

Historical Trends

Historical trend comparison was not computed in this run. The current report is a point-in-time 7-day coverage audit based on the latest downloaded run set.

Context

• workflow_runs_analyzed: 36 unique runs
• Completed runs used for coverage: 32
• In-progress runs excluded from coverage: 4
• Analysis window: Last 7 days
• Generated on: 2026-04-24

References: §24913094891, §24914380736, §24916931881

Warning

Firewall blocked 4 domains

The following domains were blocked by the firewall during workflow execution:

• ab.chatgpt.com
• api.github.com
• chatgpt.com
• github.com

💡 Tip: api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

tools:
  github:
    mode: gh-proxy

See GitHub Tools for more information on gh-proxy mode.

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"
    - "api.github.com"
    - "chatgpt.com"
    - "github.com"

See Network Configuration for more information.

Generated by Daily Observability Report for AWF Firewall and MCP Gateway · ◷

• expires on Apr 25, 2026, 11:56 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Daily Observability Report for AWF Firewall and MCP Gateway.

A newer discussion is available at Discussion #28525.