Skip to content

[aw-failures] [aw] Failure Investigation Report — 2026-04-28 (6h window) #28869

@github-actions

Description

@github-actions

Executive Summary

This investigation covers 01:35–07:35 UTC on 2026-04-28. Out of 42 total runs in the window, 6 concluded as failures across 3 root-cause clusters. No previously-open issues appear fixed or stale.

Failure Clusters

Priority Cluster Workflow(s) Failures Root Cause Existing Issue
P0 A Smoke CI 3 safeoutputs: command not found (exitCode=127) in copilot chroot #28858
P1 B Daily Hippo Learn, Code Simplifier 2 Threat detection model not emitting THREAT_DETECTION_RESULT #28866
P1 C Contribution Check 1 safe_outputs job failure (root cause unclear)

Also tracked: gpclean exit-code-22 failure in #28855 (outside 6h scope).

Evidence

Cluster A — Smoke CI: safeoutputs not on PATH in copilot chroot (3 runs)

All 3 runs fail identically on the first attempt with no partial execution; all retries exhaust:

[entrypoint][WARN] Failed to transfer /host/home/runner/work/_temp/gh-aw/safeoutputs ownership to chroot user
...
[copilot-harness] attempt 1: spawning: /tmp/gh-aw/engine-command.sh --add-dir /tmp/gh-aw/ ...
--add-dir: line 1: safeoutputs: command not found
[copilot-harness] attempt 1: process exit event exitCode=127
[copilot-harness] all 3 retries exhausted — giving up (exitCode=127)
  • Engine: GitHub Copilot CLI v1.0.36, firewall v0.25.28
  • Run timestamps: 05:21Z, 05:32Z, 05:56Z — all within 35 minutes
  • Deterministic failure: retrying never changes the outcome
Cluster B — Threat detection parse failure (2 runs)

Both workflows ran their agent successfully (18 turns, 543k and 874k tokens) but failed in the detection job:

📊 Detection log stats: 101 lines, 4885 bytes
📄 No lines containing THREAT_DETECTION_RESULT found in 101 lines
##[error]❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log.
Expected: THREAT_DETECTION_RESULT:{"prompt_injection":bool,"secret_leak":bool,"malicious_patch":bool,"reasons":[...]}

Both runs share identical detection log stats (101 lines, 4885 bytes) — suggesting the same input content or a reproducible formatting failure. GH_AW_DETECTION_CONTINUE_ON_ERROR: true was set but the detection job conclusion is still failure, propagating to workflow failure.

Cluster C — Contribution Check: safe_outputs failure (1 run)

Run §25036239526 — 29 turns, 998k tokens. Agent ran to completion and produced a safe-outputs-items.zip artifact (364 bytes). The safe_outputs job is listed as failed. Tail logs show only successful artifact upload and git cleanup, so the error is in an earlier step. Output variable names (create_discussion_error_count, code_push_failure_count) suggest a failed GitHub write operation. Root cause requires deeper log inspection.

Existing Issue Correlation

Issue Status Action
#28858 Smoke CI failed Open Covers only first run; 2 additional runs confirmed identical failure
#28866 Detection Runs Open General tracking issue; updated with specific run evidence
#28855 gpclean failed Open Pre-6h; exit code 22; no new evidence
#28861 Workflows out of sync Open Lock file drift; unrelated to current failures
#28857 Q fix Open Pending manual PR; unrelated

No issues are fixed or stale based on current evidence.

Proposed Fix Roadmap

Priority Fix Issue
P0 Fix safeoutputs binary not on PATH in copilot chroot #28871
P1 Investigate detection model format compliance — check prompt truncation or output format reinforcement
P1 Diagnose Contribution Check safe_outputs failure — inspect full step logs for the write operation error

Sub-issues Created

References:

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions