[aw-failures] Smoke suite: 4 engines failing on every run for 5+ days — no root-cause fix deployed

[github-actions]

Executive Summary

The agentic smoke test suite (engines: Gemini, Crush, OpenCode, Copilot) has been failing on every single run for at least 5 consecutive days (Apr 25–30, 2026). Each batch generates 4 auto-expiring failure issues that get closed as not_planned without fixing root causes, creating a silent feedback loop with no systemic remediation.

This issue consolidates the 6-hour failure window analysis (2026-04-29T19:19–2026-04-30T01:19 UTC) and tracks root-cause fixes.

Failure Clusters

Engine	Workflow	Failure Type	Recurrence	Current Issue
gemini	smoke-gemini	API key invalid (HTTP 400, INVALID_ARGUMENT)	Every run, 5+ days	#29211
crush	smoke-crush	EROFS read-only filesystem install failure	Every run, 5+ days	#29207
opencode	smoke-opencode	No safe outputs generated (agent never calls noop)	Every run, 5+ days	#29208
copilot	smoke-copilot	Exit code 1 (generic, varies)	Recurring	#29206

Evidence: Audit findings for representative runs

Smoke Gemini (§25138536109):

• Error: (HTTP 400, reason: , service: )
• Token usage: 0 — engine fails before any inference
• Historical pattern: Same error in every run ([aw] Smoke Gemini failed #28454, [aw] Smoke Gemini failed #28530, [aw] Smoke Gemini failed #28878, [aw] Smoke Gemini failed #29014, [aw] Smoke Gemini failed #29211)

Smoke Crush (§25138536143):

• Error:
• Identical error in every run; path is already populated by another package and is read-only
• Historical pattern: Same error in every run ([aw] Smoke Crush failed #28455, [aw] Smoke Crush failed #28531, [aw] Smoke Crush failed #28877, [aw] Smoke Crush failed #29012, [aw] Smoke Crush failed #29207)

Smoke OpenCode (§25138536100):

• GH Actions conclusion: success — runner finishes without error
• But: zero safe outputs produced; agent never calls noop
• Repeats identically across all runs ([aw] Smoke OpenCode failed #28414, [aw] Smoke OpenCode failed #28532, [aw] Smoke OpenCode failed #28879, [aw] Smoke OpenCode failed #29013, [aw] Smoke OpenCode failed #29208)

Smoke Copilot (§25138536142):

• Error: Process completed with exit code 1
• Varies in root cause between runs (previously: auth missing, now: generic exit)

Existing Issue Correlation

• [aw] Smoke Gemini failed #29211, [aw] Smoke OpenCode failed #29208, [aw] Smoke Crush failed #29207, [aw] Smoke Copilot failed #29206 — current open per-run failure trackers (expire ~2026-04-30T11:00 UTC, generated by automation)
• Prior instances were all closed as not_planned without deploying fixes
• No consolidated root-cause tracking exists — this issue fills that gap

Proposed Fix Roadmap

P0 — Blocks all smoke runs immediately

• Fix Gemini API key (credential rotation/setup) → sub-issue #aw_gemfix
• Fix Crush installation EROFS (workflow or package config) → sub-issue #aw_crfix

P1 — Workflow quality

• Ensure OpenCode smoke prompt calls noop when no action taken

P2 — Systemic

• Consider deduplicating auto-generated per-engine failure issues into a single recurring tracker to avoid issue churn

Sub-Issues Created

• #aw_gemfix — Fix invalid Gemini API key for smoke-gemini
• #aw_crfix — Fix smoke-crush EROFS installation failure

References:

• §25138536109 (Smoke Gemini failure)
• §25138536143 (Smoke Crush failure)
• §25138536100 (Smoke OpenCode failure)

Generated by [aw] Failure Investigator (6h) · ● 327.3K · ◷

• expires on May 7, 2026, 1:30 AM UTC

---

---

Update — Apr 30 2026 (run §25153298474)

New P0 Finding: MCP Tool Schema Incompatibility Blocking Smoke Claude

A new and previously untracked failure was identified in the 6-hour window (01:34–07:35 UTC):

Smoke Claude run §25149789871 (on PR branch copilot/fix-gemini-api-key-error) fails immediately with:

tools.57.custom.input_schema: input_schema does not support oneOf, allOf, or anyOf at the top level

The Anthropic API (/v1/messages) rejects the tool manifest on the first request — 0 tokens consumed. A separate Smoke Claude run (§25148772845) failed with all MCP servers dropping connection at 0s uptime (tracked in #29255).

Sub-issue created: #aw_mcpsch — fix MCP tool schema incompatibility

Status Improvements

• ✅ Smoke Copilot (§25149789919): succeeded on PR trigger (29 turns, 1M tokens) — no longer consistently failing
• ✅ Smoke OpenCode (§25149789880): concluded success on PR trigger

Persistent Failures (unchanged)

• ❌ Smoke Gemini — API key invalid, tracked in [aw-failures] Fix: Gemini API key invalid — smoke-gemini fails on every run (5+ days) #29233
• ❌ Smoke Crush — EROFS install failure, tracked in this issue

Generated by [aw] Failure Investigator (6h) · ● 484K · ◷