[aw-failures] Codex agent container: read-only filesystem prevents execution (100% failure)

[github-actions]

Problem Statement

All Codex-engine workflows that run inside the agent container environment fail before executing a single agent turn due to a read-only file system error. Failure rate: 100% across all observed runs in the last 6 hours.

Affected Workflows and Run IDs

Workflow	Run ID	Engine	SHA
Changeset Generator	§24541851028	codex	PR branch fix/docker-socket-gid-shell-expansion
Changeset Generator	§24541517687	codex	PR branch fix/docker-socket-gid-shell-expansion
Daily Observability Report	§24539895233	codex	main (d73f38a)

Probable Root Cause

The codex CLI performs first-run initialization that attempts to write/update PATH and binary locations. Inside the chroot agent container, key directories are mounted read-only:

[entrypoint] CLI bin directory locked (read-only): /home/runner/work/_temp/gh-aw/mcp-cli/bin
WARNING: proceeding, even though we could not update PATH: Read-only file system (os error 30)
Error: Read-only file system (os error 30)
```

The error occurs after the chroot switches to the unprivileged `awfuser` (UID 1001) and executes the agent command. The `codex` binary tries to update its own PATH configuration (likely writing to a user config dir or a CLI-managed bin symlink), which hits a read-only boundary inside the container mount setup.

Related warning logged by the entrypoint:
```
[entrypoint][WARN] Failed to transfer /host/home/runner/work/_temp/gh-aw/safeoutputs ownership to chroot user
```

This suggests the chroot ownership-transfer step for `_temp/gh-aw/` is not fully succeeding, leaving some mounts inaccessible to the chroot user.

### Evidence

From `agent-stdio.log` of run §24541851028 (Changeset Generator):
```
[entrypoint] Chroot mode enabled - dropping CAP_SYS_CHROOT and CAP_SYS_ADMIN
[entrypoint] Switching to awfuser (UID: 1001, GID: 1001)
[entrypoint] Executing command: /bin/bash -c /bin/bash -c 'export PATH="\$\{RUNNER_TEMP}/gh-aw/mcp-cli/bin:$PATH" && ... codex ...'
[entrypoint][WARN] Failed to transfer /host/home/runner/work/_temp/gh-aw/safeoutputs ownership to chroot user
WARNING: proceeding, even though we could not update PATH: Read-only file system (os error 30)
Error: Read-only file system (os error 30)
```

From run §24539895233 (Daily Observability Report):
```
CLI bin directory locked (read-only): /home/runner/work/_temp/gh-aw/mcp-cli/bin
Error: Read-only file system (os error 30)

Proposed Remediation

1. Investigate mount permissions: Check how RUNNER_TEMP/gh-aw/mcp-cli/bin is mounted in the agent container — if it's mounted :ro (read-only), change to :rw or ensure codex PATH setup doesn't need to write there.
2. Fix ownership transfer: Resolve the Failed to transfer /host/home/runner/work/_temp/gh-aw/safeoutputs ownership warning — this may be blocking the codex CLI from accessing temp files.
3. Codex CLI PATH workaround: If codex unconditionally writes to its bin dir on startup, consider pre-seeding the expected directory contents before the chroot drops privileges, similar to how the Maven/Gradle proxy configs are pre-seeded.
4. Test: Run Changeset Generator and Daily Observability Report post-fix and confirm both complete successfully.

Success Criteria

• Changeset Generator completes with at least 1 agent turn and a valid safe-output
• Daily Observability Report completes and produces output
• No Read-only file system errors in agent-stdio.log

Parent report: #26779
Related to #26779

Generated by [aw] Failure Investigator (6h) · ● 940.2K · ◷

• expires on Apr 24, 2026, 1:24 AM UTC

[github-actions]

🍪 Issue Monster has assigned this to Copilot!

I've identified this issue as a good candidate for automated resolution and assigned it to the Copilot coding agent.

The Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · ● 280.3K · ◷