[awf] Agent + API Proxy: P0/P1 failure report — health check intermittent + node not found in copilot container

[lpcox]

Problem

A failure report covering two active failure clusters in AWF workflows observed 2026-04-28 07:00–13:00 UTC (28 failed/cancelled out of 40 total runs):

1. P0 — awf-api-proxy health check intermittent failure: 3+ confirmed runs failed before any agent turns due to awf-api-proxy container health check timeout during docker compose up
2. P1 — node: command not found in copilot agent container: 2+ confirmed runs fail at agent invocation because node is not on PATH in the copilot engine container

27 PR-triggered cancellations are normal concurrency-cancel behavior.

Context

• Source issue: [aw-failures] P0/P1 Failure Report (2026-04-28 07:00–13:00 UTC): awf-api-proxy health check + copilot node-not-found gh-aw#28947
• P0 also tracked in [aw-failures] Fix P0: awf-api-proxy container intermittently fails health check, blocking agent startup gh-aw#28949
• P1 affected runs: 25048535353 (Daily Issues Report Generator), 25044675410 (Daily News)
• P1 failure: /bin/bash: line 1: node: command not found — containers start successfully but node binary is not accessible inside the agent chroot

Root Cause

P0 (api-proxy health check): See companion tracking issue for details. Health check start_period/retries insufficient for runner load variance.

P1 (node not found): The copilot engine requires node to be on PATH inside the agent container chroot. The agent container (containers/agent/) selectively bind-mounts host binaries under /host/ (see src/docker-manager.ts bind mounts section), but node installed on the host (e.g., via nvm or in /home/runner/.nvm/) may not be included in the whitelisted mount paths (/usr, /bin, /sbin, /lib, /lib64, /opt).

If node is installed under /home/runner/.nvm/ or a non-standard path not included in the bind mounts, it will be unavailable inside the chroot. The entrypoint.sh in containers/agent/ sets up the chroot but does not add extra PATH entries for non-standard node installations.

Proposed Solution

For P1 (node: command not found):

1. containers/agent/entrypoint.sh: Detect common Node.js installation paths on the host (/home/runner/.nvm/versions/node/*/bin, /usr/local/bin/node, /opt/hostedtoolcache/node/*/x64/bin) and, if found, add them to the PATH exported into the chroot environment.

2. src/docker-manager.ts: Add /home/runner/.nvm (or the resolved NVM_DIR) to the selective bind mounts list (read-only) so that Node.js installed via nvm is accessible inside the container at the same path.

3. containers/agent/entrypoint.sh: Alternatively, ensure node is installed directly inside the agent container image (add nodejs to the apt-get install line in containers/agent/Dockerfile) so the copilot engine always has a reliable node regardless of host configuration.

Option 3 (install node in the image) is the most robust fix since it doesn't depend on host layout.

For P0: See companion tracking issue for awf-api-proxy health check fixes.

Generated by Firewall Issue Dispatcher · ● 450K · ◷

[github-actions]

👋 Hello! I noticed this issue might be related to an existing issue:

• [awf] api-proxy: Flaky awf-api-proxy unhealthy failure during docker compose up #2266 — [awf] api-proxy: Flaky awf-api-proxy unhealthy failure during docker compose up — Tracks the same awf-api-proxy health check intermittent failure during docker compose up (your P0), including discussion of start_period/retries tuning.

The P1 (node: command not found in the copilot agent container) appears to be a new, distinct problem not previously tracked.

If #2266 fully covers your P0, you may want to reference it or consolidate. Otherwise, feel free to clarify how this issue differs!

This is an automated message from the issue duplication detector.

Generated by Issue Duplication Detector for issue #2275 · ● 131.8K · ◷

[lpcox]

Closing — both priorities addressed by merged PRs:

P0 (api-proxy health check): Fixed by PR #2255 (merged) — increased health check tolerances (start_period 30s, retries 15, interval 2s, timeout 3s). Included in firewall v0.25.29.

P1 (node not found in copilot container): Fixed by PR #2160 (merged) — added ~/.nvm to agent bind mounts and AWF_REQUIRE_NODE preflight check for Copilot workflows.