[awf] api-proxy: Intermittent health check failure blocks agent startup across multiple engines

[lpcox]

Problem

The awf-api-proxy sidecar container intermittently fails its Docker health check during docker compose up. This blocks agent startup entirely across multiple engines (copilot, claude) and workflow types. At least 3 confirmed runs failed in the 2026-04-28 07:00–13:00 UTC window.

Affected runs:

• Sub-Issue Closer (copilot) — run 25049338576
• Daily Team Evolution Insights (claude) — run 25049605437
• Smoke CI (copilot on PR trigger) — run 25052667955

Context

Parent report: github/gh-aw#28947.
Detailed issue: github/gh-aw#28949.

This is a higher-frequency occurrence of the same pattern reported in #28898.

Root Cause

The awf-api-proxy health check in the generated docker-compose.yml (produced by src/docker-manager.ts → generateDockerCompose()) fires before the Node.js proxy process has bound its ports. The start_period is likely too short to accommodate:

• Container image cold pulls
• GitHub-hosted runner CPU contention under burst load
• Node.js module initialization latency

The containers/api-proxy/ healthcheck endpoint (likely GET /healthz or GET /) must respond within the configured timeout window, and when it doesn't, Docker marks the container unhealthy, triggering depends_on: condition: service_healthy failure in the agent service.

Proposed Solution

1. Immediate (in src/docker-manager.ts): Increase api-proxy healthcheck start_period to 30–45s and retries to 5–8. This is a low-risk change that directly reduces flap frequency.
2. Medium-term: Add a lightweight /healthz route to containers/api-proxy/ if one doesn't exist, returning 200 OK as soon as the HTTP server is listening — avoiding any dependency on upstream connectivity in the health check.
3. Resilience: In containers/agent/entrypoint.sh, add a pre-flight check that polls the proxy ports before proceeding, with a configurable timeout (e.g., AWF_PROXY_STARTUP_TIMEOUT_S=60).
4. Observability: Log the proxy container startup time in debug mode so flap frequency can be correlated with runner load.

Generated by Firewall Issue Dispatcher · ● 436.3K · ◷

[github-actions]

👋 Hello! I noticed this issue might be related to several existing issues:

• [awf] api-proxy: Flaky health check causes agent startup abort in downstream workflows #2293 — [awf] api-proxy: Flaky health check causes agent startup abort in downstream workflows — Same intermittent api-proxy health check failure pattern
• [awf] API Proxy Sidecar: intermittent health check failure blocks agent startup #2274 — [awf] API Proxy Sidecar: intermittent health check failure blocks agent startup — Same root cause (health check firing before Node.js binds ports)
• [awf] api-proxy: Flaky awf-api-proxy unhealthy failure during docker compose up #2266 — [awf] api-proxy: Flaky awf-api-proxy unhealthy failure during docker compose up — Same docker compose up health check flap
• api-proxy healthcheck flaps during startup, causing Docker Compose to fail #2154 — api-proxy healthcheck flaps during startup, causing Docker Compose to fail — Earlier report of the same issue

These all describe the same core problem: the awf-api-proxy health check failing intermittently because the start_period is too short for the Node.js process to bind its ports before Docker marks it unhealthy.

If one of these addresses your concern, please consider closing this issue as a duplicate. Otherwise, feel free to clarify how your issue differs (e.g., new affected runs, different engine, or a regression after a previous fix)!

This is an automated message from the issue duplication detector.

Generated by Issue Duplication Detector for issue #2295 · ● 195.4K · ◷

[lpcox]

Closing — duplicate of #2293. Already addressed by PR #2255 (merged Apr 28) which increased api-proxy health check tolerances (start_period 30s, retries 15, interval 2s, timeout 3s). Included in firewall v0.25.29+.