fix: improve gateway lifecycle recovery

[kjw3]

Supersedes #952 and the closed #908. This branch rebuilds the same work as a single verified signed commit so it can satisfy the branch signature requirements.

Summary

• preserve a healthy shared nemoclaw gateway across repeat onboarding
• reconcile live OpenShell sandbox state during connect and status instead of trusting stale local registry entries
• classify restart/rebuild lifecycle failures so users get deterministic guidance instead of generic transport errors
• extend double-onboard coverage so creating a second sandbox does not break the first

Issues

• addresses #849
• addresses part of #859
• improves #888 by distinguishing:
  • gateway trust rotation / handshake failure
  • gateway metadata exists but the restarted API still refuses connections
  • gateway rebuilt and the old sandbox no longer exists

Security

• no secret persistence added
• no TLS downgrade or bypass added
• no destructive auto-recovery of healthy sandboxes
• no new shell-out paths beyond resolved openshell wrappers

Validation

npx vitest run test/cli.test.js test/onboard.test.js test/onboard-readiness.test.js test/registry.test.js
npx eslint bin/nemoclaw.js bin/lib/onboard.js test/cli.test.js test/onboard.test.js
npx tsc -p jsconfig.json --noEmit
bash -n test/e2e/test-double-onboard.sh
shellcheck test/e2e/test-double-onboard.sh

Brev CPU Validation

Environment:

• brev-cpu
• instance: kj-nemoclaw-cpu-20260325-155447
• branch commit tested: 43cf8eb

Validated on a real disposable Linux host:

• onboard sandbox A / onboard sandbox B path preserves the shared gateway and keeps the first sandbox reachable
• after openshell gateway stop + openshell gateway start --name nemoclaw, NemoClaw now surfaces a precise post-restart classification instead of a generic transport failure
• after destructive gateway rebuild, NemoClaw removes the stale local sandbox entry when the old sandbox is gone
• rerunning onboard after that rebuild recreates the sandbox cleanly and returns to Ready

Residual

• This PR does not make OpenShell gateway restarts durable. It makes the failure modes explicit, safer, and easier to recover from.
• On the tested Brev CPU host, I did not find a safe non-destructive recovery once the restarted gateway API entered the persistent Connection refused state. That remains a gateway/runtime limitation, not something this PR tries to bypass.

Summary by CodeRabbit

• New Features

  • Reuse healthy gateways automatically, verify live sandbox/gateway state before actions, and add a non-fatal gateway startup mode for recovery.

• Bug Fixes

  • Improved gateway health detection, smarter port/forward handling, enhanced stale registry reconciliation, and consolidated dashboard/control UI routing and URL/token behavior.

• Tests

  • Expanded unit, integration, and e2e tests for lifecycle recovery, ANSI-safe outputs, logs forwarding, and registry reconciliation.

• Chores

  • Clean staged build context (remove Python venv/cache) and add helper cleanup script.

• Documentation

  • Minor macOS first-run checklist formatting and simplified NVIDIA Endpoints label.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Detects and reuses a healthy shared NemoClaw OpenShell gateway, adds live sandbox reconciliation for connect/status flows, introduces recovery-aware gateway startup modes, runs staged build cleanup for blueprint artifacts, and expands unit/CLI/e2e tests covering gateway lifecycle and registry reconciliation.

Changes

Cohort / File(s)	Summary
Core gateway lifecycle & onboard helpers bin/lib/onboard.js	Added ANSI-tolerant parsing (stripAnsi, getActiveGatewayName) and isGatewayHealthy(). Refactored startup into startGatewayWithOptions(..., { exitOnFailure }), added startGatewayForRecovery(), updated preflight() to reuse healthy gateways (skip 8080 check) and destroy stale state only when detected. Exported new functions.
CLI — openshell invocation, sandbox reconciliation & recovery bin/nemoclaw.js	Replaced generic runner usage with getOpenshellBinary, runOpenshell, captureOpenshell. Introduced sandbox/gateway reconciliation helpers (getSandboxGatewayState, getNamedGatewayLifecycleState, getReconciledSandboxGatewayState, ensureLiveSandboxOrExit) and recoverNamedGatewayRuntime. Made sandbox commands async and recovery-aware; updated connect/status/logs/destroy flows.
Staged build cleanup & setup scripts/clean-staged-tree.sh, scripts/setup.sh	Added clean-staged-tree.sh to remove Python artifacts (.venv, .pytest_cache, __pycache__) and invoked it from scripts/setup.sh for the nemoclaw-blueprint build context.
Tests — unit, CLI, gateway-cleanup, e2e test/cli.test.js, test/onboard.test.js, test/gateway-cleanup.test.js, test/e2e/test-double-onboard.sh, test/onboard-readiness.test.js	Expanded CLI test harness (runWithEnv) and many integration tests for logs forwarding, ANSI handling, registry mutation/reconciliation, recovery flows, and lifecycle messaging. Added isGatewayHealthy unit tests and mocked startGateway() integration test. Updated gateway-cleanup test expectations and rewrote double-onboard e2e into lifecycle-recovery flow with fake OpenAI endpoint and timeout handling.
Misc docs README.md	Small formatting tweak in macOS first-run checklist (inserted an empty bullet).

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant CLI as CLI (bin/nemoclaw.js)
    participant Onboard as Onboard (bin/lib/onboard.js)
    participant OpenShell as OpenShell
    participant Registry as Local Registry

    rect rgba(200,150,100,0.5)
    Note over User,Registry: Initial onboarding may recreate gateway
    User->>CLI: onboard
    CLI->>Onboard: startGateway()
    Onboard->>OpenShell: openshell status / gateway info
    Onboard->>OpenShell: destroy/start gateway (if stale)
    OpenShell-->>Onboard: gateway ready
    Onboard-->>CLI: success (new gateway)
    end

    rect rgba(100,150,200,0.5)
    Note over User,Registry: Reuse healthy shared gateway
    User->>CLI: onboard
    CLI->>Onboard: startGateway()
    Onboard->>OpenShell: openshell status / gateway info
    OpenShell-->>Onboard: Connected & named
    Onboard->>OpenShell: select existing gateway
    Onboard-->>CLI: success (reused gateway)
    end

    rect rgba(150,200,100,0.5)
    Note over User,Registry: Sandbox connect/status reconciliation
    User->>CLI: sandbox connect / status
    CLI->>Onboard: ensureLiveSandboxOrExit
    Onboard->>OpenShell: openshell sandbox get / logs
    OpenShell-->>Onboard: live sandbox state
    Onboard->>Registry: compare with local registry
    alt sandbox live
        Onboard->>OpenShell: forward/connect
    else registry stale
        Onboard->>Registry: remove stale entry
        Onboard-->>User: guidance / recovery messaging
    end
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🐰
I sniffed the gateways, found one strong,
Reused the path and hopped along.
Two sandboxes, both intact,
No stale traces left to track,
Onboard sings a lively song!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 19.05% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix: improve gateway lifecycle recovery' directly aligns with the main objective of the PR, which is to improve how the system handles gateway lifecycle recovery and reuse.
Linked Issues check	✅ Passed	The PR comprehensively addresses all coding objectives from issue #908: preserves healthy gateways via isGatewayHealthy() and startGatewayWithOptions(), reconciles live sandbox state via getSandboxGatewayState() and ensureLiveSandboxOrExit(), classifies failure modes in sandboxStatus(), and extends test coverage for gateway reuse and recovery flows.
Out of Scope Changes check	✅ Passed	All changes are in-scope: gateway health detection, lifecycle classification, recovery flows, async sandbox operations, test coverage expansion, and supporting infrastructure like clean-staged-tree.sh align with stated objectives.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/gateway-lifecycle-recovery-final

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

🚀 Docs preview ready!

https://NVIDIA.github.io/NemoClaw/pr-preview/pr-953/

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

bin/lib/onboard.js (1)

1487-1488: Centralize this staged-tree scrubber.

The same .venv / .pytest_cache / __pycache__ cleanup now lives here and in scripts/setup.sh, so the next artifact tweak has to land in two places. A shared helper/script would keep the legacy setup path aligned with onboard.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@bin/lib/onboard.js` around lines 1487 - 1488, Duplicate cleanup commands for
removing .venv, .pytest_cache and __pycache__ are present in onboard.js
(run(...) calls) and scripts/setup.sh; extract them into a single centralized
"staged-tree scrubber" (either a small shell script like
scripts/clean-staged-tree.sh or a JS helper exported from
bin/lib/cleanStagedTree.js) and replace the inline run(...) invocations in
onboard.js and the matching snippet in scripts/setup.sh to call that central
helper. Ensure the central helper performs the same operations (rm -rf for .venv
and .pytest_cache and find ... -name __pycache__ -prune -exec rm -rf {} +) and
that onboard.js's call uses the existing run(...) wrapper semantics (preserve {
ignoreError: true } behavior) or that scripts/setup.sh invokes the script with
identical error-tolerant behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@bin/nemoclaw.js`:
- Around line 100-120: The connected check in getNamedGatewayLifecycleState uses
/Connected/i which also matches "Disconnected"; update the connected detection
to match the exact status token/line in cleanStatus (e.g. replace the
/Connected/i.test(cleanStatus) usage with a more specific regex such as one that
anchors to the status line like /^Status:\s*Connected\b/im.test(cleanStatus) or
a whole-word match for "Connected" so "Disconnected" does not match), leaving
the rest of the function logic unchanged and still using the cleanStatus
variable.

In `@test/e2e/test-double-onboard.sh`:
- Around line 411-418: The current acceptance regex for gateway_status_output
incorrectly treats "Removed stale local registry entry" as a valid gateway-stop
outcome; modify the grep pattern in the gateway status check to remove that
phrase so only lifecycle/recovery messages ("Recovered NemoClaw gateway
runtime", "gateway is no longer configured after restart/rebuild", "gateway is
still refusing connections after restart", "gateway trust material rotated after
restart") count as passes, and keep the existing pass/fail behavior
(pass(...)/fail(...)) around gateway_status_output. After this check, add an
explicit assertion that the registry still contains SANDBOX_B by calling the
registry query used elsewhere in the test (the same registry listing helper that
references SANDBOX_B) and fail the test if SANDBOX_B is missing; reference
gateway_status_output, SANDBOX_B, and the pass/fail helpers when implementing
these changes.

---

Nitpick comments:
In `@bin/lib/onboard.js`:
- Around line 1487-1488: Duplicate cleanup commands for removing .venv,
.pytest_cache and __pycache__ are present in onboard.js (run(...) calls) and
scripts/setup.sh; extract them into a single centralized "staged-tree scrubber"
(either a small shell script like scripts/clean-staged-tree.sh or a JS helper
exported from bin/lib/cleanStagedTree.js) and replace the inline run(...)
invocations in onboard.js and the matching snippet in scripts/setup.sh to call
that central helper. Ensure the central helper performs the same operations (rm
-rf for .venv and .pytest_cache and find ... -name __pycache__ -prune -exec rm
-rf {} +) and that onboard.js's call uses the existing run(...) wrapper
semantics (preserve { ignoreError: true } behavior) or that scripts/setup.sh
invokes the script with identical error-tolerant behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 17d70415-acbc-418a-b174-fa1a4f01e638

📥 Commits

Reviewing files that changed from the base of the PR and between 0acb2e4 and 70fabaa.

📒 Files selected for processing (7)

• bin/lib/onboard.js
• bin/nemoclaw.js
• scripts/setup.sh
• test/cli.test.js
• test/e2e/test-double-onboard.sh
• test/gateway-cleanup.test.js
• test/onboard.test.js

[coderabbitai]

🧹 Nitpick comments (1)

bin/lib/onboard.js (1)

188-203: Consider using regex for ANSI stripping.

The manual character-by-character approach works but is harder to maintain and has O(n²) complexity due to string concatenation. The existing isSandboxReady function at line 171 already uses a regex for ANSI stripping.

♻️ Suggested simplification using regex

 function stripAnsi(value = "") {
-  let cleaned = "";
-  for (let i = 0; i < value.length; i += 1) {
-    if (value.charCodeAt(i) === 27 && value[i + 1] === "[") {
-      i += 2;
-      while (i < value.length && /[0-9;]/.test(value[i])) {
-        i += 1;
-      }
-      if (value[i] === "m") {
-        continue;
-      }
-    }
-    cleaned += value[i] || "";
-  }
-  return cleaned;
+  // eslint-disable-next-line no-control-regex
+  return value.replace(/\x1b\[[0-9;]*m/g, "");
 }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@bin/lib/onboard.js` around lines 188 - 203, The stripAnsi function currently
iterates character-by-character and builds a string which is harder to maintain
and can be O(n²); replace its implementation with a single regex-based removal
(same approach used in isSandboxReady) to strip ANSI escape sequences and return
value.replace(ANSI_REGEX, '') where ANSI_REGEX matches CSI/escape sequences;
update stripAnsi to use that regex and ensure it defaults value to an empty
string as before.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@bin/lib/onboard.js`:
- Around line 188-203: The stripAnsi function currently iterates
character-by-character and builds a string which is harder to maintain and can
be O(n²); replace its implementation with a single regex-based removal (same
approach used in isSandboxReady) to strip ANSI escape sequences and return
value.replace(ANSI_REGEX, '') where ANSI_REGEX matches CSI/escape sequences;
update stripAnsi to use that regex and ensure it defaults value to an empty
string as before.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0b3e9fc7-003d-423c-b21f-81102d9fc253

📥 Commits

Reviewing files that changed from the base of the PR and between e560a17 and fdb20b9.

📒 Files selected for processing (1)

• bin/lib/onboard.js