[Bug] L7 egress proxy denies all CONNECT requests on Docker Desktop + WSL2 (amd64)

[davidpeden3]

Agent Diagnostic

• Pointed Claude Code at the OpenShell repo (crates/openshell-sandbox/src/)
• Read proxy.rs to trace the CONNECT handling flow: request parsing (line 301) → evaluate_opa_tcp() (line 340) → OPA deny → 403 response (line 414)
• Read procfs.rs to understand binary identity resolution: resolve_tcp_peer_identity() → parse_proc_net_tcp() → find_pid_by_socket_inode()
• Identified that parse_proc_net_tcp() only reads /proc/<pid>/net/tcp{,6} (line 178)
• Compared /proc/1/net/tcp contents between macOS Docker (arm64) and WSL2 Docker (amd64) — sandbox user connections visible on macOS, absent on WSL2
• Compared /proc/net/tcp (global view) on WSL2 — sandbox user connections ARE visible there
• Root cause: On WSL2, iptables REDIRECT/DNAT connections from the sandbox network namespace don't appear in per-PID /proc/<pid>/net/tcp, only in global /proc/net/tcp
• Fix: Added /proc/net/tcp{,6} as fallback in parse_proc_net_tcp(). Built patched binary on amd64 and verified the fix.

Description

The sandbox egress proxy at 10.200.0.1:3128 returns HTTP 403 Forbidden for all HTTP CONNECT requests when running on Docker Desktop with WSL2 (Windows 11, amd64). The same container image, policy, and binary work correctly on Docker Desktop with Apple Hypervisor (macOS, arm64).

parse_proc_net_tcp() in procfs.rs only reads /proc/<entrypoint_pid>/net/tcp{,6} to resolve the socket inode for the peer connection. On WSL2/Docker Desktop, connections from the sandbox user namespace that are redirected via iptables REDIRECT/DNAT to the proxy are not visible in the per-PID table — they only appear in the global /proc/net/tcp.

This causes resolve_tcp_peer_identity() to fail, the proxy cannot identify the calling binary, no network policy matches, and OPA denies the request.

Forward proxy requests (GET http://...) are also affected since they use the same evaluate_opa_tcp() code path.

Reproduction Steps

1. Run OpenShell cluster on Docker Desktop + WSL2 (Windows 11, amd64)
2. Create a sandbox with a network policy allowing socat and curl to connect to a non-loopback IP:

network_policies:
  my_endpoint:
    endpoints:
      - host: <gateway-ip>
        port: 18789
        enforcement: enforce
        access: full
    binaries:
      - path: /usr/bin/socat
      - path: /usr/bin/curl

3. Apply the policy and SSH into the sandbox
4. Run: curl -v -p -x http://10.200.0.1:3128 http://<gateway-ip>:18789/
5. Result: HTTP/1.1 403 Forbidden (CONNECT tunnel failed)
6. Same steps on macOS Docker Desktop (arm64): HTTP/1.1 200 Connection Established

Diagnostic from inside the pod on WSL2:

# Per-PID view — no sandbox user connections visible
cat /proc/1/net/tcp
# Only shows root-owned listeners and K8s API connections

# Global view — sandbox user connections ARE here
cat /proc/net/tcp
# Shows redirected connections from sandbox namespace

Environment

• Failing: Windows 11, Docker Desktop 4.x with WSL2/Hyper-V backend, amd64
• Working: macOS 15, Docker Desktop with Apple Hypervisor, arm64
• OpenShell: v0.0.16 (ghcr.io/nvidia/openshell/cluster:0.0.16)
• Docker: Docker Desktop (both platforms)

Logs

From sandbox pod logs on WSL2 (no deny lines visible — log level filters them):

WARN openshell_sandbox::sandbox::linux::landlock: Landlock filesystem sandbox is UNAVAILABLE

From socat inside the sandbox:
2026/03/30 15:55:37 socat[469] E CONNECT <gateway-ip>:18789: Forbidden

From curl verbose output:
> CONNECT <gateway-ip>:18789 HTTP/1.1
< HTTP/1.1 403 Forbidden
* CONNECT tunnel failed, response 403

**Note**: The OPA deny at `proxy.rs:414` uses `info!()` level logging, but the sandbox defaults to WARN level, so the deny reason (including binary path resolution failure) is not visible in pod logs.

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
• My agent could not resolve this — the diagnostic above explains why

[davidpeden3]

For anyone interested in the patch while I work through the vouch system, you can see the pr in my fork.

davidpeden3#2

[johntmyers]

I am curious why this hasn't been encountered by other WSL users. Any idea why?

[davidpeden3]

my best guess is a combination of factors.

this issue only appears when network policies include a binaries section. that setting causes the proxy to resolve the peer connection to a specific binary by looking at /proc/<pid>/net/tcp. on wsl2 with docker desktop, that code path doesn't work as expected because connections redirected via iptables/dnat don't show up in the per-pid table (they only appear in the global /proc/net/tcp).

without per-binary enforcement the identity resolution step isn't needed, so the problem doesn't surface. additionally, the failure is fairly quiet — the opa deny is logged at info!() level while the sandbox defaults to warn, so it only shows a generic 403 with no clear diagnostic information.

this might explain why it hasn't come up more often.

[johntmyers]

Ok thanks that is helpful. I will take a look at this more thoroughly tomorrow. We don't have WSL setup for CI yet but I am ok getting this in at our current stage of development if you are able to verify it working.

[davidpeden3]

@johntmyers – following up on your comment about verifying the fix.

i ran some reproduction tests on my wsl2 setup (windows 11, docker desktop 29.3.1, wsl2 kernel 6.6.87.2, openshell v0.0.16). i enabled rust_log=openshell_sandbox=debug and ran everything inside the sandbox network namespace.

what i found:

• with a properly configured policy that includes nvidia_web (allowing curl to www.nvidia.com:443), identity resolution works on the stock binary. the proxy successfully resolves the binary and opa allows the request.
• the 403s i was originally seeing were not caused by a failure in parse_proc_net_tcp(). they were caused by the active gateway policy not including the expected rules (the baked-in /etc/openshell/policy.yaml was being overridden by the gateway's own policy set).

once i added the missing policy via openshell policy set, everything worked as expected with the stock binary.

observability note:

the sandbox defaults to warn log level, but the connect allow/deny decision is only logged at info. this made the denials completely silent at the default level, which led me to assume a deeper problem in identity resolution. switching to debug level immediately showed that resolution was succeeding and the deny was purely policy-related. it might be worth considering logging connect denials at warn so they're visible by default.

regarding pr #684:

i also tested the patched binary (with the /proc/net/tcp global fallback). on my current setup, both /proc/1/net/tcp and /proc/net/tcp contain the same entries, so the fallback path never triggered.

i was unable to reproduce the per-pid vs global /proc/net/tcp discrepancy i originally reported. it's possible a recent docker desktop or wsl2 kernel update changed the behavior, or that my original 403s were always policy-related and i misattributed them.

recommendation:

i believe that the fallback added in pr #684 is defensive and harmless — it tries the per-pid path first and falls back to global if needed. however, since i can't demonstrate it fixing a real issue on my setup anymore, i'll leave it to you whether it's worth keeping as a resilience measure or if you'd prefer to close the pr.

happy to provide any additional logs or run further tests if helpful.

[johntmyers]

Thank you for the update. I've opened #704 to address the observability issue and will create a PR. At this point I am going to close out #684. My concern with falling back to the global table is that we could accidentally hit a port collision because this table contains connections from all network namespaces and all processes on the host. So it very well could return an inode matching the port but to a process outside of the sandbox. One mitigation is to match on both local port and remote port which can reduce collisions but it's not impossible. So I'd argue while the fallback makes sense it's not worth the security risk without much deeper exploration which is hard to justify now.

[johntmyers]

@davidpeden3 which logs were you looking at? The actual logs emitted from OpenShell or container logs from Kube? From what I can tell the denial logs should be available via the sandbox itself with openshell sandbox logs ... or via the TUI which you can launch with openshell term? If I'm understanding this right we'd need to update the logging so the same logs come out via container logs. But I do want to make sure you're aware of the logs we emit directly from the sandbox to the gateway.