Sandbox timeout after several minutes into training, and "422 Unprocessable Entity"

[arcyleung]

When running run_skyrl_agent_oh7b_s1.sh I am seeing timeouts on the verl pipeline side, even after increasing remote_runtime_api_timeout from 10s to 30s.

Eventually the SkyRL-OpenHands remote runtime server fails to create new containers and returns 500 error. We have plenty CPUs (224 cores) for openhands so not sure, any suggestions would be appreciated:

[SumanthRH]

@arcyleung Thanks for reporting this issue.

I see two errors here that are worth investigating. Could you provide some details on your setup?

• Remote server details. What kind of machine is this, what disk is docker using (NVMe? HDD? EBS?). Also, is this just a single machine or are you using our K8S deployment scripts?
• Image caching: Have you run image caching before starting training? When running image caching, the request concurrency should be a bit lower because there's a lot of image downloads + container builds that happen that can overload the server.
• I see a timeout error as you mentioned. For convenience, I have logged request processing times for each endpoint. For a prevoius successful request, what was the time taken for /start ?
• Remote server state at time of error: How many docker containers were running at the time of errors? Sometimes, aggressive keyboard interrupts will lead to running containers being left on the remote server (docker will not clean this up). So even at a low request concurrency, over multiple runs you could have containers accumulate if previous runs were abruptly terminated.
• Port-finding issue: Again, it would be helpful to know how many running containers there were at the time of the issue. Regardless, the port-finding logic should be sound, but I will take another look to see if we have not pushed all our changes.

[arcyleung]

Thanks for the quick reply @SumanthRH ! here is my setup:

1. Single machine, dual Xeon 8280 series w/ total of 224 cores, 8 cards, 8TB+ NVMe storage with Docker using the overlay2 storage driver. I started SkyRL-OpenHands from this command
   OPENHANDS_API_KEY="mykey" ALLHANDS_API_KEY="mykey" python -m openhands.runtime.remote_runtime_server.main --host 10.10.100.19 --port 9099 --log-level debug

2. I have not run image caching. Just started the pipeline script ./examples/sky/run_skyrl_agent_oh7b_s1.sh (with the 1 node configuration suggested in the comment)

3. The /start endpoint returns on avg 20s each time it is logged in the beginning
   
   But after ~5 minutes I start to see 500 Internal Sever Errors
   
   Eventually both the verl pipeline and openhands server hangs and doesn't output new logs to console after ~20 minutes

4. I counted over 180 with docker ps | wc -l, it appears the /stop endpoint would error with 422 so the containers didn't get stopped. I manually removed them using docker ps -a | grep xingyaoww | awk '{print$1}' | xargs docker stop | xargs docker rm before each re-run.
   

I also deleted all the docker images between re-runs, eventually I was able to reach where the first global step checkpoint saved to disk, so maybe the problem is resolved after the old images were cleared...

5. I see logs about port finding, it appears to have no errors

{'Image': 'docker.io/xingyaoww/runtime:oh_v0.25.0_b53jwub5vnz1x040', 'Cmd': ['/openhands/micromamba/bin/micromamba', 'run', '-n', 'openhands', 'poetry', 'run', 'python', '-u', '-m', 'openhands.runtime.action_execution_server', '36949', '--working-dir', '/workspace', '--username', 'root', '--user-id', '0'], 'WorkingDir': '/openhands/code/', 'Env': ['port=36949', 'PYTHONUNBUFFERED=1', 'VSCODE_PORT=41808'], 'ExposedPorts': {'36949/tcp': {}, '41808/tcp': {}, '51702/tcp': {}, '55358/tcp': {}}, 'Volumes': None, 'HostConfig': {'PortBindings': {'36949/tcp': [{'HostPort': '36949'}], '41808/tcp': [{'HostPort': '41808'}], '51702/tcp': [{'HostPort': '51702'}], '55358/tcp': [{'HostPort': '55358'}]}, 'DeviceRequests': [], 'CpuPeriod': 100000, 'CpuQuota': 100000, 'Memory': 4294967296, 'OomKillDisable': False}}
2025-05-15 19:06:45 - INFO - Port finding took 0.566624641418457 seconds for session 0d0dedf3-5c18-44c4-b71d-f87a485743af-483bb286572371d0
{'Image': 'docker.io/xingyaoww/runtime:oh_v0.25.0_0fl3i8io9be6zbj3', 'Cmd': ['/openhands/micromamba/bin/micromamba', 'run', '-n', 'openhands', 'poetry', 'run', 'python', '-u', '-m', 'openhands.runtime.action_execution_server', '39493', '--working-dir', '/workspace', '--username', 'root', '--user-id', '0'], 'WorkingDir': '/openhands/code/', 'Env': ['port=39493', 'PYTHONUNBUFFERED=1', 'VSCODE_PORT=49837'], 'ExposedPorts': {'39493/tcp': {}, '49837/tcp': {}, '50687/tcp': {}, '57809/tcp': {}}, 'Volumes': None, 'HostConfig': {'PortBindings': {'39493/tcp': [{'HostPort': '39493'}], '49837/tcp': [{'HostPort': '49837'}], '50687/tcp': [{'HostPort': '50687'}], '57809/tcp': [{'HostPort': '57809'}]}, 'DeviceRequests': [], 'CpuPeriod': 100000, 'CpuQuota': 100000, 'Memory': 4294967296, 'OomKillDisable': False}}
2025-05-15 19:06:45 - INFO - Port finding took 0.5888643264770508 seconds for session e808d7a9-f8d7-4b49-8d4b-0a94693d0e5c-fe0f1148b08f0f7b

Hope the logs help and thanks.

[SumanthRH]

Thanks for getting back!

1. Single machine, dual Xeon 8280 series w/ total of 224 cores, 8 cards, 8TB+ NVMe storage with Docker using the overlay2 storage driver. I started SkyRL-OpenHands from this command

Just to be super duper clear, what is docker data root configured to be in? i.e is docker really using the NVMe storage? Also, what is the docker runtime being used? Is there a chance you can switch to crun for lesser overhead and faster container start times? For example, you can see our daemon configuration for the k8s setup here: https://github.com/NovaSky-AI/SkyRL-OpenHands/blob/6f3ba323d8ad7d17891c1d171abc9eaf8baae252/k8s_deploy/statefulset_deploy.yaml#L63-L75

Also, when you start the docker daemon, can you also write all the logs to a log file? This is helpful to pin down the root cause (dockerd error or remote server application logic error)

I have not run image caching. Just started the pipeline script ./examples/sky/run_skyrl_agent_oh7b_s1.sh (with the 1 node configuration suggested in the comment)

Could you run image caching beforehand? this will ensure that the start times will be faster and you will be able to use much higher concurrency without issues. See: https://github.com/NovaSky-AI/SkyRL-OpenHands/blob/6f3ba323d8ad7d17891c1d171abc9eaf8baae252/openhands/runtime/remote_runtime_server/README.md#image-caching

3. But after ~5 minutes I start to see 500 Internal Sever Errors

Thanks for sharing the screenshot! I actually see a timeout error from the docker daemon (Timeout waiting for systemd....). This looks like the daemon was overloaded somehow. What is your max_parallel_agents set to? The default of 64 should be small given your machine. Perhaps after the above suggestions (updating data root, using crun and running image caching beforehand) it should be fine?

I also deleted all the docker images between re-runs, eventually I was able to reach where the first global step checkpoint saved to disk, so maybe the problem is resolved after the old images were cleared...

Interesting! Did you change any concurrency settings in the middle? Also, your NVMe storage should be enough for our subset of swe-gym (300 instances), but it's better to monitor this in case you run evaluation etc as well on the same server. You can set-up an automatic cron job to clean up images if the docker data root is close to hitting max volume.

[arcyleung]

Just to be super duper clear, what is docker data root configured to be in? i.e is docker really using the NVMe storage?

yes the docker data root has always been on our NVMe drives, and we have lots of free space.

you can switch to crun for lesser overhead and faster container start times?

I installed and switched over to crun now to match the k8s config you shared, appears to be fewer 500 Internal Server Errors now. Will continue to observe.

I occasionally get OOM, and sometimes the verl trainer hangs forever after this torch distributed step (when the "nodedup finish generate seq" log from async_rollout.py is printed), it might be NCCL related so I am still debugging that on the side.

When you start the docker daemon, can you also write all the logs to a log file

This is my last 1k lines of the journalctl -u docker.service log, I am seeing some logs about openhands-runtime failing to bind ports, not sure if related. docker_journalctl_1000.log

May 16 21:19:15 lux-2-cyber-02 dockerd[2561293]: time="2025-05-16T21:19:15.274412931Z" level=warning msg="Failed to allocate and map port: failed to bind host port for [::]:37429:172.17.0.251:37429/tcp: address already in use" eid=a712c137f7fea9616e7d5714b85bf5bceab345a635a04bf86832dfaf6a84626f ep=openhands-runtime-9c78ab60-e132-41fa-b1c8-993d450caaa0-a1f50ea10f002af0 net=bridge nid=19c0670f8452ebe95c85fd00ec42c6f948206ad5a2ed6de2da6daafc65a7e632

Could you run image caching beforehand?

I can't find the ./evaluation/benchmarks/swe_bench/scripts/run_builds.sh file in question, has it been moved?

What is your max_parallel_agents set to

I have reduced it to 32 for now, at least with 32 I was able to train the first global step.

Thanks

[arcyleung]

I got slightly further training to global step 12, after lowering MAX_PARALLEL_AGENTS=16 and after changing the runtime from "remote" to "docker" in
https://github.com/NovaSky-AI/SkyRL/blob/main/verl/workers/agentic/swe_agent/codeact.py#L854

You can set-up an automatic cron job to clean up images if the docker data root is close to hitting max volume.

To clear the accumulating stopped openhands containers, I am running the following command periodically

docker ps -a --filter "status=exited" | grep all-hands | awk '{print$1}' | xargs docker rm