Issues reproducing Text2SQL tutorial - `Watchdog caught collective operation timeout` when running `examples/text_to_sql/run_skyrl_sql.sh`

[shanghongsim]

Hi, I’m trying to reproduce the text2sql example and am running into a NCCL watchdog timeout during policy_train with FSDP on an 8×H100 node.

Environment

• Cluster: Lambda Labs, 1 node, 8× NVIDIA H100 GPUs
• Container image: novaskyai/skyrl-train-ray-2.51.1-py3.12-cu12.8
• CUDA: 12.8 (nvidia-smi confirms)
• Ray: 2.51.1
• Python: 3.12 (via uv venv --python 3.12)
• Repository: NovaSky-AI/SkyRL
  • Branch: main
  • git status: clean, up to date with origin/main

Full pip list

(skyrl-train) (base) shanghong@oumi-compute004:~/SkyRL/skyrl-train$ pip list
Package                            Version
---------------------------------- --------------
adlfs                              2023.8.0
aiofiles                           22.1.0
aiohappyeyeballs                   2.6.1
aiohttp                            3.11.16
aiohttp-cors                       0.7.0
aiosignal                          1.3.1
aiosqlite                          0.19.0
amqp                               5.3.1
annotated-types                    0.6.0
anyio                              3.7.1
anyscale                           0.26.72
archspec                           0.2.5
argon2-cffi                        23.1.0
argon2-cffi-bindings               21.2.0
arrow                              1.3.0
asttokens                          2.4.1
attrs                              25.1.0
azure-common                       1.1.28
azure-core                         1.29.5
azure-datalake-store               0.0.53
azure-identity                     1.17.1
azure-storage-blob                 12.22.0
Babel                              2.13.1
backcall                           0.2.0
beautifulsoup4                     4.11.1
billiard                           4.2.1
bleach                             6.1.0
boltons                            24.0.0
boto3                              1.29.7
botocore                           1.32.7
Brotli                             1.1.0
cachetools                         5.5.2
celery                             5.5.3
certifi                            2025.1.31
cffi                               1.16.0
charset-normalizer                 3.3.2
click                              8.1.7
click-didyoumean                   0.3.1
click-plugins                      1.1.1.2
click-repl                         0.3.0
cloudpickle                        3.1.1
colorama                           0.4.6
colorful                           0.5.5
comm                               0.2.0
conda                              24.11.3
conda-libmamba-solver              24.9.0
conda-package-handling             2.4.0
conda_package_streaming            0.11.0
cryptography                       44.0.3
cupy-cuda12x                       13.1.0
debugpy                            1.8.0
decorator                          5.1.1
defusedxml                         0.7.1
distlib                            0.3.7
distro                             1.9.0
dm-tree                            0.1.8
entrypoints                        0.4
executing                          2.0.1
Farama-Notifications               0.0.4
fastapi                            0.115.12
fastjsonschema                     2.19.0
fastrlock                          0.8.2
filelock                           3.17.0
fqdn                               1.5.1
frozendict                         2.4.6
frozenlist                         1.4.1
fsspec                             2023.12.1
gitdb                              4.0.11
GitPython                          3.1.44
google-api-core                    2.24.2
google-auth                        2.23.4
google-cloud-core                  2.4.1
google-cloud-storage               2.14.0
google-crc32c                      1.5.0
google-resumable-media             2.6.0
googleapis-common-protos           1.61.0
grpcio                             1.74.0
gymnasium                          1.1.1
h11                                0.16.0
h2                                 4.1.0
hpack                              4.0.0
httplib2                           0.20.4
httptools                          0.7.1
humanize                           4.12.1
hyperframe                         6.0.1
idna                               3.7
importlib-metadata                 6.11.0
ipykernel                          6.27.1
ipython                            8.12.3
ipython-genutils                   0.2.0
ipywidgets                         8.1.3
isodate                            0.6.1
isoduration                        20.11.0
jedi                               0.19.1
Jinja2                             3.1.6
jmespath                           1.0.1
json5                              0.9.14
jsonpatch                          1.32
jsonpointer                        2.4
jsonschema                         4.23.0
jsonschema-specifications          2024.10.1
jupyter-client                     7.3.4
jupyter_core                       5.5.0
jupyter-events                     0.6.3
jupyter-server                     1.24.0
jupyter_server_fileid              0.9.0
jupyter_server_ydoc                0.6.1
jupyter-ydoc                       0.2.5
jupyterlab                         3.6.1
jupyterlab_pygments                0.3.0
jupyterlab_server                  2.24.0
jupyterlab_widgets                 3.0.11
kombu                              5.5.4
libmambapy                         1.5.12
log-symbols                        0.0.14
lxml                               4.9.4
lz4                                4.3.3
mamba                              1.5.12
markdown-it-py                     2.2.0
MarkupSafe                         2.1.3
matplotlib-inline                  0.1.6
mdurl                              0.1.2
memray                             1.10.0
menuinst                           2.2.0
mistune                            0.8.4
msal                               1.28.1
msal-extensions                    1.2.0b1
msgpack                            1.0.7
multidict                          6.0.5
nbclassic                          1.0.0
nbclient                           0.5.13
nbconvert                          6.5.4
nbformat                           5.9.2
nest-asyncio                       1.5.8
notebook                           6.5.7
notebook_shim                      0.2.3
numpy                              1.26.4
oauth2client                       4.1.3
opencensus                         0.11.4
opencensus-context                 0.1.3
opentelemetry-api                  1.34.1
opentelemetry-exporter-prometheus  0.55b1
opentelemetry-proto                1.27.0
opentelemetry-sdk                  1.34.1
opentelemetry-semantic-conventions 0.55b1
ormsgpack                          1.7.0
packaging                          23.0
pandas                             2.3.3
pandocfilters                      1.5.0
parso                              0.8.3
pathspec                           0.11.2
pexpect                            4.8.0
pickleshare                        0.7.5
pip                                24.3.1
platformdirs                       3.11.0
pluggy                             1.5.0
polars                             1.32.3
portalocker                        2.8.2
prometheus-client                  0.19.0
prompt-toolkit                     3.0.41
propcache                          0.3.0
proto-plus                         1.22.3
protobuf                           4.25.8
psutil                             5.9.6
ptyprocess                         0.7.0
pure-eval                          0.2.2
py-spy                             0.4.1
pyarrow                            19.0.1
pyasn1                             0.5.1
pyasn1-modules                     0.3.0
pycosat                            0.6.6
pycparser                          2.21
pycurl                             7.45.3
pydantic                           2.11.7
pydantic_core                      2.33.2
Pygments                           2.18.0
PyJWT                              2.8.0
pyOpenSSL                          25.0.0
pyparsing                          3.1.1
PySocks                            1.7.1
python-dateutil                    2.8.2
python-dotenv                      1.2.1
python-json-logger                 2.0.7
pytz                               2022.7.1
PyYAML                             6.0.1
pyzmq                              26.0.3
ray                                2.51.1
referencing                        0.36.2
requests                           2.32.3
rfc3339-validator                  0.1.4
rfc3986-validator                  0.1.1
rich                               13.3.2
rpds-py                            0.22.3
rsa                                4.7.2
ruamel.yaml                        0.18.10
ruamel.yaml.clib                   0.2.8
s3transfer                         0.8.0
scipy                              1.11.4
Send2Trash                         1.8.3
setuptools                         75.8.0
six                                1.16.0
smart-open                         6.2.0
smmap                              5.0.1
sniffio                            1.3.1
soupsieve                          2.5
spinners                           0.0.24
stack-data                         0.6.3
starlette                          0.46.2
supervisor                         4.3.0
tabulate                           0.9.0
tensorboardX                       2.6.2.2
termcolor                          2.4.0
terminado                          0.18.1
tinycss2                           1.3.0
tornado                            6.5.2
tqdm                               4.67.1
traitlets                          5.14.3
truststore                         0.10.0
types-python-dateutil              2.9.0.20240316
typing_extensions                  4.12.2
typing-inspection                  0.4.1
tzdata                             2025.2
tzlocal                            5.3
uri-template                       1.3.0
urllib3                            1.26.19
uvicorn                            0.22.0
uvloop                             0.21.0
vine                               5.1.0
virtualenv                         20.29.1
watchfiles                         0.19.0
wcwidth                            0.2.13
webcolors                          24.6.0
webencodings                       0.5.1
websocket-client                   1.8.0
websockets                         11.0.3
wheel                              0.45.1
widgetsnbextension                 4.0.11
wrapt                              1.14.1
y-py                               0.6.2
yarl                               1.18.3
ypy-websocket                      0.8.4
zipp                               3.19.2
zstandard                          0.23.0

Setup / Repro Steps

On the Lambda node:

srun --gpus=8 --time=72:00:00 \
  --container-writable \
  --container-image=novaskyai/skyrl-train-ray-2.51.1-py3.12-cu12.8 \
  --container-mounts=/home/$USER/data:/workspace/data \
  --container-save=/home/$USER/data/skyrl-img.sqsh \
  --pty bash

Inside the container:

nvidia-smi # confirming cuda 12.8

ray start --head # ray status

git clone --recurse-submodules https://github.com/NovaSky-AI/SkyRL
cd SkyRL/skyrl-train

uv venv --python 3.12
source .venv/bin/activate
uv sync --active --extra vllm

export RAY_RUNTIME_ENV_HOOK=ray._private.runtime_env.uv_runtime_env_hook.hook
export WANDB_API_KEY=<wandb-api-key>

bash examples/text_to_sql/run_skyrl_sql.sh 

I did not modify the training configuration, only the data path to point to where I had stored my data files.

The error

Training proceeds through:

• convert_to_training_input
• fwd_logprobs_values_reward
• compute_advantages_and_returns
• dump_data_batch

Then during:
train_critic_and_policy -> policy_train (FSDPPolicyWorkerBase)

I get an NCCL watchdog timeout and the Ray actor dies.

Key parts of the error log:

[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:36:56.105[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mconvert_to_training_input[0m:[36m597[0m - [1m[32mNumber of sequences before padding: 1280[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:36:56.105[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mconvert_to_training_input[0m:[36m599[0m - [1m[32mNumber of sequences after padding: 1280[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:36:56.105[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m238[0m - [1m[32mNumber of sequences: 1280[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:36:56.105[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m236[0m - [1m[32mFinished: 'convert_to_training_input', time cost: 171.64s[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:36:56.105[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m241[0m - [1m[32mStarted: 'fwd_logprobs_values_reward'[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:47:09.361[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m241[0m - [1m[32mFinished: 'fwd_logprobs_values_reward', time cost: 613.26s[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:47:09.362[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m250[0m - [1m[32mStarted: 'compute_advantages_and_returns'[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:47:26.199[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mcompute_advantages_and_returns[0m:[36m781[0m - [1m[32mavg_final_rewards: -0.29609376192092896, avg_response_length: 904.32890625[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:47:26.365[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m250[0m - [1m[32mFinished: 'compute_advantages_and_returns', time cost: 17.00s[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:47:26.386[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m262[0m - [1m[32mStarted: 'dump_data_batch'[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:48:11.955[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m262[0m - [1m[32mFinished: 'dump_data_batch', time cost: 45.57s[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:48:11.955[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m267[0m - [1m[32mStarted: 'train_critic_and_policy'[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 12:48:11.956[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain_critic_and_policy[0m:[36m1038[0m - [1m[32mStarted: 'policy_train'[0m
[36m(FSDPPolicyWorkerBase pid=959049)[0m 
Policy Train epoch [1/1]:   0%|          | 0/160 [00:00<?, ?it/s]
[36m(FSDPPolicyWorkerBase pid=959049)[0m [rank0]:[E1206 13:01:11.084722089 ProcessGroupNCCL.cpp:685] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=19962, OpType=_ALLGATHER_BASE, NumelIn=68125120, NumelOut=545000960, Timeout(ms)=600000) ran for 600029 milliseconds before timing out.
[36m(FSDPPolicyWorkerBase pid=959049)[0m [rank0]:[E1206 13:01:11.138792342 ProcessGroupNCCL.cpp:2252] [PG ID 0 PG GUID 0(default_pg) Rank 0]  failure detected by watchdog at work sequence id: 19962 PG status: last enqueued work: 19964, last completed work: 19961
[36m(FSDPPolicyWorkerBase pid=959049)[0m [rank0]:[E1206 13:01:11.138832442 ProcessGroupNCCL.cpp:732] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[36m(FSDPPolicyWorkerBase pid=959049)[0m [rank0]:[E1206 13:01:11.138864267 ProcessGroupNCCL.cpp:2584] [PG ID 0 PG GUID 0(default_pg) Rank 0] First PG on this rank to signal dumping.
[36m(FSDPPolicyWorkerBase pid=959298)[0m [rank3]:[E1206 13:01:11.434681723 ProcessGroupNCCL.cpp:1806] [PG ID 0 PG GUID 0(default_pg) Rank 3] Observed flight recorder dump signal from another rank via TCPStore.
[36m(FSDPPolicyWorkerBase pid=959298)[0m [rank3]:[E1206 13:01:11.436152721 ProcessGroupNCCL.cpp:1870] [PG ID 0 PG GUID 0(default_pg) Rank 3] Received a dump signal due to a collective timeout from  rank 0 and we will try our best to dump the debug info. Last enqueued NCCL work: 19964, last completed NCCL work: 19961.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[36m(FSDPPolicyWorkerBase pid=959049)[0m [rank0]:[E1206 13:01:11.438828600 ProcessGroupNCCL.cpp:1870] [PG ID 0 PG GUID 0(default_pg) Rank 0] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 19964, last completed NCCL work: 19961.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[36m(FSDPPolicyWorkerBase pid=959049)[0m [rank0]:[E1206 13:01:11.454303498 ProcessGroupNCCL.cpp:1589] [PG ID 0 PG GUID 0(default_pg) Rank 0] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[36m(FSDPPolicyWorkerBase pid=959301)[0m [rank6]:[E1206 13:01:12.087674097 ProcessGroupNCCL.cpp:1935] [PG ID 0 PG GUID 0(default_pg) Rank 6] Could not acquire GIL within 300 ms on exit, possible GIL induced hang
[36m(FSDPPolicyWorkerBase pid=959049)[0m [rank0]:[E1206 13:02:11.138949062 ProcessGroupNCCL.cpp:746] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[36m(FSDPPolicyWorkerBase pid=959049)[0m [rank0]:[E1206 13:02:11.138971618 ProcessGroupNCCL.cpp:760] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
[36m(FSDPPolicyWorkerBase pid=959303)[0m [rank5]:[E1206 13:01:12.983163669 ProcessGroupNCCL.cpp:1806] [PG ID 0 PG GUID 0(default_pg) Rank 5] Observed flight recorder dump signal from another rank via TCPStore.[32m [repeated 6x across cluster][0m
[36m(FSDPPolicyWorkerBase pid=959303)[0m [rank5]:[E1206 13:01:12.983490603 ProcessGroupNCCL.cpp:1870] [PG ID 0 PG GUID 0(default_pg) Rank 5] Received a dump signal due to a collective timeout from  rank 0 and we will try our best to dump the debug info. Last enqueued NCCL work: 19964, last completed NCCL work: 19961.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. [32m [repeated 6x across cluster][0m
[36m(FSDPPolicyWorkerBase pid=959303)[0m [rank5]:[E1206 13:01:12.983768157 ProcessGroupNCCL.cpp:1589] [PG ID 0 PG GUID 0(default_pg) Rank 5] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1[32m [repeated 7x across cluster][0m
[36m(FSDPPolicyWorkerBase pid=959049)[0m [rank0]:[E1206 13:02:11.186390648 ProcessGroupNCCL.cpp:2068] [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=19962, OpType=_ALLGATHER_BASE, NumelIn=68125120, NumelOut=545000960, Timeout(ms)=600000) ran for 600029 milliseconds before timing out.
[36m(FSDPPolicyWorkerBase pid=959049)[0m Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:688 (most recent call first):
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x80 (0x7d1228146eb0 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libc10.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x247 (0x7ce2e4640147 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #2: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x1591 (0x7ce2e4643b61 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #3: c10d::ProcessGroupNCCL::Watchdog::run() + 0xd2 (0x7ce2e4644ec2 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #4: <unknown function> + 0xd3b65 (0x7d122ccf1b65 in /home/ray/anaconda3/bin/../lib/libstdc++.so.6)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #5: <unknown function> + 0x94ac3 (0x7d122f094ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #6: <unknown function> + 0x1268c0 (0x7d122f1268c0 in /usr/lib/x86_64-linux-gnu/libc.so.6)
[36m(FSDPPolicyWorkerBase pid=959049)[0m 
[36m(FSDPPolicyWorkerBase pid=959049)[0m [2025-12-06 13:02:11,819 E 959049 961204] logging.cc:118: Unhandled exception: N3c1016DistBackendErrorE. what(): [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=19962, OpType=_ALLGATHER_BASE, NumelIn=68125120, NumelOut=545000960, Timeout(ms)=600000) ran for 600029 milliseconds before timing out.
[36m(FSDPPolicyWorkerBase pid=959049)[0m Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:688 (most recent call first):
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x80 (0x7d1228146eb0 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libc10.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x247 (0x7ce2e4640147 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #2: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x1591 (0x7ce2e4643b61 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #3: c10d::ProcessGroupNCCL::Watchdog::run() + 0xd2 (0x7ce2e4644ec2 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #4: <unknown function> + 0xd3b65 (0x7d122ccf1b65 in /home/ray/anaconda3/bin/../lib/libstdc++.so.6)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #5: <unknown function> + 0x94ac3 (0x7d122f094ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #6: <unknown function> + 0x1268c0 (0x7d122f1268c0 in /usr/lib/x86_64-linux-gnu/libc.so.6)
[36m(FSDPPolicyWorkerBase pid=959049)[0m 
[36m(FSDPPolicyWorkerBase pid=959049)[0m Exception raised from run at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2074 (most recent call first):
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x80 (0x7d1228146eb0 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libc10.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #1: <unknown function> + 0xe1c1a1 (0x7ce2e461c1a1 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #2: <unknown function> + 0x9468e6 (0x7ce2e41468e6 in /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #3: <unknown function> + 0xd3b65 (0x7d122ccf1b65 in /home/ray/anaconda3/bin/../lib/libstdc++.so.6)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #4: <unknown function> + 0x94ac3 (0x7d122f094ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
[36m(FSDPPolicyWorkerBase pid=959049)[0m frame #5: <unknown function> + 0x1268c0 (0x7d122f1268c0 in /usr/lib/x86_64-linux-gnu/libc.so.6)
[36m(FSDPPolicyWorkerBase pid=959049)[0m  /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/ray/_raylet.so(+0x15fd74a) [0x7d122e3fd74a] ray::operator<<()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/ray/_raylet.so(+0x15fe1fc) [0x7d122e3fe1fc] ray::RayLog::operator<< <>()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/ray/_raylet.so(+0x7f1389) [0x7d122d5f1389] ray::TerminateHandler()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xb64f2) [0x7d122ccd44f2] __cxxabiv1::__terminate()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(_ZSt10unexpectedv+0) [0x7d122ccce2f3] std::unexpected()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xb64eb) [0x7d122ccd44eb] __cxxabiv1::__terminate()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so(+0x94696a) [0x7ce2e414696a] c10d::ProcessGroupNCCL::Watchdog::run()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xd3b65) [0x7d122ccf1b65] execute_native_thread_routine
[36m(FSDPPolicyWorkerBase pid=959049)[0m /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7d122f094ac3]
[36m(FSDPPolicyWorkerBase pid=959049)[0m /usr/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x7d122f1268c0]
[36m(FSDPPolicyWorkerBase pid=959049)[0m 
[36m(FSDPPolicyWorkerBase pid=959049)[0m [2025-12-06 13:02:11,830 E 959049 961204] logging.cc:125: Stack trace: 
[36m(FSDPPolicyWorkerBase pid=959049)[0m  /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/ray/_raylet.so(+0x15fd74a) [0x7d122e3fd74a] ray::operator<<()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/ray/_raylet.so(+0x15fe1fc) [0x7d122e3fe1fc] ray::RayLog::operator<< <>()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/ray/_raylet.so(+0x16007b8) [0x7d122e4007b8] ray::TerminateHandler()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xb64f2) [0x7d122ccd44f2] __cxxabiv1::__terminate()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(_ZSt10unexpectedv+0) [0x7d122ccce2f3] std::unexpected()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xb64eb) [0x7d122ccd44eb] __cxxabiv1::__terminate()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/.cache/uv/builds-v0/.tmph3Lvcm/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so(+0x94696a) [0x7ce2e414696a] c10d::ProcessGroupNCCL::Watchdog::run()
[36m(FSDPPolicyWorkerBase pid=959049)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xd3b65) [0x7d122ccf1b65] execute_native_thread_routine
[36m(FSDPPolicyWorkerBase pid=959049)[0m /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7d122f094ac3]
[36m(FSDPPolicyWorkerBase pid=959049)[0m /usr/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x7d122f1268c0]
[36m(FSDPPolicyWorkerBase pid=959049)[0m 
[36m(FSDPPolicyWorkerBase pid=959049)[0m 
[36m(FSDPPolicyWorkerBase pid=959049)[0m *** SIGABRT received at time=1765026131 on cpu 0 ***
[36m(FSDPPolicyWorkerBase pid=959049)[0m PC: @     0x7d122f0969fc  (unknown)  pthread_kill
[36m(FSDPPolicyWorkerBase pid=959049)[0m     @     0x7d122f042520  (unknown)  (unknown)
[36m(FSDPPolicyWorkerBase pid=959049)[0m [2025-12-06 13:02:11,832 E 959049 961204] logging.cc:474: *** SIGABRT received at time=1765026131 on cpu 0 ***
[36m(FSDPPolicyWorkerBase pid=959049)[0m [2025-12-06 13:02:11,832 E 959049 961204] logging.cc:474: PC: @     0x7d122f0969fc  (unknown)  pthread_kill
[36m(FSDPPolicyWorkerBase pid=959049)[0m [2025-12-06 13:02:11,833 E 959049 961204] logging.cc:474:     @     0x7d122f042520  (unknown)  (unknown)
[36m(FSDPPolicyWorkerBase pid=959049)[0m Fatal Python error: Aborted
[36m(FSDPPolicyWorkerBase pid=959049)[0m 
[36m(FSDPPolicyWorkerBase pid=959049)[0m Extension modules: msgpack._cmsgpack, psutil._psutil_linux, google._upb._message, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, yaml._yaml, uvloop.loop, ray._raylet, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, pyarrow.lib, pyarrow._json, regex._regex, markupsafe._speedups, PIL._imaging, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, PIL._imagingft, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, _cffi_backend, pyarrow._parquet, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, xxhash._xxhash, pyarrow._acero, pyarrow._csv, pyarrow._substrait, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, msgspec._core, _cbor2, setproctitle._setproctitle, zmq.backend.cython._zmq, pybase64._pybase64, cuda_utils, __triton_launcher (total: 161)
[36m(FSDPPolicyWorkerBase pid=959294)[0m [rank1]:[E1206 13:02:38.408721158 ProcessGroupNCCL.cpp:685] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=19962, OpType=_ALLGATHER_BASE, NumelIn=68125120, NumelOut=545000960, Timeout(ms)=600000) ran for 600005 milliseconds before timing out.
[36m(FSDPPolicyWorkerBase pid=959294)[0m [rank1]:[E1206 13:02:38.410314398 ProcessGroupNCCL.cpp:2252] [PG ID 0 PG GUID 0(default_pg) Rank 1]  failure detected by watchdog at work sequence id: 19962 PG status: last enqueued work: 19964, last completed work: 19961
[36m(FSDPPolicyWorkerBase pid=959294)[0m [rank1]:[E1206 13:02:38.410399804 ProcessGroupNCCL.cpp:732] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 13:03:10.478[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain_critic_and_policy[0m:[36m1038[0m - [1m[32mFinished: 'policy_train', time cost: 898.52s[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 13:03:10.483[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m267[0m - [1m[32mFinished: 'train_critic_and_policy', time cost: 898.53s[0m
[36m(skyrl_entrypoint pid=952112)[0m [32m2025-12-06 13:03:10.486[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m193[0m - [1m[32mFinished: 'step', time cost: 23727.05s[0m

[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m INFO 12-06 06:25:05 [block_pool.py:378] Successfully reset prefix cache
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m INFO 12-06 12:33:42 [block_pool.py:378] Successfully reset prefix cache
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m INFO 12-06 12:33:42 [block_pool.py:378] Successfully reset prefix cache
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m INFO 12-06 06:27:42 [executor_base.py:205] It took 0.587088 seconds to wake up tags ['kv_cache'].
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m [36m(RayWorkerWrapper pid=954392)[0m INFO 12-06 12:33:45 [cumem.py:228] CuMemAllocator: sleep freed 52.75 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 52.75 GiB is discarded directly.
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m [36m(RayWorkerWrapper pid=954392)[0m INFO 12-06 05:11:17 [gpu_worker.py:117] Sleep mode freed 69.28 GiB memory, 5.57 GiB memory is still in use.[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m [36m(RayWorkerWrapper pid=954248)[0m INFO 12-06 12:33:45 [cumem.py:228] CuMemAllocator: sleep freed 52.53 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 52.53 GiB is discarded directly.
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m [36m(RayWorkerWrapper pid=954392)[0m INFO 12-06 12:34:00 [gpu_worker.py:117] Sleep mode freed 61.54 GiB memory, 6.82 GiB memory is still in use.
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m [36m(RayWorkerWrapper pid=954388)[0m INFO 12-06 12:33:46 [cumem.py:228] CuMemAllocator: sleep freed 52.53 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 52.53 GiB is discarded directly.[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m [36m(RayWorkerWrapper pid=954248)[0m INFO 12-06 12:34:01 [gpu_worker.py:117] Sleep mode freed 62.74 GiB memory, 8.56 GiB memory is still in use.
[36m(AsyncVLLMInferenceEngine pid=953485)[0m [1;36m(EngineCore_DP0 pid=953841)[0;0m INFO 12-06 12:34:02 [executor_base.py:189] It took 20.015521 seconds to fall asleep.
[36m(AsyncVLLMInferenceEngine pid=953488)[0m [1;36m(EngineCore_DP0 pid=953826)[0;0m INFO 12-06 12:33:43 [block_pool.py:378] Successfully reset prefix cache[32m [repeated 2x across cluster][0m
[33m(raylet)[0m A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. Lease ID: 140000006e19258805c8555f61570e469502f4b732eeb39ab653a1d10c361ca6 Worker ID: 91dc7bc6b699be719f54e6992d98e61b930ffc3a90be0b9adce29202 Node ID: cd1213a83740a10a34cd25a11bb006b34272b8decaa0e0f50e7efd6b Worker IP address: 172.26.134.226 Worker port: 39485 Worker PID: 959049 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
[36m(AsyncVLLMInferenceEngine pid=953488)[0m [1;36m(EngineCore_DP0 pid=953826)[0;0m [36m(RayWorkerWrapper pid=954098)[0m INFO 12-06 12:33:47 [cumem.py:228] CuMemAllocator: sleep freed 52.53 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 52.53 GiB is discarded directly.[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=953488)[0m [1;36m(EngineCore_DP0 pid=953826)[0;0m [36m(RayWorkerWrapper pid=954097)[0m INFO 12-06 05:11:18 [gpu_worker.py:117] Sleep mode freed 68.47 GiB memory, 6.51 GiB memory is still in use.[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=953488)[0m [1;36m(EngineCore_DP0 pid=953826)[0;0m [36m(RayWorkerWrapper pid=954097)[0m INFO 12-06 12:34:02 [gpu_worker.py:117] Sleep mode freed 60.07 GiB memory, 7.78 GiB memory is still in use.[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=953488)[0m [1;36m(EngineCore_DP0 pid=953826)[0;0m [36m(RayWorkerWrapper pid=954087)[0m INFO 12-06 12:33:48 [cumem.py:228] CuMemAllocator: sleep freed 52.53 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 52.53 GiB is discarded directly.[32m [repeated 2x across cluster][0m
[36m(AsyncVLLMInferenceEngine pid=953488)[0m [1;36m(EngineCore_DP0 pid=953826)[0;0m INFO 12-06 12:34:04 [executor_base.py:189] It took 21.142570 seconds to fall asleep.
Error executing job with overrides: ['trainer.algorithm.advantage_estimator=grpo', "data.train_data=['/workspace/data/sql/train.parquet']", "data.val_data=['/workspace/data/sql/validation.parquet']", 'trainer.policy.model.path=Qwen/Qwen2.5-Coder-7B-Instruct', 'trainer.epochs=30', 'trainer.placement.colocate_all=true', 'trainer.strategy=fsdp2', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.cpu_offload=true', 'trainer.policy.optimizer_config.max_grad_norm=0.5', 'trainer.policy.sequence_parallel_size=1', 'trainer.placement.policy_num_gpus_per_node=8', 'trainer.placement.ref_num_gpus_per_node=8', 'generator.num_inference_engines=2', 'generator.inference_engine_tensor_parallel_size=4', 'trainer.train_batch_size=256', 'trainer.micro_forward_batch_size_per_gpu=8', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=6000', 'generator.max_input_length=29000', 'generator.sampling_params.max_generate_length=3000', 'trainer.policy.optimizer_config.lr=1.0e-6', 'trainer.policy_mini_batch_size=256', 'trainer.algorithm.use_kl_loss=false', 'trainer.ckpt_interval=60', 'trainer.hf_save_interval=30', 'trainer.dump_data_batch=true', 'generator.backend=vllm', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=true', 'generator.batched=false', 'environment.env_class=text2sql', 'generator.use_conversation_multi_turn=false', 'generator.n_samples_per_prompt=5', 'generator.gpu_memory_utilization=0.7', 'generator.max_turns=6', 'generator.sampling_params.temperature=0.6', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.stop=["</sql>", "</solution>"]', 'generator.eval_sampling_params.stop=["</sql>", "</solution>"]', 'environment.skyrl_gym.text2sql.db_path=/workspace/data/db/data', 'trainer.logger=wandb', 'trainer.project_name=skyrlsql', 'trainer.run_name=skyrlsql_repro', 'trainer.resume_mode=latest', 'trainer.ckpt_path=/workspace/data/ckpts/skyrl_sql_7B_ckpt', 'trainer.eval_batch_size=1024', 'trainer.eval_before_train=true', 'trainer.eval_interval=5']
Traceback (most recent call last):
  File "/home/ray/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 331, in main
    ray.get(skyrl_entrypoint.remote(cfg))
  File "/home/ray/.cache/uv/builds-v0/.tmpi9PYmt/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ray/.cache/uv/builds-v0/.tmpi9PYmt/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/.cache/uv/builds-v0/.tmpi9PYmt/lib/python3.12/site-packages/ray/_private/worker.py", line 2961, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/home/ray/.cache/uv/builds-v0/.tmpi9PYmt/lib/python3.12/site-packages/ray/_private/worker.py", line 1026, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ActorDiedError): [36mray::skyrl_entrypoint()[39m (pid=952112, ip=172.26.134.226)
  File "/home/ray/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 322, in skyrl_entrypoint
    exp.run()
  File "/home/ray/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 315, in run
    trainer.train()
  File "/tmp/ray/session_2025-12-06_00-13-20_479069_938355/runtime_resources/working_dir_files/_ray_pkg_020c09aa138b4335/skyrl_train/trainer.py", line 268, in train
    status = self.train_critic_and_policy(training_input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-12-06_00-13-20_479069_938355/runtime_resources/working_dir_files/_ray_pkg_020c09aa138b4335/skyrl_train/trainer.py", line 1040, in train_critic_and_policy
    policy_statuses = ray.get(self.policy_model.async_run_ray_method("mesh", "ppo_train", data))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^
                                  ^^^^^^^^^^^^^^^^^^^
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
	class_name: FSDPPolicyWorkerBase
	actor_id: 67075fffce7f3ff41a873b8201000000
	pid: 959049
	namespace: 73ab4044-8e64-4c73-a095-0aa288e76642
	ip: 172.26.134.226
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

What I have tried

I’m now testing with smaller batch sizes to see if this is actually an OOM-driven issue that manifests as a NCCL timeout (e.g., train_batch_size, policy_mini_batch_size, and/or max_prompt_length) while enabling key NCCL debug environment variables eg TORCH_NCCL_TRACE_BUFFER_SIZE or NCCL_DEBUG=INFO.

Questions

1. Does anyone have experience with such an error?
2. Is reducing batch size / sequence lengths the right first direction?

Any help would be greatly appreciated! Thank you!

[shanghongsim]

I’m currently testing with a simplified config (modified from examples/text_to_sql/run_skyrl_sql.sh) to rule out config / resource issues and make the problem easier to reproduce.

Key changes from the original script:

• Smaller train batch: TRAIN_BATCH_SIZE=64 (vs 256)
• Shorter sequence lengths:
  • MAX_PROMPT_LENGTH=3000
  • MAX_GENERATE_LENGTH=1000
• Still using 8 GPUs, but:
  • NUM_INFERENCE_ENGINES=4
  • generator.inference_engine_tensor_parallel_size=2
• vLLM in synchronous + batched mode:
  • generator.async_engine=false
  • generator.batched=true
• More conservative vLLM memory usage: generator.gpu_memory_utilization=0.4
• Extra debug env vars enabled for NCCL / Torch distributed

Here is the exact script I’m using now:

#!/usr/bin/env bash
set -x

DATA_DIR="/workspace/data/sql"
DB_PATH="/workspace/data/db/data"
CKPT_PATH="/workspace/data/ckpts/skyrl_sql_7B_ckpt"

NUM_GPUS=8
NUM_INFERENCE_ENGINES=4 
TP_SIZE=2 
MAX_PROMPT_LENGTH=3000
MAX_GENERATE_LENGTH=1000
TRAIN_BATCH_SIZE=64

# Debug envs
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,GRAPH,ENV
export TORCH_DISTRIBUTED_DEBUG=DETAIL
export TORCH_NCCL_TRACE_BUFFER_SIZE=1048576
export TORCH_SHOW_CPP_STACKTRACES=1
export CUDA_LAUNCH_BLOCKING=1

uv run --isolated --extra vllm -m skyrl_train.entrypoints.main_base \
  trainer.algorithm.advantage_estimator=grpo \
  "data.train_data=['$DATA_DIR/train.parquet']" \
  "data.val_data=['$DATA_DIR/validation.parquet']" \
  trainer.policy.model.path=Qwen/Qwen2.5-Coder-7B-Instruct \
  trainer.epochs=2 \
  trainer.placement.colocate_all=true \
  trainer.strategy=fsdp2 \
  trainer.policy.fsdp_config.cpu_offload=false \
  trainer.ref.fsdp_config.cpu_offload=true \
  trainer.policy.optimizer_config.max_grad_norm=0.5 \
  trainer.policy.sequence_parallel_size=1 \
  trainer.placement.policy_num_gpus_per_node=$NUM_GPUS \
  trainer.placement.ref_num_gpus_per_node=$NUM_GPUS \
  generator.num_inference_engines=$NUM_INFERENCE_ENGINES \
  generator.inference_engine_tensor_parallel_size=$TP_SIZE \
  trainer.train_batch_size=$TRAIN_BATCH_SIZE \
  trainer.micro_forward_batch_size_per_gpu=4 \
  trainer.micro_train_batch_size_per_gpu=1 \
  trainer.max_prompt_length=$MAX_PROMPT_LENGTH \
  generator.max_input_length=$MAX_PROMPT_LENGTH \
  generator.sampling_params.max_generate_length=$MAX_GENERATE_LENGTH \
  trainer.policy.optimizer_config.lr=1.0e-6 \
  trainer.policy_mini_batch_size=64 \
  trainer.algorithm.use_kl_loss=false \
  trainer.ckpt_interval=999999 \
  trainer.hf_save_interval=999999 \
  trainer.dump_data_batch=false \
  generator.backend=vllm \
  generator.run_engines_locally=true \
  generator.weight_sync_backend=nccl \
  generator.async_engine=false \
  generator.batched=true \
  environment.env_class=text2sql \
  generator.use_conversation_multi_turn=false \
  generator.n_samples_per_prompt=1 \
  generator.gpu_memory_utilization=0.4 \
  generator.max_turns=1 \
  generator.sampling_params.temperature=0.6 \
  generator.sampling_params.top_p=0.95 \
  'generator.sampling_params.stop=["</sql>", "</solution>"]' \
  environment.skyrl_gym.text2sql.db_path=$DB_PATH \
  trainer.logger=wandb \
  trainer.project_name=skyrlsql_debug \
  trainer.run_name=small_config_run \
  trainer.resume_mode=never \
  trainer.eval_before_train=false \
  trainer.eval_interval=999999 \
  "$@"

With this config, the run does start training (generation + GRPO steps are executing), so it seems the earlier timeout / NCCL issues are at least partially configuration/load-related.

[erictang000]

hey @shanghongsim, thanks for the detailed report! One thing that stands out to me is that the generation time seems unexpectedly very long?

If step time is:

'step', time cost: 23727.05s

and fwd/training are on the order of hundreds seconds, it seems like something could be off with the hardware or compute setup?

you can find a wandb report here: https://wandb.ai/sky-posttraining-uc-berkeley/skyrl-sql/reports/SkyRL-SQL---VmlldzoxMzM0MTAyMw?accessToken=vrqncoa32qcobvvpuo672yji4gweguk6tjxvaflk1zh73fn70j6l5rj8j619uvry
(from our docs here: https://skyrl.readthedocs.io/en/latest/recipes/skyrl-sql.html), for references on timing for a 7B model on 8xH100 - the total step time should be just ~450 seconds, and generation should be just ~200 seconds, even with the full 32k max context length, since we are training on an Instruct model and a lot of the trajectory length just comes from tool calls from the model.

If you're able to run training with the more conservative settings, could you help provide some wandb system metrics for your setup? (i.e. gpu utilization, gpu memory allocated) Generation metrics would also be helpful (max/min/avg num tokens), and any timing breakdowns. Thanks!

[shanghongsim]

Hi @erictang000, thank you for taking a look at this and for noticing the strange things going on with timing. I ran two debugging configs and both failed in a similar way, so I wanted to share a bit more detail and see if anything obvious jumps out.

Small config

W&B: https://wandb.ai/shanghongsim/skyrlsql/reports/Small-config--VmlldzoxNTMzNDQ1Nw
Status: failed

For this run, I reduced max lengths, batch sizes, GPU memory utilization, conversation max turns and turned off async RL training.

Small config

``` set -x

# bash examples/text_to_sql/small_config_v2.sh 2>&1 | tee /workspace/data/small_config_v2.log

DATA_DIR="/workspace/data/sql"
DB_PATH="/workspace/data/db/data"
CKPT_PATH="/workspace/data/ckpts/skyrl_sql_7B_ckpt"

NUM_GPUS=8
NUM_INFERENCE_ENGINES=4 # 2
TP_SIZE=2 # 4
MAX_PROMPT_LENGTH=3000 
MAX_GENERATE_LENGTH=1000 # 3000
TRAIN_BATCH_SIZE=64 # 256

# Debug envs
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,GRAPH,ENV
export TORCH_DISTRIBUTED_DEBUG=DETAIL
export TORCH_NCCL_TRACE_BUFFER_SIZE=1048576
export TORCH_SHOW_CPP_STACKTRACES=1
export CUDA_LAUNCH_BLOCKING=1

uv run --isolated --extra vllm -m skyrl_train.entrypoints.main_base \
trainer.algorithm.advantage_estimator="grpo" \
data.train_data="['$DATA_DIR/train.parquet']" \
data.val_data="['$DATA_DIR/validation.parquet']" \
trainer.policy.model.path="Qwen/Qwen2.5-Coder-7B-Instruct" \
trainer.epochs=2 \ # 30
trainer.placement.colocate_all=true \
trainer.strategy=fsdp2 \
trainer.policy.fsdp_config.cpu_offload=false \
trainer.ref.fsdp_config.cpu_offload=true \
trainer.policy.optimizer_config.max_grad_norm=0.5 \
trainer.policy.sequence_parallel_size=1 \
trainer.placement.policy_num_gpus_per_node=$NUM_GPUS \
trainer.placement.ref_num_gpus_per_node=$NUM_GPUS \
generator.num_inference_engines=$NUM_INFERENCE_ENGINES \
generator.inference_engine_tensor_parallel_size=$TP_SIZE \
trainer.train_batch_size=$TRAIN_BATCH_SIZE \
trainer.micro_forward_batch_size_per_gpu=4 \ # 8
trainer.micro_train_batch_size_per_gpu=1 \
trainer.max_prompt_length=$MAX_PROMPT_LENGTH \ # 6000
generator.max_input_length=$MAX_PROMPT_LENGTH \ # 29000
generator.sampling_params.max_generate_length=$MAX_GENERATE_LENGTH \
trainer.policy.optimizer_config.lr=1.0e-6 \
trainer.policy_mini_batch_size=64 \ # 256
trainer.algorithm.use_kl_loss=false \
trainer.ckpt_interval=60 \
trainer.hf_save_interval=30 \
trainer.dump_data_batch=true \
generator.backend=vllm \
generator.run_engines_locally=true \
generator.weight_sync_backend=nccl \
generator.async_engine=false \ # true
generator.batched=true \ # false
environment.env_class=text2sql \
generator.use_conversation_multi_turn=false \
generator.n_samples_per_prompt=1 \ # 5 
generator.gpu_memory_utilization=0.4 \ # 0.7
generator.max_turns=1 \ # 6
generator.sampling_params.temperature=0.6 \
generator.sampling_params.top_p=0.95 \
generator.sampling_params.stop='["</sql>", "</solution>"]' \
generator.eval_sampling_params.stop='["</sql>", "</solution>"]' \
environment.skyrl_gym.text2sql.db_path=$DB_PATH \
trainer.logger="wandb" \
trainer.project_name="skyrlsql" \
trainer.run_name="small_config_v2" \
trainer.resume_mode=latest \
trainer.ckpt_path=$CKPT_PATH \
trainer.eval_batch_size=1024 \
trainer.eval_before_train=true \
trainer.eval_interval=5 \
"$@"
```

Key observations:

• generate for a single step took ~1351s:

Finished: 'generate', time cost: 1351.50s

The slow generate is perhaps to be expected because of the following I set for debugging:

• CUDA_LAUNCH_BLOCKING=1
• Very conservative generator.gpu_memory_utilization=0.4
• Not sure if async_engine=false affects things

• Reward was consistently bad on that step:

reward/avg_pass_at_1: 0.0, reward/avg_raw_reward: -1.0

• After that, the step proceeds to fwd_logprobs_values_reward, and then Ray starts erroring with:

(raylet) ... Too many open files; RPC Error details: rpc_code: 14

• On the node:

ulimit -n
# 1024

pidof raylet
# 628221

ls /proc/628221/fd | wc -l
# 1024

so it looks like the raylet is hitting the per-process FD limit and then everything sort of stalls out.

• I noticed this because GPU utilization was 0 overnight. I verified with nvidia-smi in the morning; GPUs were basically idle but still holding memory:

GPU-Util 0%
Processes: ray::RayWorkerWrapper, ray::FSDPPolicyWorkerBase on all 8 GPUs

Relevant error chunk:

``` [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:25.544[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m209[0m - [1m[32mFinished: 'generate', time cost: 1351.50s[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.049[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m230[0m - [1m[32mStarted: 'postprocess_generator_output'[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.069[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mpostprocess_generator_output[0m:[36m686[0m - [1m[32mreward/avg_pass_at_1: 0.0, reward/avg_raw_reward: -1.0[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.079[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m230[0m - [1m[32mFinished: 'postprocess_generator_output', time cost: 0.03s[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.099[0m | [1m[32mINFO [0m | [36mskyrl_train.utils.logging_utils[0m:[36mlog_example[0m:[36m72[0m - [1m[32mExample: [36m(skyrl_entrypoint pid=645566)[0m Input: [34m[{'content': "\nTask Overview:\nYou are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question within limited turns. You should breakdown the problem, draft your reasoning process, and generate the solution.\n\nDatabase Engine:\nSQLite\n\nDatabase Schema:\n{db_details}\nThis schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints.\n\nExternal Knowledge:\n{external_knowledge}\n\nQuestion:\n{question}\n\nInstructions:\n- Make sure you only output the information that is asked in the question. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more.\n- The generated query should return all of the information asked in the question without any missing or extra information.\n- Before generating the final SQL query, please think through the steps of how to write the query. It should include detailed considerations such as analisying questions, \nsummarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, thinking of how to call SQL tools, and revisiting previous steps.\n\n\nFormat:\n- Conduct thinking inside ... blocks every time you get new observation or information. \n- You can use SQL tool written within a single your sql block to explore or verify. SQL tool output will be shown as dataframe inside .... Based on this observation, you can think again and refine.\n- The returned dataframe will be truncated in 50 rows if observation is too long. \n- If you find no further exploration is needed or reaches max turns, you MUST directly provide the final SQL query solution inside .... \n\n------------------------ START OF EXAMPLE ------------------------\nQuestion: how many pigs are in the farm? \n Database Schema:\n Table: animals\n - id (INTEGER, PRIMARY KEY)\n - species (TEXT)\n - age (INTEGER)\n - name (TEXT)\n\nI am querying how many pigs are in the farm. I will begin by checking if the 'animals' table exists and contains entries with species = 'pig'.\nSELECT COUNT(*) FROM animals WHERE species = 'pig';\n\n+----------+\n| COUNT(*) |\n+----------+\n| 12 |\n+----------+\n\nThe result indicates that there are 12 pigs in the farm. Since the question asks for how many pigs, I can now output the final SQL as the solution.\nSELECT COUNT(*) FROM animals WHERE species = 'pig';\n------------------------ END OF EXAMPLE ------------------------\n", 'role': 'system'}, {'content': "{db_details}:CREATE TABLE approach_accidents (\n approach_accident_id INTEGER, -- Unique identifier for each approach accident, example: [0, 1]\n accident_id INTEGER, -- ID of the accident to which the approach accident belongs, example: [1]\n approach_accidents INTEGER, -- Number of approach accidents, example: [3, 10]\n approach_fatalities INTEGER, -- example: [24, 73]\n approach_conditions TEXT, -- Weather and other conditions during the approach phase, example: ['Low visibility, foggy conditions', 'Strong winds, heavy rain']\n PRIMARY KEY (approach_accident_id),\n CONSTRAINT fk_approach_accidents_accident_id FOREIGN KEY (accident_id) REFERENCES accidents (accident_id)\n);\n\nCREATE TABLE entities (\n entity_id INTEGER, -- Unique identifier for each entity, example: [0, 1]\n entity_name TEXT, -- Name of the entity, example: ['World', 'United States']\n entity_type TEXT, -- example: ['Global', 'Country']\n entity_description TEXT, -- A brief description of the entity, example: ['Global entity representing all regions a', 'Country entity representing the United S']\n entity_code TEXT, -- A unique code for the entity (e.g., ISO country code), example: ['WRL', 'USA']\n entity_parent_id INTEGER, -- example: [1]\n PRIMARY KEY (entity_id)\n);\n\nCREATE TABLE landing_accidents (\n landing_accident_id INTEGER, -- example: [0, 1]\n accident_id INTEGER, -- ID of the accident to which the landing accident belongs, example: [0]\n landing_accidents INTEGER, -- example: [1, 2]\n landing_fatalities INTEGER, -- Number of fatalities in landing accidents, example: [3, 5]\n landing_location TEXT, -- example: ['John F. Kennedy International Airport', 'Los Angeles International Airport']\n landing_conditions TEXT, -- example: ['Wet runway, low visibility', 'Strong crosswinds, heavy rain']\n PRIMARY KEY (landing_accident_id),\n CONSTRAINT fk_landing_accidents_accident_id FOREIGN KEY (accident_id) REFERENCES accidents (accident_id)\n);\n\nCREATE TABLE users (\n user_id INTEGER, -- Unique identifier for each user, example: [0, 1]\n username TEXT, -- Username, example: ['admin', 'analyst']\n password TEXT, -- Password, example: ['password']\n `role` TEXT, -- Role of the user (e.g., admin, analyst), example: ['admin', 'analyst']\n user_phone TEXT, -- Phone number of the user, example: ['123-456-7890', '098-765-4321']\n user_department TEXT, -- Department of the user, example: ['IT', 'Safety Analysis']\n user_status TEXT, -- example: ['active']\n PRIMARY KEY (user_id)\n);\n\nCREATE TABLE en_route_accidents (\n en_route_accident_id INTEGER, -- example: [0, 1]\n accident_id INTEGER, -- example: [1]\n en_route_accidents INTEGER, -- example: [15, 18]\n en_route_casualties INTEGER, -- example: [196, 189]\n en_route_location TEXT, -- example: ['Atlantic Ocean', 'Pacific Ocean']\n en_route_conditions TEXT, -- example: ['Clear skies, moderate turbulence', 'Heavy rain, strong winds']\n PRIMARY KEY (en_route_accident_id),\n CONSTRAINT fk_en_route_accidents_accident_id FOREIGN KEY (accident_id) REFERENCES accidents (accident_id)\n);\n\nCREATE TABLE access_logs (\n access_id INTEGER, -- Unique identifier for each access event, example: [0, 1]\n user_id INTEGER, -- ID of the user accessing the database, example: [1, 0]\n access_date TEXT, -- example: ['2023-01-01', '2023-01-02']\n access_type TEXT, -- Type of access (e.g., read, write), example: ['read', 'write']\n user_ip TEXT, -- IP address of the user accessing the database, example: ['192.168.1.1', '192.168.1.2']\n access_duration TEXT, -- Duration of the access session, example: ['30 minutes', '1 hour']\n PRIMARY KEY (access_id),\n CONSTRAINT fk_access_logs_user_id FOREIGN KEY (user_id) REFERENCES users (user_id)\n);\n\nCREATE TABLE recommendations (\n recommendation_id INTEGER, -- example: [0, 1]\n investigation_id INTEGER, -- example: [0]\n recommendation TEXT, -- example: ['Improve engine maintenance procedures', 'Enhance pilot training programs']\n recommendation_date TEXT, -- example: ['2022-03-01', '2022-07-01']\n recommendation_category TEXT, -- A category for the recommendation (e.g., operational, regulatory), example: ['Operational']\n PRIMARY KEY (recommendation_id),\n CONSTRAINT fk_recommendations_investigation_id FOREIGN KEY (investigation_id) REFERENCES investigations (investigation_id)\n);\n\nCREATE TABLE accident_types (\n accident_type_id INTEGER, -- Unique identifier for each accident type, example: [0, 1]\n accident_type TEXT, -- Type of accident, example: ['En Route', 'Takeoff']\n accident_type_description TEXT, -- example: ['Accident occurring during the en route p', 'Accident occurring during takeoff']\n accident_type_code TEXT, -- example: ['ENR', 'TKO']\n PRIMARY KEY (accident_type_id)\n);\n\nCREATE TABLE takeoff_accidents (\n takeoff_accident_id INTEGER, -- example: [0, 1]\n accident_id INTEGER, -- ID of the accident to which the takeoff accident belongs, example: [1, 0]\n takeoff_accidents INTEGER, -- example: [4, 1]\n takeoff_casualties INTEGER, -- example: [46, 1]\n takeoff_location TEXT, -- The specific location of the takeoff accident (e.g., airport), example: ['John F. Kennedy International Airport', 'Los Angeles International Airport']\n takeoff_conditions TEXT, -- example: ['Foggy conditions, low visibility', 'Strong crosswinds']\n PRIMARY KEY (takeoff_accident_id),\n CONSTRAINT fk_takeoff_accidents_accident_id FOREIGN KEY (accident_id) REFERENCES accidents (accident_id)\n);\n\nCREATE TABLE investigations (\n investigation_id INTEGER, -- example: [0, 1]\n accident_id INTEGER, -- example: [1, 0]\n investigation_status TEXT, -- example: ['Ongoing', 'Completed']\n investigation_start_date TEXT, -- example: ['2022-01-01', '2022-02-01']\n investigation_end_date TEXT, -- example: ['2022-06-30']\n investigation_report_url TEXT, -- example: ['http://example.com/report1.pdf', 'http://example.com/report2.pdf']\n PRIMARY KEY (investigation_id),\n CONSTRAINT fk_investigations_accident_id FOREIGN KEY (accident_id) REFERENCES accidents (accident_id)\n);\n\nCREATE TABLE years (\n year_id INTEGER, -- example: [0, 1]\n `year` INTEGER, -- example: [1942, 1943]\n year_start_date TEXT, -- example: ['1942-01-01', '1943-01-01']\n year_end_date TEXT, -- example: ['1942-12-31', '1943-12-31']\n PRIMARY KEY (year_id)\n);\n\nCREATE TABLE accident_causes (\n accident_cause_id INTEGER, -- Unique identifier for each accident cause, example: [0, 1]\n accident_id INTEGER, -- ID of the accident to which the cause belongs, example: [1, 0]\n cause TEXT, -- Cause of the accident, example: ['Engine Failure', 'Human Error']\n cause_category TEXT, -- example: ['Mechanical', 'Human Error']\n PRIMARY KEY (accident_cause_id),\n CONSTRAINT fk_accident_causes_accident_id FOREIGN KEY (accident_id) REFERENCES accidents (accident_id)\n);\n\nCREATE TABLE accidents (\n accident_id INTEGER, -- example: [0, 1]\n entity_id INTEGER, -- ID of the entity to which the accident belongs, example: [0, 1]\n year_id INTEGER, -- ID of the year in which the accident occurred, example: [1, 0]\n accident_type_id INTEGER, -- ID of the type of accident, example: [0]\n accident_location TEXT, -- The location where the accident occurred, example: ['Atlantic Ocean', 'New York']\n aircraft_model TEXT, -- The model of the aircraft involved, example: ['Boeing 247', 'DC-3']\n operator TEXT, -- example: ['American Airlines', 'Delta Air Lines']\n accidents INTEGER, -- Number of accidents, example: [15, 4]\n casualties INTEGER, -- example: [196, 46]\n total_fatalities INTEGER, -- example: [200, 50]\n PRIMARY KEY (accident_id),\n CONSTRAINT fk_accidents_entity_id FOREIGN KEY (entity_id) REFERENCES entities (entity_id),\n CONSTRAINT fk_accidents_year_id FOREIGN KEY (year_id) REFERENCES years (year_id),\n CONSTRAINT fk_accidents_accident_type_id FOREIGN KEY (accident_type_id) REFERENCES accident_types (accident_type_id)\n);\n\nCREATE TABLE initial_climb_accidents (\n initial_climb_accident_id INTEGER, -- example: [0, 1]\n accident_id INTEGER, -- example: [1, 0]\n initial_climb_accidents INTEGER, -- example: [4, 6]\n initial_climb_casualties INTEGER, -- example: [51, 34]\n initial_climb_location TEXT, -- example: ['Near John F. Kennedy International Airpo', 'Near Los Angeles International Airport']\n initial_climb_conditions TEXT, -- example: ['Strong winds, low visibility', 'Heavy rain, turbulence']\n PRIMARY KEY (initial_climb_accident_id),\n CONSTRAINT fk_initial_climb_accidents_accident_id FOREIGN KEY (accident_id) REFERENCES accidents (accident_id)\n);;\n {external_knowledge}: None required for this query as it is relatively straightforward and does not require external knowledge to understand the intent behind the question.;\n {question}: What are the accident types in 1942 that resulted in any fatalities, along with their respective total fatalities and average casualties, listed in order of the highest to lowest total fatalities?", 'role': 'user'}][0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m Output (Reward: -1.0000): [36m(skyrl_entrypoint pid=645566)[0m [33mLet's break down the question and the database schema to generate the SQL query.[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m### Step-by-Step Reasoning:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m1. **Identify the Relevant Tables**:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m - `accidents`: Contains information about each accident, including the year and total fatalities.[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m - `accident_types`: Contains the types of accidents.[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m - `accident_causes`: Links accidents to their causes.[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m2. **Filter Accidents by Year and Fatalities**:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m - We need to filter accidents that occurred in 1942 and resulted in any fatalities.[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m3. **Join Tables**:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m - Join `accidents` with `accident_types` to get the type of each accident.[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m - Join `accidents` with `accident_causes` to ensure we only consider accidents with fatalities.[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m4. **Aggregate Data**:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m - Calculate the total fatalities and average casualties for each accident type.[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m5. **Order the Results**:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m - Order the results by the highest to lowest total fatalities.[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m### SQL Query Generation:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m1. **Filter Accidents by Year and Fatalities**:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m ```sql[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m SELECT a.accident_id, a.year_id, a.total_fatalities, a.casualties, at.accident_type[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m FROM accidents a[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m JOIN accident_types at ON a.accident_type_id = at.accident_type_id[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m WHERE a.year_id = (SELECT year_id FROM years WHERE year = 1942)[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m AND a.total_fatalities > 0;[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m ```[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m2. **Aggregate Data**:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m ```sql[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m SELECT at.accident_type, SUM(a.total_fatalities) AS total_fatalities, AVG(a.casualties) AS average_casualties[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m FROM accidents a[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m JOIN accident_types at ON a.accident_type_id = at.accident_type_id[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m WHERE a.year_id = (SELECT year_id FROM years WHERE year = 1942)[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m AND a.total_fatalities > 0[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m GROUP BY at.accident_type;[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m ```[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m3. **Order the Results**:[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m ```sql[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m SELECT at.accident_type, SUM(a.total_fatalities) AS total_fatalities, AVG(a.casualties) AS average_casualties[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m FROM accidents a[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m JOIN accident_types at ON a.accident_type_id = at.accident_type_id[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m WHERE a.year_id = (SELECT year_id FROM years WHERE year = 1942)[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m AND a.total_fatalities > 0[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m GROUP BY at.accident_type[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m ORDER BY total_fatalities DESC;[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m ```[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m### Final SQL Query[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33mSELECT at.accident_type, SUM(a.total_fatalities) AS total_fatalities, AVG(a.casualties) AS average_casualties FROM accidents a JOIN accident_types at ON a.accident_type_id = at.accident_type_id WHERE a.year_id = (SELECT year_id FROM years WHERE year = 1942) AND a.total_fatalities > 0 GROUP BY at.accident_type ORDER BY total_fatalities DESC;[0m[1m[32m [36m(skyrl_entrypoint pid=645566)[0m [33m[0m[1m[32m[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.120[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m242[0m - [1m[32mStarted: 'convert_to_training_input'[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.772[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mconvert_to_training_input[0m:[36m603[0m - [1m[32mNumber of sequences before padding: 64[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.773[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mconvert_to_training_input[0m:[36m605[0m - [1m[32mNumber of sequences after padding: 64[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.774[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m244[0m - [1m[32mNumber of sequences: 64[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.775[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m242[0m - [1m[32mFinished: 'convert_to_training_input', time cost: 0.66s[0m [36m(skyrl_entrypoint pid=645566)[0m [32m2025-12-10 06:25:35.781[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m247[0m - [1m[32mStarted: 'fwd_logprobs_values_reward'[0m [33m(raylet)[0m [2025-12-10 06:25:51,992 E 628221 628221] (raylet) worker.cc:236: Failed to send wait complete: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:172.26.135.196:10252: Too many open files; RPC Error details: rpc_code: 14 ```

`nvidia-smi` output

``` (base) shanghong@oumi-compute004:~$ nvidia-smi Wed Dec 10 18:58:21 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.124.06 Driver Version: 570.124.06 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA H100 80GB HBM3 On | 00000000:63:00.0 Off | 0 | | N/A 31C P0 121W / 700W | 9457MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA H100 80GB HBM3 On | 00000000:6B:00.0 Off | 0 | | N/A 33C P0 116W / 700W | 9431MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA H100 80GB HBM3 On | 00000000:71:00.0 Off | 0 | | N/A 35C P0 124W / 700W | 9413MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA H100 80GB HBM3 On | 00000000:79:00.0 Off | 0 | | N/A 31C P0 117W / 700W | 9433MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 4 NVIDIA H100 80GB HBM3 On | 00000000:7F:00.0 Off | 0 | | N/A 33C P0 120W / 700W | 9391MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 5 NVIDIA H100 80GB HBM3 On | 00000000:87:00.0 Off | 0 | | N/A 35C P0 123W / 700W | 9391MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 6 NVIDIA H100 80GB HBM3 On | 00000000:8D:00.0 Off | 0 | | N/A 34C P0 119W / 700W | 9433MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 7 NVIDIA H100 80GB HBM3 On | 00000000:95:00.0 Off | 0 | | N/A 31C P0 120W / 700W | 9203MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A          646682      C   ray::RayWorkerWrapper                  2964MiB |
|    0   N/A  N/A          651551      C   ray::FSDPPolicyWorkerBase              6438MiB |
|    1   N/A  N/A          646680      C   ray::RayWorkerWrapper                  2964MiB |
|    1   N/A  N/A          651882      C   ray::FSDPPolicyWorkerBase              6412MiB |
|    2   N/A  N/A          647174      C   ray::RayWorkerWrapper                  2964MiB |
|    2   N/A  N/A          651881      C   ray::FSDPPolicyWorkerBase              6394MiB |
|    3   N/A  N/A          647176      C   ray::RayWorkerWrapper                  2964MiB |
|    3   N/A  N/A          651880      C   ray::FSDPPolicyWorkerBase              6414MiB |
|    4   N/A  N/A          646882      C   ray::RayWorkerWrapper                  2964MiB |
|    4   N/A  N/A          651884      C   ray::FSDPPolicyWorkerBase              6372MiB |
|    5   N/A  N/A          646885      C   ray::RayWorkerWrapper                  2964MiB |
|    5   N/A  N/A          651888      C   ray::FSDPPolicyWorkerBase              6372MiB |
|    6   N/A  N/A          646887      C   ray::RayWorkerWrapper                  2964MiB |
|    6   N/A  N/A          651892      C   ray::FSDPPolicyWorkerBase              6414MiB |
|    7   N/A  N/A          646889      C   ray::RayWorkerWrapper                  2964MiB |
|    7   N/A  N/A          651898      C   ray::FSDPPolicyWorkerBase              6184MiB |
+-----------------------------------------------------------------------------------------+
```

Med config

W&B: https://wandb.ai/shanghongsim/skyrlsql/reports/Med-config--VmlldzoxNTMzNDEwNw?accessToken=e2qi6yd4thtoo2ee4svo4uti6ksxmisdjdcu45no8h0yqqjcgn50fwcw0vf6tucv
Status: failed

In this config, I increase max_length, max_turns and n_samples_per_prompt to match the original config. Things still fail with a similar pattern.

Med config

``` set -x

# bash examples/text_to_sql/med_config.sh 2>&1 | tee /workspace/data/med_config.log

DATA_DIR="/workspace/data/sql"
DB_PATH="/workspace/data/db/data"
CKPT_PATH="/workspace/data/ckpts/skyrl_sql_7B_ckpt_medium"

NUM_GPUS=8
NUM_INFERENCE_ENGINES=4 # 2
TP_SIZE=2 # 4
MAX_INPUT_LENGTH=29000
MAX_GENERATE_LENGTH=3000
TRAIN_BATCH_SIZE=64

# Debug envs
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,GRAPH,ENV
export TORCH_DISTRIBUTED_DEBUG=DETAIL
export TORCH_NCCL_TRACE_BUFFER_SIZE=1048576
export TORCH_SHOW_CPP_STACKTRACES=1
export CUDA_LAUNCH_BLOCKING=1

uv run --isolated --extra vllm -m skyrl_train.entrypoints.main_base \
trainer.algorithm.advantage_estimator="grpo" \
data.train_data="['$DATA_DIR/train.parquet']" \
data.val_data="['$DATA_DIR/validation.parquet']" \
trainer.policy.model.path="Qwen/Qwen2.5-Coder-7B-Instruct" \
trainer.epochs=30 \
trainer.placement.colocate_all=true \
trainer.strategy=fsdp2 \
trainer.policy.fsdp_config.cpu_offload=false \
trainer.ref.fsdp_config.cpu_offload=true \
trainer.policy.optimizer_config.max_grad_norm=0.5 \
trainer.policy.sequence_parallel_size=1 \
trainer.placement.policy_num_gpus_per_node=$NUM_GPUS \
trainer.placement.ref_num_gpus_per_node=$NUM_GPUS \
generator.num_inference_engines=$NUM_INFERENCE_ENGINES \
generator.inference_engine_tensor_parallel_size=$TP_SIZE \
trainer.train_batch_size=$TRAIN_BATCH_SIZE \
trainer.micro_forward_batch_size_per_gpu=4 \
trainer.micro_train_batch_size_per_gpu=1 \
trainer.max_prompt_length=6000 \
generator.max_input_length=$MAX_INPUT_LENGTH \
generator.sampling_params.max_generate_length=$MAX_GENERATE_LENGTH \
trainer.policy.optimizer_config.lr=1.0e-6 \
trainer.policy_mini_batch_size=64 \
trainer.algorithm.use_kl_loss=false \
trainer.ckpt_interval=60 \
trainer.hf_save_interval=30 \
trainer.dump_data_batch=true \
generator.backend=vllm \
generator.run_engines_locally=true \
generator.weight_sync_backend=nccl \
generator.async_engine=false \
generator.batched=true \
environment.env_class=text2sql \
generator.use_conversation_multi_turn=false \
generator.n_samples_per_prompt=5 \
generator.gpu_memory_utilization=0.4 \
generator.max_turns=6 \
generator.sampling_params.temperature=0.6 \
generator.sampling_params.top_p=0.95 \
generator.sampling_params.stop='["</sql>", "</solution>"]' \
generator.eval_sampling_params.stop='["</sql>", "</solution>"]' \
environment.skyrl_gym.text2sql.db_path=$DB_PATH \
trainer.logger="wandb" \
trainer.project_name="skyrlsql" \
trainer.run_name="med_config" \
trainer.resume_mode=latest \
trainer.ckpt_path=$CKPT_PATH \
trainer.eval_batch_size=1024 \
trainer.eval_before_train=true \
trainer.eval_interval=5 \
"$@"
```

• convert_to_training_input finishes with:

Number of sequences before padding: 320
Number of sequences after padding: 320
Finished: 'convert_to_training_input', time cost: 5.09s
Started: 'fwd_logprobs_values_reward'

• Then the raylet starts logging:

Failed to send wait complete: RpcError: ... Too many open files

• After that, there’s an NCCL collective timeout:

[Rank 5] Watchdog caught collective operation timeout:
WorkNCCL(... OpType=_ALLGATHER_BASE ... Timeout(ms)=600000) ran for 600026 milliseconds before timing out.
...
Some NCCL operations have failed or timed out... we are taking the entire process down.

• Finally, the train step fails with ActorUnavailableError from Ray, since the FSDP actors have crashed.

Relevant error chunk

``` 36m(skyrl_entrypoint pid=4054508)[0m [32m2025-12-10 04:26:22.640[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m242[0m - [1m[32mStarted: 'convert_to_training_input'[0m [36m(skyrl_entrypoint pid=4054508)[0m [32m2025-12-10 04:26:27.729[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mconvert_to_training_input[0m:[36m603[0m - [1m[32mNumber of sequences before padding: 320[0m [36m(skyrl_entrypoint pid=4054508)[0m [32m2025-12-10 04:26:27.731[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mconvert_to_training_input[0m:[36m605[0m - [1m[32mNumber of sequences after padding: 320[0m [36m(skyrl_entrypoint pid=4054508)[0m [32m2025-12-10 04:26:27.731[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m244[0m - [1m[32mNumber of sequences: 320[0m [36m(skyrl_entrypoint pid=4054508)[0m [32m2025-12-10 04:26:27.732[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m242[0m - [1m[32mFinished: 'convert_to_training_input', time cost: 5.09s[0m [36m(skyrl_entrypoint pid=4054508)[0m [32m2025-12-10 04:26:27.734[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m247[0m - [1m[32mStarted: 'fwd_logprobs_values_reward'[0m [33m(raylet)[0m [2025-12-10 04:26:32,942 E 4040530 4040530] (raylet) worker.cc:236: Failed to send wait complete: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:172.26.133.87:10239: Too many open files; RPC Error details: rpc_code: 14 [33m(raylet)[0m [2025-12-10 04:26:38,690 E 4040530 4040530] (raylet) worker.cc:236: Failed to send wait complete: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:172.26.133.87:10243: Too many open files; RPC Error details: rpc_code: 14[32m [repeated 4x across cluster][0m [36m(FSDPPolicyWorkerBase pid=4060712)[0m [rank5]:[E1210 04:36:48.172352253 ProcessGroupNCCL.cpp:685] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=24054, OpType=_ALLGATHER_BASE, NumelIn=68125120, NumelOut=545000960, Timeout(ms)=600000) ran for 600026 milliseconds before timing out. [36m(FSDPPolicyWorkerBase pid=4060712)[0m [rank5]:[E1210 04:36:48.172581690 ProcessGroupNCCL.cpp:2252] [PG ID 0 PG GUID 0(default_pg) Rank 5] failure detected by watchdog at work sequence id: 24054 PG status: last enqueued work: 24056, last completed work: 24053 [36m(FSDPPolicyWorkerBase pid=4060712)[0m [rank5]:[E1210 04:36:48.172596464 ProcessGroupNCCL.cpp:732] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value. [36m(FSDPPolicyWorkerBase pid=4060712)[0m [rank5]:[E1210 04:36:48.172640783 ProcessGroupNCCL.cpp:2584] [PG ID 0 PG GUID 0(default_pg) Rank 5] First PG on this rank to signal dumping. [36m(FSDPPolicyWorkerBase pid=4060709)[0m [rank4]:[E1210 04:36:48.252116500 ProcessGroupNCCL.cpp:1806] [PG ID 0 PG GUID 0(default_pg) Rank 4] Observed flight recorder dump signal from another rank via TCPStore. [36m(FSDPPolicyWorkerBase pid=4060709)[0m [rank4]:[E1210 04:36:48.253347916 ProcessGroupNCCL.cpp:1870] [PG ID 0 PG GUID 0(default_pg) Rank 4] Received a dump signal due to a collective timeout from rank 5 and we will try our best to dump the debug info. Last enqueued NCCL work: 24053, last completed NCCL work: 24053.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. [36m(FSDPPolicyWorkerBase pid=4060709)[0m [rank4]:[E1210 04:36:48.254164956 ProcessGroupNCCL.cpp:1589] [PG ID 0 PG GUID 0(default_pg) Rank 4] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1 [36m(FSDPPolicyWorkerBase pid=4060712)[0m [rank5]:[E1210 04:36:49.154275199 ProcessGroupNCCL.cpp:1870] [PG ID 0 PG GUID 0(default_pg) Rank 5] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 24056, last completed NCCL work: 24053.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. [36m(FSDPPolicyWorkerBase pid=4060712)[0m [rank5]:[E1210 04:37:48.174457450 ProcessGroupNCCL.cpp:746] [Rank 5] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [36m(FSDPPolicyWorkerBase pid=4060712)[0m [rank5]:[E1210 04:37:48.174509732 ProcessGroupNCCL.cpp:760] [Rank 5] To avoid data inconsistency, we are taking the entire process down. [36m(FSDPPolicyWorkerBase pid=4060712)[0m [rank5]:[E1210 04:37:48.187789741 ProcessGroupNCCL.cpp:2068] [PG ID 0 PG GUID 0(default_pg) Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=24054, OpType=_ALLGATHER_BASE, NumelIn=68125120, NumelOut=545000960, Timeout(ms)=600000) ran for 600026 milliseconds before timing out. [36m(FSDPPolicyWorkerBase pid=4060712)[0m Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:688 (most recent call first): [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x80 (0x7b38f837eeb0 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libc10.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional > >) + 0x247 (0x7b09cae40147 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #2: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x1591 (0x7b09cae43b61 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #3: c10d::ProcessGroupNCCL::Watchdog::run() + 0xd2 (0x7b09cae44ec2 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #4: + 0xd3b65 (0x7b39122f1b65 in /home/ray/anaconda3/bin/../lib/libstdc++.so.6) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #5: + 0x94ac3 (0x7b3914694ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #6: + 0x1268c0 (0x7b39147268c0 in /usr/lib/x86_64-linux-gnu/libc.so.6) [36m(FSDPPolicyWorkerBase pid=4060712)[0m [36m(FSDPPolicyWorkerBase pid=4060715)[0m [rank7]:[E1210 04:36:53.908175451 ProcessGroupNCCL.cpp:685] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=24054, OpType=_ALLGATHER_BASE, NumelIn=68125120, NumelOut=545000960, Timeout(ms)=600000) ran for 600012 milliseconds before timing out.[32m [repeated 2x across cluster][0m [36m(FSDPPolicyWorkerBase pid=4060715)[0m [rank7]:[E1210 04:36:53.908629090 ProcessGroupNCCL.cpp:2252] [PG ID 0 PG GUID 0(default_pg) Rank 7] failure detected by watchdog at work sequence id: 24054 PG status: last enqueued work: 24056, last completed work: 24053[32m [repeated 2x across cluster][0m [36m(FSDPPolicyWorkerBase pid=4060715)[0m [rank7]:[E1210 04:36:53.908672367 ProcessGroupNCCL.cpp:732] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.[32m [repeated 2x across cluster][0m [36m(FSDPPolicyWorkerBase pid=4060710)[0m [rank3]:[E1210 04:36:49.993993643 ProcessGroupNCCL.cpp:1806] [PG ID 0 PG GUID 0(default_pg) Rank 3] Observed flight recorder dump signal from another rank via TCPStore.[32m [repeated 6x across cluster][0m [36m(FSDPPolicyWorkerBase pid=4060710)[0m [rank3]:[E1210 04:36:49.008991404 ProcessGroupNCCL.cpp:1870] [PG ID 0 PG GUID 0(default_pg) Rank 3] Received a dump signal due to a collective timeout from rank 5 and we will try our best to dump the debug info. Last enqueued NCCL work: 24053, last completed NCCL work: 24053.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. [32m [repeated 6x across cluster][0m [36m(FSDPPolicyWorkerBase pid=4060712)[0m [rank5]:[E1210 04:36:49.157731906 ProcessGroupNCCL.cpp:1589] [PG ID 0 PG GUID 0(default_pg) Rank 5] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1[32m [repeated 7x across cluster][0m [36m(FSDPPolicyWorkerBase pid=4060712)[0m [2025-12-10 04:37:48,923 E 4060712 4061972] logging.cc:118: Unhandled exception: N3c1016DistBackendErrorE. what(): [PG ID 0 PG GUID 0(default_pg) Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=24054, OpType=_ALLGATHER_BASE, NumelIn=68125120, NumelOut=545000960, Timeout(ms)=600000) ran for 600026 milliseconds before timing out. [36m(FSDPPolicyWorkerBase pid=4060712)[0m Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:688 (most recent call first): [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x80 (0x7b38f837eeb0 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libc10.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional > >) + 0x247 (0x7b09cae40147 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #2: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x1591 (0x7b09cae43b61 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #3: c10d::ProcessGroupNCCL::Watchdog::run() + 0xd2 (0x7b09cae44ec2 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #4: + 0xd3b65 (0x7b39122f1b65 in /home/ray/anaconda3/bin/../lib/libstdc++.so.6) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #5: + 0x94ac3 (0x7b3914694ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #6: + 0x1268c0 (0x7b39147268c0 in /usr/lib/x86_64-linux-gnu/libc.so.6) [36m(FSDPPolicyWorkerBase pid=4060712)[0m [36m(FSDPPolicyWorkerBase pid=4060712)[0m Exception raised from run at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2074 (most recent call first): [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x80 (0x7b38f837eeb0 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libc10.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #1: + 0xe1c1a1 (0x7b09cae1c1a1 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #2: + 0x9468e6 (0x7b09ca9468e6 in /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #3: + 0xd3b65 (0x7b39122f1b65 in /home/ray/anaconda3/bin/../lib/libstdc++.so.6) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #4: + 0x94ac3 (0x7b3914694ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) [36m(FSDPPolicyWorkerBase pid=4060712)[0m frame #5: + 0x1268c0 (0x7b39147268c0 in /usr/lib/x86_64-linux-gnu/libc.so.6) [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/ray/_raylet.so(+0x15fd74a) [0x7b39139fd74a] ray::operator<<() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/ray/_raylet.so(+0x15fe1fc) [0x7b39139fe1fc] ray::RayLog::operator<< <>() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/ray/_raylet.so(+0x7f1389) [0x7b3912bf1389] ray::TerminateHandler() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xb64f2) [0x7b39122d44f2] __cxxabiv1::__terminate() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(_ZSt10unexpectedv+0) [0x7b39122ce2f3] std::unexpected() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xb64eb) [0x7b39122d44eb] __cxxabiv1::__terminate() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so(+0x94696a) [0x7b09ca94696a] c10d::ProcessGroupNCCL::Watchdog::run() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xd3b65) [0x7b39122f1b65] execute_native_thread_routine [36m(FSDPPolicyWorkerBase pid=4060712)[0m /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7b3914694ac3] [36m(FSDPPolicyWorkerBase pid=4060712)[0m /usr/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x7b39147268c0] [36m(FSDPPolicyWorkerBase pid=4060712)[0m [36m(FSDPPolicyWorkerBase pid=4060712)[0m [2025-12-10 04:37:49,057 E 4060712 4061972] logging.cc:125: Stack trace: [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/ray/_raylet.so(+0x15fd74a) [0x7b39139fd74a] ray::operator<<() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/ray/_raylet.so(+0x15fe1fc) [0x7b39139fe1fc] ray::RayLog::operator<< <>() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/ray/_raylet.so(+0x16007b8) [0x7b3913a007b8] ray::TerminateHandler() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xb64f2) [0x7b39122d44f2] __cxxabiv1::__terminate() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(_ZSt10unexpectedv+0) [0x7b39122ce2f3] std::unexpected() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xb64eb) [0x7b39122d44eb] __cxxabiv1::__terminate() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/.cache/uv/builds-v0/.tmp722kel/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so(+0x94696a) [0x7b09ca94696a] c10d::ProcessGroupNCCL::Watchdog::run() [36m(FSDPPolicyWorkerBase pid=4060712)[0m /home/ray/anaconda3/bin/../lib/libstdc++.so.6(+0xd3b65) [0x7b39122f1b65] execute_native_thread_routine [36m(FSDPPolicyWorkerBase pid=4060712)[0m /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7b3914694ac3] [36m(FSDPPolicyWorkerBase pid=4060712)[0m /usr/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x7b39147268c0] [36m(FSDPPolicyWorkerBase pid=4060712)[0m [36m(FSDPPolicyWorkerBase pid=4060712)[0m *** SIGABRT received at time=1765341469 on cpu 1 *** [36m(FSDPPolicyWorkerBase pid=4060712)[0m PC: @ 0x7b39146969fc (unknown) pthread_kill [36m(FSDPPolicyWorkerBase pid=4060712)[0m @ 0x7b3914642520 (unknown) (unknown) [36m(FSDPPolicyWorkerBase pid=4060712)[0m [2025-12-10 04:37:49,066 E 4060712 4061972] logging.cc:474: *** SIGABRT received at time=1765341469 on cpu 1 *** [36m(FSDPPolicyWorkerBase pid=4060712)[0m [2025-12-10 04:37:49,066 E 4060712 4061972] logging.cc:474: PC: @ 0x7b39146969fc (unknown) pthread_kill [36m(FSDPPolicyWorkerBase pid=4060712)[0m [2025-12-10 04:37:49,071 E 4060712 4061972] logging.cc:474: @ 0x7b3914642520 (unknown) (unknown) [36m(FSDPPolicyWorkerBase pid=4060712)[0m Fatal Python error: Aborted [36m(FSDPPolicyWorkerBase pid=4060712)[0m [36m(FSDPPolicyWorkerBase pid=4060712)[0m [36m(FSDPPolicyWorkerBase pid=4060712)[0m Extension modules: msgpack._cmsgpack, psutil._psutil_linux, google._upb._message, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, yaml._yaml, uvloop.loop, ray._raylet, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, pyarrow.lib, pyarrow._json, regex._regex, markupsafe._speedups, PIL._imaging, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, PIL._imagingft, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, _cffi_backend, pyarrow._parquet, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, xxhash._xxhash, pyarrow._acero, pyarrow._csv, pyarrow._substrait, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet [36m(FSDPPolicyWorkerBase pid=4060712)[0m , msgspec._core, _cbor2, setproctitle._setproctitle, zmq.backend.cython._zmq, pybase64._pybase64, cuda_utils, __triton_launcher (total: 161) [36m(skyrl_entrypoint pid=4054508)[0m [32m2025-12-10 04:37:51.025[0m | [1m[32mINFO [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m247[0m - [1m[32mFinished: 'fwd_logprobs_values_reward', time cost: 683.29s[0m

[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 02:42:26 [block_pool.py:378] Successfully reset prefix cache
[36m(VLLMInferenceEngine pid=4055486)[0m INFO 12-10 02:42:26 [block_pool.py:378] Successfully reset prefix cache
[36m(RayWorkerWrapper pid=4056609)[0m INFO 12-10 02:42:28 [cumem.py:228] CuMemAllocator: sleep freed 28.99 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 28.99 GiB is discarded directly.
[36m(RayWorkerWrapper pid=4056609)[0m INFO 12-10 02:42:31 [gpu_worker.py:117] Sleep mode freed 29.76 GiB memory, 6.32 GiB memory is still in use.
[36m(VLLMInferenceEngine pid=4055490)[0m INFO 12-10 02:42:27 [block_pool.py:378] Successfully reset prefix cache[32m [repeated 2x across cluster][0m
[36m(RayWorkerWrapper pid=4056462)[0m INFO 12-10 02:42:29 [cumem.py:228] CuMemAllocator: sleep freed 28.99 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 28.99 GiB is discarded directly.[32m [repeated 7x across cluster][0m
[36m(VLLMInferenceEngine pid=4055486)[0m INFO 12-10 02:42:33 [executor_base.py:189] It took 6.969628 seconds to fall asleep.
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 02:57:37 [executor_base.py:205] It took 0.562300 seconds to wake up tags ['weights'].
[36m(RayWorkerWrapper pid=4056188)[0m INFO 12-10 02:42:35 [gpu_worker.py:117] Sleep mode freed 29.76 GiB memory, 6.59 GiB memory is still in use.[32m [repeated 7x across cluster][0m
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 02:42:35 [executor_base.py:189] It took 8.172811 seconds to fall asleep.[32m [repeated 3x across cluster][0m
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 02:59:29 [block_pool.py:378] Successfully reset prefix cache
[36m(VLLMInferenceEngine pid=4055490)[0m INFO 12-10 02:57:37 [executor_base.py:205] It took 0.544679 seconds to wake up tags ['weights'].[32m [repeated 3x across cluster][0m
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 02:59:35 [executor_base.py:205] It took 0.611780 seconds to wake up tags ['kv_cache'].
[36m(VLLMInferenceEngine pid=4055490)[0m INFO 12-10 02:59:29 [block_pool.py:378] Successfully reset prefix cache[32m [repeated 3x across cluster][0m
[36m(VLLMInferenceEngine pid=4055490)[0m INFO 12-10 02:59:35 [executor_base.py:205] It took 0.581273 seconds to wake up tags ['kv_cache'].[32m [repeated 3x across cluster][0m
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 03:35:18 [block_pool.py:378] Successfully reset prefix cache
[36m(VLLMInferenceEngine pid=4055487)[0m INFO 12-10 03:35:18 [block_pool.py:378] Successfully reset prefix cache
[36m(RayWorkerWrapper pid=4056185)[0m INFO 12-10 03:35:20 [cumem.py:228] CuMemAllocator: sleep freed 28.99 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 28.99 GiB is discarded directly.
[36m(RayWorkerWrapper pid=4056185)[0m INFO 12-10 03:35:22 [gpu_worker.py:117] Sleep mode freed 29.76 GiB memory, 7.58 GiB memory is still in use.
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 03:35:24 [executor_base.py:189] It took 5.459290 seconds to fall asleep.
[36m(VLLMInferenceEngine pid=4055490)[0m INFO 12-10 03:35:18 [block_pool.py:378] Successfully reset prefix cache[32m [repeated 2x across cluster][0m
[36m(RayWorkerWrapper pid=4056609)[0m INFO 12-10 03:35:20 [cumem.py:228] CuMemAllocator: sleep freed 28.99 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 28.99 GiB is discarded directly.[32m [repeated 7x across cluster][0m
[36m(VLLMInferenceEngine pid=4055486)[0m INFO 12-10 03:49:44 [executor_base.py:205] It took 1.019664 seconds to wake up tags ['weights'].
[36m(RayWorkerWrapper pid=4056609)[0m INFO 12-10 03:35:27 [gpu_worker.py:117] Sleep mode freed 29.76 GiB memory, 6.32 GiB memory is still in use.[32m [repeated 7x across cluster][0m
[36m(VLLMInferenceEngine pid=4055486)[0m INFO 12-10 03:35:27 [executor_base.py:189] It took 8.209153 seconds to fall asleep.[32m [repeated 3x across cluster][0m
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 03:51:35 [block_pool.py:378] Successfully reset prefix cache
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 03:49:44 [executor_base.py:205] It took 1.364720 seconds to wake up tags ['weights'].[32m [repeated 3x across cluster][0m
[36m(VLLMInferenceEngine pid=4055490)[0m INFO 12-10 03:51:49 [executor_base.py:205] It took 0.461115 seconds to wake up tags ['kv_cache'].
[36m(VLLMInferenceEngine pid=4055490)[0m INFO 12-10 03:51:35 [block_pool.py:378] Successfully reset prefix cache[32m [repeated 3x across cluster][0m
[36m(VLLMInferenceEngine pid=4055486)[0m INFO 12-10 03:51:49 [executor_base.py:205] It took 0.496316 seconds to wake up tags ['kv_cache'].[32m [repeated 3x across cluster][0m
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 04:26:06 [block_pool.py:378] Successfully reset prefix cache
[36m(VLLMInferenceEngine pid=4055487)[0m INFO 12-10 04:26:06 [block_pool.py:378] Successfully reset prefix cache
[36m(RayWorkerWrapper pid=4056188)[0m INFO 12-10 04:26:07 [cumem.py:228] CuMemAllocator: sleep freed 28.99 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 28.99 GiB is discarded directly.
[36m(RayWorkerWrapper pid=4056188)[0m INFO 12-10 04:26:18 [gpu_worker.py:117] Sleep mode freed 29.76 GiB memory, 6.60 GiB memory is still in use.
[36m(VLLMInferenceEngine pid=4055490)[0m INFO 12-10 04:26:06 [block_pool.py:378] Successfully reset prefix cache[32m [repeated 2x across cluster][0m
[36m(RayWorkerWrapper pid=4056161)[0m INFO 12-10 04:26:09 [cumem.py:228] CuMemAllocator: sleep freed 28.99 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 28.99 GiB is discarded directly.[32m [repeated 7x across cluster][0m
[36m(VLLMInferenceEngine pid=4055487)[0m INFO 12-10 04:26:21 [executor_base.py:189] It took 14.837964 seconds to fall asleep.
Error executing job with overrides: ['trainer.algorithm.advantage_estimator=grpo', "data.train_data=['/workspace/data/sql/train.parquet']", "data.val_data=['/workspace/data/sql/validation.parquet']", 'trainer.policy.model.path=Qwen/Qwen2.5-Coder-7B-Instruct', 'trainer.epochs=30', 'trainer.placement.colocate_all=true', 'trainer.strategy=fsdp2', 'trainer.policy.fsdp_config.cpu_offload=false', 'trainer.ref.fsdp_config.cpu_offload=true', 'trainer.policy.optimizer_config.max_grad_norm=0.5', 'trainer.policy.sequence_parallel_size=1', 'trainer.placement.policy_num_gpus_per_node=8', 'trainer.placement.ref_num_gpus_per_node=8', 'generator.num_inference_engines=4', 'generator.inference_engine_tensor_parallel_size=2', 'trainer.train_batch_size=64', 'trainer.micro_forward_batch_size_per_gpu=4', 'trainer.micro_train_batch_size_per_gpu=1', 'trainer.max_prompt_length=6000', 'generator.max_input_length=29000', 'generator.sampling_params.max_generate_length=3000', 'trainer.policy.optimizer_config.lr=1.0e-6', 'trainer.policy_mini_batch_size=64', 'trainer.algorithm.use_kl_loss=false', 'trainer.ckpt_interval=60', 'trainer.hf_save_interval=30', 'trainer.dump_data_batch=true', 'generator.backend=vllm', 'generator.run_engines_locally=true', 'generator.weight_sync_backend=nccl', 'generator.async_engine=false', 'generator.batched=true', 'environment.env_class=text2sql', 'generator.use_conversation_multi_turn=false', 'generator.n_samples_per_prompt=5', 'generator.gpu_memory_utilization=0.4', 'generator.max_turns=6', 'generator.sampling_params.temperature=0.6', 'generator.sampling_params.top_p=0.95', 'generator.sampling_params.stop=["</sql>", "</solution>"]', 'generator.eval_sampling_params.stop=["</sql>", "</solution>"]', 'environment.skyrl_gym.text2sql.db_path=/workspace/data/db/data', 'trainer.logger=wandb', 'trainer.project_name=skyrlsql', 'trainer.run_name=med_config', 'trainer.resume_mode=latest', 'trainer.ckpt_path=/workspace/data/ckpts/skyrl_sql_7B_ckpt_medium', 'trainer.eval_batch_size=1024', 'trainer.eval_before_train=true', 'trainer.eval_interval=5']
[36m(skyrl_entrypoint pid=4054508)[0m [32m2025-12-10 04:37:51.041[0m | [1m[32mINFO    [0m | [36mskyrl_train.trainer[0m:[36mtrain[0m:[36m194[0m - [1m[32mFinished: 'step', time cost: 2761.16s[0m
Traceback (most recent call last):
File "/home/ray/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 331, in main
    ray.get(skyrl_entrypoint.remote(cfg))
File "/home/ray/.cache/uv/builds-v0/.tmpUJkO0S/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^
File "/home/ray/.cache/uv/builds-v0/.tmpUJkO0S/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
File "/home/ray/.cache/uv/builds-v0/.tmpUJkO0S/lib/python3.12/site-packages/ray/_private/worker.py", line 2961, in get
    values, debugger_breakpoint = worker.get_objects(
                                ^^^^^^^^^^^^^^^^^^^
File "/home/ray/.cache/uv/builds-v0/.tmpUJkO0S/lib/python3.12/site-packages/ray/_private/worker.py", line 1026, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ActorUnavailableError): [36mray::skyrl_entrypoint()[39m (pid=4054508, ip=172.26.133.87)
File "/home/ray/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 322, in skyrl_entrypoint
    exp.run()
File "/home/ray/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 315, in run
    trainer.train()
File "/tmp/ray/session_2025-12-10_00-17-08_935065_4040124/runtime_resources/working_dir_files/_ray_pkg_bbd4256ed22e8211/skyrl_train/trainer.py", line 248, in train
    training_input = self.fwd_logprobs_values_reward(training_input)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ray/.cache/uv/builds-v0/.tmpzpGau9/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
File "/tmp/ray/session_2025-12-10_00-17-08_935065_4040124/runtime_resources/working_dir_files/_ray_pkg_bbd4256ed22e8211/skyrl_train/trainer.py", line 910, in fwd_logprobs_values_reward
    all_rank_action_log_probs: List[TrainingOutputBatch] = ray.get(action_log_probs_refs)
                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ^^^^^^^^^^^^^^^^^^^
        ^^^^^^^^^^^^^^^^^^^^^
                                ^^^^^^^^^^^^^^^^^^^
ray.exceptions.ActorUnavailableError: The actor 5b9e3a3499db74f600d62dd402000000 is unavailable: The actor is temporarily unavailable: RpcError: RPC Error message: Socket closed; RPC Error details:  rpc_code: 14. The task may or may not have been executed on the actor.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
[36m(RayWorkerWrapper pid=4056185)[0m INFO 12-10 04:26:22 [gpu_worker.py:117] Sleep mode freed 29.76 GiB memory, 7.58 GiB memory is still in use.[32m [repeated 7x across cluster][0m
[36m(VLLMInferenceEngine pid=4055488)[0m INFO 12-10 04:26:22 [executor_base.py:189] It took 15.704998 seconds to fall asleep.[32m [repeated 3x across cluster][0m
```

Observations

• I noticed that reward unusually low (0 for all 4 steps in both runs) compared to your run. Seems like the model is not generating blocks. In your run, the reward distribution looks much healthier. This is likely unrelated to the FD/timeout issue? But it felt suspicious that all steps have reward 0/–1.

This is from med config where n_samples_per_prompt=5:

(skyrl_entrypoint pid=4054508) 2025-12-10 01:03:59.222 | INFO     | skyrl_train.utils.logging_utils:log_example:72 - Example:
(skyrl_entrypoint pid=4054508)   Input: [{'content': "\nTask Overview:\nYou are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question within limited turns. You should breakdown the problem, draft your reasoning process, and generate the solution.\n\nDatabase Engine:\nSQLite\n\nDatabase Schema:\n{db_details}\nThis schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints.\n\nExternal Knowledge:\n{external_knowledge}\n\nQuestion:\n{question}\n\nInstructions:\n- Make sure you only output the information that is asked in the question. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more.\n- The generated query should return all of the information asked in the question without any missing or extra information.\n- Before generating the final SQL query, please think through the steps of how to write the query. It should include detailed considerations such as analisying questions, \nsummarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, thinking of how to call SQL tools, and revisiting previous steps.\n\n\nFormat:\n- Conduct thinking inside <think>...</think> blocks every time you get new observation or information. \n- You can use SQL tool written within a single <sql>your sql</sql> block to explore or verify. SQL tool output will be shown as dataframe inside <observation>...</observation>. Based on this observation, you can think again and refine.\n- The returned dataframe will be truncated in 50 rows if observation is too long. \n- If you find no further exploration is needed or reaches max turns, you MUST directly provide the final SQL query solution inside <solution>...</solution>. \n\n------------------------ START OF EXAMPLE ------------------------\nQuestion: how many pigs are in the farm? \n    Database Schema:\n    Table: animals\n    - id (INTEGER, PRIMARY KEY)\n    - species (TEXT)\n    - age (INTEGER)\n    - name (TEXT)\n\n<think>I am querying how many pigs are in the farm. I will begin by checking if the 'animals' table exists and contains entries with species = 'pig'.</think>\n<sql>SELECT COUNT(*) FROM animals WHERE species = 'pig';</sql>\n<observation>\n+----------+\n| COUNT(*) |\n+----------+\n|   12     |\n+----------+\n</observation>\n<think>The result indicates that there are 12 pigs in the farm. Since the question asks for how many pigs, I can now output the final SQL as the solution.</think>\n<solution>SELECT COUNT(*) FROM animals WHERE species = 'pig';</solution>\n------------------------ END OF EXAMPLE ------------------------\n", 'role': 'system'}, {'content': '{db_details}:CREATE TABLE singer (\n    Singer_ID number, -- example: [1, 2]\n    Name text, -- example: [\'Liliane Bettencourt\', \'Christy Walton\']\n    Birth_Year number, -- example: [1944.0, 1948.0]\n    Net_Worth_Millions number, -- example: [30.0, 28.8]\n    Citizenship text, -- example: [\'France\', \'United States\']\n    PRIMARY KEY (Singer_ID)\n);\n\nCREATE TABLE song (\n    Song_ID number, -- example: [1, 2]\n    Title text, -- example: ["Do They Know It\'s Christmas", "F**k It (I Don\'t Want You Back)"]\n    Singer_ID number, -- example: [1, 2]\n    Sales number, -- example: [1094000.0, 552407.0]\n    Highest_Position number, -- example: [1.0, 3.0]\n    PRIMARY KEY (Song_ID),\n    CONSTRAINT fk_song_singer_id FOREIGN KEY (Singer_ID) REFERENCES singer (Singer_ID)\n);;\n {external_knowledge}: None;\n {question}: Show the names of singers and the total sales of their songs.', 'role': 'user'}]
(skyrl_entrypoint pid=4054508)   Output (Reward: 0.0000):
(skyrl_entrypoint pid=4054508) <think>I need to retrieve the names of singers and the total sales of their songs. This requires joining the 'singer' and 'song' tables based on the 'Singer_ID' column and then grouping the results by the singer's name to calculate the total sales.</think>
(skyrl_entrypoint pid=4054508) <sql>SELECT s.Name, SUM(so.Sales) AS Total_Sales FROM singer s JOIN song so ON s.Singer_ID = so.Singer_ID GROUP BY s.Name;</sql>

• Avg_num_tokens seem longer for my runs compared to yours
• Generation is slower for my runs but is likely impacted by the settings I changed for debugging
• GPU utilization and mem allocated does not seem too out of the ordinary

Questions

• Does SkyRL-SQL assume a higher ulimit -n than 1024 for raylet / Ray workers on this scale?
• Is there a recommended target (e.g. 4096 / 8192) and any Ray-specific knobs I should tweak to reduce FD usage with vLLM + multiple inference engines?

[erictang000]

hey @shanghongsim, thanks for all the details and debugging, that's super helpful!

async_engine=false
this flag just sets the generator to use the async llm engine for vllm: https://docs.vllm.ai/en/stable/design/arch_overview/?h=asyncllm#asyncllmengine. Async RL is configured separately.

For this task you should generally keep async_engine=true and batched=false, since we want to use the agent_loop code path for multi-turn RL training here:

SkyRL/skyrl-train/skyrl_train/generators/skyrl_gym_generator.py

Line 118 in 3c51e9a

async def agent_loop(

The slow generate is perhaps to be expected because of the following I set for debugging:

CUDA_LAUNCH_BLOCKING=1
Very conservative generator.gpu_memory_utilization=0.4
Not sure if async_engine=false affects things

I think CUDA_LAUNCH_BLOCKING=1 will slow things down significantly, and a very conservative generator.gpu_memory_utilization=0.4 will definitely too, especially for longer context.

Does SkyRL-SQL assume a higher ulimit -n than 1024 for raylet / Ray workers on this scale?
yes, I think making sure this is higher would be good, especially since we are concurrently reading from the SQL databases during generation!

I believe the hardware we ran the experiments on shows 1048576 for ulimit -n - I would try making sure is at least 8192. I am not aware of any file operations that would be happening with vLLM, I would assume open file descriptors would just be from the sqlite database writes:

SkyRL/skyrl-gym/skyrl_gym/tools/sql.py

Line 9 in 3c51e9a

class SQLCodeExecutorToolGroup(ToolGroup):

I noticed that reward unusually low (0 for all 4 steps in both runs) compared to your run. Seems like the model is not generating blocks. In your run, the reward distribution looks much healthier. This is likely unrelated to the FD/timeout issue? But it felt suspicious that all steps have reward 0/–1.

hmm yeah so if max_turns is > 1, it's expected to have initially negative reward (due to not returning an answer for example), but I would make sure initial reward looks similar to the wandb curve I sent before starting any longer training runs.

if you turn on dump_data_batch=true, you should be able to inspect the data batch (it will be dumped as a pkl file that you can load by importing skyrl_train.training_batch.TrainingInputBatch and calling .load()

let me know if increasing the ulimit resolves the FD issue and if you're able to find anything from debugging the trajectories!

[shanghongsim]

Hi @erictang000, thank you for the detailed guidance, it has been incredibly helpful! I followed your advice:

• removed CUDA_LAUNCH_BLOCKING
• set async_engine=true and batched=false
• increased gpu_memory_utilization to 0.7
• increased ulimit -n

Full config:

``` set -x

# bash examples/text_to_sql/small_config_v4.sh 2>&1 | tee examples/text_to_sql/small_config_v4.log

# ===============================
# Paths
# ===============================
DATA_DIR="/workspace/data/sql"
DB_PATH="/workspace/data/db/data"
CKPT_PATH="/workspace/data/ckpts/skyrl_sql_7B_ckpt"

# ===============================
# Hardware & Training Settings
# ===============================
NUM_GPUS=8
NUM_INFERENCE_ENGINES=4   # was 2
TP_SIZE=2                  # was 4

MAX_PROMPT_LENGTH=3000
MAX_GENERATE_LENGTH=1000   # was 3000
TRAIN_BATCH_SIZE=64        # was 256

# ===============================
# Debug Environment Variables
# ===============================
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,GRAPH,ENV
export TORCH_DISTRIBUTED_DEBUG=DETAIL
export TORCH_NCCL_TRACE_BUFFER_SIZE=1048576
export TORCH_SHOW_CPP_STACKTRACES=1
# export CUDA_LAUNCH_BLOCKING=1
export NCCL_TIMEOUT=1800

# ===============================
# SKYRL Training Command
# ===============================
uv run --isolated --extra vllm -m skyrl_train.entrypoints.main_base \
  trainer.algorithm.advantage_estimator="grpo" \
  data.train_data="['$DATA_DIR/train.parquet']" \
  data.val_data="['$DATA_DIR/validation.parquet']" \
  trainer.policy.model.path="Qwen/Qwen2.5-Coder-7B-Instruct" \
  trainer.epochs=2 \
  trainer.placement.colocate_all=true \
  trainer.strategy=fsdp2 \
  trainer.policy.fsdp_config.cpu_offload=false \
  trainer.ref.fsdp_config.cpu_offload=true \
  trainer.policy.optimizer_config.max_grad_norm=0.5 \
  trainer.policy.sequence_parallel_size=1 \
  trainer.placement.policy_num_gpus_per_node=$NUM_GPUS \
  trainer.placement.ref_num_gpus_per_node=$NUM_GPUS \
  generator.num_inference_engines=$NUM_INFERENCE_ENGINES \
  generator.inference_engine_tensor_parallel_size=$TP_SIZE \
  trainer.train_batch_size=$TRAIN_BATCH_SIZE \
  trainer.micro_forward_batch_size_per_gpu=4 \
  trainer.micro_train_batch_size_per_gpu=1 \
  trainer.max_prompt_length=6000 \
  generator.max_input_length=29000 \
  generator.sampling_params.max_generate_length=$MAX_GENERATE_LENGTH \
  trainer.policy.optimizer_config.lr=1.0e-6 \
  trainer.policy_mini_batch_size=64 \
  trainer.algorithm.use_kl_loss=false \
  trainer.ckpt_interval=60 \
  trainer.hf_save_interval=30 \
  trainer.dump_data_batch=true \
  generator.backend=vllm \
  generator.run_engines_locally=true \
  generator.weight_sync_backend=nccl \
  generator.async_engine=true \
  generator.batched=false \
  environment.env_class=text2sql \
  generator.use_conversation_multi_turn=false \
  generator.n_samples_per_prompt=5 \
  generator.gpu_memory_utilization=0.7 \
  generator.max_turns=6 \
  generator.sampling_params.temperature=0.6 \
  generator.sampling_params.top_p=0.95 \
  generator.sampling_params.stop='["</sql>", "</solution>"]' \
  generator.eval_sampling_params.stop='["</sql>", "</solution>"]' \
  environment.skyrl_gym.text2sql.db_path=$DB_PATH \
  trainer.logger="wandb" \
  trainer.export_path="/workspace/data/exports/small_config_v4" \
  trainer.project_name="skyrlsql" \
  trainer.run_name="small_config_v4" \
  trainer.resume_mode=latest \
  trainer.ckpt_path=$CKPT_PATH \
  trainer.eval_batch_size=1024 \
  trainer.eval_before_train=true \
  trainer.eval_interval=5 \
  "$@"
```

This resolves the FD issue.

However, generation is still very slow, 10x slower than what your W&B report shows. Here is the W&B report from my run:

https://wandb.ai/shanghongsim/skyrlsql/reports/Small-config-v4--VmlldzoxNTM0NTk5Nw?accessToken=o8u5vkkg8o5ldikrx2c7mkqlqjbvp37vacjqsj80m7lbxoer0ohmz98zw9hgnl8w

Any ideas on what could be causing this and whats the best way to start debugging this?

[erictang000]

@shanghongsim One thing that stands out is that everything in your current setup is slower than expected, not just generation - for example, weight syncing for a 7B model is taking ~200 seconds. I think your inter-gpu network bandwidth could be an issue here, could you try running an all reduce benchmark: https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/all_reduce_bench.py

and seeing if numbers look like what should be expected for your hardware setup?

[erictang000]

hey @shanghongsim, was this resolved? One update is that I recently merged this PR, which should reduce disk memory pressure (that may have been slowing down your generation phase or contributing to the FD issue) #814