Saving huggingface checkpoints times out after a certain number of epochs

[Harper-Hua]

Hi there,

I've been trying to run multi-turn GRPO. The training set is in size of 1027 data points and val set is 4715. Saving checkpoints and hf models was working fine until step 72, and at step 72 checkpoints took over 15 mins to save, but when it was trying to save a huggingface model at step 73 it failed with the following error message:

�[36m(skyrl_entrypoint pid=3034109)�[0m 2025-09-08 22:36:45.019 | INFO | skyrl_train.trainer:train:359 - Finished: 'save_hf_model', time cost: 601.62s
Traceback (most recent call last):
File "/home/ubuntu/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 288, in main
ray.get(skyrl_entrypoint.remote(cfg))
File "/home/ubuntu/.cache/uv/builds-v0/.tmpu9oQzg/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/uv/builds-v0/.tmpu9oQzg/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/uv/builds-v0/.tmpu9oQzg/lib/python3.12/site-packages/ray/_private/worker.py", line 2858, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/.cache/uv/builds-v0/.tmpu9oQzg/lib/python3.12/site-packages/ray/_private/worker.py", line 958, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ActorDiedError): �[36mray::skyrl_entrypoint()�[39m (pid=3034109, ip=172.31.18.65)
File "/home/ubuntu/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 279, in skyrl_entrypoint
exp.run()
File "/home/ubuntu/SkyRL/skyrl-train/skyrl_train/entrypoints/main_base.py", line 272, in run
trainer.train()
File "/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/runtime_resources/working_dir_files/_ray_pkg_18ecd9287266148b/skyrl_train/trainer.py", line 360, in train
self.save_models()
File "/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/runtime_resources/working_dir_files/_ray_pkg_18ecd9287266148b/skyrl_train/trainer.py", line 1376, in save_models
ray.get(
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
class_name: FSDPPolicyRayActorBase
actor_id: d23854da87fe7662fa512c1b01000000
pid: 3041415
namespace: 2b8a18f5-2d6c-40b4-b2b6-6ee8d3bf69df
ip: ...
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.

Here is the logged curve of timing/save_checkpoints

I feel like this error is related to the size of my training set since I tried to run the exact same config on a ~2000 data point training set and it couldn't even save the hf model on the first saving interval (step 18). And i've been consistently encountering this error.

NUM_GPUS=8
NUM_INFERENCE_ENGINES=2
TP_SIZE=4
MAX_INPUT_LENGTH=29000
MAX_GENERATE_LENGTH=3000
TRAIN_BATCH_SIZE=128

uv run --isolated --extra vllm -m skyrl_train.entrypoints.main_base
trainer.algorithm.advantage_estimator="grpo"
data.train_data="['/home/ubuntu/s3_in/skyrl/rl_training_set_1027.parquet']"
data.val_data="['/home/ubuntu/s3_in/skyrl/rl_validation_set_4715.parquet']"
trainer.policy.model.path="customized_qwen2.5-code-7B-instruct"
trainer.epochs=20
trainer.placement.colocate_all=true
trainer.strategy=fsdp2
trainer.policy.fsdp_config.cpu_offload=false
trainer.ref.fsdp_config.cpu_offload=true
trainer.policy.optimizer_config.max_grad_norm=0.5
trainer.policy.sequence_parallel_size=1
trainer.placement.policy_num_gpus_per_node=$NUM_GPUS
trainer.placement.ref_num_gpus_per_node=$NUM_GPUS
generator.num_inference_engines=$NUM_INFERENCE_ENGINES
generator.inference_engine_tensor_parallel_size=$TP_SIZE
trainer.train_batch_size=$TRAIN_BATCH_SIZE
trainer.micro_forward_batch_size_per_gpu=8
trainer.micro_train_batch_size_per_gpu=1
trainer.max_prompt_length=6000
generator.max_input_length=$MAX_INPUT_LENGTH
generator.sampling_params.max_generate_length=$MAX_GENERATE_LENGTH
trainer.policy.optimizer_config.lr=1.0e-6
trainer.policy_mini_batch_size=128
trainer.algorithm.use_kl_loss=false
trainer.ckpt_interval=18
trainer.max_ckpts_to_keep=1
trainer.hf_save_interval=19
trainer.dump_data_batch=false
generator.backend=vllm
generator.run_engines_locally=true
generator.weight_sync_backend=nccl
generator.async_engine=true
generator.batched=false
environment.env_class=text2sql
generator.use_conversation_multi_turn=true
generator.n_samples_per_prompt=6
generator.gpu_memory_utilization=0.7
generator.max_turns=10
generator.sampling_params.temperature=0.5
generator.sampling_params.top_p=0.95
generator.sampling_params.stop='["", ""]'
generator.append_eos_token_after_stop_str_in_multi_turn=true
generator.eval_sampling_params.stop='["", ""]'
generator.eval_n_samples_per_prompt=6
trainer.project_name="skyrlsql_p5"
trainer.run_name=$run_name
trainer.resume_mode=latest
trainer.ckpt_path=$HOME/ckpts/${run_name}
trainer.export_path=$HOME/skyrl_export/${run_name}/
trainer.eval_batch_size=1024
trainer.eval_before_train=false
trainer.eval_interval=18

[SumanthRH]

HI @Harper-Hua , thanks for reporting the issue.

The stack trace you provided seems a bit incomplete. This seems like the stack trace on the entrypoint task. There should be a stack trace of the exact failure in save_hf_model method in the FSDPPolicyWorker Ray actor. Do you have logs for that as well? If it is somehow missing on the driver/ your log file, you can also retrieve it by navigating to the actor on the Ray Dashboard. https://docs.ray.io/en/latest/ray-observability/getting-started.html#actors-view

[Harper-Hua]

�[36m(FSDPPolicyRayActorBase pid=3039731)�[0m [rank0]:[E908 22:36:44.137300758 ProcessGroupNCCL.cpp:1685] [PG ID 0 PG GUID 0(default_pg) Rank 0] Observed flight recorder dump signal from another rank via TCPStore.
�[36m(FSDPPolicyRayActorBase pid=3039731)�[0m [rank0]:[E908 22:36:44.137612953 ProcessGroupNCCL.cpp:1746] [PG ID 0 PG GUID 0(default_pg) Rank 0] Received a dump signal due to a collective timeout from rank 1 and we will try our best to dump the debug info. Last enqueued NCCL work: 818134, last completed NCCL work: 818134.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
�[36m(FSDPPolicyRayActorBase pid=3039731)�[0m [rank0]:[E908 22:36:44.137866214 ProcessGroupNCCL.cpp:1536] [PG ID 0 PG GUID 0(default_pg) Rank 0] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
�[36m(FSDPPolicyRayActorBase pid=3041415)�[0m [rank1]:[E908 22:36:44.115569067 ProcessGroupNCCL.cpp:632] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=818135, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600037 milliseconds before timing out.
�[36m(FSDPPolicyRayActorBase pid=3041415)�[0m [rank1]:[E908 22:36:44.115652538 ProcessGroupNCCL.cpp:2271] [PG ID 0 PG GUID 0(default_pg) Rank 1] failure detected by watchdog at work sequence id: 818135 PG status: last enqueued work: 818135, last completed work: 818134
�[36m(FSDPPolicyRayActorBase pid=3041415)�[0m [rank1]:[E908 22:36:44.115659307 ProcessGroupNCCL.cpp:670] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
�[36m(FSDPPolicyRayActorBase pid=3041415)�[0m [rank1]:[E908 22:36:44.115694797 ProcessGroupNCCL.cpp:2106] [PG ID 0 PG GUID 0(default_pg) Rank 1] First PG on this rank to signal dumping.
�[36m(FSDPPolicyRayActorBase pid=3041415)�[0m [rank1]:[E908 22:36:44.137281531 ProcessGroupNCCL.cpp:1746] [PG ID 0 PG GUID 0(default_pg) Rank 1] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 818135, last completed NCCL work: 818134.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m [rank2]:[E908 22:36:44.198153185 ProcessGroupNCCL.cpp:684] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m [rank2]:[E908 22:36:44.198155923 ProcessGroupNCCL.cpp:698] [Rank 2] To avoid data inconsistency, we are taking the entire process down.
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m [rank2]:[E908 22:36:44.198980145 ProcessGroupNCCL.cpp:1899] [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=818135, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600095 milliseconds before timing out.
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:635 (most recent call first):
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x78d3d43785e8 in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libc10.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x23d (0x78a4088f5b4d in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0xc80 (0x78a4088f78d0 in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x78a4088f8fdd in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #4: + 0xdc253 (0x78d3decdc253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #5: + 0x94ac3 (0x78d3e1694ac3 in /lib/x86_64-linux-gnu/libc.so.6)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #6: + 0x126850 (0x78d3e1726850 in /lib/x86_64-linux-gnu/libc.so.6)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m [2025-09-08 22:36:44,282 E 3041414 3042313] logging.cc:117: Unhandled exception: N3c1016DistBackendErrorE. what(): [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=818135, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600095 milliseconds before timing out.
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:635 (most recent call first):
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x78d3d43785e8 in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libc10.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x23d (0x78a4088f5b4d in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0xc80 (0x78a4088f78d0 in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x78a4088f8fdd in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #4: + 0xdc253 (0x78d3decdc253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #5: + 0x94ac3 (0x78d3e1694ac3 in /lib/x86_64-linux-gnu/libc.so.6)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #6: + 0x126850 (0x78d3e1726850 in /lib/x86_64-linux-gnu/libc.so.6)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1905 (most recent call first):
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x78d3d43785e8 in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libc10.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #1: + 0xcc7b9e (0x78a4088c7b9e in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #2: + 0x9165ed (0x78a4085165ed in /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #3: + 0xdc253 (0x78d3decdc253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #4: + 0x94ac3 (0x78d3e1694ac3 in /lib/x86_64-linux-gnu/libc.so.6)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m frame #5: + 0x126850 (0x78d3e1726850 in /lib/x86_64-linux-gnu/libc.so.6)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m [2025-09-08 22:36:44,291 E 3041414 3042313] logging.cc:124: Stack trace:
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/ray/_raylet.so(+0x151a6ea) [0x78d3e051a6ea] ray::operator<<()
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/ray/_raylet.so(+0x151dcd2) [0x78d3e051dcd2] ray::TerminateHandler()
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae20c) [0x78d3decae20c]
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae277) [0x78d3decae277]
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae1fe) [0x78d3decae1fe]
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m /home/ubuntu/.cache/uv/builds-v0/.tmpt99PwV/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so(+0x91667b) [0x78a40851667b] c10d::ProcessGroupNCCL::ncclCommWatchdog()
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x78d3decdc253]
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x78d3e1694ac3]
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x78d3e1726850]
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m *** SIGABRT received at time=1757371004 on cpu 120 ***
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m PC: @ 0x78d3e16969fc (unknown) pthread_kill
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m @ 0x78d3e1642520 (unknown) (unknown)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m [2025-09-08 22:36:44,291 E 3041414 3042313] logging.cc:501: *** SIGABRT received at time=1757371004 on cpu 120 ***
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m [2025-09-08 22:36:44,291 E 3041414 3042313] logging.cc:501: PC: @ 0x78d3e16969fc (unknown) pthread_kill
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m [2025-09-08 22:36:44,291 E 3041414 3042313] logging.cc:501: @ 0x78d3e1642520 (unknown) (unknown)
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m Fatal Python error: Aborted
�[36m(FSDPPolicyRayActorBase pid=3041414)�[0m

/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/python-core-worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103_3034109.log:41204:[2025-09-08 22:36:45,017 I 3034109 3034149] task_manager.cc:1172: Task failed: RpcError: RPC Error message: Socket closed; RPC Error details: rpc_code: 14: Type=ACTOR_TASK, Language=PYTHON, Resources: {}, function_descriptor={type=PythonFunctionDescriptor, module_name=skyrl_train.workers.fsdp.fsdp_worker, class_name=FSDPPolicyRayActorBase, function_name=save_hf_model, function_hash=}, task_id=8ba81f1d087cfcc4d23854da87fe7662fa512c1b01000000, task_name=FSDPPolicyRayActorBase.save_hf_model, job_id=01000000, num_args=4, num_returns=1, max_retries=0, depth=2, attempt_number=0, actor_task_spec={actor_id=d23854da87fe7662fa512c1b01000000, actor_caller_id=1e8ff6d236132784ffffffffffffffffffffffff01000000, seq_no=631, retry_exceptions=0}, runtime_env_hash=1596937532, eager_install=1, setup_timeout_seconds=600
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/python-core-worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103_3034109.log:41216:[2025-09-08 22:36:45,084 I 3034109 3034149] task_manager.cc:1172: Task failed: RpcError: RPC Error message: Socket closed; RPC Error details: rpc_code: 14: Type=ACTOR_TASK, Language=PYTHON, Resources: {}, function_descriptor={type=PythonFunctionDescriptor, module_name=skyrl_train.workers.fsdp.fsdp_worker, class_name=FSDPPolicyRayActorBase, function_name=save_hf_model, function_hash=}, task_id=863b70ca438bb0266ea2d4e31528e026ed12ff9401000000, task_name=FSDPPolicyRayActorBase.save_hf_model, job_id=01000000, num_args=4, num_returns=1, max_retries=0, depth=2, attempt_number=0, actor_task_spec={actor_id=6ea2d4e31528e026ed12ff9401000000, actor_caller_id=1e8ff6d236132784ffffffffffffffffffffffff01000000, seq_no=631, retry_exceptions=0}, runtime_env_hash=1596937532, eager_install=1, setup_timeout_seconds=600
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/python-core-worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103_3034109.log:41228:[2025-09-08 22:36:45,172 I 3034109 3034149] task_manager.cc:1172: Task failed: RpcError: RPC Error message: Socket closed; RPC Error details: rpc_code: 14: Type=ACTOR_TASK, Language=PYTHON, Resources: {}, function_descriptor={type=PythonFunctionDescriptor, module_name=skyrl_train.workers.fsdp.fsdp_worker, class_name=FSDPPolicyRayActorBase, function_name=save_hf_model, function_hash=}, task_id=cb84b7aab6d44a47cde77241d6df3f3849b6393301000000, task_name=FSDPPolicyRayActorBase.save_hf_model, job_id=01000000, num_args=4, num_returns=1, max_retries=0, depth=2, attempt_number=0, actor_task_spec={actor_id=cde77241d6df3f3849b6393301000000, actor_caller_id=1e8ff6d236132784ffffffffffffffffffffffff01000000, seq_no=631, retry_exceptions=0}, runtime_env_hash=1596937532, eager_install=1, setup_timeout_seconds=600
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/python-core-worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103_3034109.log:41255:[2025-09-08 22:36:45,619 I 3034109 3034149] task_manager.cc:1172: Task failed: IOError: The actor was restarted: Type=ACTOR_TASK, Language=PYTHON, Resources: {}, function_descriptor={type=PythonFunctionDescriptor, module_name=skyrl_train.workers.fsdp.fsdp_worker, class_name=FSDPPolicyRayActorBase, function_name=save_hf_model, function_hash=}, task_id=2791fcb03e87c5e17bb5cab077f637cfba9ff2c601000000, task_name=FSDPPolicyRayActorBase.save_hf_model, job_id=01000000, num_args=4, num_returns=1, max_retries=0, depth=2, attempt_number=0, actor_task_spec={actor_id=7bb5cab077f637cfba9ff2c601000000, actor_caller_id=1e8ff6d236132784ffffffffffffffffffffffff01000000, seq_no=631, retry_exceptions=0}, runtime_env_hash=1596937532, eager_install=1, setup_timeout_seconds=600
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/python-core-worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103_3034109.log:41276:[2025-09-08 22:36:45,619 I 3034109 3034149] task_manager.cc:1172: Task failed: IOError: The actor was restarted: Type=ACTOR_TASK, Language=PYTHON, Resources: {}, function_descriptor={type=PythonFunctionDescriptor, module_name=skyrl_train.workers.fsdp.fsdp_worker, class_name=FSDPPolicyRayActorBase, function_name=save_hf_model, function_hash=}, task_id=7a058f94b857c6380e6678b3f310ecce7e5a904401000000, task_name=FSDPPolicyRayActorBase.save_hf_model, job_id=01000000, num_args=4, num_returns=1, max_retries=0, depth=2, attempt_number=0, actor_task_spec={actor_id=0e6678b3f310ecce7e5a904401000000, actor_caller_id=1e8ff6d236132784ffffffffffffffffffffffff01000000, seq_no=632, retry_exceptions=0}, runtime_env_hash=1596937532, eager_install=1, setup_timeout_seconds=600
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/python-core-worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103_3034109.log:41297:[2025-09-08 22:36:45,621 I 3034109 3034149] task_manager.cc:1172: Task failed: IOError: The actor was restarted: Type=ACTOR_TASK, Language=PYTHON, Resources: {}, function_descriptor={type=PythonFunctionDescriptor, module_name=skyrl_train.workers.fsdp.fsdp_worker, class_name=FSDPPolicyRayActorBase, function_name=save_hf_model, function_hash=}, task_id=b919fcf6b4aa020e9bd892e482addd449ef196eb01000000, task_name=FSDPPolicyRayActorBase.save_hf_model, job_id=01000000, num_args=4, num_returns=1, max_retries=0, depth=2, attempt_number=0, actor_task_spec={actor_id=9bd892e482addd449ef196eb01000000, actor_caller_id=1e8ff6d236132784ffffffffffffffffffffffff01000000, seq_no=631, retry_exceptions=0}, runtime_env_hash=1596937532, eager_install=1, setup_timeout_seconds=600
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/python-core-worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103_3034109.log:41318:[2025-09-08 22:36:45,621 I 3034109 3034149] task_manager.cc:1172: Task failed: IOError: The actor was restarted: Type=ACTOR_TASK, Language=PYTHON, Resources: {}, function_descriptor={type=PythonFunctionDescriptor, module_name=skyrl_train.workers.fsdp.fsdp_worker, class_name=FSDPPolicyRayActorBase, function_name=save_hf_model, function_hash=}, task_id=325dd4ec5c029e6804e2c7c483e2fa038a35fe1701000000, task_name=FSDPPolicyRayActorBase.save_hf_model, job_id=01000000, num_args=4, num_returns=1, max_retries=0, depth=2, attempt_number=0, actor_task_spec={actor_id=04e2c7c483e2fa038a35fe1701000000, actor_caller_id=1e8ff6d236132784ffffffffffffffffffffffff01000000, seq_no=631, retry_exceptions=0}, runtime_env_hash=1596937532, eager_install=1, setup_timeout_seconds=600
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/python-core-worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103_3034109.log:41341:[2025-09-08 22:36:45,622 I 3034109 3034149] task_manager.cc:1172: Task failed: IOError: The actor was restarted: Type=ACTOR_TASK, Language=PYTHON, Resources: {}, function_descriptor={type=PythonFunctionDescriptor, module_name=skyrl_train.workers.fsdp.fsdp_worker, class_name=FSDPPolicyRayActorBase, function_name=save_hf_model, function_hash=}, task_id=fc4f673705904febbe770d7c8135222b87f2f8c001000000, task_name=FSDPPolicyRayActorBase.save_hf_model, job_id=01000000, num_args=4, num_returns=1, max_retries=0, depth=2, attempt_number=0, actor_task_spec={actor_id=be770d7c8135222b87f2f8c001000000, actor_caller_id=1e8ff6d236132784ffffffffffffffffffffffff01000000, seq_no=631, retry_exceptions=0}, runtime_env_hash=1596937532, eager_install=1, setup_timeout_seconds=600
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103-01000000-3034109.err:1464:2025-09-08 17:06:56.115 | INFO | skyrl_train.trainer:train:359 - Started: 'save_hf_model'
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103-01000000-3034109.err:1466:2025-09-08 17:07:55.564 | INFO | skyrl_train.trainer:train:359 - Finished: 'save_hf_model', time cost: 59.45s
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103-01000000-3034109.err:2723:2025-09-08 18:27:33.937 | INFO | skyrl_train.trainer:train:359 - Started: 'save_hf_model'
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103-01000000-3034109.err:2725:2025-09-08 18:28:32.889 | INFO | skyrl_train.trainer:train:359 - Finished: 'save_hf_model', time cost: 58.95s
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103-01000000-3034109.err:4378:2025-09-08 20:09:21.557 | INFO | skyrl_train.trainer:train:359 - Started: 'save_hf_model'
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103-01000000-3034109.err:4380:2025-09-08 20:10:24.184 | INFO | skyrl_train.trainer:train:359 - Finished: 'save_hf_model', time cost: 62.63s
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103-01000000-3034109.err:6715:2025-09-08 22:26:43.402 | INFO | skyrl_train.trainer:train:359 - Started: 'save_hf_model'
/tmp/ray/session_2025-09-08_15-43-10_928650_3021217/logs/worker-0149cf91e7a25f36a30f623ca89f2ca12ba8571b2cc6602861617103-01000000-3034109.err:6716:2025-09-08 22:36:45.019 | INFO | skyrl_train.trainer:train:359 - Finished: 'save_hf_model', time cost: 601.62s

I found these logs and hope they would help! (Sorry this might be a bit long...)

[SumanthRH]

Hi @Harper-Hua , sorry for the delay in getting back.

It's not obvious to me what could lead to this error. From the logs, we're looking at an ALLREDUCE op timing out.

[Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=818135, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600037 milliseconds before timing out.

NumelIn and NumelOut is 1, it's a tiny tensor. Likely something during the fsdp2_get_full_state_dict call :

SkyRL/skyrl-train/skyrl_train/distributed/fsdp_strategy.py

Line 584 in ba8e2b4

output_state_dict = fsdp2_get_full_state_dict(fsdp_model, cpu_offload=True, rank0_only=True)

I'm not too sure why timings would be normal in the first few steps and then blow up after some epochs.

Since you can repro the error consistently, could you re-run your training run and dump cprofile stats to a file at each save_hf_model call?

something like:

class FSDPStrategy(DistributedStrategy):
      ...
      def save_hf_model(self, model: Union[Actor, nn.Module], output_dir: str, tokenizer=None, **kwargs) -> None:
           import cProfile
           profiler = cProfile.Profile()
           profiler.enable()
           try: 
                # existing logic 
           finally:
               profiler.disable()
               stats = pstats.Stats(profiler)
               os.makedirs(output_dir, exist_ok=True)
               stats.dump_stats(os.path.join(output_dir, "profile_data.cprof"))

When you re-run please also use TORCH_DISTRIBUTED_DEBUG=DETAIL and NCCL_DEBUG=INFO

And btw, does the time taken for other operations look normal as training progresses? (ex: training step time, etc)

[SumanthRH]

Hi @Harper-Hua , any updates on this?