[Bug Report] NCCL "illegal memory access" error during distributed training for Isaac-Velocity-Rough-G1-v0

[LZJ910]

---

[bug]

Describe the bug

When running multi-GPU distributed training (nproc_per_node=2), the script fails for the Isaac-Velocity-Rough-G1-v0 task, but works correctly for the Isaac-Reach-Franka-v0 task.

The error appears in both ranks (rank 0 and rank 1) and is a CUDA error: an illegal memory access was encountered. This error originates from the PyTorch distributed backend (ProcessGroupNCCL) and causes the process group watchdog to terminate, ultimately leading to a ChildFailedError (Signal 6, SIGABRT).

Steps to reproduce

1. Run the following command (which works correctly) to verify the base distributed setup:

   python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Reach-Franka-v0 --headless --distributed

2. Run the failing command:

   python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Velocity-Rough-G1-v0 --headless --distributed

3. The following error occurs:

   Error executing job with overrides: []
   [rank0]:[E1113 13:57:32.273333750 ProcessGroupNCCL.cpp:1896] [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
   Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
   
   Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
   frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7189217775e8 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x71892170c4a2 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7189248a5422 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
   frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7185754e5456 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7185754f56f0 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7185754f7282 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7185754f8e8d in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #7: <unknown function> + 0xdc253 (0x718c40cdc253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
   frame #8: <unknown function> + 0x94ac3 (0x718c41e94ac3 in /lib/x86_64-linux-gnu/libc.so.6)
   frame #9: <unknown function> + 0x1268c0 (0x718c41f268c0 in /lib/x86_64-linux-gnu/libc.so.6)
   
   terminate called after throwing an instance of 'c10::DistBackendError'
     what():  [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
   Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
   
   Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
   frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7189217775e8 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x71892170c4a2 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7189248a5422 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
   frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7185754e5456 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7185754f56f0 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7185754f7282 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7185754f8e8d in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #7: <unknown function> + 0xdc253 (0x718c40cdc253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
   frame #8: <unknown function> + 0x94ac3 (0x718c41e94ac3 in /lib/x86_64-linux-gnu/libc.so.6)
   frame #9: <unknown function> + 0x1268c0 (0x718c41f268c0 in /lib/x86_64-linux-gnu/libc.so.6)
   
   Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1902 (most recent call first):
   frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7189217775e8 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #1: <unknown function> + 0xcc7a4e (0x7185754c7a4e in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #2: <unknown function> + 0x9165ed (0x7185751165ed in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #3: <unknown function> + 0xdc253 (0x718c40cdc253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
   frame #4: <unknown function> + 0x94ac3 (0x718c41e94ac3 in /lib/x86_64-linux-gnu/libc.so.6)
   frame #5: <unknown function> + 0x1268c0 (0x718c41f268c0 in /lib/x86_64-linux-gnu/libc.so.6)
   
   Error executing job with overrides: []
   [rank1]:[E1113 13:57:32.346752417 ProcessGroupNCCL.cpp:1896] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
   Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
   
   Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
   frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7e3be53c75e8 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x7e3be535c4a2 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7e3be54a5422 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
   frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7e38364e5456 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7e38364f56f0 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7e38364f7282 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7e38364f8e8d in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #7: <unknown function> + 0xdc253 (0x7e3f022dc253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
   frame #8: <unknown function> + 0x94ac3 (0x7e3f03494ac3 in /lib/x86_64-linux-gnu/libc.so.6)
   frame #9: <unknown function> + 0x1268c0 (0x7e3f035268c0 in /lib/x86_64-linux-gnu/libc.so.6)
   
   terminate called after throwing an instance of 'c10::DistBackendError'
     what():  [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
   Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
   
   Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
   frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7e3be53c75e8 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x7e3be535c4a2 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7e3be54a5422 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
   frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7e38364e5456 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7e38364f56f0 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7e38364f7282 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7e38364f8e8d in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #7: <unknown function> + 0xdc253 (0x7e3f022dc253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
   frame #8: <unknown function> + 0x94ac3 (0x7e3f03494ac3 in /lib/x86_64-linux-gnu/libc.so.6)
   frame #9: <unknown function> + 0x1268c0 (0x7e3f035268c0 in /lib/x86_64-linux-gnu/libc.so.6)
   
   Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1902 (most recent call first):
   frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7e3be53c75e8 in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libc10.so)
   frame #1: <unknown function> + 0xcc7a4e (0x7e38364c7a4e in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #2: <unknown function> + 0x9165ed (0x7e38361165ed in /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
   frame #3: <unknown function> + 0xdc253 (0x7e3f022dc253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
   frame #4: <unknown function> + 0x94ac3 (0x7e3f03494ac3 in /lib/x86_64-linux-gnu/libc.so.6)
   frame #5: <unknown function> + 0x1268c0 (0x7e3f035268c0 in /lib/x86_64-linux-gnu/libc.so.6)
   
   W1113 13:57:33.581000 2055138 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 2055222 closing signal SIGTERM
   E1113 13:57:33.846000 2055138 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -6) local_rank: 0 (pid: 2055221) of binary: /mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/bin/python
   Traceback (most recent call last):
     File "<frozen runpy>", line 198, in _run_module_as_main
     File "<frozen runpy>", line 88, in _run_code
     File "/mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/distributed/run.py", line 896, in <module>
       main()
     File "/mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
       return f(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^
     File "/mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in main
       run(args)
     File "/mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/distributed/run.py", line 883, in run
       elastic_launch(
     File "/mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
       return launch_agent(self._config, self._entrypoint, list(args))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/mnt/ssd2/homes/lzj/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
       raise ChildFailedError(
   torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
   ========================================================
   scripts/reinforcement_learning/rsl_rl/train.py FAILED
   --------------------------------------------------------
   Failures:
     <NO_OTHER_FAILURES>
   --------------------------------------------------------
   Root Cause (first observed failure):
   [0]:
     time      : 2025-11-13_13:57:33
     host      : szdx-G7666-X6
     rank      : 0 (local_rank: 0)
     exitcode : -6 (pid: 2055221)
     error_file: <N/A>
     traceback : Signal 6 (SIGABRT) received by PID 2055221
   ======================================================

System Info

Describe the characteristic of your environment:

• Commit: c32db68
• Isaac Sim Version: 5.1.0
• OS: Ubuntu 22.04.5 LTS (Jammy Jellyfish) (Kernel: 6.8.0-86-generic)
• GPU: RTX 5090
• CUDA: 12.8.93
• GPU Driver: 580.82.09

Additional context

The problem seems specific to the Isaac-Velocity-Rough-G1-v0 task environment when used in distributed mode. The fact that Isaac-Reach-Franka-v0 works suggests the base environment, PyTorch installation, and NCCL setup are likely correct, but some tensor operation or memory access specific to the G1 task is causing an illegal memory access on the GPU.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• The command python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Velocity-Rough-G1-v0 --headless --distributed runs successfully without the CUDA error: an illegal memory access was encountered.

[RandomOakForest]

Thank you for posting this. The team will review it.

[sesem738]

Is there any update on this issue? I am experiencing the same problem for my environment as well.

[sesem738]

So, it seems to be an issue with 5090s. I tried it with a workstation with RTX 8000s and they work

[dr-blitz]

I ran into this exact same issue (crashing at ProcessGroupNCCL.cpp or the first forward pass with CUDA error: an illegal memory access) when training with depth cameras via DDP.

My setup: AMD Ryzen 9800X3D, 2x RTX 5080s, Isaac Sim 5.1.0, Ubuntu 24.04.

After debugging, I found two simultaneous hardware/driver conflicts causing this:

1. Vulkan vs. CUDA device mismatch: They enumerate GPUs differently. Process 1 (on cuda:1) accidentally initializes its Vulkan render context on physical GPU 0. Reading that depth tensor across the PCIe bus triggers the illegal access.
2. AMD IOMMU blocking NCCL: Even with P2P disabled, NCCL falls back to shared system memory (/dev/shm). AMD's IOMMU isolates PCIe devices for security and blocks this transfer, which crashes the NCCL watchdog.

The Workaround

You don't need to modify the source code or change maxGpuCount. You just need to force device ordering and restrict NCCL to local sockets to bypass the IOMMU block. Export these before launching:

# Align Vulkan and CUDA device IDs
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_LAUNCH_BLOCKING=1

# Disable PCIe P2P/IB to prevent memory conflicts
export NCCL_P2P_DISABLE=1
export NCCL_IB_DISABLE=1

# Bypass the AMD IOMMU block by forcing local sockets
export NCCL_SHM_DISABLE=1
export NCCL_P2P_LEVEL=LOC

# Launch as normal
./isaaclab.sh -p -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rsl_rl/train.py --task Isaac-Velocity-G1-DepthPerception-v0 --headless --enable_cameras --distributed

(Alternatively, if you have root access, adding amd_iommu=off to your /etc/default/grub config also prevents the NCCL crash without needing the NCCL_SHM_DISABLE flag).

Hope this helps!!!!

[insung8150]

Workaround: 5-GPU G1 Distributed Training with NCCL Environment Variables

I ran into the exact same issue and found a minimal set of 3 NCCL environment variables that enables 5-GPU distributed training for G1 humanoid (23 DOF) on Isaac Lab 2.3.2 + Isaac Sim 5.1.0.

Setup

• GPU: 5× NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96GB each)
• Driver: 580.105.08
• OS: Ubuntu 24.04 (Kernel 6.17.0)
• Isaac Lab: 2.3.2
• Isaac Sim: 5.1.0
• PyTorch: 2.7.0+cu128
• rsl-rl-lib: 5.0.1

Minimal Required Environment Variables

export NCCL_SHM_DISABLE=1    # Disable shared memory transport
export NCCL_IB_DISABLE=1     # Disable InfiniBand transport
export NCCL_ALGO=Ring        # Force Ring algorithm (critical for 3+ GPUs!)

Why These 3 Variables

I systematically tested each variable by removing one at a time from a working configuration:

Variable	Required?	Without it
NCCL_SHM_DISABLE=1	YES	Crash at 2+ GPUs
NCCL_IB_DISABLE=1	YES	Crash at 3+ GPUs
NCCL_ALGO=Ring	YES	Crash at 3+ GPUs
NCCL_P2P_DISABLE=1	No	Works without it
NCCL_PROTO=Simple	No	Works without it
NCCL_NET=Socket	No	Works without it
torch.cuda.synchronize() before broadcast	No	Not needed

The key discovery is NCCL_ALGO=Ring — without it, the default Tree algorithm creates complex GPU synchronization patterns that conflict with Omniverse's CUDA contexts. Ring topology uses a simple circular pattern (GPU0→1→2→3→4→0) that avoids these conflicts.

Successful Commands

# 5-GPU G1 training (verified working)
export NCCL_SHM_DISABLE=1
export NCCL_IB_DISABLE=1
export NCCL_ALGO=Ring

python -m torch.distributed.run --nnodes=1 --nproc_per_node=5 \
  scripts/reinforcement_learning/rsl_rl/train.py \
  --task Isaac-Velocity-Flat-G1-v0 --headless --distributed \
  --num_envs 32768 --max_iterations 3000

GPU utilization: 80-90% per GPU with num_envs=32768. Memory usage: ~23% per GPU.

Additional Notes

• CUDA_VISIBLE_DEVICES must NOT be set (conflicts with Omniverse GPU enumeration)
• No code modifications needed — environment variables only
• CartPole works with 5 GPUs without any NCCL settings (lightweight physics doesn't trigger the conflict)
• @dr-blitz's workaround above is similar but missing NCCL_ALGO=Ring, which is critical for 3+ GPUs