diff --git a/docs/nsys-profiling.md b/docs/nsys-profiling.md index 7fbd7e12bb..db3f6a768a 100644 --- a/docs/nsys-profiling.md +++ b/docs/nsys-profiling.md @@ -80,7 +80,7 @@ When profiling is enabled, it generates the following logs and files: dtensor_policy_worker__.nsys-rep ``` -3. **File Location**: Profile files are saved in `/tmp/ray/session*/logs/nsight/` directory on each worker node. +3. **File Location**: Profile files are saved in `/tmp/ray/session*/logs/nsight/` directory on each worker node. Ensure you check both `ls /tmp/ray/session_[0-9]*/logs/nsight` and `ls /tmp/ray/session_latest/logs/nsight` for the profiles, since the "latest" pointer may be stale. **Note for SLURM users with `ray.sub`**: When using `ray.sub` on SLURM, set `RAY_LOG_SYNC_FREQUENCY=$NUM_SEC` (e.g., `RAY_LOG_SYNC_FREQUENCY=30`) to ensure that the nsight profile files get copied from the container's ephemeral filesystem (`/tmp/ray`) to the persistent `$SLURM_JOB_ID-logs/ray` directory.