Skip to content

fix: allow configuring ray ports in ray.sub in case conflict on cluster #173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
terrykong merged 3 commits into from
Apr 14, 2025
Merged

fix: allow configuring ray ports in ray.sub in case conflict on cluster #173

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
51650bd
fix: allow all ray ports to be configurable
terrykong Apr 14, 2025
de67729
missing upper
terrykong Apr 14, 2025
6097270
revert
terrykong Apr 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 22 additions & 8 deletions ray.sub
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,18 @@ CONTAINER=$CONTAINER
MOUNTS=$MOUNTS
COMMAND=${COMMAND:-} # This is a script relative to the SLURM_SUBMIT_DIR. If left empty, it will leave the cluster idle after it's brought up.
########################################################
# Ray ports
GCS_SERVER_PORT=${GCS_SERVER_PORT:-6379}
DASHBOARD_PORT=${DASHBOARD_PORT:-8265}
OBJECT_MANAGER_PORT=${OBJECT_MANAGER_PORT:-8076}
NODE_MANAGER_PORT=${NODE_MANAGER_PORT:-8077}
DASHBOARD_AGENT_PORT=${DASHBOARD_AGENT_PORT:-52365}
DASHBOARD_AGENT_GRPC_PORT=${DASHBOARD_AGENT_GRPC_PORT:-52366}
METRICS_PORT=${METRICS_PORT:-9002}
# NOTE: Ports start above 20000 since 10001-10257 frequently ran into conflicts
MIN_WORKER_PORT=${MIN_WORKER_PORT:-20001}
MAX_WORKER_PORT=${MAX_WORKER_PORT:-20257}
########################################################

# Defaults to placing uv cache inside the SLURM_SUBMIT_DIR
# This directory is mounted into the container at /home/ray/.cache/uv so it is shared between the head and worker nodes
Expand Down Expand Up @@ -58,8 +70,7 @@ done
head_node=${nodes_array[0]}
head_node_ip=${ip_addresses_array[0]}

port=41993
ip_head=$head_node_ip:$port
ip_head=$head_node_ip:$GCS_SERVER_PORT

# First we start the head of the ray cluster on one of the physical nodes
# Set GPU/CPU resources to 0 to avoid scheduling on the head node
Expand All @@ -75,7 +86,13 @@ ray start --head \
--num-cpus=0 \
--num-gpus=0 \
--node-ip-address="$head_node_ip" \
--port=$port \
--port=${GCS_SERVER_PORT} \
--dashboard-port=${DASHBOARD_PORT} \
--object-manager-port=${OBJECT_MANAGER_PORT} \
--node-manager-port=${NODE_MANAGER_PORT} \
--metrics-export-port=${METRICS_PORT} \
--dashboard-agent-grpc-port=${DASHBOARD_AGENT_GRPC_PORT} \
--dashboard-agent-listen-port=${DASHBOARD_AGENT_PORT} \
--block
EOFINNER
chmod +x /launch-head.sh
Expand All @@ -96,9 +113,6 @@ NUM_ACTORS=$((gpus_per_node * SLURM_JOB_NUM_NODES))
# Start Ray worker nodes
# We want 1 Ray worker node per physical node
# Worker nodes are started with ray start but without the --head flag
# NOTE: Ports start above 20000 since 10001-10257 frequently ran into conflicts
min_worker_port=20001
max_worker_port=20257
for ((i = 0; i < SLURM_JOB_NUM_NODES; i++)); do
node_i=${nodes_array[$i]}

Expand All @@ -108,8 +122,8 @@ cat <<EOFINNER | tee /launch-worker.sh
ray start --address "$ip_head" \
--disable-usage-stats \
--resources="{\"worker_units\": $gpus_per_node}" \
--min-worker-port=$min_worker_port \
--max-worker-port=$max_worker_port \
--min-worker-port=${MIN_WORKER_PORT} \
--max-worker-port=${MAX_WORKER_PORT} \
--block
EOFINNER

Expand Down
Loading