[NV] dsv4-fp4-gb200-dynamo-vllm#1163
Conversation
…-dsv4-recipes (PR #77) Repoint launch_gb200-nv.sh to NVIDIA/srt-slurm@aflowers/gb200-dsv4-recipes, which supersedes #71 and ships the vllm_numa_bind_hash_fix.py patch and sa-bench DSV4 tokenizer support — so numa-bind, benchmark.use_chat_template, and benchmark.tokenizer_mode no longer have to be stripped from recipes. 8k/1k search-space expanded from 3 topologies to 8: adds 1p4d/1p8d pure-TP decode (offload), 1p1d/2p1d/3p1d DEP-8 decode, and a 3p1d-dep16-40 wide decode shape. 1k/1k topologies unchanged (no upstream 1k/1k counterpart); 1k/1k tep8 also re-enables numa-bind + chat template to stay consistent. Local recipe deltas vs upstream are limited to: model.path alias rename deepseekv4-fp4 -> deepseek-v4-pro (matches SRT_SLURM_MODEL_PREFIX), container kept on the floating :deepseekv4-cu130 tag, slurm.time_limit added, and health_check.max_attempts bumped 360 -> 1440 for cold-cache loads.
Ankur-singh
changed the title
…amo-vllm # Conflicts: # perf-changelog.yaml
The 1k/1k tep8 numa-bind + chat-template re-enabling is rolled back — 1k/1k stays at the previous local-extrapolation tuning. Updates the perf-changelog entry to reflect that.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🔴
benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/1k1k/disagg-gb200-1p1d-dep8-tep8.yaml:140-144— Within the 1k/1k sweep, this PR updated only1k1k/disagg-gb200-1p1d-dep8-tep8.yaml(use_chat_template: true, tokenizer_mode: deepseek_v4, prefill numa-bind: true) but left the two sibling DEP-16 recipes (1k1k/disagg-gb200-1p1d-dep8-dep16.yamland1k1k/disagg-gb200-3p1d-dep8-dep16.yaml) on the old workaround path (use_chat_template: false, no tokenizer_mode, no numa-bind). All three are routed from the samedsv4-fp4-gb200-dynamo-vllm1k/1k search-space (.github/configs/nvidia-master.yaml7577-7620), so conc 1-64 now run through sa-bench's chat-template + DSV4 tokenizer path while conc 128-8192 use the legacy random-token /v1/completions path — the changelog explicitly states the tep8 update was made "to stay consistent" but the result is the opposite. Either also flip the two DEP-16 recipes (use_chat_template: true, tokenizer_mode: deepseek_v4, prefill numa-bind: true) or revert the tep8 update.Extended reasoning...
What is happening
The PR description says "drops local workarounds: numa-bind, benchmark.use_chat_template, and benchmark.tokenizer_mode are restored now that PR #77 ships vllm_numa_bind_hash_fix.py and sa-bench DSV4 tokenizer support", and the perf-changelog entry adds "1k/1k tep8 also re-enables numa-bind + chat template to stay consistent". The intent is clearly to get all 1k/1k recipes onto the same input/tokenization pipeline as the 8k/1k recipes that PR #77 ships.
But only the 1k/1k tep8 recipe was actually touched. The two DEP-16 siblings still carry the pre-PR workaround:
1k/1k recipe use_chat_template tokenizer_mode prefill numa-bind 1k1k/disagg-gb200-1p1d-dep8-tep8.yaml(modified)truedeepseek_v4true1k1k/disagg-gb200-1p1d-dep8-dep16.yaml(line 125, unchanged)false(unset) (unset) 1k1k/disagg-gb200-3p1d-dep8-dep16.yaml(line 117, unchanged)false(unset) (unset) Why it matters
.github/configs/nvidia-master.yamllines 7577-7620 route all three of these inside the samedsv4-fp4-gb200-dynamo-vllm1k/1k seq-len-config:conc [1, 4, 8, 16, 32, 64] -> 1p1d-dep8-tep8.yaml (chat_template=true, tokenizer_mode=deepseek_v4) conc [128, 256, 1024, 2048, 4096] -> 1p1d-dep8-dep16.yaml (chat_template=false, tokenizer_mode unset) conc [4096, 8192] -> 3p1d-dep8-dep16.yaml (chat_template=false, tokenizer_mode unset)sa-bench's
use_chat_template: truepath callstokenizer.apply_chat_template()and adds role markers / system prompt, so the actual on-the-wire prompt token count and request endpoint differs from the random-token/v1/completionspath used whenuse_chat_template: false. So within a single 1k/1k sweep:- conc 1-64 (tep8) measures latency through chat-template + DSV4-aware tokenization
- conc 128-8192 (DEP-16) measures throughput through raw tokens, no DSV4 tokenizer, plain completions endpoint
This breaks the comparable-curve invariant the PR was trying to restore: a low-conc TEP latency point and a higher-conc DEP-16 throughput point on the same 1k/1k pareto front are no longer apples-to-apples, because their input/tokenization pipelines differ.
Step-by-step proof
- The 1k/1k tep8 recipe (touched by this PR) ends with:
benchmark: ... tokenizer_mode: "deepseek_v4" use_chat_template: true
1k1k/disagg-gb200-1p1d-dep8-dep16.yamlline 125 (NOT in this PR's diff) ends with:Nobenchmark: ... use_chat_template: false
tokenizer_mode, no prefillnuma-bind: true.1k1k/disagg-gb200-3p1d-dep8-dep16.yamlline 117 (NOT in this PR's diff): identical situation..github/configs/nvidia-master.yamllines 7577-7620 confirm all three are in the same 1k/1k search-space, so a single benchmark sweep atisl=1024,osl=1024will hit all three recipes at different concurrencies.- Pre-PR all three 1k/1k recipes had
use_chat_template: false(consistent). Post-PR, only one was flipped, so this PR introduces the inconsistency.
How to fix
Either:
- (Most consistent with the changelog's stated intent) Update both DEP-16 recipes the same way: set
benchmark.use_chat_template: true,benchmark.tokenizer_mode: "deepseek_v4", and prefillnuma-bind: true(the worker-sidetokenizer-mode: deepseek_v4if not already there). - Or revert the 1k/1k tep8 update for now and leave all three on the old workarounds, matching pre-PR behavior. The 1k/1k recipes have no upstream PR #77 reference anyway, so this is a self-consistent fallback until 1k/1k DEP-16 also gets a chance to be validated end-to-end.
Runs will succeed either way (the recipes are individually valid), so this is a benchmark-correctness / comparability bug, not a runtime crash. Severity: normal.
| VLLM_RANDOMIZE_DP_DUMMY_INPUTS: "1" | ||
| VLLM_MOE_ROUTING_SIMULATION_STRATEGY: "uniform_random" |
There was a problem hiding this comment.
We need to comment these out in all yamls - it was added to see the most optimal engine perf.
…ION_STRATEGY These were upstream's tools for measuring most-optimal engine perf via randomized routing — disable them so the benchmark exercises the real expert routing path. Applied to every recipe that had them (all 8 new 8k/1k recipes plus the 1k/1k tep8 recipe).
Re-mirror from NVIDIA/srt-slurm aflowers/gb200-dsv4-recipes branch under recipes/vllm/deepseek-v4-pro-sa/ — the SemiAnalysis-curated subset of PR #77. 1k/1k recipes are removed (only 8k/1k is in scope now). Topology changes vs the previous mirror: * drop 1p1d-tep8, 2p1d-c256-c512-c1024, 3p1d-c2048, 3p1d-dep16-40, 7p1d * keep 1p1d-dep8-dep8-16 (concurrencies bumped to 64x128x256x512x1024), 1p4d-tp8, 1p8d-tp8 * add new c4096-offload variants: 2p1d-dep8-dep8, 3p1d-dep8-dep8, 3p1d-dep8-dep16 Other consistency fixes: * dynamo.install: false uniformly (matches -sa/ — assumes pre-installed dynamo in the container) * dynamo.hash 6a159fed... uniformly * model.container set to vllm/vllm-openai:deepseekv4-cu130-dynamo across all 6 recipes so the recipe lookup matches the alias key the launch script registers in srtslurm.yaml from nvidia-master.yaml's image: field * slurm.time_limit + health_check inserted right after setup_script: in a consistent position
Ankur-singh
changed the title
…vllm-gb200-v0.20.0 Bump container image to vllm/vllm-openai:v0.20.0-ubuntu2404@sha256:46da022c... in nvidia-master.yaml and across all 6 recipes (keeps the recipe model.container in lockstep with the alias key the launch script registers in srtslurm.yaml). Repoint launch_gb200-nv.sh from aflowers/gb200-dsv4-recipes to aflowers/vllm-gb200-v0.20.0 — the 0.20.0 branch. Per-recipe changes: * Replace dynamo.hash + dynamo.install: false with dynamo.install: true + wheel: "1.2.0.dev20260426". The new container is vanilla vLLM 0.20.0 without dynamo pre-installed, so srtctl installs from the pinned wheel. * Add benchmark.custom_tokenizer: "sa_bench_tokenizers.vllm_deepseek_v4.VLLMDeepseekV4Tokenizer" * Add identity: block at the bottom of every recipe — model repo+revision, container image (sha256), and dynamo+vllm framework versions for reproducibility tracking. * 1p8d recipe: add conc 1 (concurrencies "1x8x16x32x64x128x256x512") and rename to disagg-gb200-1p8d-dep8-tp8-c1-c8-c16-c32-c64-c128-c256-offload.yaml. CONFIG_FILE reference in nvidia-master.yaml updated; conc-list extended to [1, 8, 16, 32, 64, 128, 256, 512].
custom_tokenizer (added in the previous commit) covers sa-bench's
DSV4 tokenization; the redundant tokenizer_mode field is no longer
needed. The vllm_config.{prefill,decode}.tokenizer-mode worker-side
setting is unchanged.
Use just the tag (vllm/vllm-openai:v0.20.0-ubuntu2404) in nvidia-master.yaml image:, every recipe's model.container, every recipe's identity.container.image, and the recipe header comment lines.
The /mnt/numa1/models/deepseek-v4-pro/ stage doesn't carry HF revision metadata (no .huggingface/refs/main, no .cache/huggingface/download/ metadata), so identity.model.revision verification would fail every job with "no HF revision found at /model". Drop the block until the stage is re-populated via huggingface_hub.snapshot_download or the ref marker is planted manually. identity.container and identity.frameworks are preserved.
The compute-node-local NVMe path is not visible to the GHA runner host, so srtctl preflight (which runs there) failed with "model path unavailable". Use the Lustre copy instead so preflight resolves the alias to a path the runner can stat.
|
@Oseltamivir we were waiting for the queued job to run when you made your changes. Because of the changes it cancelled the current job, they got shuffled to the back of the queue and now we are waiting again. It now slowed down this submission cycle by another 12 hours. Can you please hold off on making changes until we are ready to merge? |
|
@Oseltamivir |
|
good morning @alec-flowers |
@alec-flowers Which queued job were you waiting for to run? |
The one that was cancelled when a new commit was pushed. |
|
@alec-flowers apologies, didn't see it as it wasn't on HEAD. I've cancelled my SGL GB200 run so this can proceed godspeed |
|
@claude please update PR description |
|
Claude finished @kedarpotdar-nv's task in 1m 15s —— View job Updated PR Description
PR description has been updated with a full summary covering:
|
alec-flowers
changed the title
| git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" | ||
| cd "$SRT_REPO_DIR" | ||
| git checkout sa-submission-q2-2026 | ||
| git checkout aflowers/vllm-gb200-v0.20.0 |
There was a problem hiding this comment.
Question on this. I made this branch off of main since it has some extra stuff I wanted to get in there.
The sa-submission-q2-2026, is that where you want these upstreamed? main I guess is a moving target. Will sa-submission-q2-2026 ever make it to main? Or it will live as a legacy fork?
There was a problem hiding this comment.
Historically it has been Nvidia's choice, and historically it will live as a legacy fork, as there has to be some changes to accommodate InferenceX runners.
Some changes in sa-submission-q2-2026 will never make it to main, e.g. several parts of lm-eval. Although I will investigate changing InfX to be usable with srt-slurm's main: NVIDIA/srt-slurm#41
6 participants
Summary
Rebase DSv4-FP4 GB200 dynamo-vLLM disaggregated recipes onto NVIDIA/srt-slurm
aflowers/vllm-gb200-v0.20.0branch, upgrading from thedeepseekv4-cu130image to vLLM v0.20.0 (vllm/vllm-openai:v0.20.0-ubuntu2404). Drops the previous 1k/1k sequence-length configs and local workarounds (numa-bind, chat template, tokenizer mode) that earlier submissions required.Supersedes the previous srt-slurm branch (
sa-submission-q2-2026) and mirrors three validated 8k/1k benchmark points from upstream.Changes
Image & branch update
vllm/vllm-openai:deepseekv4-cu130→vllm/vllm-openai:v0.20.0-ubuntu2404sa-submission-q2-2026→aflowers/vllm-gb200-v0.20.0launch_gb200-nv.sh) — derives squash filename from$IMAGEinstead of hardcodingenroot importfor the vLLM container (was previously commented out)Recipe changes (8k/1k)
Three new recipes mirrored from upstream, replacing the previous 6-topology sweep:
disagg-gb200-low-latency.yamldisagg-gb200-mid-curve.yamldisagg-gb200-max-tpt.yamlTwo existing offload recipes updated in-place:
disagg-gb200-2p1d-dep8-dep8-c4096-offload.yaml— rebased on upstream v0.20.0 recipedisagg-gb200-3p1d-dep8-dep16-c4096-offload.yaml— rebased on upstream v0.20.0 recipeDeleted recipes
1k1k/disagg-gb200-1p1d-dep8-dep16.yaml,1k1k/disagg-gb200-3p1d-dep8-dep16.yaml(no upstream NVIDIA reference for 1k/1k DSv4 disagg)8k1k/disagg-gb200-7p1d-dep8-dep16.yaml— 18-node full-cluster topology dropped in favor of validated 3P/1D max-tpt recipenvidia-master.yaml
dsv4-fp4-gb200-dynamo-vllmsearch-space from 6 concurrency bands (across 1k/1k + 8k/1k) down to 3 targeted 8k/1k pointsdeepseek-ai/DeepSeek-V4-ProOther
perf-changelog.yamlentry added/mnt/numa1/models/deepseek-v4-pro/Upstream references
aflowers/vllm-gb200-v0.20.0