[NVIDIA] Qwen3.5 B200 SGLang FP4 configs

[kedarpotdar-nv]

Summary

Add FP4 benchmark configuration and launch script for Qwen3.5-397B-A17B on NVIDIA B200 GPUs using SGLang.

Changes

New Benchmark Config (nvidia-master.yaml)

• Config key: qwen3.5-fp4-b200-sglang
• Model: nvidia/Qwen3.5-397B-A17B-NVFP4
• Image: lmsysorg/sglang:v0.5.9-cu129-amd64
• Precision: FP4 (ModelOpt NVFP4)
• Sequence length configurations:
  • 1k1k — TP4/EP1 (conc 4–32), TP8/EP1 (conc 4–64), TP8/EP8 (conc 128)
  • 1k8k — TP4/EP1 (conc 4–32), TP8/EP1 (conc 4–128)
  • 8k1k — TP4/EP1 (conc 4–32), TP8/EP1 (conc 4–128)

New Launch Script (benchmarks/single_node/qwen3.5_fp4_b200.sh)

SGLang server configuration with:

• --quantization modelopt_fp4 with --fp4-gemm-backend flashinfer_cutlass
• --kv-cache-dtype fp8_e4m3
• --attention-backend trtllm_mha / --moe-runner-backend flashinfer_trtllm
• --enable-flashinfer-allreduce-fusion
• --chunked-prefill-size 32768 / --max-prefill-tokens 32768
• --disable-radix-cache
• --mem-fraction-static 0.85

Perf Changelog

• Added entry for the new qwen3.5-fp4-b200-sglang config.

[kedarpotdar-nv]

@claude please add a PR description

[Klaud-Cold]

Claude finished @kedarpotdar-nv's task in 40s —— View job

---

Adding PR Description

• Gather context from changed files
• Draft and update PR description

PR description has been added with a summary of all changes: new qwen3.5-fp4-b200-sglang benchmark config, launch script, and perf-changelog entry.

[kedarpotdar-nv]

@claude please review the PR

[Klaud-Cold]

Claude finished @kedarpotdar-nv's task in 2m 41s —— View job

---

Reviewing PR

• Gather context from changed files and existing patterns
• Review benchmark config (nvidia-master.yaml)
• Review launch script (qwen3.5_fp4_b200.sh)
• Review perf-changelog entry
• Post review feedback

---

Review: qwen3.5-fp4-b200-sglang

Overall this is a clean addition. Config, script, and changelog all follow existing patterns. A few observations below — mostly minor, one worth discussing.

---

.github/configs/nvidia-master.yaml — Config

Looks good. Image (v0.5.9), model (nvidia/Qwen3.5-397B-A17B-NVFP4), and metadata fields are consistent with repo conventions. The search space is reasonable: TP4/EP1 for lower concurrencies, TP8/EP1 for wider sweeps, and a single TP8/EP8 point at conc=128 for 1k1k. No issues here.

---

benchmarks/single_node/qwen3.5_fp4_b200.sh — Launch Script

1. Missing --tokenizer-worker-num
The BF16 sibling script (qwen3.5_bf16_b200.sh) sets --tokenizer-worker-num 6. The new FP4 script omits this. At high concurrency (64–128), tokenization can become a bottleneck on a single worker. Was this intentionally dropped, or should it be carried over?

2. Hardcoded --scheduler-recv-interval 30
The BF16 script dynamically sets this (10 for CONC < 16, 30 for CONC ≥ 16). The FP4 script hardcodes 30. For low-concurrency runs (conc=4), a recv interval of 30 adds unnecessary scheduling latency — the scheduler polls less frequently, which can hurt latency at low load. Consider matching the BF16 pattern:

if [[ $CONC -ge 16 ]]; then
  SCHEDULER_RECV_INTERVAL=30
else
  SCHEDULER_RECV_INTERVAL=10
fi

Fix this →

3. No --served-model-name
The BF16 script passes --served-model-name "Qwen/Qwen3.5-397B-A17B". This script omits it. Not a blocker for benchmarking (the benchmark client uses the actual model path), but worth noting for consistency.

4. FP4-specific flags look correct

• --quantization modelopt_fp4 + --fp4-gemm-backend flashinfer_cutlass — correct for NVFP4
• --kv-cache-dtype fp8_e4m3 — standard for FP4 configs
• --attention-backend trtllm_mha — correct for Qwen (MHA, not MLA which is DeepSeek-specific)
• --moe-runner-backend flashinfer_trtllm — correct for MoE models
• --mem-fraction-static 0.85 — reasonable bump from BF16's 0.82 given smaller FP4 weights

5. Environment exports inherited from BF16 script
NCCL_NVLS_ENABLE=1, SGL_ENABLE_JIT_DEEPGEMM=false, SGLANG_ENABLE_FLASHINFER_GEMM=true — all look appropriate.

---

perf-changelog.yaml

Clean entry, well-formatted, correct PR link. No issues.

---

Summary

Area	Verdict
Config YAML	✅ Good
Launch script	🟡 Minor — missing --tokenizer-worker-num, hardcoded scheduler interval
Perf changelog	✅ Good

The script is functional as-is. The two items flagged (tokenizer workers and dynamic scheduler interval) are nice-to-haves that would align with the BF16 sibling script and could improve low-concurrency latency. Up to you whether to address now or in a follow-up.

View job

[kedarpotdar-nv]

@functionstackx collaborating on this existing PR to get our FP8 and FP4 configs merged upstream in sgl-cookbook. sgl-project/sgl-cookbook#168

Thanks!

[functionstackx]

@functionstackx collaborating on this existing PR to get our FP8 and FP4 configs merged upstream in sgl-cookbook. sgl-project/sgl-cookbook#168

Thanks!

Thanks! excited to see continued focus on ensuring that the end users in the ml community can benefit from these optimized flag recipes for selecting the correct MoE kernel backend & attention kernel backend

[functionstackx]

lgtm

[kedarpotdar-nv]

/sweep run only evals

[github-actions]

@kedarpotdar-nv Kicking off a sweep.

Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/22507799181
Command: run only evals
Pinned ref: c738532
Approval: not required (trusted collaborator).

[cquil11]

@kedarpotdar-nv some merge conflicts then looks ready to go?

[kedarpotdar-nv]

@ankursingh-nv will take over

[functionstackx]

@claude resolve the merge conflicts & update the PRs to follow the updated conventions. remember that qwen3.5-fp4-b200-sglang should be at the bottom of perf-changelog.yaml, remember to delete 1k/8k from nvidia-master, remember to do the EVAL_CONTEXT_ARGS thing

[Klaud-Cold]

Claude finished @functionstackx's task in 3m 16s —— View job

---

Resolving Merge Conflicts & Updating Conventions

• Gather context (check merge conflicts, current conventions)
• Reset branch to origin/main and apply changes cleanly
• Add qwen3.5-fp4-b200-sglang config to nvidia-master.yaml (1k1k and 8k1k only, no 1k8k)
• Create launch script with updated conventions (EVAL_CONTEXT_ARGS, dynamic scheduler interval, tokenizer-worker-num)
• Add perf-changelog entry at the bottom
• Commit and push changes

Summary

Resolved merge conflicts by rebasing cleanly on origin/main (single clean commit). Changes applied:

nvidia-master.yaml — Added qwen3.5-fp4-b200-sglang config with only 1k1k and 8k1k seq-len configs (removed 1k8k). Placed after qwen3.5-fp8-b200-sglang.

benchmarks/single_node/qwen3.5_fp4_b200.sh — Rewritten to follow updated conventions from the fp8 sibling script:

• Dynamic SCHEDULER_RECV_INTERVAL (10 for CONC < 16, 30 for CONC >= 16)
• EVAL_CONTEXT_ARGS pattern (setup_eval_context + EVAL_MAX_MODEL_LEN)
• --tokenizer-worker-num 6
• Conditional --enable-flashinfer-allreduce-fusion (only for TP=8)
• CONTEXT_LENGTH=$((ISL + OSL + 20)) instead of hardcoded MAX_MODEL_LEN
• MAX_RUNNING_REQUESTS=128, CUDA_GRAPH_MAX_BATCH_SIZE=$CONC
• --data-parallel-size=1

perf-changelog.yaml — Entry added at the bottom.
| Branch

[functionstackx]

@kedarpotdar-nv @ankursingh-nv the eval score is not good , it doesnt even pass an relaxed floor model eval threshold https://github.com/SemiAnalysisAI/InferenceX/actions/runs/23698363526/job/69037542640?pr=820

+viz @Oseltamivir

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://cookbook.sglang.io/
• https://docs.vllm.ai/projects/recipes/en/latest/

If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sgl-cookbook

[jgangani]

LGTM

[functionstackx]

recipes?

[kedarpotdar-nv]

https://github.com/sgl-project/sgl-cookbook/pull/230/files
PR for updated recipe

[hshrivastava-droid]

@functionstackx - could you please help reviewing this?

[cquil11]

nightly image fine for new arch

[cquil11]

https://github.com/SemiAnalysisAI/InferenceX/actions/runs/24107321633

Evals looks good, throughput looks good
Merging