[Question] GPT-OSS-120B benchmark environment requirements - Driver/CUDA version clarification needed

[k-l-lambda]

Summary

I am attempting to reproduce the InferenceMAX GPT-OSS-120B benchmarks on RunPod B200 but my vLLM results show a significant performance gap compared to SemiAnalysis benchmarks. I need clarification on the environment setup and configuration used.

My Environment

Component	Version
GPU	NVIDIA B200 (183GB VRAM, SM100)
Driver	570.195.03
CUDA	12.8.93
Platform	RunPod
vLLM	0.13.0

Performance Gap

Comparing at similar throughput levels shows a large latency gap:

Source	Output Throughput	E2E Latency	Concurrency
SemiAnalysis vLLM	~4,666 tok/s	~10s	C=128
Our vLLM	~3,663 tok/s	~27s	C=100
Our vLLM	~5,051 tok/s	~40s	C=200

At comparable latency (~10s), SemiAnalysis achieves ~4,666 tok/s while our setup would be around ~2,000 tok/s - roughly 2x performance gap.

Our Full Results

Concurrency	Output Throughput	E2E Latency
C=1	215 tok/s	4.6s
C=20	1,370 tok/s	14.6s
C=50	2,427 tok/s	20.6s
C=100	3,663 tok/s	27.3s
C=200	5,051 tok/s	39.6s
C=300	5,725 tok/s	52.4s

Questions

1. What driver version was used? RunPod B200 has driver 570.x (CUDA 12.8). Is driver 575+ (CUDA 13) required for optimal performance?

2. What cloud platform was used? Different platforms may have different driver/software stacks.

3. Is Docker required? The benchmark scripts reference Docker containers.

References

• vLLM benchmark script: https://github.com/InferenceMAX/InferenceMAX/blob/main/benchmarks/gptoss_fp4_b200_docker.sh
• TRT-LLM benchmark script: https://github.com/InferenceMAX/InferenceMAX/blob/main/benchmarks/gptoss_fp4_b200_trt_docker.sh
• NVIDIA deployment guide: https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog9_Deploying_GPT_OSS_on_TRTLLM.md

Any guidance on configuration or environment requirements would be appreciated. Thank you!

[k-l-lambda]

Comparison Chart

Here is a visual comparison of our vLLM results vs SemiAnalysis benchmarks:

(Note: Chart shows our vLLM 0.13.0 results (blue) vs SemiAnalysis vLLM (green) and TRT-LLM (red) on B200)

[functionstackx]

@k-l-lambda yes runpod is known for performance issues, have u tried running it on any csp that isnt known for perf issues?

[functionstackx]

@k-l-lambda do u have an reproducible script? typically to reprod inferencemax, ppl fork our repo and add their own nodes and run it through github actions on their own repo such that u are running all the exact configs and parallelism and container image. ur graph doesnt list the container image or parallelism for each datapoint so i dont know what u r running. i dont see an fork of inferencemax on ur profile so i assume that ur results is caused by

1. u using the correct setup of the whole inferencemax code and/or
2. ur using runpod known for perf issues

let me know if u still see any perf difference after solving #1 & #2 and will be happy to dive deeper

[cquil11]

@k-l-lambda also low priority, but we are working on publishing some system info for the runners, at some point in the future

[cquil11]

Closing as stale — maintainers responded with guidance (RunPod known perf issues, suggested forking the repo to reproduce). No follow-up from the
reporter in 3 months.