test profile b200 decode

[functionstackx]

@claude do a profile run with dsr1 fp4 on b200 with conc 4, and for 1 decode step, breakdown where the time went kernel by kernel, wall clock time & %, how long did MoE take, etc?

[Klaud-Cold]

Claude finished @functionstackx's task in 47m 33s —— View job

---

Profile Run: DSR1 FP4 on B200 (conc=4) — Decode Step Breakdown

• Trigger profile workflow (dsr1-fp4-b200-sglang, conc=4)
• Wait for profile run to complete — Run #22430968501
• Retrieve Perfetto trace link
• Analyze decode step kernel-by-kernel breakdown (wall clock time & %)
• Summarize MoE time and other key components

Perfetto Trace: Open in Perfetto

---

Config

• Model: DeepSeek-R1-0528-FP4 (61 layers, 3 dense + 58 MoE)
• Framework: SGLang v0.5.6
• Hardware: 4× B200 (TP4/EP4, single node)
• Concurrency: 4
• Profile: GPU 0 (TP0/EP0), 1 decode forward pass

---

1 Decode Step Wall Clock: 24.976 ms

The decode step has 3 phases:

Phase	Wall Clock	%
Setup (scheduling, indexing, memcpy)	3.14 ms	12.6%
NCCL AllReduce (EP all-to-all)	13.73 ms	55.0%
Transformer Layer Compute (61 layers)	8.12 ms	32.5%

---

Kernel-by-Kernel Breakdown (all streams, GPU 0)

The model uses two GPU streams in parallel during layer compute:

• Stream 39 (critical path): attention (MLA), shared expert, Lamport allreduce
• Stream 461 (parallel): routed MoE expert dispatch, FP4 BMM, routing

Stream 39 (Critical Path) — Kernel Aggregates

Kernel Category	Time (ms)	% of Wall	#Calls
NCCL AllReduce (EP dispatch/gather)	13.725	55.0%	2
FP4 GEMM — CUTLASS (shared expert)	1.522	6.1%	183
Fused AllReduce — Lamport (intra-node)	1.490	6.0%	121
Fused Absorb GEMM (MLA output proj)	1.207	4.8%	61
NVJet GEMM (QKV/O projections)	0.851	3.4%	184
FMHA Decode Attention (SM100 MLA)	0.776	3.1%	61
FP4 Weight Convert	0.423	1.7%	183
RMSNorm	0.218	0.9%	63
SiLU Activation	0.177	0.7%	61
RoPE + KV Quantize	0.137	0.5%	61
Elementwise	0.101	0.4%	64
MLA KV Buffer Setup	0.089	0.4%	61

Stream 461 (Parallel MoE Routed Experts)

Kernel Category	Time (ms)	% of Stream	#Calls
FP4 BMM — cuBLAS (routed expert GEMM)	2.092	56.9%	116
MoE Routing (DeepSeek)	0.534	14.5%	116
Router GEMM (gate projection)	0.525	14.3%	58
Block Quantize (BF16→FP4)	0.227	6.2%	58
MoE Finalize (unpermute + reduce)	0.164	4.5%	58
RMSNorm	0.135	3.7%	61

---

Per-Layer Pattern (1 of 61 layers, ~0.13ms on critical path)

 0.006ms  RMSNorm (attention input)
 0.008ms  Fused Absorb GEMM (MLA compressed projection)
 0.003ms  RMSNorm (MoE input)
 0.009ms  NVJet GEMM ×2 (Q/KV projections)
 0.002ms  RoPE + KV Quantize
 0.002ms  MLA KV Buffer Setup
 0.013ms  FMHA Decode Attention (SM100 MLA kernel)
 0.004ms  NVJet GEMM (output projection)
─── MoE Shared Expert (stream 39) ──────────────
 0.002ms  FP4 Convert
 0.009ms  FP4 GEMM (CUTLASS) — gate/up
 0.007ms  Fused AllReduce (Lamport)
 0.002ms  FP4 Convert
 0.013ms  FP4 GEMM (CUTLASS) — down
 0.002ms  SiLU Activation
 0.002ms  FP4 Convert
 0.009ms  FP4 GEMM (CUTLASS)
 0.008ms  Fused AllReduce (Lamport)
 0.011ms  Fused Absorb GEMM (residual add)
─── MoE Routed Experts (stream 461, parallel) ──
 ~0.010ms  Router GEMM + Routing + Quantize
 ~0.038ms  FP4 BMM ×2 (expert GEMM)
 ~0.003ms  MoE Finalize

---

Component Summary

Component	Time (ms)	% of Wall Clock
NCCL AllReduce (EP dispatch)	13.73	55.0%
Attention (MLA)	3.06	12.3%
MoE FFN compute	2.12	8.5%
Lamport AllReduce (intra-node)	1.49	6.0%
RMSNorm	0.22	0.9%
Setup + overhead	3.14	12.6%
Gaps/idle/other	1.22	4.9%

MoE Breakdown

Total MoE time on the critical path (stream 39 shared expert + Lamport allreduce):

• Shared expert GEMMs (FP4 GEMM + convert + SiLU): 2.12 ms (8.5%)
• Lamport intra-node AllReduce (×2 per layer): 1.49 ms (6.0%)
• Total MoE on critical path: ~3.6 ms (14.5%)

Routed expert work (stream 461, runs in parallel — hidden behind stream 39):

• Router GEMM + routing + FP4 BMM + finalize: 3.54 ms (overlapped, not on critical path)

Key Takeaways

1. NCCL AllReduce dominates at 55% (13.7ms) — this is a single massive EP all-to-all at the start of the decode step. This is the [NVIDIA] Add TRT-LLM 70B FP8 via slurm #1 bottleneck by far.
2. Attention (MLA) is 12.3% (3.06ms) — FMHA decode kernel itself is only 0.78ms, but projections add 2.06ms.
3. MoE shared expert is 8.5% (2.12ms) — 3 FP4 GEMMs per layer. Routed experts run in parallel on stream 461 and are fully hidden.
4. Lamport fused allreduce is 6.0% (1.49ms) — 2 intra-node allreduces per layer for the shared expert, runs efficiently with one-shot Lamport protocol.
5. GPU utilization is 86.4% during the layer compute phase (after NCCL). The 13.6% idle is mostly kernel launch gaps.
6. Each of the 61 layers takes only ~0.13ms on the critical path — the compute itself is highly efficient.