[Feature] MXFP4 model loading

[fxmarty-amd]

This PR allows to load MXFP4 models in sglang, using dynamic per-group OCP MXFP4 quantization for activations for linear layers.

The supported models are quantized using AMD Quark.

For now, the execution of GEMM is simulated on fp16/bf16, but in the future mxfp4 kernels will be added.

I get sensibly similar eval results vs the recent integration in vllm, on wikitext for a llama 2 70B mxfp4 model:

sglang (this PR):

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
wikitext	2	none	0	bits_per_byte	↓	0.5560	±	N/A
		none	0	byte_perplexity	↓	1.4701	±	N/A
		none	0	word_perplexity	↓	7.8516	±	N/A

vllm:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
wikitext	2	none	0	bits_per_byte	↓	0.5565	±	N/A
		none	0	byte_perplexity	↓	1.4707	±	N/A
		none	0	word_perplexity	↓	7.8674	±	N/A

To do:

• Documentation
• Test
• Support MOE models (likely in an other PR)
• Support mxfp4 * mxfp4 GEMM kernel (likely in an other PR)
• Test eval parity for Deepseek R1, Llama 4 and llama 405B with vllm

[HaiShaw]

Can we extend this to handle fp8/bf16/fp16 activations (with only weight be mxfp4)?

[zhaochenyang20]

@fxmarty-amd rebase？

[BowenBao]

are these additions needed solely by quark?

[fxmarty-amd]

This is needed for https://github.com/fxmarty-amd/sglang/blob/c0e1e1eeb12bf69ca28953d1c0960f370e7ee645/python/sglang/srt/layers/quantization/quark/quark.py#L64-L66

and

sglang/python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py

Lines 124 to 126 in 844e2f2

	if should_ignore_layer(
	prefix, ignore=self.ignore, fused_mapping=self.packed_modules_mapping
	):

to properly work.

[BowenBao]

is this how is done in vllm?

[fxmarty-amd]

No, sglang has a different logic.

vllm uses https://github.com/vllm-project/vllm/blob/d637b960994119907b41c82d79f5a71c96dd419b/vllm/model_executor/layers/quantization/quark/quark.py#L344, which does not exist for other quantization schemes in sglang.

[BowenBao]

Can we extend this to handle fp8/bf16/fp16 activations (with only weight be mxfp4)?

These are some good ideas, however I'd suggest focus the scope of this PR within just mxfp4 weight and dynamic mxfp4 activations. Folks can fill in these two spots with mxfp4 kernel call to make it work e2e.

sglang/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4.py

Line 106 in d21830c

raise NotImplementedError()

sglang/python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py

Line 815 in d21830c

raise NotImplementedError()

[fxmarty-amd]

@zhaochenyang20 I merged main, let me know what you think!

[HaiShaw]

change this to mxfp_supported()

[fxmarty-amd]

why? there are supports_custom_op, support_triton above

[HaiShaw]

@fxmarty-amd let's rebase

[fxmarty-amd]

Hi @HaiShaw, conflicts should have been solved, and tests from https://github.com/fxmarty-amd/sglang/blob/mxfp4/test/srt/models/test_quark_models.py run fine for me locally.

[fxmarty-amd]

@xutizhou

Hi @fxmarty-amd, thank you very much for your outstanding work! I was wondering, is Quark also compatible with NVIDIA GPUs?

Quark quantizer should run just fine on Nvidia GPUs, although our CI runs on AMD Instinct GPUs.

[fxmarty-amd]

Hi @HaiShaw @zhyncs @zhaochenyang20 hope you are doing well.

I solved conflicts again and made sure the added tests pass successfully as well.

Let me know if you need anything from me to get this PR merged.

Edit: actually, TestR1MXFP4Accuracy is not passing now - checking.

[fxmarty-amd]

gentle ping @zhyncs @zhaochenyang20

The added tests pass for me.

I disabled --attention-backend aiter for now as there are ongoing issues unrelated to this PR investigated by @kkHuang-amd .

[BowenBao]

@HaiShaw please kindly take a look.

[fxmarty-amd]

Closing as #8255 will land instead.