[CustomOp][MM] Register AscendMMEncoderAttention CustomOp and remove related patch

[shen-shanshan]

What this PR does / why we need it?

Register AscendMMEncoderAttention CustomOp and remove related patch.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

✅ Run Qwen2.5-VL:

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen2.5-VL-7B-Instruct \
--max_model_len 16384 \
--max-num-batched-tokens 16384 \
--tensor-parallel-size 2 \
--enforce-eager

Output:

{"id":"chatcmpl-b4e3053f30ab2442","object":"chat.completion","created":1764922950,"model":"/root/.cache/modelscope/hub/models/Qwen/Qwen2.5-VL-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The text in the image is \"TONGYI Qwen.\" The word \"TONGYI\" is written in blue, and \"Qwen\" is written in gray. The font appears to be modern and clean, with \"TONGYI\" being slightly larger than \"Qwen.\" The design includes a geometric, abstract shape on the left side of the logo, which complements the text.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null,"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":78,"total_tokens":162,"completion_tokens":84,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

✅ Run Qwen3-VL:

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-8B-Instruct \
--max_model_len 16384 \
--tensor-parallel-size 2 \
--enforce-eager

Output:

{"id":"chatcmpl-97571fbda8267bd1","object":"chat.completion","created":1764923306,"model":"/root/.cache/modelscope/hub/models/Qwen/Qwen3-VL-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The text in the illustration is **“TONGYI Qwen”**.\n\n### How it looks:\n- **“TONGYI”** is written in **uppercase letters** in a **bold, modern sans-serif font**, colored **blue**.\n- **“Qwen”** is written in **lowercase letters** in a **slightly thinner, elegant sans-serif font**, colored **dark gray**.\n- The two lines of text are stacked vertically, with “TONG","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null,"reasoning_content":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":112,"total_tokens":212,"completion_tokens":100,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

• vLLM version: v0.12.0
• vLLM main: vllm-project/vllm@ad32e3e

[gemini-code-assist]

Code Review

This pull request refactors the Ascend-specific implementation of multi-modal encoder attention by creating a new custom operator, AscendMMEncoderAttention. This improves code organization by moving specialized logic out of a model-specific patch file (patch_qwen2_5_vl.py) and into a reusable operator. The changes also include registering this new operator in the system. The refactoring is well-done, but I've found a couple of potential issues in the new custom operator that could affect correctness and robustness, particularly regarding support for Grouped-Query Attention and handling of optional arguments.

[gemini-code-assist]

The cu_seqlens tensor is used without checking if it is None. The function signature allows cu_seqlens to be None, which would cause a TypeError when torch.diff is called. This could lead to a runtime crash. Please add a check to ensure cu_seqlens is not None before using it.

Suggested change

	cu_seqlens = torch.diff(cu_seqlens).to("cpu")
	if cu_seqlens is None:
	raise ValueError("cu_seqlens cannot be None for AscendMMEncoderAttention")
	cu_seqlens = torch.diff(cu_seqlens).to("cpu")

[gemini-code-assist]

The number of key-value heads (num_kv_heads) is hardcoded to self.num_heads. This is incorrect for models that use Grouped-Query Attention (GQA) or Multi-Query Attention (MQA) where the number of key-value heads is different from the number of query heads. The num_kv_heads parameter is passed during initialization and should be available as self.num_kv_heads.

Suggested change

	num_kv_heads=self.num_heads,
	num_kv_heads=self.num_kv_heads,

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.