[Model] support other model

[zhang-prog]

No description provided.

[paddle-bot]

Thanks for your contribution!

[Copilot]

Pull Request Overview

This PR adds support for a new multimodal model called "QF VL" (QFVLForConditionalGeneration) to the FastDeploy framework. The implementation includes comprehensive multimodal processing capabilities for handling both images and videos alongside text inputs.

Key changes include:

• Addition of the QF VL model architecture with SigLIP vision transformer backbone
• Implementation of multimodal input processing pipeline for text, images, and videos
• Integration of the new model type into the existing model registry and preprocessing workflow

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
fastdeploy/worker/gpu_model_runner.py	Adds QF model-specific RoPE embedding handling and vision feature extraction
fastdeploy/multimodal/utils.py	Removes unused import and adds formatting cleanup
fastdeploy/multimodal/registry.py	Registers the new QFVLForConditionalGeneration model type
fastdeploy/model_executor/models/qf_vl/siglip.py	Implements SigLIP vision transformer with rotary embeddings and attention mechanisms
fastdeploy/model_executor/models/qf_vl/qf_vl.py	Defines the main QF VL model architecture and weight management
fastdeploy/model_executor/models/qf_vl/projector.py	Implements vision-to-text feature projection layer
fastdeploy/model_executor/models/qf_vl/config.py	Defines configuration classes for the QF VL model
fastdeploy/model_executor/models/qf_vl/init.py	Module initialization file
fastdeploy/input/qf_vl_processor/qf_vl_processor.py	Main processor for handling QF VL multimodal inputs
fastdeploy/input/qf_vl_processor/process.py	Core data processing logic for tokenization and multimodal handling
fastdeploy/input/qf_vl_processor/image_processor.py	Image and video preprocessing implementation
fastdeploy/input/qf_vl_processor/init.py	Module initialization for QF VL processor
fastdeploy/input/preprocess.py	Integrates QF VL processor into the preprocessing pipeline

Comments suppressed due to low confidence (1)

fastdeploy/input/qf_vl_processor/process.py:1

• The dtype parameter should be passed to np.concatenate as a separate argument, not as a keyword argument. The correct syntax is np.concatenate([...]).astype(np.int64).

"""

[Copilot]

The conditional logic for QF model handling should include the original code inside the else block. The current structure suggests the else block is empty, which may break existing functionality for non-QF models.

[Copilot]

The comment contains a typo. 'dubug' should be 'debug'.

Suggested change

	# todo: not dubug
	# todo: not debug

[Copilot]

This condition will always evaluate to True because 'lm_head.weight' is a non-empty string. The intended logic appears to be checking if weight_name contains 'lm_head.weight' or is empty.

Suggested change

	if "lm_head.weight" or "" in weight_name:
	if "lm_head.weight" in weight_name or weight_name == "":

[Copilot]

The docstring incorrectly refers to 'QwenVLProcessor' instead of 'QFVLProcessor'.

Suggested change

	Initialize QwenVLProcessor instance.
	Initialize QFVLProcessor instance.

[Jiang-Jia-Jun]

2025

[zhang-prog]

In #4396