Conversation
Signed-off-by: Xin He <xin3.he@intel.com>
Signed-off-by: Xin He <xin3.he@intel.com>
|
/azp run Unit-Test-CUDA-AutoRound |
|
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
Updates AutoRound’s GPTQModel integration to accommodate GPTQModel 7.0.0 API/class renames and adjusts some test helper logic used in CI/model loading.
Changes:
- Add GPTQModel 7.0.0 compatibility for Marlin kernels and ExllamaV2 class naming changes.
- Refactor GPTQModel backend linear selection to unify GPTQ/AWQ handling and dynamically resolve renamed classes.
- Simplify
get_tiny_model(..., from_config=True)config-only model construction path in tests.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
test/helpers.py |
Adjusts model-path selection and refactors config-only tiny model instantiation logic. |
auto_round_extension/cuda/gptqmodel_marlin.py |
Adds GPTQModel 7.0.0 Marlin import/argument handling. |
auto_round/inference/convert_model.py |
Switches ExllamaV2 class import based on GPTQModel version. |
auto_round/inference/backend.py |
Unifies GPTQModel GPTQ/AWQ backend routing and resolves renamed QuantLinear classes via dynamic import. |
| architectures = getattr(config, "architectures", [None])[0] | ||
| if ( | ||
| is_mllm | ||
| and architectures.endswith("Model") | ||
| and hasattr(base_lib, n := architectures.replace("Model", "ForConditionalGeneration")) | ||
| ): | ||
| model_cls = getattr(base_lib, n) | ||
| elif hasattr(base_lib, architectures): |
There was a problem hiding this comment.
architectures can be None when the config doesn’t define an architectures list (you default to [None]). In that case architectures.endswith("Model") will raise an AttributeError. Add a guard (e.g., ensure architectures is a str before calling string methods) and fall back to AutoModelForCausalLM when it’s missing/empty.
| architectures = getattr(config, "architectures", [None])[0] | |
| if ( | |
| is_mllm | |
| and architectures.endswith("Model") | |
| and hasattr(base_lib, n := architectures.replace("Model", "ForConditionalGeneration")) | |
| ): | |
| model_cls = getattr(base_lib, n) | |
| elif hasattr(base_lib, architectures): | |
| config_architectures = getattr(config, "architectures", None) | |
| architectures = ( | |
| config_architectures[0] | |
| if isinstance(config_architectures, (list, tuple)) and len(config_architectures) > 0 | |
| else None | |
| ) | |
| if ( | |
| is_mllm | |
| and isinstance(architectures, str) | |
| and architectures.endswith("Model") | |
| and hasattr(base_lib, n := architectures.replace("Model", "ForConditionalGeneration")) | |
| ): | |
| model_cls = getattr(base_lib, n) | |
| elif isinstance(architectures, str) and hasattr(base_lib, architectures): |
| if NEW_VERSION_7_0: | ||
| import gptqmodel.utils.marlin as gptqmodel_marlin_kernels | ||
| else: | ||
| try: | ||
| import gptqmodel_marlin_kernels # pylint: disable=E0401 | ||
| except ImportError as e: | ||
| marlin_import_exception = e |
There was a problem hiding this comment.
For gptqmodel>=7.0.0 you import gptqmodel.utils.marlin without handling ImportError, but older versions store the exception and raise a clear ValueError when the backend is used. Wrap the v7 import in the same try/except and set marlin_import_exception so failures produce a consistent, actionable error message.
| if "awq" in backend: | ||
| if "gptqmodel" in backend: | ||
| return get_gptqmodel_awq_infer_linear(backend) | ||
| return get_gptqmodel_infer_linear(backend, is_awq=True) | ||
| else: |
There was a problem hiding this comment.
This if "gptqmodel" in backend branch inside the generic "awq" in backend block is now unreachable because the earlier if "gptqmodel" in backend: returns first for all gptqmodel backends. Consider removing this dead branch to avoid confusion and keep backend routing maintainable.
Description
GPTQmodel 7.0.0 removes
Quantstring from Linear class nameType of Change
Bug fix
Related Issues
Fixes or relates to #1771
Checklist Before Submitting
/azp run Unit-Test-CUDA-AutoRound.