fix: Fix qwen2_vl and qwen2_5_vl monkey patch by Tcc0403 · Pull Request #738 · linkedin/Liger-Kernel

Tcc0403 · 2025-06-01T13:22:37Z

Summary

Resolve part of #729, #713

Details

For monkey patch:

Determine the class of instance and patch Liger Kernel to text model and vision model accordingly.
Add supported model type to MODEL_TYPE_TO_APPLY_LIGER_FN dictionary to correctly call the corresponding functions.

For convergence test:

video_processor is now required for Qwen2_Processor and Qwen2_5_Processor.

Testing Done

transformers/monkey_patch

❯ python3 -m pytest test/transformers/test_monkey_patch.py -k vl
================================================= test session starts ==================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/tcc/Liger-Kernel
configfile: pyproject.toml
plugins: rerunfailures-15.0, xdist-3.6.1
collected 30 items / 24 deselected / 6 selected

test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_qwen2_vl_for_conditional_generation
---------------------------------------------------- live log call -----------------------------------------------------
INFO     liger_kernel.transformers.monkey_patch:monkey_patch.py:1668 Applying Liger kernels to model instance with model type: qwen2_vl with kwargs: {}
PASSED                                                                                                           [ 16%]
test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_qwen2_vl
---------------------------------------------------- live log call -----------------------------------------------------
INFO     liger_kernel.transformers.monkey_patch:monkey_patch.py:1668 Applying Liger kernels to model instance with model type: qwen2_vl with kwargs: {}
PASSED                                                                                                           [ 33%]
test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_qwen2_vl_text
---------------------------------------------------- live log call -----------------------------------------------------
INFO     liger_kernel.transformers.monkey_patch:monkey_patch.py:1668 Applying Liger kernels to model instance with model type: qwen2_vl_text with kwargs: {}
PASSED                                                                                                           [ 50%]
test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_qwen2_5_vl
---------------------------------------------------- live log call -----------------------------------------------------
INFO     liger_kernel.transformers.monkey_patch:monkey_patch.py:1668 Applying Liger kernels to model instance with model type: qwen2_5_vl with kwargs: {}
PASSED                                                                                                           [ 66%]
test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_qwen2_5_vl_for_conditional_generation
---------------------------------------------------- live log call -----------------------------------------------------
INFO     liger_kernel.transformers.monkey_patch:monkey_patch.py:1668 Applying Liger kernels to model instance with model type: qwen2_5_vl with kwargs: {}
PASSED                                                                                                           [ 83%]
test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_qwen2_5_vl_text
---------------------------------------------------- live log call -----------------------------------------------------
INFO     liger_kernel.transformers.monkey_patch:monkey_patch.py:1668 Applying Liger kernels to model instance with model type: qwen2_5_vl_text with kwargs: {}
PASSED

convergence test

bf16/test_mini_models

❯ python3 -m pytest test/convergence/bf16/test_mini_models.py -k vl
================================================= test session starts ==================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/tcc/Liger-Kernel
configfile: pyproject.toml
plugins: rerunfailures-15.0, xdist-3.6.1
collecting ...
------------------------------------------------- live log collection --------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.7.0 available.
collected 16 items / 14 deselected / 2 selected

test/convergence/bf16/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype7-0.001-0.05-1-0.1-0.01-0.01] PASSED [ 50%]
test/convergence/bf16/test_mini_models.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype8-0.001-0.05-3-0.1-0.01-0.01] PASSED [100%]

bf16/test_mini_models_multimodal

❯ python3 -m pytest test/convergence/bf16/test_mini_models_multimodal.py -k qwen2
================================================= test session starts ==================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/tcc/Liger-Kernel
configfile: pyproject.toml
plugins: rerunfailures-15.0, xdist-3.6.1
collecting ...
------------------------------------------------- live log collection --------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.7.0 available.
collected 7 items / 5 deselected / 2 selected

test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype0-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [ 50%]
test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_5_vl-32-0.0001-dtype2-0.001-0.01-0.1-0.01-0.01-0.01] PASSED [100%]

fp32/test_mini_models

❯ python3 -m pytest test/convergence/fp32/test_mini_models.py -k vl
================================================= test session starts ==================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/tcc/Liger-Kernel
configfile: pyproject.toml
plugins: rerunfailures-15.0, xdist-3.6.1
collecting ...
------------------------------------------------- live log collection --------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.7.0 available.
collected 17 items / 15 deselected / 2 selected

test/convergence/fp32/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype7-1e-05-0.1-1-0.1-0.005-1e-05] PASSED [ 50%]
test/convergence/fp32/test_mini_models.py::test_mini_model[mini_qwen2_5_vl-32-0.0001-dtype8-1e-05-0.1-3-0.1-0.005-1e-05] PASSED [100%]

fp32/test_mini_models_multimodal

❯ python3 -m pytest test/convergence/fp32/test_mini_models_multimodal.py -k vl
================================================= test session starts ==================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/tcc/Liger-Kernel
configfile: pyproject.toml
plugins: rerunfailures-15.0, xdist-3.6.1
collecting ...
------------------------------------------------- live log collection --------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.7.0 available.
collected 7 items / 5 deselected / 2 selected

test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype0-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [ 50%]
test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_5_vl-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED [100%]

Hardware Type:
run make test to ensure correctness
run make checkstyle to ensure code style
run make test-convergence to ensure convergence

Resolve part of #729 format Signed-off-by: Tcc0403 <76503978+Tcc0403@users.noreply.github.com>

Signed-off-by: Tcc0403 <76503978+Tcc0403@users.noreply.github.com>

…ence test Signed-off-by: Tcc0403 <76503978+Tcc0403@users.noreply.github.com>

Tcc0403 · 2025-06-07T01:44:48Z

@shimizust @vaibhavjindal I saw you have a disscusion about transformers version in https://github.com/linkedin/Liger-Kernel/pull/743/files#r2130545912.
Do I need to cover 4.49.0<=transformers<=4.52.0 as well?

Also, most lce_deprecated_forward was made for transformers==4.44.2 (the only old version tested in ci). Perhaps we need a complete support matrix and add it to ci and readme for users.

shimizust · 2025-06-09T22:48:12Z

@Tcc0403 Yeah, we are planning to update the lowest compatible transformers version to 4.49 in CI, so if you can make it compatible that would be ideal. If not, we can add a check specifically for this model that says it's only compatible with transformers >= 4.52.?

The plan is to have the testing run against transformers 4.49, current release, and latest main branch (non-blocking).

We have an issue open with transformers to have model layers itself be responsible for patching with liger kernels to avoid all these issues, but discussion is still ongoing. In the meantime, we should update the documentation clearly with what models/compatible transformers versions are supported.

Tcc0403 · 2025-06-09T23:14:55Z

@shimizust ~~Gotcha! I'll make it compatible with 4.49.0.~~

Update:
To support transformers 4.49.0, there are so many version dispatching issue (no TextModel, video_processor, helper funcs, etc) across files. I suggest limiting qwen2_vl and qwen2_5_vl models to transformers >=4.52.4 which has text_config bug fix covered.

Signed-off-by: Tcc0403 <76503978+Tcc0403@users.noreply.github.com>

shimizust

I see, that makes sense. Yeah, in this case it doesn't seem worth the effort to support earlier versions given all the changes/bugfix

## Summary  `Qwen2VLVideoProcessor` requires torchvision. Related to #738 #723 (comment)  ## Testing Done   - Hardware Type: <BLANK> - [ ] run `make test` to ensure correctness - [ ] run `make checkstyle` to ensure code style - [ ] run `make test-convergence` to ensure convergence

fix: Fix qwen2_vl and qwen2_5_vl monkey patch

a20da9e

Resolve part of #729 format Signed-off-by: Tcc0403 <76503978+Tcc0403@users.noreply.github.com>

Tcc0403 force-pushed the tcc/qwen2-5-vl-monkeypatch branch from ac10b92 to a20da9e Compare June 1, 2025 13:26

Tcc0403 mentioned this pull request May 31, 2025

CI failure tracker #729

Open

17 tasks

Tcc0403 changed the title ~~fix: Fix qwen2_vl and qwen2_5_vl monkey patch~~ [WIP] fix: Fix qwen2_vl and qwen2_5_vl monkey patch Jun 1, 2025

Tcc0403 added 3 commits June 1, 2025 23:04

Add video processor

1267399

Signed-off-by: Tcc0403 <76503978+Tcc0403@users.noreply.github.com>

Fix lce_forward for qwen2_vl and qwen2_5_vl

04e089b

Signed-off-by: Tcc0403 <76503978+Tcc0403@users.noreply.github.com>

Add missing video processor for qwen2_5_vl in fp32 multimodal converg…

adb981c

…ence test Signed-off-by: Tcc0403 <76503978+Tcc0403@users.noreply.github.com>

Tcc0403 changed the title ~~[WIP] fix: Fix qwen2_vl and qwen2_5_vl monkey patch~~ fix: Fix qwen2_vl and qwen2_5_vl monkey patch Jun 1, 2025

Merge branch 'main' into tcc/qwen2-5-vl-monkeypatch

7993258

Tcc0403 requested review from shimizust and vaibhavjindal June 7, 2025 01:28

Tcc0403 force-pushed the tcc/qwen2-5-vl-monkeypatch branch from 11ca284 to 3deecfc Compare June 7, 2025 01:39

Tcc0403 added 2 commits June 10, 2025 08:26

Merge branch 'main' into tcc/qwen2-5-vl-monkeypatch

78fd22e

Add version check for qwen2-vl and qwen2.5-vl

55eb80c

Signed-off-by: Tcc0403 <76503978+Tcc0403@users.noreply.github.com>

Tcc0403 force-pushed the tcc/qwen2-5-vl-monkeypatch branch 2 times, most recently from 60909f1 to 55eb80c Compare June 10, 2025 03:23

shimizust approved these changes Jun 10, 2025

View reviewed changes

shimizust merged commit 53a37a5 into main Jun 10, 2025
8 of 14 checks passed

shimizust deleted the tcc/qwen2-5-vl-monkeypatch branch June 10, 2025 17:46

Tcc0403 mentioned this pull request Jun 11, 2025

build(dev-dependency): Add torchvision for multimodal test #755

Merged

3 tasks

Tcc0403 mentioned this pull request Jun 13, 2025

CI failure due to transformers VLM change #723

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Fix qwen2_vl and qwen2_5_vl monkey patch#738

fix: Fix qwen2_vl and qwen2_5_vl monkey patch#738
shimizust merged 7 commits intomainfrom
tcc/qwen2-5-vl-monkeypatch

Tcc0403 commented Jun 1, 2025 •

edited

Loading

Uh oh!

Tcc0403 commented Jun 7, 2025 •

edited

Loading

Uh oh!

shimizust commented Jun 9, 2025

Uh oh!

Tcc0403 commented Jun 9, 2025 •

edited

Loading

Uh oh!

shimizust left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Tcc0403 commented Jun 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

For monkey patch:

For convergence test:

Testing Done

Uh oh!

Tcc0403 commented Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shimizust commented Jun 9, 2025

Uh oh!

Tcc0403 commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shimizust left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Tcc0403 commented Jun 1, 2025 •

edited

Loading

Tcc0403 commented Jun 7, 2025 •

edited

Loading

Tcc0403 commented Jun 9, 2025 •

edited

Loading