Add old InternVL2-1B/2B support to the InternVL conversion script #45092

[baonudesifeizhai]

What does this PR do?

This PR extends the InternVL conversion script to support the old OpenGVLab/InternVL2-1B and OpenGVLab/InternVL2-2B checkpoints.
These checkpoints currently rely on remote code and are problematic for downstream users on Transformers v5. Instead of instantiating the original remote-code models, the converter now reads the original config and weights directly and emits HF-native InternVLForConditionalGeneration checkpoints.

Fixes # (issue)
#45092
38425

before that : on vllm main branch:
pytest
tests/models/multimodal/generation/test_common.py
-k 'intern_vl2-hf-local and test_multi_image_models' -vv broken;

this branch:

VLLM_TEST_INTERNVL2_HF_MODEL=/tmp/InternVL2-1B-hf /root/venv/bin/python -m pytest
tests/models/multimodal/generation/test_common.py
-k 'intern_vl2-hf-local and test_multi_image_models' -vv passed

Validation

Ran the following with source ~/venv/bin/activate:

python src/transformers/models/internvl/convert_internvl_weights_to_hf.py \
  --input_dir OpenGVLab/InternVL2-1B \
  --output_dir /tmp/InternVL2-1B-hf

python src/transformers/models/internvl/convert_internvl_weights_to_hf.py \
  --input_dir OpenGVLab/InternVL2-2B \
  --output_dir /tmp/InternVL2-2B-hf

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

[ArthurZucker]

Happy to have!
Dynamic Weight Loader + attribute map + TokenizersBackend could potentially work OOB as well? if we add the model type to

transformers/src/transformers/models/auto/tokenization_auto.py

Line 64 in 013f76a

TOKENIZER_MAPPING_NAMES = OrderedDict[str, str | None](

it will skip remote code for the tokenizer at least

[hmellor]

I don't think we can use this to solve the vLLM side issue because it requires recreating the checkpoints

[baonudesifeizhai]

k i will try t dynamic Weight Loader + attribute map + TokenizersBackend...

I don't think we can use this to solve the vLLM side issue because it requires recreating the checkpoints

[baonudesifeizhai]

PYTHONPATH=/root/zdj/transformers/src
VLLM_TEST_INTERNVL2_HF_MODEL=/tmp/InternVL2-1B-hf
python -m pytest tests/models/multimodal/generation/test_common.py
-k 'intern_vl2-hf-local and test_single_image_models'
-vv -x -s

and

PYTHONPATH=/root/zdj/transformers/src
VLLM_TEST_INTERNVL2_HF_MODEL=/tmp/InternVL2-1B-hf
python -m pytest tests/models/multimodal/generation/test_common.py
-k 'intern_vl2-hf-local and test_multi_image_models'
-vv -x -s

passed on vllm

[baonudesifeizhai]

God ... https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45097&sha=c2e006 is that relative?

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, internvl

[baonudesifeizhai]

python -m pytest tests/models/multimodal/generation/test_common.py::test_single_image_models -vv -s -x -k "intern_vl-test_case25 or intern_vl-test_case26 or intern_vl-test_case27 or intern_vl-test_case28 or intern_vl-test_case29 or intern_vl-test_case30" passed on vllm side

[baonudesifeizhai]

https://paste.ubuntu.com/p/wJWpJwgvst/ passed python -m pytest tests/models/multimodal/generation/test_common.py::test_single_image_models -vv -s -x -k "intern_vl-test_case25 or intern_vl-test_case26 or intern_vl-test_case27 or intern_vl-test_case28 or intern_vl-test_case29 or intern_vl-test_case30"

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45097&sha=da3790

[baonudesifeizhai]

god will fix it tomorrow...