Chat-MedGen: Omni-Adaptation of Multi-Modal Large Language Models for Diverse Biomedical Tasks

(Knowledge-Based Systems 2025)

Qinyue Tong¹ · Ziqian Lu^2* · Zhe-ming Lu^1*
Yunlong Yu¹ · Jun Liu¹ · Yangming Zheng^1
1Zhejiang University, ²Zhejiang Sci-Tech University, ^*Corresponding author

📣 News

• 2025.9.30: We released the MedVision-MT benchmark! You can use it in your research or evaluate Chat-MedGen on it.
• 2025.9.27: We released the inference codes. Feel free to chat with Chat-MedGen on your own data!
• 2025.9.26: We released the pretrained weights of Chat-MedGen. You can find it in the Hugging Face.
• 2025.7.25: We created the Chat-MedGen project and upload the demo video of Chat-MedGen.

📝 ToDo List

• Release training codes.
• Release inference codes and Chat-MedGen model pre-trained weights.
• Release MedVision-MT benchmark.
• Release more visual results and video demo.

📷 Video Demo

MedVision-MT Benchmark

We release our MedVision-MT benchmark here to provide a standard evaluation and inspire further research in medical MLLMs.

Getting Started

Installation

1. Prepare the code and the environment

Git clone our repository, creating a python environment and activate it via the following command

git clone https://github.com/Edisonhimself/Chat-MedGen.git
cd Chat-MedGen
conda env create -f environment.yml
conda activate chatmedgen

2. Prepare the pretrained LLM weights

Chat-MedGen is based on Llama2-7B. Please first download the LLM weights from the following huggingface space: Download.

Then, set the variable llama_model in the model config file to the LLM weight path. Specifically, please set the LLM path here at Line 14.

3. Prepare our Chat-MedGen pretrained model checkpoints

Download the pretrained model checkpoints at Download.

Then, set the variable ckpt in the eval config file to the pretrained Chat-MedGen weight path. Specifically, please set the ckpt path here at Line 11.

Launching Chat-MedGen Locally and get a quick start

Please set your image and your query in here and run:

torchrun chat_medgen_eval.py --cfg-path eval_configs/chat_medgen_eval.yaml --gpu-id 0

We provide two examples in the examples folder to help you get started quickly.

Evaluate Chat-MedGen on MedVision-MT

You can first download the MedVision-MT dataset from here.

Notice: The downloaded MedVision-MT dataset only contains image metadata and corresponding annotations. To obtain the actual image files, please download them from the respective sub-datasets. We will provide a consolidated list of all sub-dataset download links in the near future to facilitate easier access.

Then, you can modify the TestEvalData class in the eval_demo.py file to point to the dataset you want to test. This will allow you to run the evaluation on your chosen dataset.

The structure of MedVision-MT is shown below:

MedVision-MT/
├── chestxray/
│   ├── train.json
│   └── test.json
├── iu_xray/
│   ├── train.json
│   └── test.json
├── slake/
│   ├── train.json
│   └── test.json
├── ...  # Other folders (folder4, folder5, ..., folder12)

✌️ More Examples

👏 Acknowledgements

This project is developed on the codebase of MiniGPT-4. We appreciate its valuable contributions!