Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detection

CVPR 2025

Wenqiao Li✶¹, Yao Gu✶¹, Xintao Chen✶¹, Xiaohao Xu², Ming Hu³, Xiaonan Huang² Yingna Wu¹

✶ indicates equal contribution

¹ShanghaiTech University    ²University of Michigan, Annor bor    ³Monash University

Website | Arxiv | Data

Table of Contents

• Overview
• Preparation
• Installation
• How to run
• Links to methods
• PAEval
• Contact
• Citation
• License

Overview

This repository is a benchmark for Phys-AD dataset, including unsupervised methods (MemAE, MNAD, MPN, SVM), weakly-supervised(MGFN, S3R, VadCLIP) and LLM based methods (VideoChatgpt, VideoLLaMA, VideoLLaVA, LAVAD, ZSCLIP, ZSImageBind)

Preparation

For all algorithms, in addition to the original data, we also need to prepare the following forms of data:

• frames
• clip features
• i3d features

For details and pre-trained weights downloading, please refer to here.

For some methods, some extra pre-process should be applied, please refer to here.

Installation

Install Dependencies

The environmental differences for the algorithm are quite significant. We have provided three environments for the algorithm, corresponding to the following algorithms:

For most methods included:

# Install Python dependencies
pip install -r requirements/requirements.txt

While there are 2 exceptions:

For LAVAD:

# Install Python dependencies
pip install -r requirements/requirements_lavad.txt

For Video-LLaVA:

# Install Python dependencies
pip install -r requirements/requirements_llava.txt

How to run

Make sure you have installed the right environment and all of the pretrained weights(especially for the LLM methods), and you can run the algorithms from the scripts under scripts folder.
For most of the methods there are related option files to set the parameters under the options folder. Among the params the data path and the object to be detected are two parameters you should modified according to your own setting.

Note: The script for LAVAD is a little different, where you need to modified the parameters in the script directly (data path and object are at the very beginning).

cd scripts
sh script_of_method_you_want_to_run.sh

Unsurpervised methods

For example, if you want to train and test MemAE method:

1. Switch to the scripts folder.

cd scripts

2. Find the scripts of the method you want to run. You may want to modify the flag --obj in the script to specify the object you want to train or test.
3. For training:

sh memae_trainer.sh

4. For testing:

sh memae_tester.sh

Weakly-surpervised methods

For example, if you want to train and test VadCLIP method:

0. Randomly put some abnormal samples into the training set (10% of the total abnormal samples in our experiments).
1. Switch to the scripts folder.

cd scripts

2. Find the scripts of the method you want to run. You may want to modify the flag --obj in the script to specify the object you want to train or test.
3. For training:

sh vadclip_trainer.sh

4. For testing:

sh vadclip_tester.sh

Video-understanding methods

For example, if you want to test VideoLLaVA method:

1. Switch to the scripts folder.

cd scripts

2. Find the scripts of the method you want to run. You may want to modify the flag --obj in the script to specify the object you want to test. Check the cuda device setting.
3. For testing:

sh videollava_tester.sh

Note:

1. In this project we use '_' to connect the name of an object, e.g.: 'rolling_bearing' for 'rolling bearing'.
2. Video understanding methods have only tester and no need to train.
3. All the results will be saved to results file and the trained models to checkpoints file.

PAEval

Please refer to here to check more instructions on PAEval experiments.

Links to methods

MemAE
MNAD
MPN
MGFN
S3R
VadCLIP
ZSCLIP
ZSImageBind
VideoChatgpt
VideoLLaMA
VideoLLaVA
LAVAD

Contact

Question about Data: chenxt12024@shanghaitech.edu.cn Question about Code: guyao2023@shanghaitech.edu.cn

Citation

Please cite the following paper if this work helps your project:

@article{li2025towards,
  title={Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detection},
  author={Li, Wenqiao and Gu, Yao and Chen, Xintao and Xu, Xiaohao and Hu, Ming and Huang, Xiaonan and Wu, Yingna},
  journal={arXiv preprint arXiv:2503.03562},
  year={2025}
}

License

MIT License

Copyright (c) 2025 Phys-AD

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.