[Paper]
Taolin Zhang, Jinpeng Wang, Hang Guo, Tao Dai, Bin Chen and Shu-Tao Xia
Abstract: Adaptation of pretrained vision-language models such as CLIP to various downstream tasks have raised great interest in recent researches. Previous works have proposed a variety of test-time adaptation (TTA) methods to achieve strong generalization without any knowledge of the target domain. However, existing training-required TTA approaches like TPT necessitate entropy minimization that involves large computational overhead, while training-free methods like TDA overlook the potential for information mining from the test samples themselves. In this paper, we break down the design of existing popular training-required and training-free TTA methods and bridge the gap between them within our framework. Specifically, we maintain a light-weight key-value memory for feature retrieval from instance-agnostic historical samples and instance-aware boosting samples. The historical samples are filtered from the testing data stream and serve to extract useful information from the target distribution, while the boosting samples are drawn from regional bootstrapping and capture the knowledge of the test sample itself. We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets, showcasing its applicability in real-world situations.
This codebase was tested with the following environment configurations. It may work with other versions.
- Python 3.9
- CUDA 11.8
- PyTorch 2.2.0 + cu118
Please follow Datasets to download the OOD and Cross-Domain benchmarks.
Run the following script to run BoostAdapter:
bash scripts/run.sh $GPU $EXP_NAME
For example, you can run the a demo with the following command:
bash scripts/run.sh 0 demo
Please cite us if our work is useful for your research.
@article{zhang2024boostadapter,
title={Boostadapter: Improving vision-language test-time adaptation via regional bootstrapping},
author={Zhang, Taolin and Wang, Jinpeng and Guo, Hang and Dai, Tao and Chen, Bin and Xia, Shu-Tao},
journal={Advances in Neural Information Processing Systems},
volume={37},
pages={67795--67825},
year={2024}
}
This project is released under the MIT license.
This code is based on TDA. Thanks for their awesome work.
If you have any questions, feel free to approach me at zhangtlin3@gmail.com.