Skip to content

Latest commit

 

History

History
121 lines (72 loc) · 4.92 KB

README.md

File metadata and controls

121 lines (72 loc) · 4.92 KB
Raw

✂️ TRIM

Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs

Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, Benyou Wang*

Python 3.10+ Pytorch 2.1.1 transformers accelerate

🤗 Model | 📖 arXiv | GitHub

🌈 Update

  • [2024.11.30] 🎉🎉🎉 TRIM is accepted by COLING25!🎉🎉🎉
  • [2024.9.30] 🎉🎉🎉 TRIM is public!🎉🎉🎉

Contents

Introduction

We introduce new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance. Inspired by human attention patterns in Visual Question Answering (VQA) tasks, TRIM presents a fresh perspective on the selection and reduction of image tokens. The TRIM method has been extensively tested across 12 datasets, and the results demonstrate a significant reduction in computational overhead while maintaining a consistent level of performance. This research marks a critical stride in efficient MLLM development, promoting greater accessibility and sustainability of high-performing models.

Our approach

Preparation

🤖 Environment Setup

Please refer to LLaVA. 😊

Run

Step.0: Set the environment the same as LLaVA-1.5

Note that the core of our proposed module is here in the CLIP image encoder.

Step.1: Model preparation

Train model with TRIM

If you want to reproduce the result of the model trained with TRIM, configure the dataset path, vision_tower path, projecter path and LLM checkpoint path in the training script.

Please set reduce_func as TRIM, reduce_func_param as -1 for automatic selection.

bash scripts/finetune_8gpu_TRIM.sh

or Download checkpoints

If you want to use TRIM without training the model, please download the checkpoints from Huggingface liuhaotian/llava-v1.5-7b or Our fine-tuned model with TRIM.

Step.2 (for inference): Change the methods (TRIM).

If you wish to implement TRIM in another model, such as liuhaotian/llava-v1.5-7b in Huggingface, please add the following line to the config.json file in the model's directory.

    "mm_vision_token_reduce_func": "TRIM:-1",

Step.3 (for evaluation): Run the evaluation script.

If you want to reproduce the result in our paper, for all benchmark，the evaluation script is：

bash eval_all_benchmarks.sh

For example, the evaluation for TextVQA is:

bash scripts/v1_5/eval/testvqa.sh

For other inference scripts, refer to LLaVA Evaluation.

License

Code License Data License

All software is licensed under the Apache License, Version 2.0 (Apache 2.0). All other materials are licensed under the Creative Commons Attribution 4.0 International License (CC-BY).

Contact

Citation

If you find this repository helpful, please consider citing it:

@misc{song2024moresimpleeffectivetoken,
      title={Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs},
      author={Dingjie Song and Wenjun Wang and Shunian Chen and Xidong Wang and Michael Guan and Benyou Wang},
      year={2024},
      eprint={2409.10994},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.10994},
}