This repository is the official implementation for the paper titled "UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models".
pip install -r requirements.txt
conda create -n unimel python==3.8.18
conda activate unimelYou may download WikiMEL and RichpediaMEL from https://github.com/seukgcode/MELBench and WikiDiverse from https://github.com/wangxw5/wikiDiverse.
We provide a trained checkpoint "checkpoint-2200.zip", you just need to unzip it to the current directory and record its path.
If you want to train a new checkpoint, please refer to peft (https://github.com/huggingface/peft) or swift (https://github.com/modelscope/swift).
cd UniMEL
bash run.sh 0 wikidiverse # for wikidiverse
├─code
│ │ main.py
│ │
│ └─untils
│ │ dataset.py
│ │ functions.py
│ │
│ └─__pycache__
│ dataset.cpython-38.pyc
│ functions.cpython-38.pyc
│
└─config
wikidiverse.yaml
│ framework.png
│ README.md
│ requirements.txt
│ run.sh| Model | Richpedia | WikiMEL | Wikidiverse | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Top-1 | Top-5 | Top-10 | Top-20 | Top-1 | Top-5 | Top-10 | Top-20 | Top-1 | Top-5 | Top-10 | Top-20 | |
| BERT | 31.6 | 42.0 | 47.6 | 57.3 | 31.7 | 48.8 | 57.8 | 70.3 | 22.2 | 53.8 | 69.8 | 82.8 |
| BLINK | 30.8 | 38.8 | 44.5 | 53.6 | 30.8 | 44.6 | 56.7 | 66.4 | 22.4 | 50.5 | 68.4 | 76.6 |
| ARNN | 31.2 | 39.3 | 45.9 | 54.5 | 32.0 | 45.8 | 56.6 | 65.0 | - | - | - | - |
| DZMNED | 29.5 | 41.6 | 45.8 | 55.2 | 30.9 | 50.7 | 56.9 | 65.1 | - | 39.1 | - | - |
| JMEL | 29.6 | 42.3 | 46.6 | 54.1 | 31.3 | 49.4 | 57.9 | 64.8 | 21.9 | 54.5 | 69.9 | 76.3 |
| MEL-HI | 34.9 | 43.1 | 50.6 | 58.4 | 38.7 | 55.1 | 65.2 | 75.7 | 27.1 | 60.7 | 78.7 | 89.2 |
| HieCoAtt | 37.2 | 46.8 | 54.2 | 62.4 | 40.5 | 57.6 | 69.6 | 78.6 | 28.4 | 63.5 | 84.0 | 92.6 |
| GHMFC | 38.7 | 50.9 | 58.5 | 66.7 | 43.6 | 64.0 | 74.4 | 85.8 | - | - | - | - |
| MMEL | - | - | - | - | 71.5 | 91.7 | 96.3 | 98.0 | - | - | - | - |
| CLIP | 60.4 | 96.1 | 98.3 | 99.2 | 36.1 | 81.3 | 92.8 | 98.3 | 42.4 | 80.5 | 91.7 | 96.6 |
| DRIN | - | - | - | - | 65.5 | 91.3 | 95.8 | 97.7 | - | - | - | - |
| DWE | 67.6 | 97.1 | 98.6 | 99.5 | 44.7 | 65.9 | 80.8 | 93.2 | 47.5 | 81.3 | 92.0 | 96.9 |
| DWE+ | 72.5 | 97.3 | 98.8 | 99.6 | 72.8 | 97.5 | 98.9 | 99.7 | 51.2 | 91.0 | 96.3 | 98.9 |
| UniMEL (ours) | 94.8 | 97.9 | 98.3 | 98.8 | 94.1 | 97.2 | 98.4 | 98.9 | 92.9 | 97.0 | 99.5 | 99.8 |