Which One Are You Referring To? Multimodal Object Identification in Situated Dialogue

Holy Lovenia, Samuel Cahyawijaya

---

Abstract

The demand for multimodal dialogue system has been rising in various domains, emphasizing the importance of interpreting multimodal inputs from conversational and situational context. One main challenge in multimodal dialogue understanding is multimodal object identification, which constitutes the ability to identify objects relevant to a multimodal user-system conversation. We explore three methods to tackle this problem and evaluate them on SIMMC 2.1. Our best method, scene-dialogue alignment, improves the performance by ~20% F1-score compared to the SIMMC 2.1 baselines. We provide analysis and discussion regarding the limitation of our methods and the potential directions for future works.

Setup

• Pull the latest simmc2 and detr git submodules
• Install all required python dependency

pip install -r requirements.txt

• To obtain the preprocessed dialogue data, run ./run_data_preprocessing.sh
• To obtain the preprocessed visual embeddings from DETR, run ./run_object_detection_finetuning.sh and ./run_object_detection_prediction.sh
• To obtain the preprocessed visual embeddings from ResNet-50 for the SIMMC 2.1 baselines, download from the Google Drive provided by the organizers, then place the file under ./preprocessed_data/visual_features/
• To obtain all fine-tuned models, download from this OneDrive, then place the folder at ./save.

File Structure

• git submodules
  • simmc2
  • detr
• fonts -> store the fonts for visualizing detection results
• model
  • clipper.py -> implementation of the CLIPPER model
  • holy_detr.py -> implementation of the SitCoM-DETR model
  • model_utils.py -> modeling utility functions
• trainer
  • detr_trainer.py -> a customized HuggingFace trainer class for DETR-derived model fine-tuning
• utils
  • args_helper.py -> argument parser used in the main script
  • data_utils.py -> data loading functions for various model fine-tuning formats (CLIP, DETR, SitCoM-DETR, etc)
  • utils.py -> common utility functions
• contextualized_object_detection_finetuning.py -> fine-tuning script for the SitCoM-DETR model
• contextualized_object_detection_prediction.py -> prediction script for the SitCoM-DETR model
• mdetr_zero_shot.py -> zero-shot prediction script for MDETR
• object_detection_finetuning.py -> fine-tuning script for the DETR-based model
• object_detection_prediction.py -> prediction script for the DETR-based model
• conv_image_alignment_finetuning.py -> fine-tuning script for the CLIP-based model using conversation data
• text_image_alignment_finetuning.py -> fine-tuning script for the CLIP-based model using scene-generated text data
• text_image_alignment_prediction.py -> prediction script for the CLIP-based model
• run_.sh -> all the bash scripts to run the corresponding () experiment
• *.ipynb -> notebook files for sandbox and visualization

Experiment Results

All experiment results are recorded on the following Google Sheets.

LICENSE

The source code for the site is licensed under the MIT license, which you can find in the LICENSE file.