CLIP4CirDemo

CLIP for Conditioned image retrieval Demo

Live Demo available here

Training code available at

• Demo paper at CVPR 2022
  

• Follow-up paper presented at CVPR 2022 workshop O-DRUM
  

Table of Contents

• About the Project
  • Built With
• Getting Started
  • Prerequisites
  • Installation
• Usage
  • Data Preparation
  • Feature Extraction
  • Run the Demo
• Demo Overview
• Authors
• Citation

About The Project

This is the official repository of the paper Effective conditioned and composed image retrieval combining CLIP-based features accepted for the Demo Track at CVPR 2022.

If you are interested in Conditioned and Composed image retrieval take a look at our follow-up work Conditioned and composed image retrieval combining and partially fine-tuning CLIP-based features accepted at CVPR 2022 workshop O-DRUM

Conditioned and composed image retrieval extend CBIR systems by combining a query image with an additional text that expresses the intent of the user, describing additional requests w.r.t. the visual content of the query image. This type of search is interesting for e-commerce applications, e.g. to develop interactive multimodal searches and chatbots.

In this demo, we present an interactive system based on a combiner network, trained using contrastive learning, that combines visual and textual features obtained from the OpenAI CLIP network to address conditioned CBIR. The system can be used to improve e-shop search engines. For example, considering the fashion domain it lets users search for dresses, shirts and toptees using a candidate start image and expressing some visual differences w.r.t. its visual content, e.g. asking to change color, pattern or shape.

The proposed network obtains state-of-the-art performance on the FashionIQ dataset and on the more recent CIRR dataset, showing its applicability to the fashion domain for conditioned retrieval, and to more generic content considering the more general task of composed image retrieval.

Built With

• Python
• PyTorch
• Torchvision
• CLIP
• Flask
• Bootstrap

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

We strongly recommend the use of the Anaconda package manager in order to avoid dependency/reproducibility problems. A conda installation guide for linux systems can be found here

Installation

1. Clone the repo

git clone https://github.com/ABaldrati/CLIP4CirDemo

2. Install Python dependencies

conda create -n clip4cir -y python=3.8
conda activate clip4cir
conda install -y -c pytorch pytorch=1.7.1 torchvision=0.8.2
pip install flask==2.0.2
pip install git+https://github.com/openai/CLIP.git

3. Download FashionIQ and CIRR datasets

Usage

Here's a brief description of each and every file and folder in the repo:

• utils.py: Utils file
• model.py: Combiner model definition file
• data_utils.py: Dataset loading and preprocessing utils file
• extract_features.py: Feature extraction file
• hubconf.py: Torch Hub config file
• app.py: Flask server file
• static: Flask static files folder
• templates: Flask templates folder

Data Preparation

To properly work with the codebase FashionIQ and CIRR datasets should have the following structure:

project_base_path
└───  fashionIQ_dataset
      └─── captions
            | cap.dress.test.json
            | cap.dress.train.json
            | cap.dress.val.json
            | ...
            
      └───  images
            | B00006M009.jpg
            | B00006M00B.jpg
            | B00006M6IH.jpg
            | ...
            
      └─── image_splits
            | split.dress.test.json
            | split.dress.train.json
            | split.dress.val.json
            | ...

└───  cirr_dataset       
       └─── dev
            | dev-0-0-img0.png
            | dev-0-0-img1.png
            | dev-0-1-img0.png
            | ...
       
       └─── test1
            | test1-0-0-img0.png
            | test1-0-0-img1.png
            | test1-0-1-img0.png 
            | ...
       
       └─── cirr
            └─── captions
                | cap.rc2.test1.json
                | cap.rc2.train.json
                | cap.rc2.val.json
                
            └─── image_splits
                | split.rc2.test1.json
                | split.rc2.train.json
                | split.rc2.val.json

Feature Extraction

Before launching the demo it is necessary to extract the features using the following command

python extract_features.py

Run the Demo

Start the server and run the demo using the following command

python app.py

By default, the server run on port 5000 of localhost address: http://127.0.0.1:5000/

Demo overview

• Initially choose the dataset you want to experience with. As the image shown by the image you can experience with CIRR dataset or FashionIQ dataset

• Choose the reference image

• Choose or manually insert the relative caption

• Check out the results. By clicking on a retrieved image you can use such image as reference image in a new query

Authors

• Alberto Baldrati
• Marco Bertini
• Tiberio Uricchio
• Alberto Del Bimbo

Citation

If you find this code to be useful for your research, please consider citing this DEMO paper

@inproceedings{baldrati2022effective,
  title={Effective Conditioned and Composed Image Retrieval Combining CLIP-Based Features},
  author={Baldrati, Alberto and Bertini, Marco and Uricchio, Tiberio and Del Bimbo, Alberto},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={21466--21474},
  year={2022}
}

If you are interested in Conditioned and Composed image retrieval take a look at our follow-up work: Conditioned and Composed Image Retrieval Combining and Partially Fine-Tuning CLIP-Based Features

@inproceedings{baldrati2022conditioned,
  title={Conditioned and Composed Image Retrieval Combining and Partially Fine-Tuning CLIP-Based Features},
  author={Baldrati, Alberto and Bertini, Marco and Uricchio, Tiberio and Del Bimbo, Alberto},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={4959--4968},
  year={2022}
}

Take also a look to our most recent work: Zero-Shot Composed Image Retrieval with Textual Inversion

@misc{baldrati2023zeroshot,
      title={Zero-Shot Composed Image Retrieval with Textual Inversion}, 
      author={Alberto Baldrati and Lorenzo Agnolucci and Marco Bertini and Alberto Del Bimbo},
      year={2023},
      eprint={2303.15247},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgements

This work was partially supported by the European Commission under European Horizon 2020 Programme, grant number 101004545 - ReInHerit.