ImageRetriever

[ZanSara]

[Part of #2418]

What
ImageRetriever is a node that is capable of performing two operations:

• Creating comparable embeddings for text and images
• Comparing such embeddings to retrieve images that correspond to a given description.

How
ImageRetriever could be based on CLIP, which is available through sentence-transformers.

Given that ImageRetriever has clearly two responsibilities, we can consider splitting them between ImageEmbedder (only creates and stores the embeddings) and ImageRetriever (computes the embeddings for the query only and uses the pre-existing image embeddings from the document stores) #2403

Expected results

• A pipeline should be capable to take a string as input and output a list of paths to images.
• All document stores should be able to support image embeddings

NOT expected (yet):

• Pipelines capable of returning mixed images and text results with comparable scores.

This new node is likely to be implemented in a single PR.

[julian-risch]

@ZanSara I think we could make use of Data2Vec from transformers: https://huggingface.co/docs/transformers/main/en/model_doc/data2vec That would allow to do the implementation of the embedding calculation in the EmbeddingRetriever in just a few lines of code.

[julian-risch]

Data2VecVision has been recently added: https://github.com/huggingface/transformers/pull/16760/files
and Data2VecText and Data2VecAudio are already there.

[julian-risch]

@ZanSara There was a transformers release and Data2VecVision has also been part of the release: https://github.com/huggingface/transformers/releases

[masci]

Superseded by #2418