Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify compatibility between Data2VecVision models and existing retrievers #2865

Closed
ZanSara opened this issue Jul 21, 2022 · 2 comments · Fixed by #2891
Closed

Verify compatibility between Data2VecVision models and existing retrievers #2865

ZanSara opened this issue Jul 21, 2022 · 2 comments · Fixed by #2891
Assignees
Labels
topic:images topic:modeling topic:retriever type:feature New feature or request type:refactor Not necessarily visible to the users

Comments

@ZanSara
Copy link
Contributor

ZanSara commented Jul 21, 2022

Context

  • Part of Add support for images #2418
  • After the simplification of language_model.py and tokenization.py, adding new supported model types in Haystack has been heavily simplified
  • The entire framework is still oriented heavily towards question answering on text, and this assumption is embedded into the code in many parts of the stack

Goal

  • Verify if any existing retriever can load a image retrieval model such as Data2VecVision with minor changes along the way
    • If it can, consider a small refactoring to make the code paths more generic (change get_tokenizer into get_feature_extractor and so on)
    • If it cannot in its current state, even with minor adaptation, consider creating a separate ImageRetriever class that can do that. Also evaluate if the underlying stack (Inferencer, Processor, AdaptiveModel etc) can be leveraged or not, and to which degree.
@ZanSara ZanSara added type:feature New feature or request topic:modeling type:refactor Not necessarily visible to the users topic:retriever topic:images labels Jul 21, 2022
@ZanSara ZanSara self-assigned this Jul 21, 2022
@ZanSara ZanSara mentioned this issue Jul 21, 2022
8 tasks
@ZanSara ZanSara changed the title Verify compatibility between Data2VecVision models and Embedding Retriever Verify compatibility between Data2VecVision models and existing retrievers Jul 21, 2022
@ZanSara ZanSara mentioned this issue Jul 27, 2022
3 tasks
@ZanSara
Copy link
Contributor Author

ZanSara commented Jul 27, 2022

An attempt to generalize TableTextRetriever to work with images quickly proved too complex for the scope of this issue.

Rather than modifying an existing Retriever with the risk of breaking working code, I opted for cloning TableTextRetriever and its stack of supporting classes and perform the changes needed to support N models rather than just 3 (query, text and tables).

The goal of this issue then changes to the following:

  • Create a multi modal retriever called MultiModalRetriever by generalizing the concepts introduced by TableTextRetriever
  • It introduces a stack of new subclasses to support such retriever, such as:
    • MultiAdaptiveModel (from TriAdaptiveModel)
    • EmbeddingSimilarityHead (from TextSimilarityHead)
    • MultiModalSimilarityProcessor (from TableTextSimilarityProcessor)

Note that this Retriever will NOT be tested for working in pipelines, but only to work in isolation. It will also, most likely, stay undocumented. See #2418 for the rationale.

@ZanSara
Copy link
Contributor Author

ZanSara commented Aug 30, 2022

Continues in #2857

@ZanSara ZanSara closed this as completed Aug 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic:images topic:modeling topic:retriever type:feature New feature or request type:refactor Not necessarily visible to the users
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant