Introduce SentenceTransformer Reranker

[machatschek]

Description

This PR introduces support for document reranking, specifically leveraging SentenceTransformer cross-encoders. The addition aims to provide a lightweight and optimized approach to reranking, enhancing the model's response quality and speed.

Motivation

The integration of a reranking feature addresses the need for more relevant and accurate responses by pre-filtering documents before answer generation. The choice of SentenceTransformer's cross-encoder as the reranker is motivated by its efficiency and effectiveness in identifying the most relevant documents, compared to traditional LLM-based reranking methods.

Changes Made

• Added an optional SentenceTransformerRerank node_postprocessor to ChatService, which facilitates the reranking process using the SentenceTransformer cross-encoder.
• Updated settings to include rerank-specific configurations, allowing users to enable and customize reranking according to their specific needs. This includes parameters such as model and top_n, where the latter controls the selection process of documents for final response generation.
• Documentation updates to guide users through enabling reranking, installing necessary dependencies (poetry install --extras rerank-sentence-transformers), and configuring rerank settings effectively for optimal performance.

The reranking feature is disabled by default to accommodate the additional dependencies and the need for users to adjust configurations based on their unique use cases.

Future Considerations

Looking ahead, there's potential to further refine the reranking functionality by:

• Creating a dedicated reranker component: This would facilitate the incorporation of different reranker types, enhancing flexibility and customization for users.
• Expanding reranker support: Exploring and integrating additional reranker methods beyond the SentenceTransformer cross-encoder could provide users with more options to tailor the reranking process to their specific requirements.

[pabloogc]

nice job, useful feature too 👏

[imartinez]

Nice contribution!

[danielgallegovico]

Neat contribution!

[icsy7867]

Does llamacpp-python need to be rebuilt for gpu? I would assume so. I've offloaded all of my gpu stuff, so I might have to update my pipeline if so @machatschek

[machatschek]

Does llamacpp-python need to be rebuilt for gpu? I would assume so. I've offloaded all of my gpu stuff, so I might have to update my pipeline if so @machatschek

The SentenceTransformer reranker does not rely on llamacpp-python. If you want to run the reranker on GPU, you would need to install GPU-enabled PyTorch version. SentenceTransfromer will then use GPU by default if one is available. If you want to overwrite this behaviour, you can set the device parameter in SentenceTransformerRerank to "cpu".