Issue: Pinecone Vector Store incorrectly assigns score in similarity_search_with_relevance_scores

[dangoldbj]

Issue you'd like to raise.

With the Pinecone index setup with cosine metric, so the scores received from the vector database are already in thecosine metric.

However, the _select_relevance_score_fn of langchain pinecone vector store returns the base vector store's _cosine_relevance_score_fn for the cosine distance strategy.

langchain/libs/langchain/langchain/vectorstores/pinecone.py

Line 231 in b213850

def _select_relevance_score_fn(self) -> Callable[[float], float]:

In the base vector store implementation of _cosine_relevance_score_fn the score is the complement.

https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/schema/vectorstore.py#L187

So the cosine metric of 0.81 returned by the pinecone vector database indicating a higher similarity is turned into 0.19 in langchain and processed as very low similarity.

Suggestion:

No response

[dosubot]

🤖

Hello @dangoldbj,

I'm here to assist you with your issue. I'm an AI bot designed to help solve bugs, answer questions, and guide you in becoming a contributor while you're waiting for a human maintainer.

From your description, it seems like the issue is with the _cosine_relevance_score_fn method in the Pinecone vector store of LangChain. This method is used to calculate the relevance score of a document in relation to a query when the distance strategy is COSINE. However, as you've pointed out, the score is being complemented, which is causing a higher similarity score to be processed as a very low similarity.

There have been similar issues raised in the past. For instance, Issue #11587 was about the similarity_search_with_relevance_scores not working properly with Pinecone. The suggested solution was to provide a custom relevance_score_fn during the initialization of the ScaNN class that transforms the raw scores in a way that suits your needs. Here's an example of how you can do this:

class PineconeCustom(Pinecone):
    def _cosine_relevance_score_fn(distance: float) -> float:
        """Normalize the distance to a score on a scale [0, 1]."""
        return (1.0 + distance)/2

vectorstore = PineconeCustom(index, embed.embed_query, text_field)
....

This code creates a new class PineconeCustom that inherits from the Pinecone class. It overrides the _cosine_relevance_score_fn method to return a score on a scale from 0 to 1. This score is calculated by adding 1.0 to the distance and dividing by 2. This should give you higher similarity scores.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Best,
Dosu

Sources

• When search_type="similarity_score_threshold, retriever returns negative scores
• similarity_search_with_relevance_scores is not working properly with Pinecone
• libs/langchain/langchain/vectorstores/pinecone.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot]

Hi, @dangoldbj,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, you raised an issue regarding the incorrect assignment of scores in the similarity_search_with_relevance_scores function in the Pinecone Vector Store. There was a suggestion for a potential solution involving the customization of the relevance_score_fn during the initialization of the ScaNN class. However, it seems that the issue remains unresolved.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, kindly let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!