.Net: Bug: IVectorStore::GetCollection throws away HasNamedVectors

[lilhoser]

Describe the bug
When retrieving an existing collection, the GetCollection API creates a new QdrantVectorStoreRecordCollectionOptions object that passes the caller's VectorStoreRecordDefinition parameter but does not consider HasNamedVectors which is a crucial collection option for Qdrant:

var recordCollection = new QdrantVectorStoreRecordCollection<TRecord>(this._qdrantClient, name, new QdrantVectorStoreRecordCollectionOptions<TRecord>() { VectorStoreRecordDefinition = vectorStoreRecordDefinition });

Source

This also throws away other options that might have been specified when the caller created the collection, such as PointStructCustomMapper.

As a result, the default value of HasNamedVectors is used, which is false. Later vector searches that try to search on a specific named vector causes an exception:

One or more errors occurred. (Call to vector store failed.)

To Reproduce
Steps to reproduce the behavior:

1. Create a collection:

var collection = new QdrantVectorStoreRecordCollection<MyRecord>(
    new QdrantClient("localhost"),
    "mycollection",
    new()
    {
        HasNamedVectors = true,
        PointStructCustomMapper = new Mapper(),
        VectorStoreRecordDefinition =RecordDefinition
    });
await collection.CreateCollectionAsync();

2. Use IVectorStore.GetCollection to retrieve that collection - notice that the retrieved collection has HasNamedVectors set to false even though it was created with true:

var store = m_Kernel.GetRequiredService<IVectorStore>();
var collection = store.GetCollection<ulong, MyRecord>("mycollection", RecordDefinition);

3. Issue a vector search:

var embedding = m_Kernel.GetRequiredService<ITextEmbeddingGenerationService>();
var searchVector = await embedding.GenerateEmbeddingAsync(query);
var searchOptions = new VectorSearchOptions
{
    VectorPropertyName = nameof(RecordDefinition.MyEmbedding)
};
var collection = store.GetCollection<ulong, MyRecord>("mycollection", RecordDefinition);
var searchResult = await collection.VectorizedSearchAsync(searchVector, searchOptions);

Expected behavior
GetCollection should allow the caller to pass in the QdrantVectorStoreRecordCollectionOptions it used to create the collection, or better yet, this should not be a required parameter at all and SK should figure out how to retrieve the collection instead of creating new parameters that alters the behavior of the collection.

Platform

• OS: Windows 11
• IDE: VS 2022
• Language: C#

[westey-m]

@lilhoser, thanks for filing a bug, I've created a PR to fix the issue where HasNamedVectors was not passed from VectorStore To Collection. With regards to your design suggestion:

GetCollection should allow the caller to pass in the QdrantVectorStoreRecordCollectionOptions it used to create the collection

Since GetCollection is part of the abstraction and doesn't know anything about Qdrant it cannot be modified to allow Qdrant options be provided. There is of course the option of adding overloads of GetCollection that are only on the concrete implementations e.g. QdrantVectorStore and not on IVectorStore, which would allow passing Qdrant options, but this would live outside of the abstraction. We have been considering this and if there is interest we can certainly add these.

this should not be a required parameter at all and SK should figure out how to retrieve the collection instead of creating new parameters that alters the behavior of the collection

While it is possible to make a request to Qdrant on first use, each time a collection object was created, to determine whether it is using named or unnamed vectors, it isn't good to subject all consumers to this overhead. This also doesn't address the scenario where a developer wants to use a specific mode, and are using SK to create the collection. In this case there isn't a way to know which mode the developer prefers without the developer specifying it.

[lilhoser]

Appreciate those details!

It's unfortunate that to get a collection object out of Qdrant, you have to specify its configuration (QdrantVectorStoreRecordCollectionOptions), because when/if this class ever changes, all places that instantiate this class will need to be updated to propagate correct values to the new field(s).

Perhaps the overload is the way to go (a generic GetCollection<T> ?), but I'm not quite familiar enough with some of the SK abstractions to offer better suggestions.

[westey-m]

Created a feature for adding concrete implementation overloads #9537