A tested demonstration of C# floating vector insertion

[RockNHawk]

Important .NET version is implemented differently than the Java version, sometimes you can refer to the java version code and sometimes it doesn't work, you may need to check the .NET SDK source code to determine how to use it.

You should use DataType.FloatVector in Field Definition, but use BinaryVectorField when inserting.
Field<List<float>> is not worked.

    public static readonly FieldType TextVector = FieldType.Create("TextVector", DataType.FloatVector, new Dictionary<string, string>()
    {
        ["dim"] = TextVectorDim.ToString(),
    }, false);

List<Field> fields = new();
fields.Add(Field.Create(TextSpanCollectionDef.Id.Name, idArray));
fields.Add(new BinaryVectorField()
{
    FieldName = TextSpanCollectionDef.TextVector.Name,
    Datas = vectorArray.ToList(),
});

these folling code is not work for float vector:

fields.Add(Field.Create(TextSpanCollectionDef.TextVector.Name, book_intro_array));

         fields.Add(new Field<List<float>>()
        {
            FieldName = TextSpanCollectionDef.TextVector.Name,
            Datas = book_intro_array.ToList(),
        });

---- full codes ----

class TextSpanCollectionDef
{
    public const string CollectionName = "test";


    public static readonly FieldType Id = FieldType.Create("Id", DataType.VarChar, new Dictionary<string, string>()
    {
        ["max_length"] = "40",
    }, true);

    public const int TextVectorDim = 512;
    public static readonly FieldType TextVector = FieldType.Create("TextVector", DataType.FloatVector, new Dictionary<string, string>()
    {
        ["dim"] = TextVectorDim.ToString(),
    }, false);
    
    public static FieldType[] FieldDefs = new FieldType[]
    {
        Id,
        TextVector
    };

    static TextSpanCollectionDef()
    {
    }
}

#nullable enable
using System;
using System.Text.Json.Serialization;
using System.Threading.Tasks;
using Antlr4.Runtime.Misc;
using IO.Milvus.Client;
using IO.Milvus.Grpc;
using IO.Milvus.Param;
using IO.Milvus.Param.Collection;
using IO.Milvus.Param.Dml;
using IO.Milvus.Param.Index;
using IO.Milvus.Param.Dml.Field;
using Xunit.Abstractions;

namespace Xxxxx;

public class MilvusTest
{
    private readonly ITestOutputHelper _testOutputHelper;

    public MilvusTest(ITestOutputHelper testOutputHelper)
    {
        _testOutputHelper = testOutputHelper;
    }

    [Fact]
    public async Task MilvusConect()
    {
        var milvusClient = new MilvusServiceClient(ConnectParam.Create(host: "127.0.0.1", port: 19530));

        milvusClient.DropCollection(TextSpanCollectionDef.CollectionName);
        var createCollectionResult = milvusClient.CreateCollection(CreateCollectionParam.Create(TextSpanCollectionDef.CollectionName, 1, TextSpanCollectionDef.FieldDefs));
        _testOutputHelper.WriteLine(createCollectionResult.ToString());

        Random ran = new Random();
        List<string> idArray = new();
        List<List<float>> vectorArray = new();
        for (int i = 0; i < 2000; ++i)
        {
            idArray.Add(i.ToString());
            List<float> vector = new();
            for (int k = 0; k < TextSpanCollectionDef.TextVectorDim; ++k)
            {
                vector.Add(ran.Next());
            }

            vectorArray.Add(vector);
        }

        List<Field> fields = new();
        fields.Add(Field.Create(TextSpanCollectionDef.Id.Name, idArray));
        fields.Add(new BinaryVectorField()
        {
            FieldName = TextSpanCollectionDef.TextVector.Name,
            Datas = vectorArray.ToList(),
        });
        var insertRet = await milvusClient.InsertAsync(InsertParam.Create(TextSpanCollectionDef.CollectionName, "", fields));
        _testOutputHelper.WriteLine(insertRet.ToString());
    }
}

[weianweigan]

Sorry, there is an error here.

I released a new version(2.2.1-alpha.4) and created a pr for this.I tested it in notebooks, this is the code based on your code.

Create a client.

#r "nuget: IO.Milvus, 2.2.1-alpha.4"
#!import config/Settings.cs

using IO.Milvus;
using IO.Milvus.Client;
using IO.Milvus.Client.gRPC;
using IO.Milvus.Client.REST;
using InteractiveKernel = Microsoft.DotNet.Interactive.Kernel;

//Connect to milvus
(string endpoint,int port,string userName,string password) = Settings.LoadFromFile();

IMilvusClient milvusClient = default;

milvusClient = new MilvusGrpcClient(endpoint,port,userName,password);
milvusClient

Create a collection

using System.Diagnostics;

string collectionName = "test";

//Check if this collection exists
var hasCollection = await milvusClient.HasCollectionAsync(collectionName);

if(hasCollection){
    await milvusClient.DropCollectionAsync(collectionName);
    Console.WriteLine("Drop collection {0}",collectionName);
}

var fieldTypes = new[] {
    FieldType.CreateVarchar("Id",40,isPrimaryKey:true),
    FieldType.CreateFloatVector("TextVector",512)};

await milvusClient.CreateCollectionAsync(
            collectionName,
            fieldTypes
        );

Insert vectors

Random ran = new Random();
List<string> idArray = new();
List<List<float>> vectorArray = new();
for (int i = 0; i < 2000; ++i)
{
    idArray.Add(i.ToString());
    List<float> vector = new();
    for (int k = 0; k < 512; ++k)
    {
        vector.Add(ran.Next());
    }

    vectorArray.Add(vector);
}

MilvusMutationResult result = await milvusClient.InsertAsync(collectionName,
    new Field[]
    {
        Field.Create(fieldTypes[0].Name,idArray),
        Field.CreateFloatVector(fieldTypes[1].Name,vectorArray),
    });

result

Result

Source File

This is the notebook i used.

test.zip

[weianweigan]

@RockNHawk You can find the notebooks here https://github.com/milvus-io/milvus-sdk-csharp/tree/dev-milvus2.x_muti_client/docs/notebooks

[RockNHawk]

Sorry, there is an error here.

I released a new version(2.2.1-alpha.4) and created a pr for this.I tested it in notebooks, this is the code based on your code.

Create a client.

#r "nuget: IO.Milvus, 2.2.1-alpha.4"
#!import config/Settings.cs

using IO.Milvus;
using IO.Milvus.Client;
using IO.Milvus.Client.gRPC;
using IO.Milvus.Client.REST;
using InteractiveKernel = Microsoft.DotNet.Interactive.Kernel;

//Connect to milvus
(string endpoint,int port,string userName,string password) = Settings.LoadFromFile();

IMilvusClient milvusClient = default;

milvusClient = new MilvusGrpcClient(endpoint,port,userName,password);
milvusClient

Create a collection

using System.Diagnostics;

string collectionName = "test";

//Check if this collection exists
var hasCollection = await milvusClient.HasCollectionAsync(collectionName);

if(hasCollection){
    await milvusClient.DropCollectionAsync(collectionName);
    Console.WriteLine("Drop collection {0}",collectionName);
}

var fieldTypes = new[] {
    FieldType.CreateVarchar("Id",40,isPrimaryKey:true),
    FieldType.CreateFloatVector("TextVector",512)};

await milvusClient.CreateCollectionAsync(
            collectionName,
            fieldTypes
        );

Insert vectors

Random ran = new Random();
List<string> idArray = new();
List<List<float>> vectorArray = new();
for (int i = 0; i < 2000; ++i)
{
    idArray.Add(i.ToString());
    List<float> vector = new();
    for (int k = 0; k < 512; ++k)
    {
        vector.Add(ran.Next());
    }

    vectorArray.Add(vector);
}

MilvusMutationResult result = await milvusClient.InsertAsync(collectionName,
    new Field[]
    {
        Field.Create(fieldTypes[0].Name,idArray),
        Field.CreateFloatVector(fieldTypes[1].Name,vectorArray),
    });

result

Result

Source File

This is the notebook i used.

test.zip

Chinese ? Milvus 是国产吗，我看到 gitee 上有主页。

[weianweigan]

@RockNHawk 是的 made in china.

[RockNHawk]

@RockNHawk 是的 made in china.

It's so cool !!

[FrasWasim]

Hi, currently testing the code given above (for the REST client) with this deployment:

Milvus Version : 2.2.8 locally deployed using docker compose
Milvus C# SDK Version : 2.2.1-alpha.4

I am getting the following errors for:

1. GetLoadingProgressAsync - 404 Error
2. SearchAsync - 400 Bad Request

I tested the search directly with Attu visual client and works correctly.

I assume it's just because of the version difference, could you state in docs which milvus version this SDK supports or is tested on. Can I assume its correlated, as in I can only use Milvus Version 2.2.1 at the moment?

Thanks in advance!

[weianweigan]

@FrasWasim There is not a restful api that support GetLoadingProgressAsync method,so 404 throws.I will note it in the next version.About searchasync error, I don not any idea now.Can you give me more information about this error.

[FrasWasim]

@weianweigan Thanks for the quick response - since I was using custom code to test, I cloned this C# SDK source code, checked out your branch [dev-milvus2.x_muti_client], and ran the unit tests:

SearchTest fails for same reason - 400 bad request. The milvus I am using is the standalone:
https://github.com/milvus-io/milvus/releases/download/v2.2.8/milvus-standalone-docker-compose.yml

I tried v2.2.1 but no luck either.

[weianweigan]

@FrasWasim, I checked the code and found an issue with the JSON converter. The parameters in Milvus use the { key: "key", value: "value" } format, but C# does not. I forgot to add the [JsonConverter] attribute in SearchParameter, which caused a 400 error.

This is the error message:

{
    "error_code": 5,
    "reason": "bad request: parse body failed: json: cannot unmarshal object into Go struct field SearchRequest.search_params of type []*commonpb.KeyValuePair"
}

To fix this, I will add a custom JSON converter attribute in SearchRequest.cs:

    /// <summary>
    /// Search parameters
    /// </summary>
    [JsonPropertyName("search_params")]
    [JsonConverter(typeof(MilvusDictionaryConverter))]
    public IDictionary<string, string> SearchParams { get; set; } = new Dictionary<string, string>();

There is no relationship between the C# SDK version and Milvus version now. The C# SDK is currently targeting Milvus v2.2.x, and I am also using v2.2.8.

You can use MilvusGRPCClient now. The JSON schema that the RESTful API uses is not very clear, so I need to do more testing to ensure the response JSON format.