Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: Fix for Chroma boolean values processing #2072

Merged
merged 6 commits into from
Jul 20, 2023

Conversation

dmytrostruk
Copy link
Member

@dmytrostruk dmytrostruk commented Jul 19, 2023

Motivation and Context

With latest updates in Chroma (Release 0.4.0), boolean values are automatically converted to numeric representation (False = 0, True = 1). This PR contains changes to handle this case and convert boolean values properly.

Fixes #2049

Description

  1. Implemented ChromaBooleanConverter to be used during Chroma response deserialization.
  2. Added more fixes based on new Chroma version - changed error message when trying to delete non-existent collection.
  3. Added integration test to verify boolean values processing.

Contribution Checklist

@dmytrostruk dmytrostruk requested a review from a team as a code owner July 19, 2023 17:50
@dmytrostruk dmytrostruk self-assigned this Jul 19, 2023
@shawncal shawncal added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory connector labels Jul 19, 2023
@dmytrostruk dmytrostruk added bug Something isn't working PR: ready for review All feedback addressed, ready for reviews ai connector Anything related to AI connectors labels Jul 19, 2023
@dmytrostruk dmytrostruk removed the bug Something isn't working label Jul 19, 2023
@shawncal shawncal added this pull request to the merge queue Jul 20, 2023
Merged via the queue into microsoft:main with commit b79f3a0 Jul 20, 2023
11 checks passed
@paolosalvatori
Copy link
Member

It looks like the problem is still there. I use the latest Microsoft.SemanticKernel.Connectors.Memory.Chroma, 0.24.230911.2-preview nuget package in the 09-memory-with-chroma.ipynb sample, and I get the following error:

**Error: System.Text.Json.JsonException: The JSON value could not be converted to Microsoft.SemanticKernel.Memory.MemoryRecordMetadata. Path: $.is_reference | LineNumber: 0 | BytePositionInLine: 102.
---> System.InvalidOperationException: Cannot get the value of a token type 'False' as a number.
at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_ExpectedNumber(JsonTokenType tokenType)
at System.Text.Json.Utf8JsonReader.TryGetInt16(Int16& value)
at Microsoft.SemanticKernel.Connectors.Memory.Chroma.ChromaBooleanConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryReadAsObject(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state, Object& value)
at System.Text.Json.Serialization.Converters.LargeObjectWithParameterizedConstructorConverter`1.ReadAndCacheConstructorArgument(ReadStack& state, Utf8JsonReader& reader, JsonParameterInfo jsonParameterInfo)
at System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
--- End of inner exception stack trace ---
at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
at Microsoft.SemanticKernel.Connectors.Memory.Chroma.ChromaMemoryStore.GetMetadataForMemoryRecord(List`1 metadatas, Int32 recordIndex)
at Microsoft.SemanticKernel.Connectors.Memory.Chroma.ChromaMemoryStore.GetMemoryRecordFromModel(List`1 metadatas, List`1 embeddings, List`1 ids, Int32 recordIndex)
at Microsoft.SemanticKernel.Connectors.Memory.Chroma.ChromaMemoryStore.GetMemoryRecordFromQueryResultModel(ChromaQueryResultModel queryResultModel, Int32 recordIndex)
at Microsoft.SemanticKernel.Connectors.Memory.Chroma.ChromaMemoryStore.GetNearestMatchesAsync(String collectionName, ReadOnlyMemory`1 embedding, Int32 limit, Double minRelevanceScore, Boolean withEmbeddings, CancellationToken cancellationToken)+MoveNext()
at Microsoft.SemanticKernel.Connectors.Memory.Chroma.ChromaMemoryStore.GetNearestMatchesAsync(String collectionName, ReadOnlyMemory`1 embedding, Int32 limit, Double minRelevanceScore, Boolean withEmbeddings, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Microsoft.SemanticKernel.Memory.SemanticTextMemory.SearchAsync(String collection, String query, Int32 limit, Double minRelevanceScore, Boolean withEmbeddings, CancellationToken cancellationToken)+MoveNext()
at Microsoft.SemanticKernel.Memory.SemanticTextMemory.SearchAsync(String collection, String query, Int32 limit, Double minRelevanceScore, Boolean withEmbeddings, CancellationToken cancellationToken)+MoveNext()
at Microsoft.SemanticKernel.Memory.SemanticTextMemory.SearchAsync(String collection, String query, Int32 limit, Double minRelevanceScore, Boolean withEmbeddings, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.Linq.AsyncEnumerable.<TryGetFirst>g__Core|95_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/FirstOrDefault.cs:line 130
at System.Linq.AsyncEnumerable.<TryGetFirst>g__Core|95_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/FirstOrDefault.cs:line 132
at System.Linq.AsyncEnumerable.<FirstOrDefaultAsync>g__Core|91_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/FirstOrDefault.cs:line 30
at Submission#6.<<Initialize>>d__0.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)**

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ai connector Anything related to AI connectors kernel Issues or pull requests impacting the core kernel memory connector .NET Issue or Pull requests regarding .NET code PR: ready for review All feedback addressed, ready for reviews
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Chroma .NET MemoryRecordMetadata field 'is_reference' is boolean but is saved as number in ChromaDB
4 participants