universal-search: Query Group API and local mode #690

agourlay · 2024-07-12T09:54:33Z

This PR adds the client API and local mode for the new group query API.

I took the opportunity to increase the gRPC coverage but it seems to create issues with newest Python versions.

netlify · 2024-07-12T09:55:00Z

✅ Deploy Preview for poetic-froyo-8baba7 ready!

Name	Link
🔨 Latest commit	`f84b0c1`
🔍 Latest deploy log	https://app.netlify.com/sites/poetic-froyo-8baba7/deploys/66b62f8a8bcfc80008a7012c
😎 Deploy Preview	https://deploy-preview-690--poetic-froyo-8baba7.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

agourlay · 2024-07-15T18:25:18Z

qdrant_client/conversions/conversion.py

@@ -2523,8 +2529,8 @@ def convert_vector_input(cls, model: rest.VectorInput) -> grpc.VectorInput:
    @classmethod
    def convert_recommend_input(cls, model: rest.RecommendInput) -> grpc.RecommendInput:
        return grpc.RecommendInput(
-            positive=[cls.convert_vector_input(vector) for vector in model.positive],
-            negative=[cls.convert_vector_input(vector) for vector in model.negative],
+            positive=[cls.convert_vector_input(vector) for vector in model.positive] if model.positive is not None else None,


fix existing bug

coszio · 2024-07-18T14:42:28Z

tests/congruence_tests/test_query.py

I think it is still valuable to test grpc too. To edit this easier though, we could set the pytestmark global variable instead of repeating it for each test

pytestmark = pytest.mark.parametrize("prefer_grpc", [False, True])

I agree, just wanted to take it out of this PR because it triggers issues unrelated to this PR 👍

As the date time bug has been merged I re-pushed a bit of gRPC testing.
7f4f843

It found right away a bug in the with_lookup implementation :)

qdrant_client/async_qdrant_client.py

qdrant_client/client_base.py

qdrant_client/qdrant_client.py

qdrant_client/qdrant_remote.py

qdrant_client/conversions/conversion.py

qdrant_client/local/local_collection.py

qdrant_client/local/qdrant_local.py

joein · 2024-07-18T14:19:29Z

qdrant_client/local/local_collection.py

+                prefetch=[set_prefetch_limit_recursively(p, limit) for p in prefetch.prefetch],
+            )
+        else:
+            return types.Prefetch(limit=limit, prefetch=list())


nit:

Suggested change

return types.Prefetch(limit=limit, prefetch=list())

return rest_models.Prefetch(limit=limit, prefetch=list())

this does not work, it can't find such module 🤔

joein · 2024-07-18T14:20:56Z

qdrant_client/local/local_collection.py

+def set_prefetch_limit_recursively(prefetch: types.Prefetch, limit: int) -> types.Prefetch:
+    if prefetch is not None:
+        if isinstance(prefetch.prefetch, list):
+            return types.Prefetch(


nit:

Suggested change

return types.Prefetch(

return rest_models.Prefetch(

qdrant_client/http/models/models.py

coszio · 2024-07-18T20:43:10Z

qdrant_client/local/qdrant_local.py

+                with_lookup_collection = self._get_collection(with_lookup)
+            else:
+                with_lookup_collection = self._get_collection(with_lookup.collection)
+


we need to resolve ids into vectors at this point, considering that we add lookup_from parameter first

added and tested in 2e52841 :)

qdrant_client/qdrant_remote.py

agourlay · 2024-07-22T11:30:41Z

Dear reviewers, I believe I have addressed all your comments 🫡

agourlay · 2024-07-23T08:06:14Z

Let's keep this PR open to not make cherry picking too difficult in case of patch release.

We could stack other PRs on top of it for the next release.

coszio

Happy to approve after the description edit :)

qdrant_client/qdrant_client.py

Co-authored-by: Luis Cossío <[email protected]>

* pre-implement dbsf * add dbsf congruence tests * mypy lints * add conversions * tests: add test for dbsf conversion --------- Co-authored-by: George Panchuk <[email protected]>

* pre-implement random sampling * generate models * add conversions and tests * fix mypy lints * tests: add test for sample random conversion * use camelcase Sample.Random * review fixes * fix mypy --------- Co-authored-by: George Panchuk <[email protected]>

agourlay · 2024-08-09T07:51:12Z

@joein Thanks for handling the merges 👏

* add parametrized fixture for using grpc too * compare grpc and http without running each setup twice * fix: fix exception types in invalid types test * fix: remove random seed which led to a erroneous sequence --------- Co-authored-by: Luis Cossío <[email protected]>

* universal-search: Query Group API and local mode * requires different logic for gRPC error assertion * you come to me at runtime for a compile time issue * suddenly throwing a different error * test more group key types * extend limit of prefetches during group_by * rescoring is the issue * one problem at a time please * code review * regen clients * code review * add lookup_from to query_points_groups * test with_lookup * test and fix gRPC * drop dedicated conversion * Update qdrant_client/qdrant_client.py Co-authored-by: Luis Cossío <[email protected]> * regen async * Distribution-based score fusion in local mode (#703) * pre-implement dbsf * add dbsf congruence tests * mypy lints * add conversions * tests: add test for dbsf conversion --------- Co-authored-by: George Panchuk <[email protected]> * Random sample in local mode (#705) * pre-implement random sampling * generate models * add conversions and tests * fix mypy lints * tests: add test for sample random conversion * use camelcase Sample.Random * review fixes * fix mypy --------- Co-authored-by: George Panchuk <[email protected]> * fix: add type ignore for mypy * fix: fix type hints for 3.8 * fix: do not run mypy on async client generator in CI, simplify condition * Grpc comparison in tests (#726) * add parametrized fixture for using grpc too * compare grpc and http without running each setup twice * fix: fix exception types in invalid types test * fix: remove random seed which led to a erroneous sequence --------- Co-authored-by: Luis Cossío <[email protected]> --------- Co-authored-by: Luis Cossío <[email protected]> Co-authored-by: George Panchuk <[email protected]>

agourlay commented Jul 15, 2024

View reviewed changes

agourlay force-pushed the universal-search-query-group-API branch from f15898f to 9715a26 Compare July 16, 2024 07:07

agourlay marked this pull request as ready for review July 17, 2024 13:48

agourlay requested a review from joein July 17, 2024 13:48

agourlay force-pushed the universal-search-query-group-API branch 4 times, most recently from 4c05a1f to de1c9ed Compare July 18, 2024 11:18

agourlay requested a review from coszio July 18, 2024 13:39

coszio reviewed Jul 18, 2024

View reviewed changes

qdrant_client/async_qdrant_client.py Outdated Show resolved Hide resolved

joein reviewed Jul 18, 2024

View reviewed changes

coszio requested changes Jul 18, 2024

View reviewed changes

agourlay force-pushed the universal-search-query-group-API branch from d5e5e00 to 0f782d5 Compare July 22, 2024 10:11

agourlay commented Jul 22, 2024

View reviewed changes

qdrant_client/qdrant_remote.py Show resolved Hide resolved

agourlay requested review from joein and coszio July 22, 2024 11:29

coszio mentioned this pull request Jul 22, 2024

Distribution-based score fusion in local mode #703

Merged

joein approved these changes Jul 22, 2024

View reviewed changes

coszio reviewed Jul 23, 2024

View reviewed changes

qdrant_client/qdrant_client.py Outdated Show resolved Hide resolved

coszio approved these changes Jul 26, 2024

View reviewed changes

agourlay added 6 commits August 8, 2024 22:46

universal-search: Query Group API and local mode

fb56f26

requires different logic for gRPC error assertion

6613e10

you come to me at runtime for a compile time issue

df99117

suddenly throwing a different error

00d5a3e

test more group key types

c2b016d

extend limit of prefetches during group_by

8695972

agourlay and others added 13 commits August 8, 2024 22:46

rescoring is the issue

acee32a

one problem at a time please

377893b

code review

e753d61

regen clients

ed2352d

code review

290b96f

add lookup_from to query_points_groups

b9d1c56

test with_lookup

38e01a8

test and fix gRPC

70602ba

drop dedicated conversion

4875dc7

Update qdrant_client/qdrant_client.py

d9aeae3

Co-authored-by: Luis Cossío <[email protected]>

regen async

d969a30

Distribution-based score fusion in local mode (#703)

5f11d89

* pre-implement dbsf * add dbsf congruence tests * mypy lints * add conversions * tests: add test for dbsf conversion --------- Co-authored-by: George Panchuk <[email protected]>

joein force-pushed the universal-search-query-group-API branch from 9f31c70 to dc585b3 Compare August 8, 2024 21:32

joein added 3 commits August 8, 2024 23:47

fix: add type ignore for mypy

2bd182a

fix: fix type hints for 3.8

008c683

fix: do not run mypy on async client generator in CI, simplify condition

dde2edb

joein merged commit 44bbded into dev Aug 9, 2024
13 of 14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

universal-search: Query Group API and local mode #690

universal-search: Query Group API and local mode #690

agourlay commented Jul 12, 2024 •

edited

Loading

netlify bot commented Jul 12, 2024 •

edited

Loading

agourlay Jul 15, 2024

coszio Jul 18, 2024

agourlay Jul 18, 2024

agourlay Jul 22, 2024

joein Jul 18, 2024

agourlay Jul 22, 2024

joein Jul 18, 2024

coszio Jul 18, 2024

agourlay Jul 22, 2024

agourlay commented Jul 22, 2024

agourlay commented Jul 23, 2024

coszio left a comment

agourlay commented Aug 9, 2024

	return types.Prefetch(limit=limit, prefetch=list())
	return rest_models.Prefetch(limit=limit, prefetch=list())

universal-search: Query Group API and local mode #690

universal-search: Query Group API and local mode #690

Conversation

agourlay commented Jul 12, 2024 • edited Loading

netlify bot commented Jul 12, 2024 • edited Loading

✅ Deploy Preview for poetic-froyo-8baba7 ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agourlay commented Jul 22, 2024

agourlay commented Jul 23, 2024

coszio left a comment

Choose a reason for hiding this comment

agourlay commented Aug 9, 2024

agourlay commented Jul 12, 2024 •

edited

Loading

netlify bot commented Jul 12, 2024 •

edited

Loading