[Bug]: [Nightly] Range search iterator with radius only will fail reporting range_filter < radius #29305

NicoYuan1986 · 2023-12-19T02:41:30Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: 95ada1b
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):     kafka
- SDK version(e.g. pymilvus v2.0.0rc2): 2.3.3.post1.dev8
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Range search iterator with radius only will fail reporting range_filter < radius.

[2023-12-18T19:56:01.589Z] __________ TestSearchIterator.test_range_search_iterator_only_radius ___________
[2023-12-18T19:56:01.589Z] [gw3] linux -- Python 3.8.17 /usr/local/bin/python3
[2023-12-18T19:56:01.589Z] 
[2023-12-18T19:56:01.589Z] self = <test_search.TestSearchIterator object at 0x7fb15e512a30>
[2023-12-18T19:56:01.589Z] 
[2023-12-18T19:56:01.589Z]     @pytest.mark.tags(CaseLabel.L2)
[2023-12-18T19:56:01.589Z]     def test_range_search_iterator_only_radius(self):
[2023-12-18T19:56:01.589Z]         """
[2023-12-18T19:56:01.589Z]         target: test search iterator normal
[2023-12-18T19:56:01.589Z]         method: 1. search iterator
[2023-12-18T19:56:01.589Z]                 2. check the result, expect pk not repeat and meet the expr requirements
[2023-12-18T19:56:01.589Z]         expected: search successfully
[2023-12-18T19:56:01.589Z]         """
[2023-12-18T19:56:01.589Z]         # 1. initialize with data
[2023-12-18T19:56:01.589Z]         batch_size = 100
[2023-12-18T19:56:01.589Z]         collection_w = self.init_collection_general(prefix, True, is_index=False)[0]
[2023-12-18T19:56:01.589Z]         collection_w.create_index(field_name, {"metric_type": "L2"})
[2023-12-18T19:56:01.589Z]         collection_w.load()
[2023-12-18T19:56:01.589Z]         # 2. search iterator
[2023-12-18T19:56:01.589Z]         search_params = {"metric_type": "L2", "params": {"radius": 35.0}}
[2023-12-18T19:56:01.589Z] >       collection_w.search_iterator(vectors[:1], field_name, search_params, batch_size,
[2023-12-18T19:56:01.589Z]                                      check_task=CheckTasks.check_search_iterator,
[2023-12-18T19:56:01.589Z]                                      check_items={"metric_type": "L2",
[2023-12-18T19:56:01.589Z]                                                   "radius": 35.0})
[2023-12-18T19:56:01.589Z] 
[2023-12-18T19:56:01.589Z] testcases/test_search.py:9506: 
[2023-12-18T19:56:01.589Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2023-12-18T19:56:01.589Z] utils/wrapper.py:33: in inner_wrapper
[2023-12-18T19:56:01.589Z]     res, result = func(*args, **kwargs)
[2023-12-18T19:56:01.589Z] base/collection_wrapper.py:188: in search_iterator
[2023-12-18T19:56:01.589Z]     check_result = ResponseChecker(res, func_name, check_task, check_items, check,
[2023-12-18T19:56:01.589Z] check/func_check.py:61: in run
[2023-12-18T19:56:01.589Z]     result = self.check_search_iterator(self.response, self.func_name, self.check_items)
[2023-12-18T19:56:01.589Z] check/func_check.py:366: in check_search_iterator
[2023-12-18T19:56:01.589Z]     res = search_iterator.next()
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/orm/iterator.py:484: in next
[2023-12-18T19:56:01.589Z]     new_page = self.__try_search_fill()
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/orm/iterator.py:502: in __try_search_fill
[2023-12-18T19:56:01.589Z]     new_page = self.__execute_next_search(next_params, next_expr, True)
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/orm/iterator.py:521: in __execute_next_search
[2023-12-18T19:56:01.589Z]     res = self._conn.search(
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:129: in handler
[2023-12-18T19:56:01.589Z]     raise e from e
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:125: in handler
[2023-12-18T19:56:01.589Z]     return func(*args, **kwargs)
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:164: in handler
[2023-12-18T19:56:01.589Z]     return func(self, *args, **kwargs)
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:104: in handler
[2023-12-18T19:56:01.589Z]     raise e from e
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:68: in handler
[2023-12-18T19:56:01.589Z]     return func(*args, **kwargs)
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py:748: in search
[2023-12-18T19:56:01.589Z]     return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py:709: in _execute_search
[2023-12-18T19:56:01.589Z]     raise e from e
[2023-12-18T19:56:01.589Z] /usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py:702: in _execute_search
[2023-12-18T19:56:01.589Z]     check_status(response.status)
[2023-12-18T19:56:01.589Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2023-12-18T19:56:01.589Z] 
[2023-12-18T19:56:01.589Z] status = error_code: UnexpectedError
[2023-12-18T19:56:01.589Z] reason: "fail to search on QueryNode 3: worker(3) query failed: UnknownError: Assert \"ran...re/src/common/RangeSearchHelper.cpp:125\n => range_filter must be less than radius for L2/HAMMING/JACCARD"
[2023-12-18T19:56:01.589Z] code: 65535
[2023-12-18T19:56:01.589Z] 
[2023-12-18T19:56:01.589Z] 
[2023-12-18T19:56:01.589Z]     def check_status(status: Status):
[2023-12-18T19:56:01.589Z]         if status.code != 0 or status.error_code != 0:
[2023-12-18T19:56:01.589Z] >           raise MilvusException(status.code, status.reason, status.error_code)
[2023-12-18T19:56:01.589Z] E           pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=fail to search on QueryNode 3: worker(3) query failed: UnknownError: Assert "range_filter < radius" at /go/src/github.com/milvus-io/milvus/internal/core/src/common/RangeSearchHelper.cpp:125
[2023-12-18T19:56:01.590Z] E            => range_filter must be less than radius for L2/HAMMING/JACCARD)>
[2023-12-18T19:56:01.590Z]

Expected Behavior

pass

Steps To Reproduce

No response

Milvus Log

link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/591/pipeline/205
log: artifacts-milvus-distributed-kafka-nightly-591-pymilvus-e2e-logs.tar.gz
collection name: search_collection_KDtq5PRY
failed time: [2023-12-18T19:55:16.842Z] [gw3] [ 99%] FAILED testcases/test_search.py::TestSearchIterator::test_range_search_iterator_only_radius

Anything else?

No response

The text was updated successfully, but these errors were encountered:

yanliang567 · 2023-12-19T02:56:03Z

/unassign

czs007 · 2023-12-31T10:08:23Z

17.768Z] self = <test_search.TestSearchIterator object at 0x7f9444560cd0>

[2023-12-31T10:02:17.768Z]

[2023-12-31T10:02:17.768Z] @pytest.mark.tags(CaseLabel.L1)

[2023-12-31T10:02:17.768Z] def test_range_search_iterator_COSINE(self):

[2023-12-31T10:02:17.768Z] """

[2023-12-31T10:02:17.768Z] target: test iterator range search

[2023-12-31T10:02:17.768Z] method: 1. search iterator

[2023-12-31T10:02:17.768Z] 2. check the result, expect pk not repeat and meet the expr requirements

[2023-12-31T10:02:17.768Z] expected: search successfully

[2023-12-31T10:02:17.768Z] """

[2023-12-31T10:02:17.768Z] # 1. initialize with data

[2023-12-31T10:02:17.768Z] batch_size = 100

[2023-12-31T10:02:17.768Z] collection_w = self.init_collection_general(prefix, True, is_index=False)[0]

[2023-12-31T10:02:17.768Z] collection_w.create_index(field_name, {"metric_type": "COSINE"})

[2023-12-31T10:02:17.768Z] collection_w.load()

[2023-12-31T10:02:17.768Z] # 2. search iterator

[2023-12-31T10:02:17.768Z] search_params = {"metric_type": "COSINE", "params": {"radius": 0.8, "range_filter": 1}}

[2023-12-31T10:02:17.768Z] > collection_w.search_iterator(vectors[:1], field_name, search_params, batch_size,

[2023-12-31T10:02:17.768Z] check_task=CheckTasks.check_search_iterator,

[2023-12-31T10:02:17.768Z] check_items={"metric_type": "COSINE",

[2023-12-31T10:02:17.768Z] "radius": 0.8,

[2023-12-31T10:02:17.768Z] "range_filter": 1})

[2023-12-31T10:02:17.768Z]

[2023-12-31T10:02:17.768Z] testcases/test_search.py:9486:

[2023-12-31T10:02:17.768Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

[2023-12-31T10:02:17.768Z] utils/wrapper.py:33: in inner_wrapper

[2023-12-31T10:02:17.768Z] res, result = func(*args, **kwargs)

[2023-12-31T10:02:17.768Z] base/collection_wrapper.py:188: in search_iterator

[2023-12-31T10:02:17.768Z] check_result = ResponseChecker(res, func_name, check_task, check_items, check,

[2023-12-31T10:02:17.768Z] check/func_check.py:61: in run

[2023-12-31T10:02:17.768Z] result = self.check_search_iterator(self.response, self.func_name, self.check_items)

[2023-12-31T10:02:17.768Z] check/func_check.py:366: in check_search_iterator

[2023-12-31T10:02:17.768Z] res = search_iterator.next()

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/orm/iterator.py:487: in next

[2023-12-31T10:02:17.768Z] new_page = self.__try_search_fill()

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/orm/iterator.py:505: in __try_search_fill

[2023-12-31T10:02:17.768Z] new_page = self.__execute_next_search(next_params, next_expr, True)

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/orm/iterator.py:524: in __execute_next_search

[2023-12-31T10:02:17.768Z] res = self._conn.search(

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:129: in handler

[2023-12-31T10:02:17.768Z] raise e from e

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:125: in handler

[2023-12-31T10:02:17.768Z] return func(*args, **kwargs)

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:164: in handler

[2023-12-31T10:02:17.768Z] return func(self, *args, **kwargs)

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:104: in handler

[2023-12-31T10:02:17.768Z] raise e from e

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/decorators.py:68: in handler

[2023-12-31T10:02:17.768Z] return func(*args, **kwargs)

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py:748: in search

[2023-12-31T10:02:17.768Z] return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py:709: in _execute_search

[2023-12-31T10:02:17.768Z] raise e from e

[2023-12-31T10:02:17.768Z] /usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py:702: in _execute_search

[2023-12-31T10:02:17.768Z] check_status(response.status)

[2023-12-31T10:02:17.768Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

[2023-12-31T10:02:17.768Z]

[2023-12-31T10:02:17.768Z] status = error_code: UnexpectedError

[2023-12-31T10:02:17.768Z] reason: "fail to search on QueryNode 1: worker(1) query failed: Assert "range_filter > ra...cpp:122\n => range_filter must be greater than radius for IP/COSINE: service internal error: UnknownError"

[2023-12-31T10:02:17.768Z] code: 65535

[2023-12-31T10:02:17.768Z]

[2023-12-31T10:02:17.768Z] def check_status(status: Status):

[2023-12-31T10:02:17.768Z] if status.code != 0 or status.error_code != 0:

[2023-12-31T10:02:17.768Z] > raise MilvusException(status.code, status.reason, status.error_code)

[2023-12-31T10:02:17.768Z] E pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=fail to search on QueryNode 1: worker(1) query failed: Assert "range_filter > radius" at /go/src/github.com/milvus-io/milvus/internal/core/src/common/RangeSearchHelper.cpp:122

[2023-12-31T10:02:17.768Z] E => range_filter must be greater than radius for IP/COSINE: service internal error: UnknownError)>

same behavior for test case : test_range_search_iterator_COSINE

NicoYuan1986 · 2024-01-11T06:57:27Z

reproduce:
https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/614/pipeline/237

stale · 2024-02-10T19:14:09Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

yanliang567 · 2024-02-17T12:15:14Z

keep it

xiaofan-luan · 2024-02-18T16:46:11Z

@cydrain this might be the similar case

NicoYuan1986 · 2024-03-22T06:43:00Z

The issue has not reproduced for a long time. So close it first.
I will reopen it once it occurs.
latest image: master-20240321-09281a07-amd64

NicoYuan1986 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 19, 2023

NicoYuan1986 added this to the 2.4.0 milestone Dec 19, 2023

NicoYuan1986 assigned yanliang567 and MrPresent-Han Dec 19, 2023

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 19, 2023

yanliang567 removed their assignment Dec 19, 2023

MrPresent-Han mentioned this issue Jan 5, 2024

[Bug]: Zero width cause search failure for iterator milvus-io/pymilvus#1856

Open

1 task

stale bot added the stale indicates no udpates for 30 days label Feb 10, 2024

stale bot removed the stale indicates no udpates for 30 days label Feb 17, 2024

NicoYuan1986 closed this as completed Mar 22, 2024

PiercarloSlavazza mentioned this issue Apr 29, 2024

Iterators and range_filter error (when no range filter is specified) milvus-io/milvus-sdk-java#881

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: [Nightly] Range search iterator with radius only will fail reporting range_filter < radius #29305

[Bug]: [Nightly] Range search iterator with radius only will fail reporting range_filter < radius #29305

NicoYuan1986 commented Dec 19, 2023

yanliang567 commented Dec 19, 2023

czs007 commented Dec 31, 2023

NicoYuan1986 commented Jan 11, 2024

stale bot commented Feb 10, 2024

yanliang567 commented Feb 17, 2024

xiaofan-luan commented Feb 18, 2024

NicoYuan1986 commented Mar 22, 2024

[Bug]: [Nightly] Range search iterator with radius only will fail reporting range_filter < radius #29305

[Bug]: [Nightly] Range search iterator with radius only will fail reporting range_filter < radius #29305

Comments

NicoYuan1986 commented Dec 19, 2023

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

yanliang567 commented Dec 19, 2023

czs007 commented Dec 31, 2023

NicoYuan1986 commented Jan 11, 2024

stale bot commented Feb 10, 2024

yanliang567 commented Feb 17, 2024

xiaofan-luan commented Feb 18, 2024

NicoYuan1986 commented Mar 22, 2024