Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: cpu-e2e failure: failed to set up mvccTs for search iterator with newest #36809

Closed
1 task done
zhengbuqian opened this issue Oct 12, 2024 · 2 comments
Closed
1 task done

[Bug]: cpu-e2e failure: failed to set up mvccTs for search iterator with newest #36809

zhengbuqian opened this issue Oct 12, 2024 · 2 comments
Assignees
@MrPresent-Han
Labels
ci/e2e kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@zhengbuqian
Copy link
Collaborator

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master
- SDK version(e.g. pymilvus v2.0.0rc2): since pymilvus-2.5.0rc87

Current Behavior

I tried to update pymilvus used by CI to 2.5.0rc93 in #36780 and the ci failed on test test_search_iterator_binary. Did a binary search and it should fail since milvus-io/pymilvus#2279.

Error:

(.venv) ➜  python_client git:(pymilvus-empty-sparse-check) ✗ pytest -k test_search_iterator_binary -s
[get_env_variable] failed to get environment variables : 'CI_LOG_PATH', use default path : /tmp/ci_logs
[create_path] folder(/tmp/ci_logs) is not exist.
[create_path] create path now...
Test session starts (platform: linux, Python 3.8.10, pytest 7.2.0, pytest-sugar 0.9.5)
cachedir: .pytest_cache
metadata: {'Python': '3.8.10', 'Platform': 'Linux-5.15.0-122-generic-x86_64-with-glibc2.29', 'Packages': {'pytest': '7.2.0', 'pluggy': '1.5.0'}, 'Plugins': {'metadata': '3.1.1', 'sugar': '0.9.5', 'repeat': '0.8.0', 'Faker': '19.2.0', 'rerunfailures': '9.1.1', 'print': '0.2.1', 'forked': '1.6.0', 'allure-pytest': '2.7.0', 'html': '3.1.1', 'assume': '2.4.3', 'tags': '1.8.1', 'xdist': '2.5.0', 'cov': '2.8.1', 'random-order': '1.1.1', 'level': '0.1.1', 'parallel': '0.1.1', 'timeout': '1.3.3'}}
tags: all
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
rootdir: /home/zilliz/milvus/tests/python_client, configfile: pytest.ini
plugins: metadata-3.1.1, sugar-0.9.5, repeat-0.8.0, Faker-19.2.0, rerunfailures-9.1.1, print-0.2.1, forked-1.6.0, allure-pytest-2.7.0, html-3.1.1, assume-2.4.3, tags-1.8.1, xdist-2.5.0, cov-2.8.1, random-order-1.1.1, level-0.1.1, parallel-0.1.1, timeout-1.3.3
selected 1 items


――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestSearchIterator.test_search_iterator_binary ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

self = <test_search.TestSearchIterator object at 0x7fb63696eee0>

    @pytest.mark.tags(CaseLabel.L1)
    def test_search_iterator_binary(self):
        """
        target: test search iterator binary
        method: 1. search iterator
                2. check the result, expect pk
        expected: search successfully
        """
        # 1. initialize with data
        batch_size = 200
        collection_w = self.init_collection_general(
            prefix, True, is_binary=True)[0]
        # 2. search iterator
        _, binary_vectors = cf.gen_binary_vectors(2, ct.default_dim)
>       collection_w.search_iterator(binary_vectors[:1], binary_field_name,
                                     ct.default_search_binary_params, batch_size,
                                     check_task=CheckTasks.check_search_iterator,
                                     check_items={"batch_size": batch_size})

testcases/test_search.py:10111: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
utils/wrapper.py:33: in inner_wrapper
    res, result = func(*args, **kwargs)
base/collection_wrapper.py:204: in search_iterator
    check_result = ResponseChecker(res, func_name, check_task, check_items, check,
check/func_check.py:61: in run
    result = self.check_search_iterator(self.response, self.func_name, self.check_items)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

search_res = Error(code=1, message=failed to set up mvccTs for search iterator), func_name = 'search_iterator', check_items = {'batch_size': 200}

    @staticmethod
    def check_search_iterator(search_res, func_name, check_items):
        """
        target: check the search iterator results
        method: 1. check the iterator number
                2. check the limit(topK) and ids
                3. check the distance
        expected: check the search is ok
        """
        log.info("search_iterator_results_check: checking the searching results")
        if func_name != 'search_iterator':
            log.warning("The function name is {} rather than {}".format(func_name, "search_iterator"))
        search_iterator = search_res
        pk_list = []
        while True:
>           res = search_iterator.next()
E           AttributeError: 'Error' object has no attribute 'next'

check/func_check.py:388: AttributeError
------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured log setup --------------------------------------------------------------------------------------------------------------------------------------------------------------
[2024-10-12 13:36:44 - INFO - ci_test]: ################################################################################ (conftest.py:232)
[2024-10-12 13:36:44 - INFO - ci_test]: [initialize_milvus] Log cleaned up, start testing... (conftest.py:233)
[2024-10-12 13:36:44 - INFO - ci_test]: [setup_class] Start setup class... (client_base.py:40)
[2024-10-12 13:36:44 - INFO - ci_test]: *********************************** setup *********************************** (client_base.py:46)
[2024-10-12 13:36:44 - INFO - ci_test]: [setup_method] Start setup test case test_search_iterator_binary. (client_base.py:47)
-------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------------------------------------------------------
[2024-10-12 13:36:44 - INFO - ci_test]: Test case of search interface: initialize before test case (client_base.py:257)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : False  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [Connections.connect] args: ['default', '', '', 'default', ''], kwargs: {'host': 'localhost', 'port': 19530} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['int64', <DataType.INT64: 5>, ''], kwargs: {'is_primary': False, 'is_partition_key': False, 'nullable': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>}  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['varchar', <DataType.VARCHAR: 21>, ''], kwargs: {'max_length': 65535, 'is_primary': False, 'is_partition_key': False, 'nullable': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'name': 'varchar', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}}  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['float_vector', <DataType.FLOAT_VECTOR: 101>, ''], kwargs: {'dim': 128, 'is_primary': False, 'nullable': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['float', <DataType.FLOAT: 10>, ''], kwargs: {'is_primary': False, 'nullable': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['json_field', <DataType.JSON: 23>, ''], kwargs: {'is_primary': False, 'nullable': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'name': 'json_field', 'description': '', 'type': <DataType.JSON: 23>}  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [CollectionSchema] args: [[{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}}, {'name': 'json_field', 'description': '', 'type': <DataTyp......, kwargs: {'primary_field': 'int64', 'auto_id': False, 'enable_dynamic_field': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params......  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['int64', <DataType.INT64: 5>, ''], kwargs: {'is_primary': False, 'nullable': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>}  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['float', <DataType.FLOAT: 10>, ''], kwargs: {'is_primary': False, 'nullable': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['varchar', <DataType.VARCHAR: 21>, ''], kwargs: {'max_length': 65535, 'is_primary': False, 'nullable': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'name': 'varchar', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}}  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['binary_vector', <DataType.BINARY_VECTOR: 100>, ''], kwargs: {'dim': 128, 'is_primary': False, 'nullable': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'name': 'binary_vector', 'description': '', 'type': <DataType.BINARY_VECTOR: 100>, 'params': {'dim': 128}}  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [CollectionSchema] args: [[{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}}, {'name': 'binary_vector', 'description': '', 'type': <Data......, kwargs: {'primary_field': 'int64', 'auto_id': False} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params......  (api_request.py:37)
[2024-10-12 13:36:44 - INFO - ci_test]: init_collection_general: collection creation (client_base.py:301)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : True  (api_request.py:37)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['search_collection_GWDSaKuz', {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': ......, kwargs: {'consistency_level': 'Strong'} (api_request.py:62)
[2024-10-12 13:36:44 - DEBUG - ci_test]: (api_response) : <Collection>:
-------------
<name>: search_collection_GWDSaKuz
<description>: 
<schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FL......  (api_request.py:37)
[2024-10-12 13:36:44 - INFO - ci_test]: inserting 2000 data into collection search_collection_GWDSaKuz (common_func.py:2577)
[2024-10-12 13:36:44 - DEBUG - ci_test]: Dynamic field is enabled: False (common_func.py:2582)
[2024-10-12 13:36:45 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [      int64   float varchar                                      binary_vector
0         0     0.0       0  b'!"\xcb\x85\xa5\xf3\x81\xf5))X\xb7T\xc5\xc1\xfe'
1         1     1.0       1    b'-\xbdf\xd2j\xb6\x07D\xabZ\xa22\xc8\x17\n\x84'
2         2     2.0       2  b'\xa7\xe7<\xa5\x16H\xfdZ\x17/\xb......, kwargs: {'timeout': 180} (api_request.py:62)
[2024-10-12 13:36:45 - DEBUG - ci_test]: (api_response) : (insert count: 2000, delete count: 0, upsert count: 0, timestamp: 453171322579058692, success count: 2000, err count: 0  (api_request.py:37)
[2024-10-12 13:36:45 - INFO - ci_test]: inserted 2000 data into collection search_collection_GWDSaKuz (common_func.py:2647)
[2024-10-12 13:36:45 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)
[2024-10-12 13:36:48 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-10-12 13:36:48 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)
[2024-10-12 13:36:55 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-10-12 13:36:55 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['binary_vector', {'index_type': 'BIN_FLAT', 'params': {}, 'metric_type': 'JACCARD'}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-10-12 13:36:56 - DEBUG - ci_test]: (api_response) : Status(code=0, message=)  (api_request.py:37)
[2024-10-12 13:36:56 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-10-12 13:36:57 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-10-12 13:36:57 - DEBUG - ci_test]: (api_request)  : [Collection.search_iterator] args: [[b'\tJ\xffA\xd2\xe8Ls\xab\x8e:\x01\x949\xc2f'], 'binary_vector', {'metric_type': 'JACCARD', 'params': {'nprobe': 32}}, 200, -1, None, None, None, 180, -1], kwargs: {} (api_request.py:62)
[2024-10-12 13:36:57 - ERROR - ci_test]: Traceback (most recent call last):
  File "/home/zilliz/milvus/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
    res = func(*args, **_kwargs)
  File "/home/zilliz/milvus/tests/python_client/utils/api_request.py", line 63, in api_request
    return func(*arg, **kwargs)
  File "/home/zilliz/milvus/tests/python_client/.venv/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 976, in search_iterator
    return SearchIterator(
  File "/home/zilliz/milvus/tests/python_client/.venv/lib/python3.8/site-packages/pymilvus/orm/iterator.py", line 425, in __init__
    self.__init_search_iterator()
  File "/home/zilliz/milvus/tests/python_client/.venv/lib/python3.8/site-packages/pymilvus/orm/iterator.py", line 431, in __init_search_iterator
    raise MilvusException(message="failed to set up mvccTs for search iterator")
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to set up mvccTs for search iterator)>
 (api_request.py:45)
[2024-10-12 13:36:57 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=failed to set up mvccTs for search iterator)> (api_request.py:46)
[2024-10-12 13:36:57 - INFO - ci_test]: search_iterator_results_check: checking the searching results (func_check.py:382)

 testcases/test_search.py::TestSearchIterator.test_search_iterator_binary ⨯                                                                                                                                                                                                                                                     100% ██████████
-------------------------------------------------------------------------------------------------------------------------------------------- generated html file: file:///tmp/ci_logs/report.html ---------------------------------------------------------------------------------------------------------------------------------------------
=========================================================================================================================================================== short test summary info ===========================================================================================================================================================
FAILED testcases/test_search.py::TestSearchIterator::test_search_iterator_binary - AttributeError: 'Error' object has no attribute 'next'

Results (20.19s):
       1 passed
       1 failed
         - testcases/test_search.py:10097 TestSearchIterator.test_search_iterator_binary
    8585 deselected

Expected Behavior

No response

Steps To Reproduce

in `tests/python_client`, run `pytest -k test_search_iterator_binary -s`

Milvus Log

No response

Anything else?

No response
@zhengbuqian zhengbuqian added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 12, 2024
@zhengbuqian
Copy link
Collaborator Author

cc @XuanYang-cn

@zhengbuqian zhengbuqian mentioned this issue Oct 12, 2024
Closed
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. ci/e2e and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 12, 2024
@yanliang567 yanliang567 added this to the 2.5.0 milestone Oct 12, 2024
@MrPresent-Han
Copy link
Contributor

fixed: milvus-io/pymilvus#2297

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MrPresent-Han MrPresent-Han

Labels
ci/e2e kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
3 participants
@zhengbuqian @yanliang567 @MrPresent-Han