Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: The full-text search function allows inserting empty strings, but it does not support using empty strings as search data during search operations. #37022

Closed
1 task done
zhuwenxing opened this issue Oct 21, 2024 · 6 comments
Closed
1 task done

[Bug]: The full-text search function allows inserting empty strings, but it does not support using empty strings as search data during search operations. #37022

zhuwenxing opened this issue Oct 21, 2024 · 6 comments
Assignees
@zhuwenxing
Labels
feature/full text search kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:346510e-dev
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc101
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The current error is "fail to search on QueryNode 1: can't build BM25 IDF for data not varchar". However, in my personal opinion, for searches with a token count of 0 after tokenization, it's better to directly return an empty result.

[2024-10-21 16:09:01 - INFO - ci_test]: dataframe
        id       word                                           sentence                                          paragraph                                               text                                                emb
0        0     career                                      them at some.  stage us ok office sit rate. think cold marria...  election new risk along. start admit parent be...  [0.4934216178138785, 0.39034774645470205, 0.40...
1        1       same                       whose idea expect party far.  nothing water bank full close. drop strong fiv...  could popular world clearly lot. method star o...  [0.16044665878800168, 0.7060933017329347, 0.95...
2        2    mention  try offer citizen because discuss station arti...  three order rather network fund none. owner co...  month something their. all side focus once onl...  [0.3457558758390712, 0.236007238316436, 0.5572...
3        3  necessary        discuss share month establish they account.  day financial red ahead watch design. notice r...  special moment fire loss best pick. mr full pl...  [0.9124762384965902, 0.9715284102187233, 0.337...
4        4       that                       account guess live continue.  worry page night design. discussion will road ...  field full include five middle goal. specific ...  [0.18660411943766408, 0.6210301154146072, 0.01...
...    ...        ...                                                ...                                                ...                                                ...                                                ...
4995  4995    mention  method finish show present of money everything...  none keep stage at him others herself enjoy. c...  say traditional view term. per admit ability e...  [0.2784927986209893, 0.6326771858929588, 0.457...
4996  4996  agreement                       continue probably per class.  season structure pull defense concern pay figu...  happen what guess and personal year three. fou...  [0.8610058541963198, 0.3990729934883801, 0.009...
4997  4997       game       mrs trial choice evening economy first drug.  word value nation past race have happen. toget...  force go along represent skin. meet threat fly...  [0.41181370519675986, 0.45152238537780975, 0.8...
4998  4998       read                    mean image western detail also.  agent night skill our boy. down real power ite...  themselves writer themselves list realize appr...  [0.43636499668438733, 0.5333805669754456, 0.79...
4999  4999    country  type conference become career value sense scor...  hundred matter tend ground anyone guy now baby...  pass adult effect school while benefit east he...  [0.8620305090004367, 0.7880380167528181, 0.926...

[5000 rows x 6 columns] (test_full_text_search.py:2874)
[2024-10-21 16:09:02 - INFO - ci_test]: Analyze document cost time: 0.0665740966796875 (common_func.py:169)
[2024-10-21 16:09:28 - INFO - ci_test]: search data: ['', ''] (test_full_text_search.py:2915)
[2024-10-21 16:09:29 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=65535, message=fail to search on QueryNode 1: can't build BM25 IDF for data not varchar)>, <Time:{'RPC start': '2024-10-21 16:09:28.160243', 'RPC error': '2024-10-21 16:09:29.053421'}> (decorators.py:140)
[2024-10-21 16:09:29 - ERROR - ci_test]: Traceback (most recent call last):
  File "/Users/zilliz/workspace/milvus/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
    res = func(*args, **_kwargs)
  File "/Users/zilliz/workspace/milvus/tests/python_client/utils/api_request.py", line 63, in api_request
    return func(*arg, **kwargs)
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 801, in search
    resp = conn.search(
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/decorators.py", line 141, in handler
    raise e from e
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/decorators.py", line 137, in handler
    return func(*args, **kwargs)
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/decorators.py", line 176, in handler
    return func(self, *args, **kwargs)
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/decorators.py", line 116, in handler
    raise e from e
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/decorators.py", line 86, in handler
    return func(*args, **kwargs)
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 805, in search
    return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 746, in _execute_search
    raise e from e
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 735, in _execute_search
    check_status(response.status)
  File "/Users/zilliz/opt/anaconda3/envs/full_text_search/lib/python3.8/site-packages/pymilvus/client/utils.py", line 63, in check_status
    raise MilvusException(status.code, status.reason, status.error_code)
pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=fail to search on QueryNode 1: can't build BM25 IDF for data not varchar)>
 (api_request.py:45)

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 21, 2024
@zhuwenxing
Copy link
Contributor Author

/assign @zhengbuqian

@yanliang567
Copy link
Contributor

/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 22, 2024
@yanliang567 yanliang567 added this to the 2.5.0 milestone Oct 22, 2024
@xiaofan-luan
Copy link
Collaborator

the error message is a bit confusing

@zhengbuqian
Copy link
Collaborator

Yes the expected result for empty string in BM25 search, similar to searching an empty string, should be empty results. Working on a fix.

I'll optimize the error message as well

This was referenced Oct 23, 2024
Merged
Merged
sre-ci-robot pushed a commit to milvus-io/pymilvus that referenced this issue Oct 24, 2024
@zhengbuqian
fix: entity_is_sparse_matrix: each row should be dict or list (#2309)
377ad60 
issue: milvus-io/milvus#37022 and
milvus-io/milvus#37021

list of empty string is erroneously treated as sparse vector

adding a check to explicitly require that, if the input data will be
seen as sparse vector, its data type, if not scipy format, should be
list of list or list of dict.

Signed-off-by: Buqian Zheng <[email protected]>
@zhengbuqian
Copy link
Collaborator

/assign @zhuwenxing
/unassign
please verify, thanks!

error message is being updated in #37074.

sre-ci-robot pushed a commit that referenced this issue Oct 25, 2024
@zhengbuqian
fix: optimize BM25 err message (#37074)
088d5d7 
issue: #37022

Signed-off-by: Buqian Zheng <[email protected]>
@zhuwenxing
Copy link
Contributor Author

verified and fixed in master-20241025-80aa9ab4-amd64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhuwenxing zhuwenxing

Labels
feature/full text search kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
4 participants
@zhengbuqian @zhuwenxing @yanliang567 @xiaofan-luan