[Bug]: [benchmark][cluster] search raises error fail to search on QueryNode 7: distribution is not servcieable: channel not available in concurrent dql and dml(multi-partition) scene
#33550
Assignees
Labels
kind/bug
Issues or changes related a bug
test/benchmark
benchmark test
triage/accepted
Indicates an issue or PR is ready to be actively worked on.
Milestone
Comments
wangting0128
added
kind/bug
|
/assign @zhagnlu |
yanliang567
added
triage/accepted
same error, different sceneargo task: multi-vector-corn-1-1718892000 server: client pod name: multi-vector-corn-1-1718892000-3432833946 test steps: test result: |
Recurrentargo task: multi-vector-corn-1-1722866400 server: client log: test result: |
Recurrent on 2.5 branchargo task: fouramf-85cth server: client log: test steps: test result: |
sre-ci-robot
pushed a commit
that referenced
this issue
Sep 5, 2024
) issue: #33550 cause concurrent issue may occur between remove parition in target manager and sync segment list to delegator. when it happens, some segment may be released in delegator, and those segment may also be synced to delegator, which cause delegator become unserviceable due to lack of necessary segments, then search/query fails. this PR make sure that all write access to target_manager will be executed in serial to avoid the concurrent issues. Signed-off-by: Wei Liu <[email protected]>
weiliu1031
added a commit
to weiliu1031/milvus
that referenced
this issue
Sep 5, 2024
…vus-io#35919) issue: milvus-io#33550 cause concurrent issue may occur between remove parition in target manager and sync segment list to delegator. when it happens, some segment may be released in delegator, and those segment may also be synced to delegator, which cause delegator become unserviceable due to lack of necessary segments, then search/query fails. this PR make sure that all write access to target_manager will be executed in serial to avoid the concurrent issues. Signed-off-by: Wei Liu <[email protected]>
sre-ci-robot
pushed a commit
that referenced
this issue
Sep 6, 2024
) (#36019) issue: #33550 pr: #35919 cause concurrent issue may occur between remove parition in target manager and sync segment list to delegator. when it happens, some segment may be released in delegator, and those segment may also be synced to delegator, which cause delegator become unserviceable due to lack of necessary segments, then search/query fails. this PR make sure that all write access to target_manager will be executed in serial to avoid the concurrent issues. Signed-off-by: Wei Liu <[email protected]>
sre-ci-robot
pushed a commit
that referenced
this issue
Oct 23, 2024
issue: #33550 cause wrong impl of UpdateCollectionNextTarget, if ReleaseCollection and UpdateCollectionNextTarget happens at same time, the the released partition's segment list may be add to target again, and delegator will be marked as unserviceable due to lack of segment. This PR fix the impl of UpdateCollectionNextTarget Signed-off-by: Wei Liu <[email protected]>
weiliu1031
added a commit
to weiliu1031/milvus
that referenced
this issue
Oct 24, 2024
issue: milvus-io#33550 cause wrong impl of UpdateCollectionNextTarget, if ReleaseCollection and UpdateCollectionNextTarget happens at same time, the the released partition's segment list may be add to target again, and delegator will be marked as unserviceable due to lack of segment. This PR fix the impl of UpdateCollectionNextTarget Signed-off-by: Wei Liu <[email protected]>
sre-ci-robot
pushed a commit
that referenced
this issue
Oct 24, 2024
issue: #33550 pr: #37049 cause wrong impl of UpdateCollectionNextTarget, if ReleaseCollection and UpdateCollectionNextTarget happens at same time, the the released partition's segment list may be add to target again, and delegator will be marked as unserviceable due to lack of segment. This PR fix the impl of UpdateCollectionNextTarget Signed-off-by: Wei Liu <[email protected]>
reproduceargo task: memory-opt-scenes-2x7j4 server: client logs: test steps: @weiliu1031 please help to check |
|
should be fixed by #37694, please verify this with latest image |
|
/assign @wangting0128 |
Recurrent on 2.5 branchargo task: fouramf-bitmap-scenes-h7r46 
server: client log: test step: /assign @weiliu1031 please help on this |
czs007
pushed a commit
that referenced
this issue
Nov 21, 2024
issue: #33550 balance segment and balance segment execute at same time, which will cause bounch of corner case. This PR disable simultaneous balance of segments and channels Signed-off-by: Wei Liu <[email protected]>
weiliu1031
added a commit
to weiliu1031/milvus
that referenced
this issue
Nov 22, 2024
…#37850) issue: milvus-io#33550 balance segment and balance segment execute at same time, which will cause bounch of corner case. This PR disable simultaneous balance of segments and channels Signed-off-by: Wei Liu <[email protected]>
sre-ci-robot
pushed a commit
that referenced
this issue
Nov 26, 2024
…37939) issue: #33550 pr: #37850 balance segment and balance segment execute at same time, which will cause bounch of corner case. This PR disable simultaneous balance of segments and channels Signed-off-by: Wei Liu <[email protected]>
JsDove
pushed a commit
to JsDove/milvus
that referenced
this issue
Nov 26, 2024
…#37850) issue: milvus-io#33550 balance segment and balance segment execute at same time, which will cause bounch of corner case. This PR disable simultaneous balance of segments and channels Signed-off-by: Wei Liu <[email protected]>
Did you verify it on master? |
yes, master image, still verifying this issue
|
verification passed, not reproduce recentlyargo task： fouramf-concurrent-xcq |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Environment
Current Behavior
argo task: multi-vector-corn-1-1717336800
test case name: test_hybrid_search_locust_dql_dml_partition_hybrid_search_cluster
server:
client pod name: multi-vector-corn-1-1717336800-161976730
client logs:
hybrid search request raises error
Expected Behavior
No response
Steps To Reproduce
concurrent test and calculation of RT and QPS :purpose: `DQL & DML(partition)` verify concurrent DQL & DML(partition) scenario, which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1` :test steps: 1. create collection with fields: 'float_vector': 128dim, 'float_vector_1': 128dim, 'float_vector_2': 128dim, 'float_vector_3': 128dim, scalar field: int64_1, varchar_1 2. build indexes: IVF_FLAT: 'float_vector' HNSW: 'float_vector_1', DISKANN: 'float_vector_2' IVF_SQ8: 'float_vector_3' INVERTED: 'int64_1', 'varchar_1' default scalar index: 'id' 3. insert 1 million data into 10 partitions 4. flush collection 5. build indexes again using the same params 6. load collection replica: 1 7. concurrent request: - scene_test_partition_hybrid_search (partition: create->insert->flush->index again->load->hybrid_search->release->hybrid_search failed->drop) - search - hybrid_search - queryMilvus Log
No response
Anything else?
test result:
The text was updated successfully, but these errors were encountered: