[Bug]: e2e case test_compact_during_insert is not stable

[congqixia]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 531092c
- Deployment mode(standalone or cluster): both
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

e2e case test_compact_during_insert is not stable

Expected Behavior

e2e case shall pass stably

Steps To Reproduce

No response

Milvus Log

[2024-07-12T04:15:07.918Z]         # search

[2024-07-12T04:15:07.918Z]         search_res, _ = collection_w.search([df[ct.default_float_vec_field_name][0]],

[2024-07-12T04:15:07.918Z]                                             ct.default_float_vec_field_name,

[2024-07-12T04:15:07.918Z]                                             ct.default_search_params, ct.default_limit)

[2024-07-12T04:15:07.918Z] >       assert len(search_res[0]) == ct.default_limit

[2024-07-12T04:15:07.918Z] E       AssertionError: assert 9 == 10

[2024-07-12T04:15:07.918Z] E        +  where 9 = len(['id: 4000, distance: 0.9999669194221497, entity: {}', 'id: 1776, distance: 0.8461944460868835, entity: {}', 'id: 1563, distance: 0.8287100195884705, entity: {}', 'id: 5415, distance: 0.8226537108421326, entity: {}', 'id: 2236, distance: 0.8202782869338989, entity: {}', 'id: 523, distance: 0.8194977641105652, entity: {}', 'id: 1070, distance: 0.8174188137054443, entity: {}', 'id: 1955, distance: 0.8171544671058655, entity: {}', 'id: 976, distance: 0.8167256712913513, entity: {}'])

[2024-07-12T04:15:07.918Z] E        +  and   10 = ct.default_limit

[2024-07-12T04:15:07.918Z] 

[2024-07-12T04:15:07.918Z] testcases/test_compaction.py:1209: AssertionError

Anything else?

https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20HA%20CI/detail/PR-34632/1/pipeline

[bigsheeper]

I think this case is normal. This case inserted 6000 entries, then built an ivfpq index (nlist=128), search with nprob=32, and top 10. When ivf clustering is uneven, getting only 9 results is possible.

[congqixia]

@bigsheeper after some confirmation with Knowhere team. It's possible to get an empty bucket when doing KMeans clustering but the chance is extremely low. It is almost impossible to meet 32 buckets having only 9 candidates. Even then situation is met, Knowhere would skip empty buckets. It looks that some bug still exists here.
/cc @XuanYang-cn