[Bug]: The proxy CPU usage reaches 30+ cores when concurrent searching from 10w-128d data with nq=1

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4-20240506-0fc6b224
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus==2.5.0rc17
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

1. deploy milvus with proxy and queryNode

spec:
  components:
    dataCoord:
      replicas: 1
    dataNode:
      replicas: 1
      resources:
        limits:
          cpu: "8"
          memory: 16Gi
        requests:
          cpu: "5"
          memory: 9Gi
    image: harbor.milvus.io/milvus/milvus:2.4-20240506-0fc6b224
    imageUpdateMode: rollingUpgrade
    indexCoord:
      paused: false
      replicas: 1
    indexNode:
      replicas: 1
      resources:
        limits:
          cpu: "8"
          memory: 16Gi
        requests:
          cpu: "5"
          memory: 9Gi
    proxy:
      replicas: 1
      resources:
        limits:
          cpu: "60"
        requests:
          cpu: "32"
      serviceType: ClusterIP
    queryNode:
      replicas: 1
      resources:
        limits:
          cpu: "20"
          memory: 6Gi
        requests:
          cpu: "20"
          memory: 6Gi
    rootCoord:
      replicas: 1
  config:
    log:
      level: debug

2. create collection coll_hnsw with 1 shard -> hnsw index {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 30, 'efConstruction': 360}} -> insert 10w-128d sift data -> flush (only 1 segment) -> load
3. concurrent search by go:

                                 'search_params': {'timeout': 1200,
                                                   'nq': 1,
                                                   'param': {'metric_type': 'L2',
                                                             'params': {'ef': 1000}},
                                                   'top_k': 1000,
                                                   'expr': 'id >= 200 and id '
                                                           'not in [100 random ids] '},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 30,
                                                                  'efConstruction': 360}},
                                 'go_search_params': {'concurrent_number': 200,
                                                      'during_time': '10m',
                                                      'interval': 20}},

4. cpu usage
   

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

• argo workflow: https://argo-workflows.zilliz.cc/archived-workflows/qa/5be3eccd-48f2-47a3-8883-bad555735bd2?nodeId=smartnews-qps-hnsw-go-200-1
• grafana: metrics of zong-smart-hnsw
• pods:

zong-smart-hnsw-etcd-0                                            1/1     Running       0              24h     10.104.30.216   4am-node38   <none>           <none>
zong-smart-hnsw-etcd-1                                            1/1     Running       0              24h     10.104.24.60    4am-node29   <none>           <none>
zong-smart-hnsw-etcd-2                                            1/1     Running       0              24h     10.104.18.175   4am-node25   <none>           <none>
zong-smart-hnsw-milvus-datacoord-fc475898c-r7lpm                  1/1     Running       0              24h     10.104.33.117   4am-node36   <none>           <none>
zong-smart-hnsw-milvus-datanode-5d5f455596-xnwnf                  1/1     Running       0              24h     10.104.14.202   4am-node18   <none>           <none>
zong-smart-hnsw-milvus-indexcoord-7486b764df-sjcxd                1/1     Running       0              24h     10.104.33.116   4am-node36   <none>           <none>
zong-smart-hnsw-milvus-indexnode-84ffdb8c5b-72vrb                 1/1     Running       0              24h     10.104.14.201   4am-node18   <none>           <none>
zong-smart-hnsw-milvus-proxy-c794c774b-5hgdh                      1/1     Running       0              3h13m   10.104.30.26    4am-node38   <none>           <none>
zong-smart-hnsw-milvus-querycoord-69d586d765-cbw5w                1/1     Running       0              24h     10.104.33.115   4am-node36   <none>           <none>
zong-smart-hnsw-milvus-querynode-1-6b6f644d8-6m7vb                1/1     Running       0              24h     10.104.17.6     4am-node23   <none>           <none>
zong-smart-hnsw-milvus-rootcoord-5798fc4f5-45v2z                  1/1     Running       0              24h     10.104.33.113   4am-node36   <none>           <none>
zong-smart-hnsw-minio-0                                           1/1     Running       0              24h     10.104.30.215   4am-node38   <none>           <none>
zong-smart-hnsw-minio-1                                           1/1     Running       0              24h     10.104.24.61    4am-node29   <none>           <none>
zong-smart-hnsw-minio-2                                           1/1     Running       0              24h     10.104.18.176   4am-node25   <none>           <none>
zong-smart-hnsw-minio-3                                           1/1     Running       0              24h     10.104.27.226   4am-node31   <none>           <none>
zong-smart-hnsw-pulsar-bookie-0                                   1/1     Running       0              24h     10.104.24.63    4am-node29   <none>           <none>
zong-smart-hnsw-pulsar-bookie-1                                   1/1     Running       0              24h     10.104.21.187   4am-node24   <none>           <none>
zong-smart-hnsw-pulsar-bookie-2                                   1/1     Running       0              24h     10.104.27.229   4am-node31   <none>           <none>
zong-smart-hnsw-pulsar-bookie-init-qt65f                          0/1     Completed     0              24h     10.104.9.72     4am-node14   <none>           <none>
zong-smart-hnsw-pulsar-broker-0                                   1/1     Running       0              24h     10.104.6.33     4am-node13   <none>           <none>
zong-smart-hnsw-pulsar-proxy-0                                    1/1     Running       0              24h     10.104.4.218    4am-node11   <none>           <none>
zong-smart-hnsw-pulsar-pulsar-init-l6pgk                          0/1     Completed     0              24h     10.104.9.71     4am-node14   <none>           <none>
zong-smart-hnsw-pulsar-recovery-0                                 1/1     Running       0              24h     10.104.9.74     4am-node14   <none>           <none>
zong-smart-hnsw-pulsar-zookeeper-0                                1/1     Running       0              24h     10.104.18.178   4am-node25   <none>           <none>
zong-smart-hnsw-pulsar-zookeeper-1                                1/1     Running       0              24h     10.104.24.66    4am-node29   <none>           <none>
zong-smart-hnsw-pulsar-zookeeper-2                                1/1     Running       0              24h     10.104.33.87    4am-node36   <none>           <none>

Anything else?

No response

[ThreadDao]

Pyroscope cpu of zong-smart-hnsw-proxy

[ThreadDao]

#32871 and #32869

[xiaofan-luan]

also upgrade antlr see if we can get some benefit #32977

[ThreadDao]

The last test concluded that upgrading antlr seems to have no benefit