Question about query performance

[artesby]

Hello.

Our milvus cluster have about 20 Upsert RPS into 3 collections, avg vector size is 768.
Current collection size is ~5 million each.
Each collection uses FLAT index with consistency "bounded".

When we started quering these collections we noticed that query latency is 60s.

We tried to stop inserting (~30min) and restarted all components, without success.
Finally, setting search consistency to "strong" in pymilvus client reduced latency to ~500ms.

However, i am still wondering, what could make bounded consistency work so long? And what is expected search latency on 5-10mill FLAT index?

- Milvus version: v2.4.13-hotfix
- Deployment mode: cluster
- MQ type:  kafka external
- Storage: external s3
- resources: as suggested by calculator https://milvus.io/tools/sizing

[xiaofan-luan]

usually it takes 10-100 ms to do search.
The weird part is strong is usually gonna to be slower than bounded staleness.

Questions:

1. what is index we use?
2. if diskann or mmap used, what kind of disk we use.
3. Are you running batch query? what is the batch size? If NQ(number of query) is huge this might be explainable.
4. what is the segcore latency?

We would definitely like to offer further help. Feel free to reach me out at [email protected]

[artesby]

Thanks for your reply.

1. we use IVF_FLAT with nlist=3162, nprobe=128. Default index consistency level is bounded staleness.
2. we don't use mmap or disk. Regarding configuration, almost everything is default. Cluster is deployed using official helm-chart and our values is

cluster:
  enabled: true

service: ...

extraConfigFiles:
  user.yaml: |+
    common:
      security:
        authorizationEnabled: true
    kafka:
      saslUsername: ...
      saslPassword: ...
      producer:
        ssl.ca.location: ...
      consumer:
        ssl.ca.location: ...

metrics:
  enabled: true

proxy:
  enabled: true
  replicas: 1
  resources:
    requests:
      memory: 4Gi
      cpu: 500m
    limits:
      memory: 4Gi
      cpu: 1

queryNode:
  enabled: true
  replicas: 3
  disk:
    enabled: false
  resources:
    requests:
      memory: 56Gi
      cpu: 4
    limits:
      memory: 56Gi
      cpu: 8

indexNode:
  enabled: true
  replicas: 2
  disk:
    enabled: false
  resources:
    requests:
      memory: 56Gi
      cpu: 16
    limits:
      memory: 56Gi
      cpu: 32

dataNode:
  enabled: true
  replicas: 1
  resources:
    requests:
      memory: 4Gi
      cpu: 500m
    limits:
      memory: 4Gi
      cpu: 1

mixCoordinator:
  enabled: true
  replicas: 1
  resources:
    requests:
      memory: 8Gi
      cpu: 1
    limits:
      memory: 8Gi
      cpu: 2

minio:
  enabled: false

etcd:
  enabled: true
  name: etcd
  replicaCount: 3
  pdb:
    create: false # deprecated api version
  resources:
    requests:
      memory: 4Gi
      cpu: 1
    limits:
      memory: 4Gi
      cpu: 2
  persistence:
    enabled: true
    storageClass: network-ssd-retain
    accessMode: ReadWriteOnce
    size: 30Gi
  metrics:
    enabled: true

pulsar:
  enabled: false

kafka:
  enabled: false

externalS3:
  enabled: true
  host: ...
  port: ...
  useSSL: true
  bucketName: ...

externalKafka:
  enabled: true
  brokerList: ...
  securityProtocol: SASL_SSL
  sasl:
    mechanisms: SCRAM-SHA-512

3. we don't use batches, we upsert single vectors with ~20rps. search rps is < 10
4. segcore latency is <100ms.
   

I've done some more diagnostics, here the observations:

1. query node repeatedly receiving delete operations. i believe this is ok, given we are not using batches?

{"level":"INFO","time":"2024/11/21 12:02:14.377 +00:00","caller":"pipeline/delete_node.go:57","message":"pipeline fetch delete msg","collectionID":453584804738224339,"partitionID":453584804738224340,"deleteRowNum":1,"timestampMin":454083353651445766,"timestampMax":454083353651445766}
{"level":"INFO","time":"2024/11/21 12:02:14.381 +00:00","caller":"pipeline/delete_node.go:57","message":"pipeline fetch delete msg","collectionID":453584804738224297,"partitionID":453584804738224298,"deleteRowNum":1,"timestampMin":454083353651445765,"timestampMax":454083353651445765}
{"level":"INFO","time":"2024/11/21 12:02:14.384 +00:00","caller":"pipeline/delete_node.go:57","message":"pipeline fetch delete msg","collectionID":453584804738224249,"partitionID":453584804738224250,"deleteRowNum":1,"timestampMin":454083353612124174,"timestampMax":454083353612124174}

2. I found that datanode have some warnings with one specific topic. restart or pause on writing does not help.

{"level":"WARN","time":"2024/11/21 11:52:46.057 +00:00","caller":"flowgraph/node.go:82","message":"some node(s) haven't received input","list":["nodeCtxTtChecker-writeNode-by-dev-rootcoord-dml_3_453584804738020460v0","nodeCtxTtChecker-ttNode-by-dev-rootcoord-dml_3_453584804738020460v0","nodeCtxTtChecker-dmInputNode-data-by-dev-rootcoord-dml_3_453584804738020460v0","nodeCtxTtChecker-ddNode-by-dev-rootcoord-dml_3_453584804738020460v0"],"duration ":"2m0s"}
{"level":"WARN","time":"2024/11/21 11:52:47.962 +00:00","caller":"kafka/kafka_consumer.go:139","message":"consume msg failed","topic":"by-dev-rootcoord-dml_3","groupID":"datanode-160-by-dev-rootcoord-dml_3_453584804738020460v0-true","error":"Local: Timed out"}

3. When I use attu to search collection with bounded concistency, i receive error, and find warnings in logs:

proxy:

{"level":"WARN","time":"2024/11/21 12:00:50.179 +00:00","caller":"grpcclient/client.go:404","message":"call received grpc error","traceID":"b245bd4a6d589fe179e00940630448eb","clientRole":"querynode-152","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
{"level":"WARN","time":"2024/11/21 12:00:50.179 +00:00","caller":"retry/retry.go:106","message":"retry func failed","traceID":"b245bd4a6d589fe179e00940630448eb","retried":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
{"level":"WARN","time":"2024/11/21 12:00:50.180 +00:00","caller":"grpcclient/client.go:556","message":"ClientBase Call grpc call get error","traceID":"b245bd4a6d589fe179e00940630448eb","role":"querynode-152","address":"10.100.146.7:21123","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.180 +00:00","caller":"proxy/task_search.go:727","message":"QueryNode search return error","traceID":"b245bd4a6d589fe179e00940630448eb","collection":453584804738224339,"partitionIDs":[],"nodeID":152,"channel":"by-dev-rootcoord-dml_6_453584804738224339v0","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"INFO","time":"2024/11/21 12:00:50.180 +00:00","caller":"proxy/meta_cache.go:1035","message":"clearing shard cache for collection","collectionName":"cv1"}
{"level":"WARN","time":"2024/11/21 12:00:50.180 +00:00","caller":"proxy/lb_policy.go:180","message":"search/query channel failed","collectionID":453584804738224339,"channelName":"by-dev-rootcoord-dml_6_453584804738224339v0","nodeID":152,"error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.180 +00:00","caller":"retry/retry.go:46","message":"retry func failed","traceID":"b245bd4a6d589fe179e00940630448eb","retried":0,"error":"failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:186\n  | [...repeated from below...]\nWraps: (2) failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0\nWraps: (3) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (4) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (5) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.181 +00:00","caller":"proxy/task_search.go:551","message":"search execute failed","traceID":"b245bd4a6d589fe179e00940630448eb","nq":1,"error":"failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:186\n  | [...repeated from below...]\nWraps: (2) failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0\nWraps: (3) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (4) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (5) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.181 +00:00","caller":"proxy/task_scheduler.go:477","message":"Failed to execute task: ","traceID":"b245bd4a6d589fe179e00940630448eb","error":"failed to search: failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"failed to search: failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Execute\n  | \t/workspace/source/internal/proxy/task_search.go:552\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:474\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:553\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\nWraps: (2) failed to search\nWraps: (3) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:186\n  | [...repeated from below...]\nWraps: (4) failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0\nWraps: (5) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (6) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (7) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.182 +00:00","caller":"proxy/impl.go:3015","message":"Search failed to WaitToFinish","traceID":"b245bd4a6d589fe179e00940630448eb","role":"proxy","db":"cv_product_search","collection":"cv1","partitions":[],"dsl":"","len(PlaceholderGroup)":2060,"OutputFields":["id"],"search_params":[{"key":"anns_field","value":"vector"},{"key":"params","value":"{\"nprobe\":128}"},{"key":"topk","value":"50"},{"key":"offset","value":"0"},{"key":"metric_type"},{"key":"ignore_growing","value":"false"}],"ConsistencyLevel":"Bounded","useDefaultConsistency":false,"nq":1,"error":"proxy TaskCondition context Done: context deadline exceeded","errorVerbose":"proxy TaskCondition context Done: context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*TaskCondition).WaitToFinish\n  | \t/workspace/source/internal/proxy/condition.go:45\n  | github.com/milvus-io/milvus/internal/proxy.(*Proxy).search\n  | \t/workspace/source/internal/proxy/impl.go:3014\n  | github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search.func1\n  | \t/workspace/source/internal/proxy/impl.go:2889\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search\n  | \t/workspace/source/internal/proxy/impl.go:2887\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search\n  | \t/workspace/source/internal/distributed/proxy/service.go:846\n  | github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler.func1\n  | \t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/[email protected]/milvuspb/milvus_grpc.pb.go:2009\n  | github.com/milvus-io/milvus/internal/proxy/connection.KeepActiveInterceptor\n  | \t/workspace/source/internal/proxy/connection/util.go:60\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy.TraceLogInterceptor\n  | \t/workspace/source/internal/proxy/trace_log_interceptor.go:36\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryUpdateAccessInfoInterceptor\n  | \t/workspace/source/internal/proxy/accesslog/util.go:47\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.RateLimitInterceptor.func5\n  | \t/workspace/source/internal/proxy/rate_limit_interceptor.go:56\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n  | \t/workspace/source/pkg/util/logutil/grpc_interceptor.go:23\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func4\n  | \t/workspace/source/internal/proxy/privilege_interceptor.go:112\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy.HookInterceptor\n  | \t/workspace/source/internal/proxy/hook_interceptor.go:49\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerHookInterceptor.func3\n  | \t/workspace/source/internal/proxy/hook_interceptor.go:19\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.DatabaseInterceptor.func2\n  | \t/workspace/source/internal/proxy/database_interceptor.go:15\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:47\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLogInterceptor\n  | \t/workspace/source/internal/proxy/accesslog/util.go:38\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34\n  | github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler\n  | \t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/[email protected]/milvuspb/milvus_grpc.pb.go:2011\n  | google.golang.org/grpc.(*Server).processUnaryRPC\n  | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:1343\n  | google.golang.org/grpc.(*Server).handleStream\n  | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:1737\nWraps: (2) proxy TaskCondition context Done\nWraps: (3) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) context.deadlineExceededError"}
{"level":"INFO","time":"2024/11/21 12:00:50.182 +00:00","caller":"proxy/impl.go:2978","message":"Search slow","traceID":"b245bd4a6d589fe179e00940630448eb","role":"proxy","db":"cv_product_search","collection":"cv1","partitions":[],"dsl":"","len(PlaceholderGroup)":2060,"OutputFields":["id"],"search_params":[{"key":"anns_field","value":"vector"},{"key":"params","value":"{\"nprobe\":128}"},{"key":"topk","value":"50"},{"key":"offset","value":"0"},{"key":"metric_type"},{"key":"ignore_growing","value":"false"}],"ConsistencyLevel":"Bounded","useDefaultConsistency":false,"guarantee_timestamp":454083328129105920,"nq":1,"duration":"15.001946834s"}
{"level":"INFO","time":"2024/11/21 12:00:50.240 +00:00","caller":"proxy/meta_cache.go:960","message":"no shard cache for collection, try to get shard leaders from QueryCoord","traceID":"5eeb37dbaea36971c84af1e115fd5ea8","collectionName":"cv1","collectionID":453584804738224339}

query:

{"level":"WARN","time":"2024/11/21 12:19:01.265 +00:00","caller":"delegator/delegator.go:291","message":"delegator search failed to wait tsafe","traceID":"665be0a78bc033466ab070661b8831bd","collectionID":453584804738224339,"channel":"by-dev-rootcoord-dml_6_453584804738224339v0","replicaID":453584804871405581,"error":"context deadline exceeded"}
{"level":"WARN","time":"2024/11/21 12:19:01.265 +00:00","caller":"querynodev2/handlers.go:376","message":"failed to search on delegator","traceID":"665be0a78bc033466ab070661b8831bd","msgID":454035955343126262,"collectionID":453584804738224339,"channel":"by-dev-rootcoord-dml_6_453584804738224339v0","scope":"All","nq":1,"error":"context deadline exceeded"}

this sounds to me that there is some issue with mix coord, isnt it? it has warnings as well, but timings are not matching with search attempts.

{"level":"WARN","time":"2024/11/21 12:24:30.717 +00:00","caller":"datacoord/meta.go:699","message":"meta update: get segment failed - segment not found","segmentID":454035955324283695,"segment nil":true,"segment unhealthy":true}
{"level":"WARN","time":"2024/11/21 12:24:37.410 +00:00","caller":"datacoord/task_index.go:204","message":"failed to get dim from field type params","field type":"VarChar","error":"dim not found in params","errorVerbose":"dim not found in params\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n  | \t/workspace/source/internal/storage/utils.go:233\n  | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n  | \t/workspace/source/internal/datacoord/task_index.go:202\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:244\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:211\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).schedule\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:179\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"}
{"level":"WARN","time":"2024/11/21 12:24:37.532 +00:00","caller":"datacoord/task_index.go:204","message":"failed to get dim from field type params","field type":"VarChar","error":"dim not found in params","errorVerbose":"dim not found in params\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n  | \t/workspace/source/internal/storage/utils.go:233\n  | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n  | \t/workspace/source/internal/datacoord/task_index.go:202\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:244\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:211\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).schedule\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:179\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"}
{"level":"WARN","time":"2024/11/21 12:24:38.394 +00:00","caller":"datacoord/task_index.go:204","message":"failed to get dim from field type params","field type":"VarChar","error":"dim not found in params","errorVerbose":"dim not found in params\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n  | \t/workspace/source/internal/storage/utils.go:233\n  | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n  | \t/workspace/source/internal/datacoord/task_index.go:202\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:244\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:211\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).schedule\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:183\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"}
{"level":"WARN","time":"2024/11/21 12:24:39.396 +00:00","caller":"datacoord/task_index.go:204","message":"failed to get dim from field type params","field type":"VarChar","error":"dim not found in params","errorVerbose":"dim not found in params\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n  | \t/workspace/source/internal/storage/utils.go:233\n  | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n  | \t/workspace/source/internal/datacoord/task_index.go:202\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:244\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:211\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).schedule\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:183\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"}
{"level":"WARN","time":"2024/11/21 12:24:39.397 +00:00","caller":"grpcclient/client.go:404","message":"call received grpc error","clientRole":"indexnode-157","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:24:39.397 +00:00","caller":"retry/retry.go:106","message":"retry func failed","retried":0,"error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:24:39.397 +00:00","caller":"grpcclient/client.go:556","message":"ClientBase Call grpc call get error","role":"indexnode-157","address":"10.100.143.8:21121","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n/workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:92\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:140\n  | github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | \t/workspace/source/internal/datacoord/indexnode_manager.go:129\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | /workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:24:39.397 +00:00","caller":"datacoord/indexnode_manager.go:131","message":"get IndexNode slots failed","nodeID":157,"error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n/workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:92\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:140\n  | github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | \t/workspace/source/internal/datacoord/indexnode_manager.go:129\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | /workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:24:41.395 +00:00","caller":"grpcclient/client.go:404","message":"call received grpc error","clientRole":"indexnode-157","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:24:41.395 +00:00","caller":"retry/retry.go:106","message":"retry func failed","retried":0,"error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:24:41.396 +00:00","caller":"grpcclient/client.go:556","message":"ClientBase Call grpc call get error","role":"indexnode-157","address":"10.100.143.8:21121","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n/workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:92\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:140\n  | github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | \t/workspace/source/internal/datacoord/indexnode_manager.go:129\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | /workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:24:41.396 +00:00","caller":"datacoord/indexnode_manager.go:131","message":"get IndexNode slots failed","nodeID":157,"error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n/workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:92\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:140\n  | github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | \t/workspace/source/internal/datacoord/indexnode_manager.go:129\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | /workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:25:08.390 +00:00","caller":"utils/types.go:67","message":"delta position is quite stale","collectionID":453584804738224339,"segmentID":454035955324476843,"channel":"by-dev-rootcoord-dml_6_453584804738224339v0","posTs":454083554204975120,"posTime":"2024/11/21 12:14:52.591 +00:00","tsLag":"10m15.799799912s"}
{"level":"WARN","time":"2024/11/21 12:25:37.426 +00:00","caller":"datacoord/task_scheduler.go:300","message":"task running time is too long","taskID":454035955344814714,"running time(ms)":9223372036854}
{"level":"WARN","time":"2024/11/21 12:25:38.404 +00:00","caller":"datacoord/task_scheduler.go:300","message":"task running time is too long","taskID":454035955344814718,"running time(ms)":9223372036854}
{"level":"WARN","time":"2024/11/21 12:26:18.391 +00:00","caller":"utils/types.go:67","message":"delta position is quite stale","collectionID":453584804738224249,"segmentID":454035955324283695,"channel":"by-dev-rootcoord-dml_4_453584804738224249v0","posTs":454083573878620166,"posTime":"2024/11/21 12:16:07.640 +00:00","tsLag":"10m10.751654025s"}
{"level":"WARN","time":"2024/11/21 12:26:28.388 +00:00","caller":"utils/types.go:67","message":"delta position is quite stale","collectionID":453584804738224249,"segmentID":454035955324476922,"channel":"by-dev-rootcoord-dml_4_453584804738224249v0","posTs":454083573878620166,"posTime":"2024/11/21 12:16:07.640 +00:00","tsLag":"10m20.748952996s"}
{"level":"WARN","time":"2024/11/21 12:26:34.889 +00:00","caller":"grpcclient/client.go:404","message":"call received grpc error","clientRole":"datacoord","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:26:34.889 +00:00","caller":"retry/retry.go:106","message":"retry func failed","retried":0,"error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:26:34.889 +00:00","caller":"grpcclient/client.go:556","message":"ClientBase Call grpc call get error","role":"datacoord","address":"10.100.139.33:13333","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:610\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:609\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).getLoadInfo\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:697\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:196\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:161\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:124\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:26:34.890 +00:00","caller":"meta/coordinator_broker.go:335","message":"failed to get segment index info","collectionID":453584804738224249,"segmentID":454035955324476922,"error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:610\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:609\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).getLoadInfo\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:697\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:196\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:161\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:124\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:26:34.890 +00:00","caller":"task/executor.go:700","message":"failed to get index of segment","level":"L1","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:610\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:609\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).getLoadInfo\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:697\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:196\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:161\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:124\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:26:37.431 +00:00","caller":"datacoord/task_scheduler.go:300","message":"task running time is too long","taskID":454035955344819731,"running time(ms)":9223372036854}
{"level":"WARN","time":"2024/11/21 12:26:38.389 +00:00","caller":"utils/types.go:67","message":"delta position is quite stale","collectionID":453584804738224249,"segmentID":454035955324283783,"channel":"by-dev-rootcoord-dml_4_453584804738224249v0","posTs":454083573878620166,"posTime":"2024/11/21 12:16:07.640 +00:00","tsLag":"10m30.749302353s"}
{"level":"WARN","time":"2024/11/21 12:26:38.398 +00:00","caller":"datacoord/task_scheduler.go:300","message":"task running time is too long","taskID":454035955344819734,"running time(ms)":9223372036854}
{"level":"WARN","time":"2024/11/21 12:26:46.889 +00:00","caller":"meta/coordinator_broker.go:413","message":"failed to fetch index meta","collectionID":453584804738224249,"error":"context canceled"}
{"level":"WARN","time":"2024/11/21 12:26:46.889 +00:00","caller":"task/executor.go:709","message":"fail to get index meta of collection","level":"L1","error":"context canceled"}

[xiaofan-luan]

Thanks for your reply.

1. we use IVF_FLAT with nlist=3162, nprobe=128. Default index consistency level is bounded staleness.
2. we don't use mmap or disk. Regarding configuration, almost everything is default. Cluster is deployed using official helm-chart and our values is

cluster:
  enabled: true

service: ...

extraConfigFiles:
  user.yaml: |+
    common:
      security:
        authorizationEnabled: true
    kafka:
      saslUsername: ...
      saslPassword: ...
      producer:
        ssl.ca.location: ...
      consumer:
        ssl.ca.location: ...

metrics:
  enabled: true

proxy:
  enabled: true
  replicas: 1
  resources:
    requests:
      memory: 4Gi
      cpu: 500m
    limits:
      memory: 4Gi
      cpu: 1

queryNode:
  enabled: true
  replicas: 3
  disk:
    enabled: false
  resources:
    requests:
      memory: 56Gi
      cpu: 4
    limits:
      memory: 56Gi
      cpu: 8

indexNode:
  enabled: true
  replicas: 2
  disk:
    enabled: false
  resources:
    requests:
      memory: 56Gi
      cpu: 16
    limits:
      memory: 56Gi
      cpu: 32

dataNode:
  enabled: true
  replicas: 1
  resources:
    requests:
      memory: 4Gi
      cpu: 500m
    limits:
      memory: 4Gi
      cpu: 1

mixCoordinator:
  enabled: true
  replicas: 1
  resources:
    requests:
      memory: 8Gi
      cpu: 1
    limits:
      memory: 8Gi
      cpu: 2

minio:
  enabled: false

etcd:
  enabled: true
  name: etcd
  replicaCount: 3
  pdb:
    create: false # deprecated api version
  resources:
    requests:
      memory: 4Gi
      cpu: 1
    limits:
      memory: 4Gi
      cpu: 2
  persistence:
    enabled: true
    storageClass: network-ssd-retain
    accessMode: ReadWriteOnce
    size: 30Gi
  metrics:
    enabled: true

pulsar:
  enabled: false

kafka:
  enabled: false

externalS3:
  enabled: true
  host: ...
  port: ...
  useSSL: true
  bucketName: ...

externalKafka:
  enabled: true
  brokerList: ...
  securityProtocol: SASL_SSL
  sasl:
    mechanisms: SCRAM-SHA-512

3. we don't use batches, we upsert single vectors with ~20rps. search rps is < 10
4. segcore latency is <100ms.
   

I've done some more diagnostics, here the observations:

1. query node repeatedly receiving delete operations. i believe this is ok, given we are not using batches?

{"level":"INFO","time":"2024/11/21 12:02:14.377 +00:00","caller":"pipeline/delete_node.go:57","message":"pipeline fetch delete msg","collectionID":453584804738224339,"partitionID":453584804738224340,"deleteRowNum":1,"timestampMin":454083353651445766,"timestampMax":454083353651445766}
{"level":"INFO","time":"2024/11/21 12:02:14.381 +00:00","caller":"pipeline/delete_node.go:57","message":"pipeline fetch delete msg","collectionID":453584804738224297,"partitionID":453584804738224298,"deleteRowNum":1,"timestampMin":454083353651445765,"timestampMax":454083353651445765}
{"level":"INFO","time":"2024/11/21 12:02:14.384 +00:00","caller":"pipeline/delete_node.go:57","message":"pipeline fetch delete msg","collectionID":453584804738224249,"partitionID":453584804738224250,"deleteRowNum":1,"timestampMin":454083353612124174,"timestampMax":454083353612124174}

2. I found that datanode have some warnings with one specific topic. restart or pause on writing does not help.

{"level":"WARN","time":"2024/11/21 11:52:46.057 +00:00","caller":"flowgraph/node.go:82","message":"some node(s) haven't received input","list":["nodeCtxTtChecker-writeNode-by-dev-rootcoord-dml_3_453584804738020460v0","nodeCtxTtChecker-ttNode-by-dev-rootcoord-dml_3_453584804738020460v0","nodeCtxTtChecker-dmInputNode-data-by-dev-rootcoord-dml_3_453584804738020460v0","nodeCtxTtChecker-ddNode-by-dev-rootcoord-dml_3_453584804738020460v0"],"duration ":"2m0s"}
{"level":"WARN","time":"2024/11/21 11:52:47.962 +00:00","caller":"kafka/kafka_consumer.go:139","message":"consume msg failed","topic":"by-dev-rootcoord-dml_3","groupID":"datanode-160-by-dev-rootcoord-dml_3_453584804738020460v0-true","error":"Local: Timed out"}

3. When I use attu to search collection with bounded concistency, i receive error, and find warnings in logs:

proxy:

{"level":"WARN","time":"2024/11/21 12:00:50.179 +00:00","caller":"grpcclient/client.go:404","message":"call received grpc error","traceID":"b245bd4a6d589fe179e00940630448eb","clientRole":"querynode-152","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
{"level":"WARN","time":"2024/11/21 12:00:50.179 +00:00","caller":"retry/retry.go:106","message":"retry func failed","traceID":"b245bd4a6d589fe179e00940630448eb","retried":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
{"level":"WARN","time":"2024/11/21 12:00:50.180 +00:00","caller":"grpcclient/client.go:556","message":"ClientBase Call grpc call get error","traceID":"b245bd4a6d589fe179e00940630448eb","role":"querynode-152","address":"10.100.146.7:21123","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.180 +00:00","caller":"proxy/task_search.go:727","message":"QueryNode search return error","traceID":"b245bd4a6d589fe179e00940630448eb","collection":453584804738224339,"partitionIDs":[],"nodeID":152,"channel":"by-dev-rootcoord-dml_6_453584804738224339v0","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"INFO","time":"2024/11/21 12:00:50.180 +00:00","caller":"proxy/meta_cache.go:1035","message":"clearing shard cache for collection","collectionName":"cv1"}
{"level":"WARN","time":"2024/11/21 12:00:50.180 +00:00","caller":"proxy/lb_policy.go:180","message":"search/query channel failed","collectionID":453584804738224339,"channelName":"by-dev-rootcoord-dml_6_453584804738224339v0","nodeID":152,"error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.180 +00:00","caller":"retry/retry.go:46","message":"retry func failed","traceID":"b245bd4a6d589fe179e00940630448eb","retried":0,"error":"failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:186\n  | [...repeated from below...]\nWraps: (2) failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0\nWraps: (3) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (4) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (5) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.181 +00:00","caller":"proxy/task_search.go:551","message":"search execute failed","traceID":"b245bd4a6d589fe179e00940630448eb","nq":1,"error":"failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:186\n  | [...repeated from below...]\nWraps: (2) failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0\nWraps: (3) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (4) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (5) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.181 +00:00","caller":"proxy/task_scheduler.go:477","message":"Failed to execute task: ","traceID":"b245bd4a6d589fe179e00940630448eb","error":"failed to search: failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n/workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n/workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2: rpc error: code = DeadlineExceeded desc = context deadline exceeded","errorVerbose":"failed to search: failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Execute\n  | \t/workspace/source/internal/proxy/task_search.go:552\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:474\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:553\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\nWraps: (2) failed to search\nWraps: (3) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:186\n  | [...repeated from below...]\nWraps: (4) failed to search/query delegator 152 for channel by-dev-rootcoord-dml_6_453584804738224339v0\nWraps: (5) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:202\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | \t/workspace/source/internal/proxy/task_search.go:725\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | \t/workspace/source/internal/proxy/lb_policy.go:178\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | \t/workspace/source/internal/proxy/lb_policy.go:147\n  | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n  | \t/workspace/source/internal/proxy/lb_policy.go:215\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (6) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:202 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n  | /workspace/source/internal/proxy/task_search.go:725 github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n  | /workspace/source/internal/proxy/lb_policy.go:178 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/proxy/lb_policy.go:147 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n  | /workspace/source/internal/proxy/lb_policy.go:215 github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\nWraps: (7) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:00:50.182 +00:00","caller":"proxy/impl.go:3015","message":"Search failed to WaitToFinish","traceID":"b245bd4a6d589fe179e00940630448eb","role":"proxy","db":"cv_product_search","collection":"cv1","partitions":[],"dsl":"","len(PlaceholderGroup)":2060,"OutputFields":["id"],"search_params":[{"key":"anns_field","value":"vector"},{"key":"params","value":"{\"nprobe\":128}"},{"key":"topk","value":"50"},{"key":"offset","value":"0"},{"key":"metric_type"},{"key":"ignore_growing","value":"false"}],"ConsistencyLevel":"Bounded","useDefaultConsistency":false,"nq":1,"error":"proxy TaskCondition context Done: context deadline exceeded","errorVerbose":"proxy TaskCondition context Done: context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*TaskCondition).WaitToFinish\n  | \t/workspace/source/internal/proxy/condition.go:45\n  | github.com/milvus-io/milvus/internal/proxy.(*Proxy).search\n  | \t/workspace/source/internal/proxy/impl.go:3014\n  | github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search.func1\n  | \t/workspace/source/internal/proxy/impl.go:2889\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search\n  | \t/workspace/source/internal/proxy/impl.go:2887\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search\n  | \t/workspace/source/internal/distributed/proxy/service.go:846\n  | github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler.func1\n  | \t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/[email protected]/milvuspb/milvus_grpc.pb.go:2009\n  | github.com/milvus-io/milvus/internal/proxy/connection.KeepActiveInterceptor\n  | \t/workspace/source/internal/proxy/connection/util.go:60\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy.TraceLogInterceptor\n  | \t/workspace/source/internal/proxy/trace_log_interceptor.go:36\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryUpdateAccessInfoInterceptor\n  | \t/workspace/source/internal/proxy/accesslog/util.go:47\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.RateLimitInterceptor.func5\n  | \t/workspace/source/internal/proxy/rate_limit_interceptor.go:56\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n  | \t/workspace/source/pkg/util/logutil/grpc_interceptor.go:23\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func4\n  | \t/workspace/source/internal/proxy/privilege_interceptor.go:112\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy.HookInterceptor\n  | \t/workspace/source/internal/proxy/hook_interceptor.go:49\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerHookInterceptor.func3\n  | \t/workspace/source/internal/proxy/hook_interceptor.go:19\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.DatabaseInterceptor.func2\n  | \t/workspace/source/internal/proxy/database_interceptor.go:15\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:47\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLogInterceptor\n  | \t/workspace/source/internal/proxy/accesslog/util.go:38\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34\n  | github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler\n  | \t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/[email protected]/milvuspb/milvus_grpc.pb.go:2011\n  | google.golang.org/grpc.(*Server).processUnaryRPC\n  | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:1343\n  | google.golang.org/grpc.(*Server).handleStream\n  | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:1737\nWraps: (2) proxy TaskCondition context Done\nWraps: (3) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) context.deadlineExceededError"}
{"level":"INFO","time":"2024/11/21 12:00:50.182 +00:00","caller":"proxy/impl.go:2978","message":"Search slow","traceID":"b245bd4a6d589fe179e00940630448eb","role":"proxy","db":"cv_product_search","collection":"cv1","partitions":[],"dsl":"","len(PlaceholderGroup)":2060,"OutputFields":["id"],"search_params":[{"key":"anns_field","value":"vector"},{"key":"params","value":"{\"nprobe\":128}"},{"key":"topk","value":"50"},{"key":"offset","value":"0"},{"key":"metric_type"},{"key":"ignore_growing","value":"false"}],"ConsistencyLevel":"Bounded","useDefaultConsistency":false,"guarantee_timestamp":454083328129105920,"nq":1,"duration":"15.001946834s"}
{"level":"INFO","time":"2024/11/21 12:00:50.240 +00:00","caller":"proxy/meta_cache.go:960","message":"no shard cache for collection, try to get shard leaders from QueryCoord","traceID":"5eeb37dbaea36971c84af1e115fd5ea8","collectionName":"cv1","collectionID":453584804738224339}

query:

{"level":"WARN","time":"2024/11/21 12:19:01.265 +00:00","caller":"delegator/delegator.go:291","message":"delegator search failed to wait tsafe","traceID":"665be0a78bc033466ab070661b8831bd","collectionID":453584804738224339,"channel":"by-dev-rootcoord-dml_6_453584804738224339v0","replicaID":453584804871405581,"error":"context deadline exceeded"}
{"level":"WARN","time":"2024/11/21 12:19:01.265 +00:00","caller":"querynodev2/handlers.go:376","message":"failed to search on delegator","traceID":"665be0a78bc033466ab070661b8831bd","msgID":454035955343126262,"collectionID":453584804738224339,"channel":"by-dev-rootcoord-dml_6_453584804738224339v0","scope":"All","nq":1,"error":"context deadline exceeded"}

this sounds to me that there is some issue with mix coord, isnt it? it has warnings as well, but timings are not matching with search attempts.

{"level":"WARN","time":"2024/11/21 12:24:30.717 +00:00","caller":"datacoord/meta.go:699","message":"meta update: get segment failed - segment not found","segmentID":454035955324283695,"segment nil":true,"segment unhealthy":true}
{"level":"WARN","time":"2024/11/21 12:24:37.410 +00:00","caller":"datacoord/task_index.go:204","message":"failed to get dim from field type params","field type":"VarChar","error":"dim not found in params","errorVerbose":"dim not found in params\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n  | \t/workspace/source/internal/storage/utils.go:233\n  | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n  | \t/workspace/source/internal/datacoord/task_index.go:202\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:244\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:211\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).schedule\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:179\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"}
{"level":"WARN","time":"2024/11/21 12:24:37.532 +00:00","caller":"datacoord/task_index.go:204","message":"failed to get dim from field type params","field type":"VarChar","error":"dim not found in params","errorVerbose":"dim not found in params\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n  | \t/workspace/source/internal/storage/utils.go:233\n  | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n  | \t/workspace/source/internal/datacoord/task_index.go:202\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:244\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:211\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).schedule\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:179\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"}
{"level":"WARN","time":"2024/11/21 12:24:38.394 +00:00","caller":"datacoord/task_index.go:204","message":"failed to get dim from field type params","field type":"VarChar","error":"dim not found in params","errorVerbose":"dim not found in params\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n  | \t/workspace/source/internal/storage/utils.go:233\n  | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n  | \t/workspace/source/internal/datacoord/task_index.go:202\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:244\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:211\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).schedule\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:183\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"}
{"level":"WARN","time":"2024/11/21 12:24:39.396 +00:00","caller":"datacoord/task_index.go:204","message":"failed to get dim from field type params","field type":"VarChar","error":"dim not found in params","errorVerbose":"dim not found in params\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n  | \t/workspace/source/internal/storage/utils.go:233\n  | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n  | \t/workspace/source/internal/datacoord/task_index.go:202\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:244\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:211\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).schedule\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:183\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"}
{"level":"WARN","time":"2024/11/21 12:24:39.397 +00:00","caller":"grpcclient/client.go:404","message":"call received grpc error","clientRole":"indexnode-157","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:24:39.397 +00:00","caller":"retry/retry.go:106","message":"retry func failed","retried":0,"error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:24:39.397 +00:00","caller":"grpcclient/client.go:556","message":"ClientBase Call grpc call get error","role":"indexnode-157","address":"10.100.143.8:21121","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n/workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:92\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:140\n  | github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | \t/workspace/source/internal/datacoord/indexnode_manager.go:129\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | /workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:24:39.397 +00:00","caller":"datacoord/indexnode_manager.go:131","message":"get IndexNode slots failed","nodeID":157,"error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n/workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:92\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:140\n  | github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | \t/workspace/source/internal/datacoord/indexnode_manager.go:129\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | /workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:24:41.395 +00:00","caller":"grpcclient/client.go:404","message":"call received grpc error","clientRole":"indexnode-157","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:24:41.395 +00:00","caller":"retry/retry.go:106","message":"retry func failed","retried":0,"error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:24:41.396 +00:00","caller":"grpcclient/client.go:556","message":"ClientBase Call grpc call get error","role":"indexnode-157","address":"10.100.143.8:21121","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n/workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:92\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:140\n  | github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | \t/workspace/source/internal/datacoord/indexnode_manager.go:129\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | /workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:24:41.396 +00:00","caller":"datacoord/indexnode_manager.go:131","message":"get IndexNode slots failed","nodeID":157,"error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n/workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:92\n  | github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | \t/workspace/source/internal/distributed/indexnode/client/client.go:140\n  | github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | \t/workspace/source/internal/datacoord/indexnode_manager.go:129\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/indexnode/client/client.go:92 github.com/milvus-io/milvus/internal/distributed/indexnode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/indexnode/client/client.go:140 github.com/milvus-io/milvus/internal/distributed/indexnode/client.(*Client).GetJobStats\n  | /workspace/source/internal/datacoord/indexnode_manager.go:129 github.com/milvus-io/milvus/internal/datacoord.(*IndexNodeManager).PickClient.func1\n  | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:25:08.390 +00:00","caller":"utils/types.go:67","message":"delta position is quite stale","collectionID":453584804738224339,"segmentID":454035955324476843,"channel":"by-dev-rootcoord-dml_6_453584804738224339v0","posTs":454083554204975120,"posTime":"2024/11/21 12:14:52.591 +00:00","tsLag":"10m15.799799912s"}
{"level":"WARN","time":"2024/11/21 12:25:37.426 +00:00","caller":"datacoord/task_scheduler.go:300","message":"task running time is too long","taskID":454035955344814714,"running time(ms)":9223372036854}
{"level":"WARN","time":"2024/11/21 12:25:38.404 +00:00","caller":"datacoord/task_scheduler.go:300","message":"task running time is too long","taskID":454035955344814718,"running time(ms)":9223372036854}
{"level":"WARN","time":"2024/11/21 12:26:18.391 +00:00","caller":"utils/types.go:67","message":"delta position is quite stale","collectionID":453584804738224249,"segmentID":454035955324283695,"channel":"by-dev-rootcoord-dml_4_453584804738224249v0","posTs":454083573878620166,"posTime":"2024/11/21 12:16:07.640 +00:00","tsLag":"10m10.751654025s"}
{"level":"WARN","time":"2024/11/21 12:26:28.388 +00:00","caller":"utils/types.go:67","message":"delta position is quite stale","collectionID":453584804738224249,"segmentID":454035955324476922,"channel":"by-dev-rootcoord-dml_4_453584804738224249v0","posTs":454083573878620166,"posTime":"2024/11/21 12:16:07.640 +00:00","tsLag":"10m20.748952996s"}
{"level":"WARN","time":"2024/11/21 12:26:34.889 +00:00","caller":"grpcclient/client.go:404","message":"call received grpc error","clientRole":"datacoord","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:26:34.889 +00:00","caller":"retry/retry.go:106","message":"retry func failed","retried":0,"error":"rpc error: code = Canceled desc = context canceled"}
{"level":"WARN","time":"2024/11/21 12:26:34.889 +00:00","caller":"grpcclient/client.go:556","message":"ClientBase Call grpc call get error","role":"datacoord","address":"10.100.139.33:13333","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:610\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:609\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).getLoadInfo\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:697\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:196\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:161\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:124\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:26:34.890 +00:00","caller":"meta/coordinator_broker.go:335","message":"failed to get segment index info","collectionID":453584804738224249,"segmentID":454035955324476922,"error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:610\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:609\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).getLoadInfo\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:697\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:196\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:161\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:124\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:26:34.890 +00:00","caller":"task/executor.go:700","message":"failed to get index of segment","level":"L1","error":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo: rpc error: code = Canceled desc = context canceled","errorVerbose":"stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:610\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:609\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\n  | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).getLoadInfo\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:697\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:196\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:161\n  | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n  | \t/workspace/source/internal/querycoordv2/task/executor.go:124\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/datacoord/client/client.go:610 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/distributed/datacoord/client/client.go:609 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetIndexInfos\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:323 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo.func1\n  | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n  | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:322 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetIndexInfo\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"}
{"level":"WARN","time":"2024/11/21 12:26:37.431 +00:00","caller":"datacoord/task_scheduler.go:300","message":"task running time is too long","taskID":454035955344819731,"running time(ms)":9223372036854}
{"level":"WARN","time":"2024/11/21 12:26:38.389 +00:00","caller":"utils/types.go:67","message":"delta position is quite stale","collectionID":453584804738224249,"segmentID":454035955324283783,"channel":"by-dev-rootcoord-dml_4_453584804738224249v0","posTs":454083573878620166,"posTime":"2024/11/21 12:16:07.640 +00:00","tsLag":"10m30.749302353s"}
{"level":"WARN","time":"2024/11/21 12:26:38.398 +00:00","caller":"datacoord/task_scheduler.go:300","message":"task running time is too long","taskID":454035955344819734,"running time(ms)":9223372036854}
{"level":"WARN","time":"2024/11/21 12:26:46.889 +00:00","caller":"meta/coordinator_broker.go:413","message":"failed to fetch index meta","collectionID":453584804738224249,"error":"context canceled"}
{"level":"WARN","time":"2024/11/21 12:26:46.889 +00:00","caller":"task/executor.go:709","message":"fail to get index meta of collection","level":"L1","error":"context canceled"}

From all the information we have, the problem might be kafka consume too slow.

It seems that milvus consumes kafka hit timeout。

This could significantly slow down search under some of the consistency. Do you have monitoring metrics about your kafka consumption time?

[artesby]

looks weird to me that log prints "topic":"by-dev-rootcoord-dml_3", but according to logs\metrics, this topic is not used -- only _4, _5, _6 are used.
also i dont see replication lag. and p95 fetch seems reasonable to me

may be i can force milvus to use some hardcoded topic?