[Bug]: The stat operation encountered an error during the internal execution of a BulkInsert.

[lentitude2tk]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka): --   
- SDK version(e.g. pymilvus v2.0.0rc2): milvus-java-sdk
- OS(Ubuntu or CentOS): --
- CPU/Memory: --
- GPU: --
- Others: Cloud

Current Behavior

1. Use the latest master image to create a bulkInsert task.
2. The bulkInsert task ultimately fails without providing a failure reason.

[2024/12/10 08:24:33.996 +00:00] [INFO] [datacoord/services.go:1782] ["GetImportProgress done"] [jobID=454491344001365247] [resp="status:{}  state:Failed  collection_name:\"MultiFieldCollection\"  task_progresses:{file_name:\"[bulkload/in01-dcd8e3875a6162b/job-016881078e8tilww9au69t/1/MultiFieldCollection.json]\"  file_size:27822  complete_time:\"2024-12-10T07:57:15Z\"  state:\"Failed\"  total_rows:10}  task_progresses:{file_name:\"[bulkload/in01-dcd8e3875a6162b/job-016881078e8tilww9au69t/0/MultiFieldCollection.json]\"  file_size:27822  complete_time:\"2024-12-10T07:57:15Z\"  state:\"Failed\"  total_rows:10}  start_time:\"2024-12-10T07:57:05Z\""] |  
  |  

https://grafana-vdc-dev.zilliz.cc/explore?orgId=1&panes=%7B%22QXf%22:%7B%22datasource%22:%22Zgl22j0Vk%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bcluster%3D%5C%22sit-vdc-test%5C%22,namespace%3D%5C%22milvus-in01-dcd8e3875a6162b%5C%22,pod%3D~%5C%22in01-dcd8e3875a6162b-milvus-.%2A%5C%22%7D%7C%3D%5C%22454491344001365247%5C%22%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22Zgl22j0Vk%22%7D,%22editorMode%22:%22code%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1

Expected Behavior

The bulkInsert is expected to succeed.

Steps To Reproduce

Create a bulkInsert task using the latest master image.

Milvus Log

[2024/12/10 07:57:14.460 +00:00] [INFO] [importv2/task_import.go:225] ["sync import data done"] [taskID=454491344001365282] [jobID=454491344001365247] [collectionID=454483308634540295] [type=ImportTask] [segmentInfo="segmentID:454491344001365283 imported_rows:10 binlogs:{fieldID:105 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/105/454491344001365316" log_size:476 memory_size:41}} binlogs:{fieldID:107 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/107/454491344001365317" log_size:541 memory_size:81}} binlogs:{fieldID:100 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/100/454491344001365321" log_size:469 memory_size:81}} binlogs:{fieldID:102 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/102/454491344001365322" log_size:638 memory_size:402}} binlogs:{fieldID:103 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/103/454491344001365323" log_size:476 memory_size:11}} binlogs:{binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/0/454491344001365327" log_size:509 memory_size:81}} binlogs:{fieldID:101 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/101/454491344001365315" log_size:7567 memory_size:5124}} binlogs:{fieldID:108 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/108/454491344001365318" log_size:352 memory_size:11}} binlogs:{fieldID:109 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/109/454491344001365319" log_size:1048 memory_size:1013}} binlogs:{fieldID:1 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/1/454491344001365320" log_size:468 memory_size:81}} binlogs:{fieldID:104 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/104/454491344001365324" log_size:476 memory_size:21}} binlogs:{fieldID:106 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/106/454491344001365325" log_size:467 memory_size:41}} binlogs:{fieldID:110 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/insert_log/454483308634540295/454483308634540296/454491344001365283/110/454491344001365326" log_size:393 memory_size:181}} statslogs:{fieldID:100 binlogs:{entries_num:10 timestamp_from:454509837275627521 timestamp_to:454509837275627521 log_path:"dcd8e3875a6162b/stats_log/454483308634540295/454483308634540296/454491344001365283/100/454491344001365328" log_size:189 memory_size:189}}"]

[2024/12/10 07:57:42.483 +00:00] [INFO] [datacoord/task_stats.go:255] ["query stats task result success"] [taskID=454491344001365352] [segmentID=454491344001365283] ["result state"=JobStateFailed] [failReason="The specified key does not exist.: key not found[key=insert_log/454483308634540295/454483308634540296/454491344001365283/105/454491344001365316]"]

https://grafana-vdc-dev.zilliz.cc/explore?orgId=1&panes=%7B%22QXf%22:%7B%22datasource%22:%22Zgl22j0Vk%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bcluster%3D%5C%22sit-vdc-test%5C%22,namespace%3D%5C%22milvus-in01-dcd8e3875a6162b%5C%22,pod%3D~%5C%22in01-dcd8e3875a6162b-milvus-.%2A%5C%22%7D%7C%3D%5C%22454491344001365284%5C%22%20%7C~%20%5C%22datacoord%2F%5C%22%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22Zgl22j0Vk%22%7D,%22editorMode%22:%22code%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1

Anything else?

No response

[xiaocai2333]

/assign
/unassign @yanliang567

[xiaocai2333]

[2024/12/10 07:57:42.483 +00:00] [INFO] [datacoord/task_stats.go:255] ["query stats task result success"] [taskID=454491344001365352] [segmentID=454491344001365283] ["result state"=JobStateFailed] [failReason="The specified key does not exist.: key not found[key=insert_log/454483308634540295/454483308634540296/454491344001365283/105/454491344001365316]"]

Decompress binlogs path with wrong RootPath in stats tasks.

[xiaocai2333]

It has been fixed. Please verify on it. @lentitude2tk
/assign @lentitude2tk
/unassign

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.