Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Milvus-standalone 2.3 suddenly crashed #26723

Closed
1 task done
darkerin opened this issue Aug 30, 2023 · 7 comments
Closed
1 task done

[Bug]: Milvus-standalone 2.3 suddenly crashed #26723

darkerin opened this issue Aug 30, 2023 · 7 comments
Assignees
@yah01 @darkerin
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.3

Comments

@darkerin
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: v2.3.0
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):  rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS):  Ubuntu18.04
- CPU/Memory:  4C 8G
- GPU: 
- Others:

Current Behavior

Milvus-standalone container crashed when i query data use attu，this happened several times

goroutine 91588 [select]:
runtime.gopark(0xc06c5a2f28?, 0x2?, 0x98?, 0x2d?, 0xc06c5a2ed4?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc06c5a2d50 sp=0xc06c5a2d30 pc=0x1543096
runtime.selectgo(0xc06c5a2f28, 0xc06c5a2ed0, 0x47?, 0x0, 0x1?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc06c5a2e90 sp=0xc06c5a2d50 pc=0x1553bbe
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).work(0xc05972aec0)
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:68 +0x1d6 fp=0xc06c5a2fc8 sp=0xc06c5a2e90 pc=0x359cb56
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start.func1()
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x26 fp=0xc06c5a2fe0 sp=0xc06c5a2fc8 pc=0x359c8e6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc06c5a2fe8 sp=0xc06c5a2fe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x7b

goroutine 92020 [select]:
runtime.gopark(0xc097009750?, 0x2?, 0xe0?, 0x95?, 0xc097009714?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc097009598 sp=0xc097009578 pc=0x1543096
runtime.selectgo(0xc097009750, 0xc097009710, 0x19?, 0x0, 0x1?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0970096d8 sp=0xc097009598 pc=0x1553bbe
github.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).work(0xc05657c2a0)
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/stream_pipeline.go:52 +0xe5 fp=0xc0970097c8 sp=0xc0970096d8 pc=0x359d645
github.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).Start.func1.1()
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/stream_pipeline.go:92 +0x26 fp=0xc0970097e0 sp=0xc0970097c8 pc=0x359e166
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0970097e8 sp=0xc0970097e0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).Start.func1
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/stream_pipeline.go:92 +0xc5

goroutine 91916 [select]:
runtime.gopark(0xc003713f28?, 0x2?, 0x98?, 0x3d?, 0xc003713ed4?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc003713d50 sp=0xc003713d30 pc=0x1543096
runtime.selectgo(0xc003713f28, 0xc003713ed0, 0x47?, 0x0, 0x1?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc003713e90 sp=0xc003713d50 pc=0x1553bbe
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).work(0xc0466c5b80)
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:68 +0x1d6 fp=0xc003713fc8 sp=0xc003713e90 pc=0x359cb56
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start.func1()
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x26 fp=0xc003713fe0 sp=0xc003713fc8 pc=0x359c8e6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc003713fe8 sp=0xc003713fe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x7b

goroutine 92114 [select]:
runtime.gopark(0xc002e74f28?, 0x2?, 0x88?, 0x4d?, 0xc002e74ed4?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc002e74d50 sp=0xc002e74d30 pc=0x1543096
runtime.selectgo(0xc002e74f28, 0xc002e74ed0, 0x48?, 0x0, 0x1?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc002e74e90 sp=0xc002e74d50 pc=0x1553bbe
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).work(0xc05b3ce5c0)
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:68 +0x1d6 fp=0xc002e74fc8 sp=0xc002e74e90 pc=0x359cb56
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start.func1()
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x26 fp=0xc002e74fe0 sp=0xc002e74fc8 pc=0x359c8e6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc002e74fe8 sp=0xc002e74fe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x7b

goroutine 91917 [select]:
runtime.gopark(0xc06c611f28?, 0x2?, 0x88?, 0x1d?, 0xc06c611ed4?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc06c611d50 sp=0xc06c611d30 pc=0x1543096
runtime.selectgo(0xc06c611f28, 0xc06c611ed0, 0x47?, 0x0, 0x1?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc06c611e90 sp=0xc06c611d50 pc=0x1553bbe
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).work(0xc0466c5bc0)
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:68 +0x1d6 fp=0xc06c611fc8 sp=0xc06c611e90 pc=0x359cb56
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start.func1()
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x26 fp=0xc06c611fe0 sp=0xc06c611fc8 pc=0x359c8e6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc06c611fe8 sp=0xc06c611fe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x7b

goroutine 91905 [select]:
runtime.gopark(0xc002b9cf28?, 0x2?, 0x88?, 0xcd?, 0xc002b9ced4?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc002b9cd50 sp=0xc002b9cd30 pc=0x1543096
runtime.selectgo(0xc002b9cf28, 0xc002b9ced0, 0x47?, 0x0, 0x1?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc002b9ce90 sp=0xc002b9cd50 pc=0x1553bbe
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).work(0xc05c95ac40)
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:68 +0x1d6 fp=0xc002b9cfc8 sp=0xc002b9ce90 pc=0x359cb56
github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start.func1()
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x26 fp=0xc002b9cfe0 sp=0xc002b9cfc8 pc=0x359c8e6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc002b9cfe8 sp=0xc002b9cfe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/util/pipeline.(*nodeCtx).Start
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/node.go:51 +0x7b

goroutine 91918 [select]:
runtime.gopark(0xc0061faf50?, 0x2?, 0xe0?, 0xad?, 0xc0061faf14?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0061fad98 sp=0xc0061fad78 pc=0x1543096
runtime.selectgo(0xc0061faf50, 0xc0061faf10, 0x19?, 0x0, 0x1?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0061faed8 sp=0xc0061fad98 pc=0x1553bbe
github.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).work(0xc05657c310)
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/stream_pipeline.go:52 +0xe5 fp=0xc0061fafc8 sp=0xc0061faed8 pc=0x359d645
github.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).Start.func1.1()
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/stream_pipeline.go:92 +0x26 fp=0xc0061fafe0 sp=0xc0061fafc8 pc=0x359e166
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0061fafe8 sp=0xc0061fafe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).Start.func1
	/go/src/github.com/milvus-io/milvus/internal/util/pipeline/stream_pipeline.go:92 +0xc5

goroutine 245445 [semacquire]:
runtime.gopark(0x60?, 0x60?, 0x40?, 0xda?, 0x7fec75abd398?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0887ab7b0 sp=0xc0887ab790 pc=0x1543096
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.semacquire1(0xc07fb78bf8, 0x30?, 0x1, 0x0, 0x51?)
	/usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc0887ab818 sp=0xc0887ab7b0 pc=0x1554e6f
sync.runtime_Semacquire(0xc08009b0e0?)
	/usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc0887ab850 sp=0xc0887ab818 pc=0x1574d07
sync.(*WaitGroup).Wait(0xc04d3ca890?)
	/usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc0887ab878 sp=0xc0887ab850 pc=0x1588f4b
github.com/milvus-io/milvus/internal/querynodev2/segments.searchSegments({0x44b3c98?, 0xc07ff06af0}, {0xc08009ede0, 0x1, 0xc07ff06af0?}, 0x2, 0xc080083140)
	/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/search.go:76 +0x385 fp=0xc0887ab9f0 sp=0xc0887ab878 pc=0x3557985
github.com/milvus-io/milvus/internal/querynodev2/segments.SearchStreaming({0x44b3c98, 0xc07ff06af0}, 0x11?, 0xc04d3cab00?, 0x44a0db8?, {0x0?, 0x2710?, 0xc04d3cab00?}, {0xc07ff104d0, 0x1, ...})
	/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/search.go:118 +0xc8 fp=0xc0887aba70 sp=0xc0887ab9f0 pc=0x3558548
github.com/milvus-io/milvus/internal/querynodev2/tasks.(*SearchTask).Execute(0xc07fd6dbc0)
	/go/src/github.com/milvus-io/milvus/internal/querynodev2/tasks/task.go:141 +0x437 fp=0xc0887abeb8 sp=0xc0887aba70 pc=0x359a3f7
github.com/milvus-io/milvus/internal/querynodev2/tasks.(*scheduler).exec.func1()
	/go/src/github.com/milvus-io/milvus/internal/querynodev2/tasks/concurrent_safe_scheduler.go:193 +0xec fp=0xc0887abf28 sp=0xc0887abeb8 pc=0x35977cc
github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1()
	/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:78 +0xbc fp=0xc0887abf88 sp=0xc0887abf28 pc=0x30a7c3c
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
	/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67 +0x97 fp=0xc0887abfe0 sp=0xc0887abf88 pc=0x28cb777
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0887abfe8 sp=0xc0887abfe0 pc=0x1579301
created by github.com/panjf2000/ants/v2.(*goWorker).run
	/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0x65

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@darkerin darkerin added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 30, 2023
@yanliang567
Copy link
Contributor

@darkerin Could you please refer this doc to export the whole Milvus logs for investigation?
/assign @darkerin

@yanliang567 yanliang567 added triage/needs-information Indicates an issue needs more information in order to work on it. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 30, 2023
@darkerin
Copy link
Author

darkerin commented Aug 30, 2023 •
edited
Loading

@yanliang567

milvus.log

@yanliang567
Copy link
Contributor

/assign @jiaoew1991
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed triage/needs-information Indicates an issue needs more information in order to work on it. labels Aug 30, 2023
@xiaofan-luan xiaofan-luan added this to the 2.3 milestone Aug 30, 2023
@jiaoew1991
Copy link
Contributor

/assign @yah01
/unassign

@sre-ci-robot sre-ci-robot assigned yah01 and unassigned jiaoew1991 Aug 31, 2023
@darkerin
Copy link
Author

New problem causes container to exit abnormally

#26748

@darkerin
Copy link
Author

Did not appear again for the time being, Maybe it has something to do with the old version of milvus.yml

@locustbaby
Copy link
Contributor

Did not appear again for the time being, Maybe it has something to do with the old version of milvus.yml

Hi @darkerin, Would you mind sharing the milvus.yml with us?
And something want to confirm with you, is your log level debug?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yah01 yah01

@darkerin darkerin

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.3
Development

No branches or pull requests
6 participants
@jiaoew1991 @yah01 @locustbaby @yanliang567 @xiaofan-luan @darkerin