runtime: unexpected return pc for golang.gopark #47003
Labels
compiler/runtime
Issues related to the Go compiler and/or runtime.
FrozenDueToAge
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
WaitingForInfo
Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Comments
|
This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector . |
dmitshur
added
the
NeedsInvestigation
yes I've tried using the race-detector and there is no complaints from the tests Just to make sure I merely need to run |
|
The race detector can only detect races if your test code exercises them. The litmus test is to pass -race to go build and deploy the application. |
|
@davecheney oh, thanks, that's something I missed out, I will deploy the race-enabled version to see if we can catch anything and report back |
|
Unfortunately ... enabled race detector does not give me more information regarding the issue. |
|
Since you can reproduce the issue, can you provide complete instructions to let others do so? |
|
@networkimprov sorry for the late reply, I cannot really reproduce it , I just keep it running in our Staging environment, and the occurrence seems random to me. In addition to enable race on the build, I've enabled core dump this time, hope I can get more info and share here later. Just my feeling, it could be a runtime issue with memory management |
|
Luckily I caught a coredump this time, but the backtrace is mysterious to me also |
|
Corresponding stderr goes like this |
|
Output from |
|
perhaps related, I got another stack from a user as below: |
|
Feels like starting from go 1.16 the new GC may accidentally corrupt some mem area of parked or running goroutines ? |
|
@imcom Anything is possible, but as far as I can tell you are the only person reporting this specific problem. That makes it seem more likely to be due to memory corruption in your program rather than in the garbage collector. Memory corruption could come, for example, from uses of unsafe, or calls to C code, or memory races. Unfortunately it's going to be hard for us to analyze this if we can't reproduce it. Is it possible for you to share the code with us, even if the problem doesn't happen very often? |
|
Hi @ianlancetaylor! Glad to hear from you. Actually I am not the only one who experienced this, there are 3 ppl that I know of. But indeed, we are working with the same program GoBGP. I am pretty sure there is no CGO involved, but since the project uses reflection so unsafe is there. I tried race-detector enabled build but it did not produce any meaningful dump if any. I am also writing a fuzzing tool on Side note on ppl experienced this, we are not in the same organisation and our use cases/setup are different |
|
Was this fixed by 08ecdf7 (Go 1.18)? That commit links to #49686 which has a similar looking stack trace: https://build.golang.org/log/c443a4442d00c324be6e09f70d5e3bc401493531 |
|
@tomfitzhenry , I don't think this was fixed by 08ecdf7. That commit only affects weak memory architectures, and this issue is being reporting on amd64, which is not a weak memory architecture. Other than being panics on the runtime stack, the two stack traces look pretty dissimilar to me. |
gopherbot
added
the
compiler/runtime
seankhliao
added
the
WaitingForInfo
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Did not try
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
I was running gobgpd:v2.23 in production environment and I have no clue how to troubleshoot or reproduce this issue.
This only happened once in my production environment, no applicable logs when the crash happened
What did you expect to see?
I do not see why there is this unexpected return pc fatal error. My main concern is that since I do not know why it happened, I cannot tell when it will happen again if ever. Also I need help on interpreting the error dump and how to troubleshoot a runtime error like this. Thanks in advance.
What did you see instead?
runtime: unexpected return pc for runtime.gopark called from 0xc00a1946f0 stack: frame={sp:0xc000e694c8, fp:0xc000e694e8} stack=[0xc000e68000,0xc000e6a000) 000000c000e693c8: 0000000000000000 0000000000000000 000000c000e693d8: 000000c000d3f400 000000000048b4dc 000000c000e693e8: 000000c0000428d8 000000c000190800 000000c000e693f8: 0000000000000004 000000c000d3f428 000000c000e69408: 000000000075e9a7 000000c0000428d8 000000c000e69418: 0000000000d02820 000000c0042ae000 000000c000e69428: 000000c000d3f470 000000000075ec4c 000000c000e69438: 000000c0000428a0 000000c0042ae000 000000c000e69448: 000000c001497ae0 0000000000000010 000000c000e69458: 000000c0042ae000 000000c0000428a0 000000c000e69468: 0000000000d4e9e8 000000c000d3fe58 000000c000e69478: 0000000000b21f45 000000c001497ae0 000000c000e69488: 000000c000d3f588 0000000000000001 000000c000e69498: 0000000000000001 000000c000d3f9c0 000000c000e694a8: 0000000000000000 0000000000000000 000000c000e694b8: 0000000000000000 0000000000000000 000000c000e694c8: <000000c009302408 000000c00483f230 000000c000e694d8: 000000c007c172d8 !000000c00a1946f0 000000c000e694e8: >0000000000e238c8 000000c00d9b66d8 000000c000e694f8: 000000c00a8af8c0 0000000000000013 000000c000e69508: 0000000000000013 000000c004e5fce0 000000c000e69518: 000000c0073908b8 000000c00e24e468 000000c000e69528: 000000c000d3ff10 000000c00045b2c0 000000c000e69538: 000000c00a079008 000000c000464c40 000000c000e69548: 000000c00045b2b0 000000c000f7e420 000000c000e69558: 000000c0074ac7f8 000000c000139968 000000c000e69568: 000000c0035c1fc8 000000c000caefb0 000000c000e69578: 0000000000000000 0000000000000000 000000c000e69588: 0000000000bba9a0 0000000000df9f50 000000c000e69598: 000000c00dcce078 000000c0030e15d8 000000c000e695a8: 000000c00d8b3758 000000c009d647e0 000000c000e695b8: 000000c00901eae0 000000c00f5aff38 000000c000e695c8: 000000c00853ae88 000000c007897a28 000000c000e695d8: 000000c00de05e78 000000c011ce25a0 fatal error: unknown caller pc fatal error: unexpected signal during runtime execution runtime: unexpected return pc for runtime.gopark called from 0x0 stack: frame={sp:0xc000e20dc0, fp:0xc000e20de0} stack=[0xc000e20800,0xc000e21000) 000000c000e20cc0: 0000000000000000 0000000000000000 000000c000e20cd0: 0000000000000000 0000000000000000 000000c000e20ce0: 0000000000000000 0000000000000000 000000c000e20cf0: 0000000000000000 0000000000000000 000000c000e20d00: 0000000000000000 0000000000000000 000000c000e20d10: 0000000000000000 0000000000000000 000000c000e20d20: 0000000000000000 0000000000000000 000000c000e20d30: 0000000000000000 0000000000000000 000000c000e20d40: 0000000000000000 0000000000000000 000000c000e20d50: 0000000000000000 0000000000000000 000000c000e20d60: 0000000000000000 0000000000000000 000000c000e20d70: 0000000000000000 0000000000000000 000000c000e20d80: 0000000000000000 0000000000000000 000000c000e20d90: 0000000000000000 0000000000000000 000000c000e20da0: 0000000000000000 0000000000000000 000000c000e20db0: 0000000000000000 0000000000000000 000000c000e20dc0: <0000000000000000 0000000000000000 000000c000e20dd0: 0000000000000000 !0000000000000000 000000c000e20de0: >0000000000000000 0000000000000000 000000c000e20df0: 0000000000000000 0000000000000000 000000c000e20e00: 0000000000000000 0000000000000000 000000c000e20e10: 0000000000000000 0000000000000000 000000c000e20e20: 0000000000000000 0000000000000000 000000c000e20e30: 0000000000000000 0000000000000000 000000c000e20e40: 0000000000000000 0000000000000000 000000c000e20e50: 0000000000000000 0000000000000000 000000c000e20e60: 0000000000000000 0000000000000000 000000c000e20e70: 0000000000000000 0000000000000000 000000c000e20e80: 0000000000000000 0000000000000000 000000c000e20e90: 0000000000000000 0000000000000000 000000c000e20ea0: 0000000000000000 0000000000000000 000000c000e20eb0: 0000000000000000 0000000000000000 000000c000e20ec0: 0000000000000000 0000000000000000 000000c000e20ed0: 0000000000000000 0000000000000000 fatal error: unknown caller pc runtime: unexpected return pc for runtime.gopark called from 0x0 stack: frame={sp:0xc000e5d4b8, fp:0xc000e5d4d8} stack=[0xc000e5c000,0xc000e5e000) 000000c000e5d3b8: 0000000000000000 0000000000000000 000000c000e5d3c8: 0000000000000000 0000000000000000 000000c000e5d3d8: 0000000000000000 0000000000000000 000000c000e5d3e8: 0000000000000000 0000000000000000 000000c000e5d3f8: 0000000000000000 0000000000000000 000000c000e5d408: 0000000000000000 0000000000000000 000000c000e5d418: 0000000000000000 0000000000000000 000000c000e5d428: 0000000000000000 0000000000000000 000000c000e5d438: 0000000000000000 0000000000000000 000000c000e5d448: 0000000000000000 0000000000000000 000000c000e5d458: 0000000000000000 0000000000000000 000000c000e5d468: 0000000000000000 0000000000000000 000000c000e5d478: 0000000000000000 0000000000000000 000000c000e5d488: 0000000000000000 0000000000000000 000000c000e5d498: 0000000000000000 0000000000000000 000000c000e5d4a8: 0000000000000000 0000000000000000 000000c000e5d4b8: <0000000000000000 0000000000000000 000000c000e5d4c8: 0000000000000000 !0000000000000000 000000c000e5d4d8: >0000000000000000 0000000000000000 000000c000e5d4e8: 0000000000000000 0000000000000000 000000c000e5d4f8: 0000000000000000 0000000000003023 000000c000e5d508: 0000000000e23e58 000000c000396600 000000c000e5d518: 0000000000000000 0000000000e1b280 000000c000e5d528: 000000c01468d350 000000c0013ec520 000000c000e5d538: 0000000000000000 000000c000a55920 000000c000e5d548: 000000c002f53a00 000000000000001f 000000c000e5d558: 0000000000000000 0000000000000000 000000c000e5d568: 0000000000000000 0000000000000000 000000c000e5d578: 000000c012edf680 0000000000e09f40 000000c000e5d588: 000000c01468d380 000000c000623560 000000c000e5d598: 000000c000b75700 000000c0102fd140 000000c000e5d5a8: 0000000000000000 0000000000000000 000000c000e5d5b8:The text was updated successfully, but these errors were encountered: