Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Suppress Panic in Emitter #419

Merged
merged 1 commit into from
Sep 28, 2023
Merged

Suppress Panic in Emitter #419

merged 1 commit into from
Sep 28, 2023

Conversation

wangzlei
Copy link
Contributor

Issue #, if available:

The potential concurrency risk could lead to Default_Emitter.Emit() Panic during JSON marshaling. We would like to use recover() to prevent X-ray SDK crashes in the user's application before addressing the root cause.

  • fatal error 1:
panic: runtime error: index out of range [13] with length 13 [recovered]
    panic: runtime error: index out of range [13] with length 13

goroutine 41146499 [running]:
encoding/json.(*encodeState).marshal.func1()
    /usr/local/go/src/encoding/json/encode.go:291 +0x80
panic({0x1162f60?, 0x4001aa78a8?})
    /usr/local/go/src/runtime/panic.go:914 +0x218
encoding/json.mapEncoder.encode({0x40031e77a8?}, 0x40044c8340, {0x102d760?, 0x4001410478?, 0xffffffffffffffff?}, {0xe?, 0x0?})
    /usr/local/go/src/encoding/json/encode.go:745 +0x5b0
encoding/json.structEncoder.encode({{{0x4002bd6000, 0x19, 0x23}, 0x4000521d10, 0x4000521ec0}}, 0x40044c8340, {0x1205d60?, 0x4001410340?, 0x4002bd6000?}, {0x19?, ...})
    /usr/local/go/src/encoding/json/encode.go:706 +0x190
encoding/json.ptrEncoder.encode({0x40031e79c8?}, 0x40044c8340, {0x11f52e0?, 0x4001410340?, 0x40031e79d8?}, {0x60?, 0x72?})
    /usr/local/go/src/encoding/json/encode.go:878 +0x1c4
encoding/json.(*encodeState).reflectValue(0x40031e7a78?, {0x11f52e0?, 0x4001410340?, 0x1258e0?}, {0x40?, 0x65?})
    /usr/local/go/src/encoding/json/encode.go:323 +0x70
encoding/json.(*encodeState).marshal(0x40031e7b28?, {0x11f52e0?, 0x4001410340?}, {0xb4?, 0x8?})
    /usr/local/go/src/encoding/json/encode.go:295 +0xb8
encoding/json.Marshal({0x11f52e0, 0x4001410340})
    /usr/local/go/src/encoding/json/encode.go:162 +0x94
github.com/aws/aws-xray-sdk-go/xray.(*DefaultStreamingStrategy).StreamCompletedSubsegments(0x40031e7d38?, 0x4001410000)
    /go/pkg/mod/github.com/aws/[email protected]/xray/default_streaming_strategy.go:74 +0x2b4
github.com/aws/aws-xray-sdk-go/xray.packSegments.func1(0x4001410000)
    /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:97 +0xb4
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x400281f040, {0x4002767500, 0x6, 0x8})
    /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:110 +0xcc
github.com/aws/aws-xray-sdk-go/xray.(*DefaultEmitter).Emit(0x40006bc168, 0x18a44?)
    /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:66 +0x80
github.com/aws/aws-xray-sdk-go/xray.(*Segment).emit(0x4002f12fc0?)
    /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:468 +0x84
github.com/aws/aws-xray-sdk-go/xray.(*Segment).flush(0x1e89f98?)
    /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:510 +0x224
github.com/aws/aws-xray-sdk-go/xray.(*Segment).send(0x400281f040)
    /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:490 +0x60
github.com/aws/aws-xray-sdk-go/xray.(*Segment).handleContextDone(0x400281f040)
    /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:476 +0x54
github.com/aws/aws-xray-sdk-go/xray.BeginSegmentWithSampling.func1()
    /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:152 +0x40
created by github.com/aws/aws-xray-sdk-go/xray.BeginSegmentWithSampling in goroutine 40912008
    /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:150 +0xd2c
  • fatal error2 :
fatal error: concurrent map iteration and map write

goroutine 25931611 [running]:
reflect.mapiterinit(0x200?, 0x10e40e0?, 0x40021cd578?)
    /usr/local/go/src/runtime/map.go:1392 +0x1c
reflect.(*MapIter).Next(0x102d760?)
    /usr/local/go/src/reflect/value.go:1935 +0x60
encoding/json.mapEncoder.encode({0x40021cd7e8?}, 0x40025153c0, {0x102d760?, 0x4003cd2138?, 0x40021cd8a8?}, {0xe?, 0x0?})
    /usr/local/go/src/encoding/json/encode.go:744 +0x28c
encoding/json.structEncoder.encode({{{0x4000772000, 0x19, 0x23}, 0x4000512de0, 0x4000512e10}}, 0x40025153c0, {0x1205d60?, 0x4003cd2000?, 0x40021cd908?}, {0x28?, ...})
    /usr/local/go/src/encoding/json/encode.go:706 +0x190
encoding/json.ptrEncoder.encode({0x40021cda08?}, 0x40025153c0, {0x11f52e0?, 0x4003cd2000?, 0x40021cda18?}, {0x7c?, 0xf0?})
    /usr/local/go/src/encoding/json/encode.go:878 +0x1c4
encoding/json.(*encodeState).reflectValue(0x40021cdab8?, {0x11f52e0?, 0x4003cd2000?, 0x1258e0?}, {0x40?, 0x65?})
    /usr/local/go/src/encoding/json/encode.go:323 +0x70
encoding/json.(*encodeState).marshal(0x0?, {0x11f52e0?, 0x4003cd2000?}, {0xc8?, 0xf6?})
    /usr/local/go/src/encoding/json/encode.go:295 +0xb8
encoding/json.Marshal({0x11f52e0, 0x4003cd2000})
    /usr/local/go/src/encoding/json/encode.go:162 +0x94
github.com/aws/aws-xray-sdk-go/xray.packSegments.func1(0x4003cd2000)
    /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:100 +0x1a4
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x4000e03860, {0x400361f200, 0x5, 0x8})
    /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:110 +0xcc
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x400276d040, {0x400361f200, 0x5, 0x8})
    /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:109 +0xb0
github.com/aws/aws-xray-sdk-go/xray.(*DefaultEmitter).Emit(0x40007021c8, 0x18a44?)
    /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:66 +0x80
github.com/aws/aws-xray-sdk-go/xray.(*Segment).emit(0x40029fe480?)
    /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:468 +0x84
github.com/aws/aws-xray-sdk-go/xray.(*Segment).flush(0x50cf98?)
    /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:510 +0x224

Description of changes:

Adding defer recover() in Default_Emitter.Emit() function.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@wangzlei wangzlei requested a review from a team as a code owner September 27, 2023 18:31
@wangzlei wangzlei merged commit 065bcb0 into aws:master Sep 28, 2023
11 checks passed
@wangzlei wangzlei changed the title suppress Panic in Emitter Suppress Panic in Emitter Sep 29, 2023
@hxy1991
Copy link

hxy1991 commented Mar 5, 2024

@wangzlei @srprash

hi all, fatal error 2 still occurs in 1.8.2

fatal error: concurrent map iteration and map write

goroutine 3216017 [running]:
reflect.mapiterinit(0x40?, 0x14ae160?, 0x4003537568?)
 /usr/local/go/src/runtime/map.go:1392 +0x1c
reflect.(*MapIter).Next(0x13ad9e0?)
 /usr/local/go/src/reflect/value.go:1935 +0x60
encoding/json.mapEncoder.encode({0x40035377d8?}, 0x400379a700, {0x13ad9e0?, 0x4001770600?, 0x165acea?}, {0x6?, 0x0?})
 /usr/local/go/src/encoding/json/encode.go:744 +0x28c
encoding/json.structEncoder.encode({{{0x40008fa000, 0x19, 0x23}, 0x4000e400c0, 0x4000e400f0}}, 0x400379a700, {0x1628840?, 0x40017704e0?, 0x12c67350000000f?}, {0x5c?, ...})
 /usr/local/go/src/encoding/json/encode.go:706 +0x190
encoding/json.ptrEncoder.encode({0x40035379f8?}, 0x400379a700, {0x1614ca0?, 0x40017704e0?, 0x1000003537a38?}, {0x1c?, 0xd4?})
 /usr/local/go/src/encoding/json/encode.go:878 +0x1c4
encoding/json.(*encodeState).reflectValue(0x4003537aa8?, {0x1614ca0?, 0x40017704e0?, 0x12d2e0?}, {0xb8?, 0x7a?})
 /usr/local/go/src/encoding/json/encode.go:323 +0x70
encoding/json.(*encodeState).marshal(0x4003537b18?, {0x1614ca0?, 0x40017704e0?}, {0xd8?, 0x5e?})
 /usr/local/go/src/encoding/json/encode.go:295 +0xb8
encoding/json.Marshal({0x1614ca0, 0x40017704e0})
 /usr/local/go/src/encoding/json/encode.go:162 +0x94
github.com/aws/aws-xray-sdk-go/xray.packSegments.func1(0x40017704e0)
 /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:106 +0x1a4
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x4004535ba0, {0x400373f920, 0x2, 0x2})
 /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:116 +0xcc
github.com/aws/aws-xray-sdk-go/xray.packSegments(0x4000103860, {0x400373f920, 0x2, 0x2})
 /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:115 +0xb0
github.com/aws/aws-xray-sdk-go/xray.(*DefaultEmitter).Emit(0x4000a141b0, 0x18a44?)
 /go/pkg/mod/github.com/aws/[email protected]/xray/default_emitter.go:72 +0x9c
github.com/aws/aws-xray-sdk-go/xray.(*Segment).emit(0x4003d90600?)
 /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:468 +0x84
github.com/aws/aws-xray-sdk-go/xray.(*Segment).flush(0x1cf4f98?)
 /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:510 +0x224
github.com/aws/aws-xray-sdk-go/xray.(*Segment).send(0x4000103860)
 /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:490 +0x60
github.com/aws/aws-xray-sdk-go/xray.(*Segment).handleContextDone(0x4000103860)
 /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:476 +0x54
github.com/aws/aws-xray-sdk-go/xray.BeginSegmentWithSampling.func1()
 /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:152 +0x40
created by github.com/aws/aws-xray-sdk-go/xray.BeginSegmentWithSampling in goroutine 3030832
 /go/pkg/mod/github.com/aws/[email protected]/xray/segment.go:150 +0xd2c

@wangzlei wangzlei mentioned this pull request Mar 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srprash srprash srprash approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wangzlei @hxy1991 @srprash