Panic in honeycomb extension

[roryj-vendia]

Hi there! We use have been using the lambda honeycomb extension for a while now, and this is the first time I have seen a panic from the extension. I have posted below the output of our cloudwatch logs, which unfortunately doesn't have a lot of information besides the fact that the code segfaulted.

START RequestId: 66fa2fb8-8e4e-4fc9-9fbd-a89a466d675c Version: $LATEST
--
panic:
runtime error: invalid memory address or nil pointer dereference
 
[signal
SIGSEGV: segmentation violation
code=
0x1
addr=
0x8
pc=
0x4e6786
]
 
goroutine
16
[
running
]:
bytes.(*Reader).Read
(
0x0
,
0xc0007c4000
,
0x80000
,
0x80000
,
0x0
,
0x0
,
0x0
)
 
/usr/local/go/src/bytes/reader.go
:
41
+
0x26
 
net/http.(*readTrackingBody).Read
(
0xc0009d44c8
,
0xc0007c4000
,
0x80000
,
0x80000
,
0x1000
,
0x12
,
0x0
)
 
/usr/local/go/src/net/http/transport.go
:
632
+
0x55
 
net/http.(*http2clientStream).writeRequestBody
(
0xc000087e00
,
0xc0008a0400
,
0x0
,
0x0
)
 
/usr/local/go/src/net/http/h2_bundle.go
:
8099
+
0x534
 
net/http.(*http2clientStream).writeRequest
(
0xc000087e00
,
0xc0008a0400
,
0x0
,
0x0
)
 
/usr/local/go/src/net/http/h2_bundle.go
:
7836
+
0x5c5
 
net/http.(*http2clientStream).doRequest
(
0xc000087e00
,
0xc0008a0400
)
 
/usr/local/go/src/net/http/h2_bundle.go
:
7722
+
0x35
 
net/http.(*http2ClientConn).RoundTrip
 
/usr/local/go/src/net/http/h2_bundle.go
:
7651

+
0x29f
 
END RequestId: 66fa2fb8-8e4e-4fc9-9fbd-a89a466d675c
REPORT RequestId: 66fa2fb8-8e4e-4fc9-9fbd-a89a466d675c	Duration: 37.79 ms	Billed Duration: 38 ms	Memory Size: 768 MB	Max Memory Used: 127 MB
RequestId: 66fa2fb8-8e4e-4fc9-9fbd-a89a466d675c Error: exit status 2Extension.Crash

Versions

• Lambda runtime: python 3.9
• Lambda extension: arn:aws:lambda:us-east-1:702835727665:layer:honeycomb-lambda-extension-x86_64:8

Steps to reproduce

I am not sure! We have been using the extension in a lot of our Lambda functions up to this point, and this is the first example we have seen panic.

Additional context

We do have two additional layers as part of our lambda function, but these layers are python libraries that we use as part of our function code.

[MikeGoldsmith]

Hey @roryj-vendia - thanks for the report. We'll take a look and see what we can find.

[gavinyue-vendia]

2022-04-14T12:33:31.311-07:00 [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x4e6786]
2022-04-14T12:33:31.311-07:00
2022-04-14T12:33:31.311-07:00 goroutine 45 [running]:
2022-04-14T12:33:31.311-07:00 bytes.(*Reader).Read(0x0, 0xc00085a000, 0x80000, 0x80000
2022-04-14T12:33:31.312-07:00 ,
2022-04-14T12:33:31.312-07:00 0x0
2022-04-14T12:33:31.312-07:00 ,
2022-04-14T12:33:31.312-07:00 0x0
2022-04-14T12:33:31.312-07:00 , 0x0)
2022-04-14T12:33:31.312-07:00 /usr/local/go/src/bytes/reader.go:41 +0x26
2022-04-14T12:33:31.312-07:00 net/http.(*readTrackingBody).Read(0xc0005df158, 0xc00085a000, 0x80000, 0x80000, 0x1000, 0x12, 0x0)
2022-04-14T12:33:31.312-07:00 /usr/local/go/src/net/http/transport.go
2022-04-14T12:33:31.312-07:00 :632 +0x55
2022-04-14T12:33:31.312-07:00
2022-04-14T12:33:31.312-07:00 net/http.(*http2clientStream).writeRequestBody
2022-04-14T12:33:31.312-07:00 (
2022-04-14T12:33:31.312-07:00 0xc000083500
2022-04-14T12:33:31.312-07:00 ,
2022-04-14T12:33:31.312-07:00 0xc0000d3100
2022-04-14T12:33:31.312-07:00 ,
2022-04-14T12:33:31.312-07:00 0x0
2022-04-14T12:33:31.312-07:00 ,
2022-04-14T12:33:31.312-07:00 0x0
2022-04-14T12:33:31.312-07:00 )
2022-04-14T12:33:31.312-07:00
2022-04-14T12:33:31.312-07:00 /usr/local/go/src/net/http/h2_bundle.go
2022-04-14T12:33:31.312-07:00 :
2022-04-14T12:33:31.312-07:00 8099
2022-04-14T12:33:31.313-07:00 +
2022-04-14T12:33:31.313-07:00 0x534
2022-04-14T12:33:31.313-07:00
2022-04-14T12:33:31.313-07:00 net/http.(*http2clientStream).writeRequest
2022-04-14T12:33:31.313-07:00 (
2022-04-14T12:33:31.313-07:00 0xc000083500
2022-04-14T12:33:31.313-07:00 ,
2022-04-14T12:33:31.313-07:00 0xc0000d3100
2022-04-14T12:33:31.313-07:00 ,
2022-04-14T12:33:31.313-07:00 0x0
2022-04-14T12:33:31.313-07:00 ,
2022-04-14T12:33:31.313-07:00 0x0
2022-04-14T12:33:31.313-07:00 )
2022-04-14T12:33:31.313-07:00
2022-04-14T12:33:31.313-07:00 /usr/local/go/src/net/http/h2_bundle.go
2022-04-14T12:33:31.313-07:00 :
2022-04-14T12:33:31.313-07:00 7836
2022-04-14T12:33:31.313-07:00 +
2022-04-14T12:33:31.313-07:00 0x5c5
2022-04-14T12:33:31.313-07:00
2022-04-14T12:33:31.313-07:00 net/http.(*http2clientStream).doRequest
2022-04-14T12:33:31.313-07:00 (
2022-04-14T12:33:31.313-07:00 0xc000083500
2022-04-14T12:33:31.313-07:00 ,
2022-04-14T12:33:31.313-07:00 0xc0000d3100
2022-04-14T12:33:31.313-07:00 )
2022-04-14T12:33:31.313-07:00
2022-04-14T12:33:31.313-07:00 /usr/local/go/src/net/http/h2_bundle.go
2022-04-14T12:33:31.313-07:00 :
2022-04-14T12:33:31.313-07:00 7722
2022-04-14T12:33:31.313-07:00 +0x35
2022-04-14T12:33:31.313-07:00 created by net/http.(*http2ClientConn).RoundTrip
2022-04-14T12:33:31.313-07:00 /usr/local/go/src/net/http/h2_bundle.go:7651 +0x29f
2022-04-14T12:33:31.336-07:00 END RequestId: 5fd5d180-b25c-4c58-8be8-2ca3b6787a8a

[gavinyue-vendia]

Hey,

We saw the same issue for the third time recently.

Please take a look. Thanks.

[roryj-vendia]

Apparently this issue has already been root caused in honeycombio/libhoney-go#154 and fixed in honeycombio/libhoney-go#156

It was unclear from the documentation here and in the releases that there was a newer version of the lambda extension available, which there appears to be version 9 now. Can you update the releases and documentation to let us know what is the latest and greatest for this extension?

Thanks!

[JamieDanielson]

Ah! You are absolutely right @roryj-vendia , looks like we totally missed a step in here. The fix was in libhoney-go version 1.15.7, and the latest lambda extension (layer version 9, v10.2.0) is on libhoney-go 1.15.8. I'll look at updating those docs now and will report back.

[JamieDanielson]

@roryj-vendia , @gavinyue-vendia , the docs and README have been updated to reflect the latest lambda layer.

We'll close out this issue with this update but if it turns out it doesn't solve the problem by using the new version, feel free to comment back on here and/or open a new issue if it's a different error.