Random timeout while invoking long running lambda (cont.)

[michaeltarleton]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
Hi,

We are having this random timeout problem in our amazon Lambda that runs a node script that in turn uses aws-sdk (2.706.0) to invoke another lambda which takes ~6min.

Our lambda 1 will randomly timeout after 15 mins (900s) despite the lambda 2 having a log saying it ran successfully:

So looks like Lambda 2 completes successfully but Lambda 1 never gets the response (only if Lambda 2 takes ~6 mins or longer).

(seems to be related to #2558)

Is the issue in the browser/Node.js?
Node.js

If on Node.js, are you running this on AWS Lambda?

Details of the browser/Node.js version
Node 12_x (using cloudformation through CDK)

SDK version number
Example: v2.706.0

To Reproduce (observed behavior)

1. Create Lambda 1 that calls a Lambda 2
2. Create a 6 minute delay in Lambda 2, then respond w/ success
3. Run Lambda 1 (with timeout set to 15 mins)

Expected behavior

Lambda 1 should get the success response from Lambda 2.

Additional context

We have tried:

• adding keepAlive: true
• increasing timeouts

[vudh1]

Hi @michaeltarleton thanks for reaching out. Are you still experiencing this with the latest version of the SDK?

[michaeltarleton]

I have since changed companies but at the time I left this was still an issue.

[vudh1]

Feel free to reopen this issue if the problem is still persisting with the latest version of SDK.

[bbright024]

turns out it's a problem with node itself, not the aws sdk.
nodejs/node#47137

@michaeltarleton and all others impacted by this bug -

there's a simple hack you can use to avoid this issue: complete a guaranteed fast request with the client, before attempting the >5 minute request.

e.g. if there's a lambda invoke that may take >5 minutes, first use the client to make an invoke request with the DryRun parameter set to true, then attempt the >5 minute invoke.

DryRun – Validate parameter values and verify that the user or role has permission to invoke the function.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html

this works because the keepalive parameters are not applied until after the first request is completed. you can see it in this code -

state.keepAliveTimeoutSet = true;

I highly recommend adding this workaround to your code - bnoordhuis mentions that this is intended behavior in node, so there's no guarantee it will ever be fixed:

The keepAliveMsecs option isn't applied until the socket is added to the connection pool, i.e., after the first request. That's by design, as far as I'm aware.
(2) is a policy change. Best way forward is to open a pull request and see how it's received.
nodejs/node#47137 (comment)