Botocore endpoint timeout not the same as the lambda timeout

[lucasdf]

Issue:

I have an AWS Lambda function that is expected to run for a few minutes and it has a timeout of 4 minutes. I noticed that when I invoke this function using the CLI the function is always invoked multiple times, basically once every one minute.

By using the "--debug" flag I noticed that botocore is setting the timeout of the endpoint as 60 seconds. So this is the cause of the multiple invocations: since the CLI reaches the timeout after one minute it sends a new request every minute, despite the fact that the lambda function's own timeout is higher than this value and the first invocation is still being processed.

Questions:

1. What is the expected behaviour of botocore/aws-cli: is it to use 60 seconds as timeout no matter what the lambda function timeout is, or is the expected behaviour to get the timeout value from the function definition? It seems to me that using the same timeout as the function should be the expected behaviour. If indeed it is then there seems to be something wrong when setting this value.
2. Is there a way to set the timeout value while invoking aws lambda invoke? aws lambda invoke help does not display any information regarding this.

Commands:
I am invoking the function using the following command:
aws lambda invoke --function-name FUNCTION_NAME --payload 'JSON_PAYLOAD' output.json
The debug flag shows the following message:
MainThread - botocore.endpoint - DEBUG - Setting lambda timeout as (60, 60)

Versions:
aws-cli/1.11.93 Python/2.7.12 Linux/4.4.0-79-generic botocore/1.5.56
My lambda function is using Python 3.6

Reproducing the issue:
It should be possible to reproduce it by creating a lambda function that just sleeps for more than 60 seconds and invoking it by using the CLI.

tl;dr:
My function is being invoked multiple times because botocore is using 60 seconds as the timeout instead of using the four minutes that is defined in my lambda function. The expected behaviour should be to use the same timeout used by the lambda function. At least there should be a way for me to define the timeout value as a parameter when using aws lambda invoke.

[jamesls]

The expected behavior is to use the default of 60 seconds for all operations. Changing the defaults is tricky, especially for people that may be relying on the existing behavior. To control the timeout per-invocation, you can specify the --cli-read-timeout option to the number of seconds to wait before timing out.

aws lambda invoke --cli-read-timeout 240 --function-name FUNCTION_NAME --payload 'JSON_PAYLOAD' output.json

Let me know if you have any more questions.

[lucasdf]

@jamesls oh I see. It took me a while to understand why the function was being invoked multiple times.

Do you think it would make sense to have an option such as "--use-resource-timeout" that would get the resource configuration, parse the timeout and use it instead of using the default of 60 seconds?

[jamesls]

It's an interesting idea, that would require an extra API call to get that value before invoking the function. You could write an CLI alias (https://github.com/awslabs/awscli-aliases) that did that. It would look something like:

invoke-function = 
  !f() {
   aws lambda invoke ... \
     --cli-read-timeout $(aws lambda get-function-configuration --function-name $1 --query Timeout --output text)
  }; f

[lucasdf]

I wrote an alias to do that, but it seems that bash does not like arguments with quotes or double quotes, so this only works if the payload is loaded from a file and it won't work when the JSON is written inline.

aws lambda-invoke --function-name LAMBDA_NAME --payload file://FILE_NAME output.temp

lambda-invoke =
  !f() {
    set -e
    args="$*"

    for i in "$@"; do
      if [ "$i" = "--function-name" ]; then
        shift
        lambda_name="$1"
        break
      fi
      shift
    done
    timeout=$(aws lambda get-function-configuration --function-name $lambda_name --query Timeout --output text)
    aws lambda invoke --cli-read-timeout $timeout $args
  }; f

Another option, which is the one that I am using, is to use an alias to get the timeout only.

aws lambda invoke --function-name LAMBDA_NAME --payload PAYLOAD --cli-read-timeout $(aws lambda-timeout LAMBDA_NAME) output.temp

lambda-timeout =
  !f() {
    aws lambda get-function-configuration --function-name "$1" --query Timeout --output text
  }; f

[AlJohri]

@jamesls just got burned by this- couldn't tell for the life of me while the lambda kept "restarting" even though it was in RequestResponse mode. I realized that since the request id's weren't the same it wasn't "restarting" but instead multiple invocations exactly 1 minute spaced apart. Like @lucasdf mentioned, --debug helped me figure it out for sure.

I believe this should be mentioned in the aws lambda invoke help page along with the boto3 documentation as suggested here (boto/boto3#883 (comment)).

[AlJohri]

Wow, I just got burned by this again. Tried to invoke a long running lambda from the CLI and it fired twice. Ended up sending emails to all my users twice. This is definitely not obvious from reading the command. Please change the default!

[jmehnle]

This issue was closed by a commit that "quote[s] arguments in aliases" (#2689). This merely facilitates one particular work-around for this issue but in no way actually fixes the issue. At the very least the aws lambda invoke help documentation should be updated to very prominently call out that if your synchronous lambda takes more than 60 seconds to return a response, it will be implicitly reinvoked every 60 seconds.

I inherited a lambda function that rotates the master user password shared by multiple RDS clusters, and this generally takes on the order of 75-120 seconds, causing an implicit reinvocation by the aws CLI. And it just so happened that this reinvocation interfered with the initial run, which was still in the process of updating the first cluster's password, which then makes both invocations fail and retry internally, which makes the aws CLI retry even more times. Granted, this may not be the best designed lambda (again, I inherited this), but it took me hours to figure out that it's the aws CLI that's quietly making multiple retries.

Honestly, though, this issue should be fixed regardless of documentation.

[kdaily]

@jmehnle, thanks for your comments. I'm reopening to review the behavior.

I would agree that documentation improvements would be welcome - I'll open a separate issue for that.

[hack2learn]

it would be great if the cli command could use the lambda's configured timeout.

[irvintim]

Wouldn't the easiest thing to do be to just make the default --cli-read-timeout be 900 (or maybe 910) when calling "lambda invoke"? 900 is the max lambda timeout, so it will end when the Lambda completes, or when it timeouts.

[frenchtoastbeer]

I don't want to increase the timeout, I want to disable the retry.

[jdsampayo]

@frenchtoastbeer in order to disable the retry you can use:

AWS_MAX_ATTEMPTS=1 aws lambda invoke ...