Cloudfront 503 ERROR

[cristiam86]

I'm getting a 503 error from CloudFront from my app deployed with this plugin.
The error seems to occur when a page takes longer than 10 seconds to respond. I have already increased the limit of the lambda and invalidated CloudFront cache, but it still happens.
Since we are catching the response of the API queried to obtain the data needed to show the page, if the user refreshes the page, the page gets showed without an error.

You can try it out at: https://data.enantio.com with any random search. If the result page is showed instantly, then try for a different search because that one would be cached for sure.

Here is my response header

content-type: text/html
date: Wed, 02 Oct 2019 12:40:25 GMT
server: CloudFront
status: 503
via: 1.1 9b51ea0192df87ecb2185cd2247e4a1b.cloudfront.net (CloudFront)
x-amz-cf-id: S4ARsV1e_Oj6NX-eZJAeb8RKadcruVN_Fd8UHHkUgmkidL7SCLFMZg==
x-amz-cf-pop: MAD51-C1
x-cache: LambdaExecutionError from cloudfront

And the message obtained from CloudFront:

503 ERROR
The request could not be satisfied.
The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions.
If you received this error while trying to use an app or access a website, please contact the provider or website owner for assistance.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by following steps in the CloudFront documentation (http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-503-service-unavailable.html).
Generated by cloudfront (CloudFront)
Request ID: 0X8O19E4s2kbWJSgrZ7I0zAI0sXHqhNdJ6e0Njq93JUVreeHSWl8Fg==

[danielcondemarin]

@cristiam86 Could you post any errors in your Lambda CloudWatch logs as well? You can find these in the CloudWatch service page. Also, make sure you look at the right region where your Lambda executed:

[cristiam86]

Sure @danielcondemarin Thanks!

This is an error record

And this is a successful record

Also the lambda timeout is increased to 30 seconds but the error persists.

Regarding the regions configuration, I see that the lambda is deployed on us-east-1 but the logs are getting recorded on London (I've verified this refreshing the page and seeing more logs added to CloudWatch). Is there a way to deploy the Lamda to a specific region?

[danielcondemarin]

Do you know or suspect what might be causing the timeout? i.e. outbound network request issued from the Lambda?

The master lambda is in us-east-1, but is replicated globally to many regions so it can't be deployed to one specific region. This is by design 🙂

[cristiam86]

The website calls a serverless API (https://rctf69ejt3.execute-api.eu-central-1.amazonaws.com/prod/data?flow=imp&product=220410&reporter=251&partner=all) that may take longer than 10 seconds depending on the parameters and catching policies.

About the master lambda region, perfect, I understand.

[cristiam86]

Hello @danielcondemarin if there is something I can help with in order to solve this issue please just let me know.

[danielcondemarin]

@cristiam86 This is happening because the default timeout for the Lambda is 10 seconds. I believe it could be increased to 30 seconds. However, I don't think it is a good idea to increase it because this blocks the page load which is really poor ux.

I would suggest you gracefully handle timeouts with a value lower than 10 seconds. A few examples of how to do that using fetch here. If it times out (maybe after 5 secs) defer the API call to the client.

I don't think there is anything else the component can do here to help. Does that sound fair?

[cristiam86]

hey @danielcondemarin thanks for the response. The problem here is that when the query is not catched it depends on a 3rd party API and we can't improve its performance. Also the SSR is key for SEO purposes so deferring the call to client wouldn't work for us.
Also, I increased the timeout for the lambda to 30 seconds on the AWS console but it seems to don't have any effect. If you can just point out where to change this limit it would be awesome.

[danielcondemarin]

@cristiam86 The timeout can be configured in the Console here:

I generally don't recommend updating your infrastructure via the console as is not repeatable. Also, you might have to publish a new version of the Lambda for this to work but to be honest, I don't know if it will get overridden by the component on subsequent deployments

In my opinion increasing the timeout to 30 seconds won't solve much, I'd be surprised if users stick around that much waiting for the page load. You could add a cache layer in front of the third party service and cache popular requests.

[cristiam86]

@danielcondemarin like I said in my first post, I have already increased the limit to 30 seconds but the error persists. Here is my configuration:

Also I have already implemented a cache layer in front of the 3rd party service to cache popular requests so I don't care if one query takes 30 seconds to one user since it's going to be fast to next users. The 503 error pushes away more users than a 30 seconds waiting time.

[danielcondemarin]

@cristiam86 Did you also publish a new version of the function?

Also the new published version would have to be set in the CloudFront Cache Behaviour (example is set to 23):

[danielcondemarin]

@cristiam86 Manually increasing the timeout like that is probably not going to play nice with the component deployments.
I'm happy to accept a PR to pass a custom timeout for the Lambda functions, which defaults to 10 seconds.

Something like:

myNextApp:
  component: "serverless-next.js"
  inputs:
    timeout: 30

Could you verify first setting the 30seconds manually works?

I've not got a lot of free time atm so if you're happy to, feel free to submit the PR, otherwise might take me sometime to land this :)

[cristiam86]

@danielcondemarin IT WORKS!

As you say, it should be updated on every deploy so adding that configuration to the yml would be perfect.

I'm also overwhelmed with work and I don't have any experience with CloudFormation but I can give it a try.

Thank you very much again.

[Mrt3383x38]

Binance kaldırdım indirdim eror veriyo