Figure out how to serve an AWS Lambda function with a Function URL from a custom subdomain

[simonw]

[This issue thread provides a blow-by-blow account of how I figured out the way to serve an AWS Lambda function from a custom domain, using CloudFront and ACM]

I think Cloudfront is the way to do this:

• https://aws.amazon.com/blogs/networking-and-content-delivery/using-amazon-cloudfront-with-aws-lambda-as-origin-to-accelerate-your-web-applications/
• https://serverlessland.com/patterns/cloudfront-lambda-urls

My previous notes on how I shipped the Lambda Function URL are here: https://til.simonwillison.net/awslambda/asgi-mangum

[simonw]

I'm going to first try setting up a CloudFront distribution. If I can get that running in front of a Lambda URL I'll have solved half the problem - the other half will be figuring out how to point a custom domain at CloudFront.

Some useful docs:

• https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudfront/create-distribution.html
• https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-overview-required-fields.html
• https://github.com/cliqz-oss/browser-f/blob/d55ff3b6d92fcf72e2e2206851f0c09e75d899fe/mozilla-release/python/boto/docs/source/cloudfront_tut.rst#working-with-cloudfront-distributions
• https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudfront.html
• https://aws.amazon.com/cloudfront/pricing/

[simonw]

I'm going to skim through and make notes on the whole Amazon CloudFront developer guide: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html

[simonw]

It's basically a caching CDN like Fastly. You don't pay for transfer between your Amazon-hosted application servers and CloudFront, but you do pay for bandwidth served between CloudFront and end users.

[simonw]

• Origin servers: where the data is hosted. I guess in my case that's going to be Lambda functions.
• Distribution: configuration that "tells CloudFront which origin servers to get your files from when users request the files through your web site or application".

Optionally, you can configure your origin server to add headers to the files, to indicate how long you want the files to stay in the cache in CloudFront edge locations. By default, each file stays in an edge location for 24 hours before it expires.

Sounds like max-age or s-maxage is pretty important then!

[simonw]

CloudFront can also run "Lambda@Edge" functions, but these can currently only be written in JavaScript.

[simonw]

You can chose a cheaper plan that doesn't have POP locations in some countries: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PriceClass.html

[simonw]

https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/ is the function URL I want to use.

[simonw]

Trying this on https://us-east-1.console.aws.amazon.com/cloudfront/v3/home?region=us-east-1&skipRegion=true#/distributions/create

[simonw]

https://us-east-1.console.aws.amazon.com/cloudfront/v3/home?region=us-east-1&skipRegion=true#/distributions/E14OFPHUGRRDXW

[simonw]

https://d81numqdze872.cloudfront.net/ is not working yet.

[simonw]

OK it works now.

[simonw]

Hitting the same page twice with curl:

~ % time curl 'https://d81numqdze872.cloudfront.net/fixtures/compound_three_primary_keys'
...
</html>curl   0.02s user 0.02s system 4% cpu 0.764 total
~ % time curl 'https://d81numqdze872.cloudfront.net/fixtures/compound_three_primary_keys'
...
</html>curl   0.02s user 0.01s system 26% cpu 0.117 total

[simonw]

I don't think querystring parameters are being passed through correctly.

https://d81numqdze872.cloudfront.net/fixtures/no_primary_key?_facet=a ignores the facet request.

But https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/fixtures/no_primary_key?_facet=a works correctly.

[simonw]

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesQueryString

I think I want "Forward all, cache based on all" for query strings, somewhere under "Cache behavior settings".

[simonw]

Looks like that's hidden away under "legacy cache settings":

[simonw]

That seemed to take effect pretty fast, https://d81numqdze872.cloudfront.net/fixtures/sortable?_facet=pk2 works now.

[simonw]

Got to that edit interface by clicking:

[simonw]

I can't figure out how to enable the x-cache header for Cloudfront.

[simonw]

Weird. I'm getting those headers now!

I tried adding and then removing a Response Headers Policy to the distribution's behaviour, but I don't know if that made a difference here or not. Maybe it was working anyway?

Janky video trying to show where I found that option:

[simonw]

OK, I have that header now and it seems to confirm my 5s TTL is working as it should

Next job: point a custom domain at https://d81numqdze872.cloudfront.net/ !

[simonw]

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CNAMEs.html shows how to do that for HTTP (I'll add HTTPS later, following https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/using-https-alternate-domain-names.html ):

1. Sign in to the AWS Management Console and open the CloudFront console at https://console.aws.amazon.com/cloudfront/v3/home
2. Choose the ID for the distribution that you want to update.
3. On the General tab, choose Edit.
4. Update the following values:
   Alternate Domain Names (CNAMEs)
   Add your alternate domain names. Separate domain names with commas, or type each domain name on a new line.

[simonw]

Now in my DNS provider:

Use the method provided by your DNS service provider to add a CNAME record for your domain. This new CNAME record will redirect DNS queries from your alternate domain name (for example, www.example.com) to the CloudFront domain name for your distribution (for example, d111111abcdef8.cloudfront.net). For more information, see the documentation provided by your DNS service provider.

[simonw]

Did this in Vercel:

Just noticed I got this error in AWS when I tried to save that though:

[simonw]

Decided to click "Request certificate" and see what happens:

Sent me to https://us-east-1.console.aws.amazon.com/acm/home?region=us-east-1#/certificates/request

Looks like this has the validation bit: https://us-east-1.console.aws.amazon.com/acm/home?region=us-east-1#/certificates/98a4b28c-d7d1-4ad6-ae28-4db269e03ccc

[simonw]

In Vercel:

[simonw]

A few seconds later:

Annoyingly that certificate is not shown in this drop-down menu:

Maybe because this now says "Failed": https://us-east-1.console.aws.amazon.com/acm/home?region=us-east-1#/certificates/98a4b28c-d7d1-4ad6-ae28-4db269e03ccc

[simonw]

I'm deleting the certificate and trying again.

[simonw]

I think I didn't put the right values into Vercel.

Trying with the full copy of these:

• CNAME name: _9150f9c8c1bc531c257114dc32c51bff.lambda-demo.datasette.io.
• CNAME value: _fd09657d57266a937c50b8b626fe9a4d.gbycpywhzv.acm-validations.aws.

[simonw]

That came back "Failed" too.

[simonw]

I'm going to try email validation.

[simonw]

That deploy worked but it did not fix the bug - even though https://lambda-demo.datasette.io/-/plugins shows the new plugin.

[simonw]

https://lambda-demo.datasette.io/-/asgi-scope seems to indicate that x-forwarded-host is not provided by Cloudfront:

 'headers': [[b'sec-fetch-mode', b'navigate'],
             [b'x-amzn-tls-version', b'TLSv1.2'],
             [b'sec-fetch-site', b'none'],
             [b'x-forwarded-proto', b'https'],
             [b'x-forwarded-port', b'443'],
             [b'dnt', b'1'],
             [b'x-forwarded-for', b'24.5.172.176'],
             [b'sec-fetch-user', b'?1'],
             [b'via',
              b'2.0 549a5eaa264d3b997d6acfdba72f56d0.cloudfront.net (CloudFr'
              b'ont)'],
             [b'x-amzn-tls-cipher-suite', b'ECDHE-RSA-AES128-GCM-SHA256'],
             [b'x-amzn-trace-id', b'Root=1-633a1410-62bde31d28b9714b06d45b27'],
             [b'host',
              b'fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws'],
             [b'upgrade-insecure-requests', b'1'],
             [b'accept-encoding', b'gzip'],
             [b'x-amz-cf-id',
              b'kfdCiG-Q0XGPjb-Efmws8N2BbCJF-slwm8f_ZbSvQVGWDMP2bj02Iw=='],
             [b'sec-fetch-dest', b'document'],
             [b'user-agent', b'Amazon CloudFront']],

[simonw]

https://aws.amazon.com/premiumsupport/knowledge-center/configure-cloudfront-to-forward-headers/ says:

1. Follow the steps to create a cache policy using the CloudFront console.
2. Under Cache key settings, for Headers, select Include the following headers. From the Add header dropdown list, select Host.

[simonw]

Tried editing behavior and adding this:

[simonw]

That seemed to break everything: https://lambda-demo.datasette.io/ now returns:

{"Message":null}

[simonw]

AccessDeniedException looks relevant.

[simonw]

Removing Host fixed it. Adding Host broke it again.

[simonw]

Definitely proven to myself that having Host as a header that gets sent through breaks the application and causes every page to return a AccessDeniedException error.

[simonw]

I turned on a whole bunch of those extra headers:

I tried to do even more but it gave me an error (tucked away at the bottom of the page where I initially missed it) complaining I had selected too many.

Now https://lambda-demo.datasette.io/-/asgi-scope has this:

 'headers': [[b'cloudfront-viewer-country', b'US'],
             [b'x-amzn-tls-version', b'TLSv1.2'],
             [b'sec-fetch-site', b'none'],
             [b'cloudfront-viewer-postal-code', b'94018'],
             [b'x-forwarded-port', b'443'],
             [b'sec-fetch-user', b'?1'],
             [b'via',
              b'2.0 e2d7efb4a6fe4a49c212c47079f43f9c.cloudfront.net (CloudFr'
              b'ont)'],
             [b'x-amzn-tls-cipher-suite', b'ECDHE-RSA-AES128-GCM-SHA256'],
             [b'cloudfront-viewer-country-name', b'United States'],
             [b'host',
              b'fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws'],
             [b'upgrade-insecure-requests', b'1'],
             [b'cloudfront-viewer-city', b'El Granada'],
             [b'sec-fetch-mode', b'navigate'],
             [b'x-forwarded-proto', b'https'],
             [b'dnt', b'1'],
             [b'x-forwarded-for', b'24.5.172.176'],
             [b'cloudfront-viewer-country-region', b'CA'],
             [b'cloudfront-viewer-time-zone', b'America/Los_Angeles'],
             [b'x-amzn-trace-id', b'Root=1-633a1788-3196e3a46e240cf84721ba58'],
             [b'cloudfront-viewer-longitude', b'-122.47390'],
             [b'cloudfront-viewer-latitude', b'37.51410'],
             [b'cloudfront-viewer-country-region-name', b'California'],
             [b'accept-encoding', b'gzip'],
             [b'x-amz-cf-id',
              b'g9dQ7qE_E8mL3_-7A118ZbnSlEfkNb0jRHzDM59JY3ehcnir4GD4qA=='],
             [b'sec-fetch-dest', b'document'],
             [b'user-agent', b'Amazon CloudFront']],

That latitude/longitude is on the edge of the small town where I live, so pretty accurate!

[simonw]

These look relevant to my Host problem:

• https://serverfault.com/questions/888714/send-custom-host-header-with-cloudfront
• https://brianschiller.com/blog/2018/10/24/cloudfront-host-header

[simonw]

Aha! Here's a Twitter thread where someone notes that Host breaks things, but X-Forwarded-Host is not supported (when using Cloudfront and API Gateway): https://twitter.com/matthieunapoli/status/1288444807011065856

[simonw]

From this PR it looks like the answer may be to set x-forwarded-host using a CloudFront function.

• Server-side Website construct getlift/lift#44

[simonw]

Apparenty CloudFront Functions are NOT the same thing as Lambda@Edge: https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/

[simonw]

OK I'm going to make a function.

function handler(event) {
    event.request.headers["x-forwarded-host"] = event.request.headers["host"];
    return request;
}

[simonw]

ARN is:

arn:aws:cloudfront::462092780466:function/Set-X-Forwarded-Host-Header

[simonw]

Then you have to "publish" it (I forgot to do this):

Now you can associate it from the functions page:

I hope "Viewer Request" (as opposed to "Viewer Response") was the right guess!

[simonw]

https://lambda-demo.datasette.io/ now says:

% curl -i 'https://lambda-demo.datasette.io/'
HTTP/2 503 
server: CloudFront
date: Sun, 02 Oct 2022 23:21:57 GMT
content-type: text/html
content-length: 999
x-cache: FunctionExecutionError from cloudfront
via: 1.1 f85d379725bf31eb2428acfa2b9da6e6.cloudfront.net (CloudFront)
x-amz-cf-pop: SFO5-P1
x-amz-cf-id: rYuOIEz9U6vm04p5xFfqri6xSJ3FihQKrupKTIP-CZ_Nh1aKJ3TtSw==

So it didn't like my new function.

[simonw]

I tried the "Test" feature and it showed a useful error:

Fixed the implementation to:

function handler(event) {
    event.request.headers["x-forwarded-host"] = event.request.headers["host"];
    return event.request;
}

[simonw]

That fixed the error but https://lambda-demo.datasette.io/-/asgi-scope still shows no x-forwarded-host header.

[simonw]

Trying this instead:

That caused a 503 error again.

[simonw]

I switched it back to Viewer Request.

[simonw]

Saw this in the "Test" view:

The CloudFront function returned an invalid value: request.headers is malformed. invalid request header, key 'x-forwarded-host', keyMultiValue is not an object

[simonw]

I think this might be it:

function handler(event) {
    event.request.headers["x-forwarded-host"] = {
        value: event.request.headers["host"].value
    };
    return event.request;
}

Clue was here: https://www.honeybadger.io/blog/aws-cloudfront-functions/

That seems to work in the test mode:

That worked!!

 'headers': [(b'cloudfront-viewer-country', b'US'),
             (b'x-amzn-tls-version', b'TLSv1.2'),
             (b'sec-fetch-site', b'none'),
             (b'cloudfront-viewer-postal-code', b'94018'),
             (b'x-forwarded-port', b'443'),
             (b'sec-fetch-user', b'?1'),
             (b'via',
              b'2.0 f7aef728fd226cb808d34cb93114336c.cloudfront.net (CloudFr'
              b'ont)'),
             (b'x-amzn-tls-cipher-suite', b'ECDHE-RSA-AES128-GCM-SHA256'),
             (b'x-forwarded-host', b'lambda-demo.datasette.io'),
             (b'cloudfront-viewer-country-name', b'United States'),
             (b'host', b'lambda-demo.datasette.io'),
             (b'upgrade-insecure-requests', b'1'),
             (b'cloudfront-viewer-city', b'El Granada'),
             (b'sec-fetch-mode', b'navigate'),
             (b'x-forwarded-proto', b'https'),
             (b'dnt', b'1'),
             (b'x-forwarded-for', b'24.5.172.176'),
             (b'cloudfront-viewer-country-region', b'CA'),
             (b'cloudfront-viewer-time-zone', b'America/Los_Angeles'),
             (b'x-amzn-trace-id', b'Root=1-633a1fcd-3d0392966fa468423e62cd20'),
             (b'cloudfront-viewer-longitude', b'-122.47390'),
             (b'cloudfront-viewer-latitude', b'37.51410'),
             (b'cloudfront-viewer-country-region-name', b'California'),
             (b'accept-encoding', b'gzip'),
             (b'x-amz-cf-id',
              b'4n8H9CDFEVLWnnCAV41QZaTAGzT4T52bqF3Dy4GZHDUBdxu8rScnBA=='),
             (b'sec-fetch-dest', b'document'),
             (b'user-agent', b'Amazon CloudFront')],

[simonw]

And now the "suggested facets" links on https://lambda-demo.datasette.io/fixtures/compound_three_primary_keys go to the right place.

[simonw]

It looks like the homepage / doesn't set a cache-control: max-age= header at all, and as a result Cloudfront defaults to caching it for 24 hours.

% curl -s -i https://lambda-demo.datasette.io/ | head -n 15
HTTP/2 200 
content-type: text/html; charset=utf-8
content-length: 2675
date: Sun, 02 Oct 2022 23:35:07 GMT
x-amzn-requestid: de5693c7-47f4-403b-8606-83fab9827158
link: https://lambda-demo.datasette.io/.json; rel="alternate"; type="application/json+datasette"
x-amzn-trace-id: root=1-633a202b-251a20ca5abdeb9f59160250;sampled=0
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 f85d379725bf31eb2428acfa2b9da6e6.cloudfront.net (CloudFront)
x-amz-cf-pop: SFO5-P1
x-amz-cf-id: aaAXEqtcMhgfOnbQgD8ur8HmR1jxZNfYW2_r1OTnZTgK6RlV3FNdpQ==
age: 131

<!DOCTYPE html>

Note the age: 131 header there, showing it was cached 131 seconds ago.

Other pages on the site with cache-control headers max out at 5 seconds as they should:

% curl -s -i 'https://lambda-demo.datasette.io/fixtures/facetable' | head -n 15
HTTP/2 200 
content-type: text/html; charset=utf-8
content-length: 34484
date: Sun, 02 Oct 2022 23:37:46 GMT
x-amzn-requestid: b1ce10b8-4275-4df6-9680-b9dcab21ef3f
referrer-policy: no-referrer
cache-control: max-age=5
link: https://lambda-demo.datasette.io/fixtures/facetable.json; rel="alternate"; type="application/json+datasette"
x-amzn-trace-id: root=1-633a20ca-0539b22874f7aa8443da0663;sampled=0
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 52a50599e55838e3cced4f5e481dca9e.cloudfront.net (CloudFront)
x-amz-cf-pop: SFO5-P1
x-amz-cf-id: hQGjCckm7Je4yN6oktuf57vgTGD87mgfii4PKjKaq1UOucRcXwntaQ==
age: 3