Setting up Lambda function not working

[SumOverHistories]

Hey,

I'm trying to set up the AWS Lambda function, but the README you provided doesn't offer much information on how to do it.

Do I need an API Gateway? If so, how should that be configured?

Currently, when I call an image, it seems to find the file, but it's only downloading an empty file instead of displaying the image in the browser.

I'm not sure what I'm doing wrong, and I don't have a way to debug it.

It would be really helpful if the instructions were clearer. For example, I guess that the URL function has to be set to return a stream. Otherwise, it returns an internal server error. These things could be better documented.

For now, I can't proceed with the installation. Some help would be greatly appreciated.

Thank you.

[rmccue]

Apologies for the missing docs here!

Do I need an API Gateway? If so, how should that be configured?

We previously used API Gateway prior to Tachyon v3, but have now converted to use Lambda Function URLs to add streaming response support; see #162 for some of the history behind that.

The relevant Terraform code we use to provision the Lambda function and function URL is:

resource "aws_lambda_function" "default" {
  function_name = local.function_name
  runtime       = "nodejs18.x"
  handler       = "dist/lambda-handler.handler"
  # ...

  environment {
    variables = {
      S3_BUCKET = var.s3_bucket
      S3_REGION = var.s3_bucket_region
      # Force the S3 endpoint to use the global s3 hostnames. This is beacuse the origin in CloudFront
      # also uses the global hostname, and we need presigned requests to S3 to use the same hostname
      # whether the file is being references via Tachyon or via the S3 cloudfront origin.
      S3_ENDPOINT = "https://s3.amazonaws.com"
    }
  }
}

resource "aws_lambda_function_url" "default" {
  count         = var.enabled
  function_name = aws_lambda_function.default[0].function_name
  authorization_type = "NONE"
  invoke_mode = "RESPONSE_STREAM"
}

[SumOverHistories]

Thank you for the quick response.

This is indeed what I've configured so far. I've also added the S3 Permissions to the AWSLambdaBasicExecutionRole.

But still, whenever I call the URL to an image its only downloading a 0 bytes file instead of showing the file.
Calling a file that doesn't exist returns "File not found". So something is indeed working. Only when the file should be delivered it's not.

Any ideas on that? Cloudwatch isn't showing any errors.

Also it took me a while to figure out that the handler is: dist/lambda-handler.js
Could be added to the docs as well. :)

[rmccue]

I've added some additional docs in #207 which has now been merged.

Not sure exactly what could be happening with a zero-byte response, apart from potentially missing some configuration elsewhere; the above configuration is verbatim directly from our production configuration, with only irrelevant details removed.

Are the response headers set correctly for what you expect? And what is the format of your URL?

[SumOverHistories]

As far as I know I've configured everything like it should be. I've done it several times with a new function, just to be sure.

https://xxxxxxxxxxxxxx.lambda-url.eu-central-1.on.aws/uploads/1/2024/04/5.png
Is the URL. Trying it without query params for now.

These are the response headers:

connection: keep-alive
content-type: application/octet-stream
date: Wed, 31 Jul 2024 16:58:10 GMT
transfer-encoding: chunked
x-amzn-requestid: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x-amzn-trace-id: root=1-xxxxxxxxx-xxxxxxxxxxxxxxxx;parent=xxxxxxxxxxxxxxxxx;sampled=0;lineage=xxxxxxxxx:0

[rmccue]

Successful responses should have Cache-Control, Last-Modified, and Content-Type (with image/...) set, so it looks like it's not getting to this part of the code:

tachyon/src/lambda-handler.ts

Lines 93 to 95 in 662e4ee

	'Cache-Control': `max-age=${ maxAge }`,
	'Last-Modified': ( new Date() ).toUTCString(),
	'Content-Type': 'image/' + info.format,

For your URL, I believe it should be /tachyon/1/2024/04/5.png rather than /uploads/...; tachyon-plugin dynamically replaces this and that's typically how it's routed. Without this, I think it will try and pull the wrong object from S3 - that should trigger an error appearing in CloudWatch Logs, but maybe there's a separate issue there?

Does the same thing occur with this changed URL? Also, are there any other entries in CW Logs that indicate it's routing the request at all?

[SumOverHistories]

My URL with /upload is correct this way. I'm not using the tachyon-plugin but a custom solution.
So my files are within /uploads. But for this testing im not using a client plugin just to make sure its not an issue with that.
Calling the mentioned URL does start the download. Changing the URL to a path which doesn't exist leads to the "File not found" error. Therefore the files seem to be found on the S3.

None the less I have just tested it by moving the files to /tachyon instead of /uploads. It's resulting in the same behaviour.

But I now figured out that it's running into a timeout. I've added some logs to it:

let buffer = Buffer.from(await s3_response.Body.transformToByteArray());
console.log('Buffer size:', buffer.length);

let { info, data } = await resizeBuffer(buffer, args);
console.log('Resized image info:', info);

Buffer size is being logged then it times out right before "Resized image info".
I've set the timeout to 5 sec.

[rmccue]

Ah, I stripped the timeout from the config above because I thought it was irrelevant here; we have it set to 60s, although most requests are lower than that.

With the timeout increased, is it now working?

[SumOverHistories]

With the timeout set to 1 minute it's working!

But it's kinda slow. I was using version 2.6.x before, which i didn't setup myself on AWS and remember it being very very fast.

Shouldn't the new version be even faster? Or do i have to respec the lambda function?

Thanks for all the help. :)

[rmccue]

There shouldn't be a substantial difference in speed that I'm aware of, but most of Tachyon's performance comes from heavy caching in most scenarios.

@joehoyle Have you observed any slowdowns with v3?

[joehoyle]

I have not. How much resources are you giving the lambda function @SumOverHistories, and how slow is "slow", could you also attach an example image you're resizing.

[SumOverHistories]

It's getting 128MB RAM and 512MB for the /tmp.

Take this image for example: https://bn2wgdb4t5rkwgzeommlkkwnfm0tdqev.lambda-url.eu-central-1.on.aws/uploads/1/2024/05/DriveWorks-Einsatzmoeglichkeit-Design-Automation.png?resize=100,100

Change the resize values to avoid the cache.
It's fast when it's chached, I agree. But shouldn't it be faster if it's not cached than what it is right now?

The original image size is 238KB.

[rmccue]

That's definitely slower than what I'm observing with our setup; using the same image on our cloud, an uncached load takes 300ms (0.326s), cached is 100ms so ~200ms generation time. Using sharp directly locally, I get 70-90ms.

Note that we have memory set much higher at 2048MB, because CPU allocation scales with memory - 2GB is approx 1vCPU, whereas 128MB is only getting you 5% of a vCPU. I suspect that speed is due to Lambda's provisioning limits.

(I also tested locally using node --max-old-space-size=128 to limit Node's memory and this didn't have much impact, so I suspect it's the CPU limit rather than memory directly.)

[SumOverHistories]

That must be it. Im going to try that out.
Does it matter if it runs under x86 or arm?

This issue can be closed then. It would be nice if all these infos find their way into the readme. :)

Thank you very much for the great work. Tachyon makes life so much better.

One more thing: Support for focal points would be nice too. Currently I'm calculating dimensions in a custom plugin in order to use focal points., which works so far.
If tachyon itself would accept args like "fp=50,50" (percentages or px) it would be great. :)

Best Regards

[rmccue]

Does it matter if it runs under x86 or arm?

I'm not certain about it, but generally sharp has better performance on x86 (likely due to AVX-512/etc instructions); we use x86_64 in production.

If you want to spin up a pull request to add any info, please do, otherwise I'll get to it in time :)

Support for focal points would be nice too

Cropping uses .extract() which doesn't support this alas.

Currently I'm calculating dimensions in a custom plugin in order to use focal points., which works so far.

You might want to check out https://github.com/humanmade/smart-media/ too, either to use it or to see how we're doing the same thing with Tachyon there.