HTTPError: Response code 403 (Forbidden)

[Vacilando]

Above a certain number of images (a few hundred, perhaps) during build (or develop) I get a barrage of errors like this:

error failed to process https://s3.amazonaws.com/MYBUCKET/path/to/image.jpg?AWSAccessKeyId=XXX&Expires=1605742170&Signature=zzz
HTTPError: Response code 403 (Forbidden)

The AWS setup is just like in https://github.com/robinmetral/gatsby-source-s3#aws-setup ... granting s3:ListBucket and s3:GetObject for the bucket.
First just for the accessing user's IAM role access policy, then also for the given S3 bucket policy (not sure what's the difference, perhaps it could be clarified in the setup instructions).
I even experimented with making the bucket list and items accessible to anonymous users... no help.

The S3 settings might be a red herring since things work for small number of images. The largest number I got to work was 175, anything above produces a shower of these errors.
Why would it work for a small but not a larger number of images?

Happens locally as well as on the server (Netlify).

[robinmetral]

Hi @Vacilando! Thanks for reporting this 🙂

Unfortunately I'm not sure how to debug this because I can't reproduce on my end. I have a Gatsby site with 500+ large photos being pulled, and I've never has permission issues - only the occasional ECONNREFUSED coming back from AWS (and in that case, re-building normally fixes it).

Please let us know if any tweaks to your policy fixes this! In the meantime I'll add the help wanted label here, maybe someone can help out.

[Vacilando]

Hi @robinmetral,

S3 setting indeed were a red herring... they were OK and did not need tweaking. ( Incidentally, if anyone is confused by the various overlapping permission levels for S3, read this page attentively, it explains it all: https://aws.amazon.com/blogs/security/iam-policies-and-bucket-policies-and-acls-oh-my-controlling-access-to-s3-resources/ )

The problem was that the S3 files were in my cased processed so long that they exceeded the "Expires" period set in this plugin to be 60 seconds.

Do you think you could make this value configurable in plugin options? I guess you can do it very quickly yourself, it's a small change. Or do you prefer me to create a a PR?

[robinmetral]

Ah, this makes sense! Thanks a lot for investigating this and for the update @Vacilando 🙏

I think we can just extend the expiration period by default, what do you think? But we can also make this a configurable option if it makes sense.

And I'd love a PR! 😊

[Vacilando]

I definitely plead for a configuration option for this because the expiration time may differ greatly from one project to another.

On a large site I had to increase it to 1800 to make sure everything gets generated, on other sites the default value of 60 will be fully satisfactory.

Working on a quick PR.

[robinmetral]

🎉 This issue has been resolved in version 2.1.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀