Skip to content
This repository has been archived by the owner on Feb 1, 2022. It is now read-only.

adds ContinuationToken and extra error handling #20

Merged
merged 12 commits into from
Apr 14, 2020
Merged

adds ContinuationToken and extra error handling #20

merged 12 commits into from
Apr 14, 2020

Conversation

LpmRaven
Copy link
Contributor

@LpmRaven LpmRaven commented Apr 10, 2020

S3 has a limit of 1000 objects that can be retrieved in a single request. AWS docs

  • Handles NextContinuationToken for multiple buckets
  • Extra error handling

@robinmetral I'm not familiar with writing TS (I should have learnt it by now 🤓) so I would appreciate any input.

Copy link
Contributor

@robinmetral robinmetral left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for your PR! I've added a few comments related to TypeScript mostly. Looks great ✨

Regarding the feature, unfortunately I cannot test it on my end because I don't have a large enough bucket available 🙈 I'm planning on building a suite of E2E tests to automate this (see #21), but for now I'm testing manually.

Just confirming: did you manage to use this version to fetch objects from your bucket with over 1000 objects? I can confirm that the plugin is still behaving as expected for my use cases (multiple buckets with about 500-1000 objects)

src/gatsby-node.ts Outdated Show resolved Hide resolved
src/gatsby-node.ts Outdated Show resolved Hide resolved
src/gatsby-node.ts Outdated Show resolved Hide resolved
src/gatsby-node.ts Outdated Show resolved Hide resolved
src/gatsby-node.ts Outdated Show resolved Hide resolved
src/gatsby-node.ts Outdated Show resolved Hide resolved
src/gatsby-node.ts Show resolved Hide resolved
src/gatsby-node.ts Outdated Show resolved Hide resolved
src/gatsby-node.ts Show resolved Hide resolved
src/gatsby-node.ts Outdated Show resolved Hide resolved
@LpmRaven
Copy link
Contributor Author

Thanks for the comments, I've committed those changes. If you find anything else, I'll happily change it.

I've tested the code with 1 bucket containing 1500 items.

Copy link
Contributor

@robinmetral robinmetral left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for addressing the comments @LpmRaven!

Looks great, the TypeScript compiler is just complaining about two small things. After this I'll quickly test again on my end and we'll be good to go!

src/gatsby-node.ts Outdated Show resolved Hide resolved
src/gatsby-node.ts Outdated Show resolved Hide resolved
LpmRaven and others added 2 commits April 14, 2020 12:13
@robinmetral robinmetral merged commit ec9fc17 into gatsby-uc:master Apr 14, 2020
@robinmetral
Copy link
Contributor

🎉 This PR is included in version 1.2.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@robinmetral
Copy link
Contributor

Thank you again for your contribution @LpmRaven! Update to version 1.2.0 to use the feature 🙂

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants