diff --git a/README.md b/README.md index f7dd64a0..61a04f20 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,11 @@ A Gatsby plugin to source objects and images from AWS S3. -## Install +## Getting started + +### Gatsby setup + +Install the plugin: ```bash # with npm @@ -11,18 +15,13 @@ npm install @robinmetral/gatsby-source-s3 yarn add @robinmetral/gatsby-source-s3 ``` -## Configure - -Declare the plugin in your `gatsby-config.js`, taking care to pass your AWS -credentials as +Declare it in your `gatsby-config.js`, making sure to pass your AWS credentials +as [environment variables](https://www.gatsbyjs.org/docs/environment-variables/): ```javascript -// configure dotenv -// see https://www.gatsbyjs.org/docs/environment-variables -require("dotenv").config({ - path: `.env.${process.env.NODE_ENV}` -}); +// gatsby-config.js +require("dotenv").config(); module.exports = { plugins: [ @@ -32,56 +31,50 @@ module.exports = { aws: { accessKeyId: process.env.AWS_ACCESS_KEY_ID, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, - region: process.env.AWS_REGION + region: process.env.AWS_REGION, }, - buckets: ["my-bucket", "my-second-bucket"] - } - } - ] + buckets: ["my-bucket", "my-second-bucket"], + }, + }, + ], }; ``` -Currently, your buckets will need to be configured for public access with this -access policy: (add your bucket name under `Statement.Resource`) +### AWS setup + +You can use the plugin both with private and public buckets. ```json -{ - "Version": "2008-10-17", - "Statement": [ - { - "Sid": "AllowPublicRead", - "Effect": "Allow", - "Principal": { - "AWS": "*" - }, - "Action": "s3:GetObject", - "Resource": "arn:aws:s3:::my-bucket/*" - } - ] -} + ``` -## Query +## Querying S3 objects -S3 objects can be queried in GraphQL as "s3Object" of "allS3Object": +S3 objects can be queried in GraphQL as "s3Object" or "allS3Object": ```graphql query AllObjectsQuery { allS3Object { nodes { - Key - Url + Key # the object's key, i.e. file name + Bucket # the object's bucket name on S3 + LastModified # the date the object was last modified + Size # the object's size in bytes + localFile # the local file node for image objects processed with sharp (see below) } } } ``` -## Image processing +### Processing images with sharp Any images in your bucket(s) will be downloaded by the plugin and stored as -local files, to be processed with `gatsby-plugin-sharp` and +local file nodes, to be processed with `gatsby-plugin-sharp` and `gatsby-transformer-sharp`. +If you don't have them yet, you will need to add the sharp plugin and +transformer to your Gatsby site: + ```bash # with npm npm install gatsby-plugin-sharp gatsby-transformer-sharp @@ -89,6 +82,19 @@ npm install gatsby-plugin-sharp gatsby-transformer-sharp yarn add gatsby-plugin-sharp gatsby-transformer-sharp ``` +```javascript +// gatsby-config.js +module.exports = { + plugins: [ + // ... + `gatsby-plugin-sharp`, + `gatsby-transformer-sharp`, + ], +}; +``` + +You can then query the processed images with GraphQL: + ```graphql query AllImagesQuery { images: allS3Object { @@ -106,9 +112,19 @@ query AllImagesQuery { } ``` +And use them with `gatsby-image`: + +```javascript +import Img from "gatsby-image"; + +const Image = ({ s3Object }) => ( + +); +``` + ## Thanks -This plugin was based on Dustin Schau's +This plugin was initially based on Dustin Schau's [`gatsby-source-s3`](https://github.com/DSchau/gatsby-source-s3/) and influenced -by Jesse Stuart's Typescript +by Jesse Stuart's TypeScript [`gatsby-source-s3-image`](https://github.com/jessestuart/gatsby-source-s3-image). diff --git a/cypress.json b/cypress.json index 710a205e..3216a812 100644 --- a/cypress.json +++ b/cypress.json @@ -1,5 +1,5 @@ { - "integrationFolder": "tests", + "integrationFolder": "tests/e2e", "screenshotsFolder": "tests/screenshots", "fixturesFolder": false, "supportFile": false, diff --git a/examples/gatsby-starter-source-s3/README.md b/examples/gatsby-starter-source-s3/README.md new file mode 100644 index 00000000..5e3aede7 --- /dev/null +++ b/examples/gatsby-starter-source-s3/README.md @@ -0,0 +1,79 @@ +# gatsby-starter-source-s3 + +This starter is an example of how to source objects from AWS S3 in a Gatsby site +at build time, using `@robinmetral/gatsby-source-s3`. + +It uses a local version of the plugin located in `/src`, and it can be used for +local development and testing. + +To run it locally, you'll need to add the following environment variables in a +`.env` file: + +```bash +AWS_ACCESS_KEY_ID="" +AWS_SECRET_ACCESS_KEY="" +AWS_REGION="" +``` + +## AWS S3 setup + +This site sources images from three separate buckets: + +1. gatsby-source-s3-example (public) +2. gatsby-source-s3-example-2 (public) +3. gatsby-source-s3-continuation-token (private) + +The first two buckets are set up for public access with the following policy: + +```json +{ + "Version": "2008-10-17", + "Statement": [ + { + "Sid": "AllowPublicRead", + "Effect": "Allow", + "Principal": { + "AWS": "*" + }, + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::gatsby-source-s3-example/*" + } + ] +} +``` + +_Note: the resource is the bucket's arn with the `/*` scope._ + +The third bucket is private, its policy is the default for S3 (i.e. nothing was +changed when creating the bucket). + +## AWS IAM setup + +The AWS access keys used by this example are for a `gatsby-source-s3` user to +which I attached the following access policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": ["s3:ListBucket"], + "Resource": [ + "arn:aws:s3:::gatsby-source-s3-example", + "arn:aws:s3:::gatsby-source-s3-example-2", + "arn:aws:s3:::gatsby-source-s3-continuation-token" + ] + }, + { + "Effect": "Allow", + "Action": ["s3:GetObject"], + "Resource": [ + "arn:aws:s3:::gatsby-source-s3-example/*", + "arn:aws:s3:::gatsby-source-s3-example-2/*", + "arn:aws:s3:::gatsby-source-s3-continuation-token/*" + ] + } + ] +} +``` diff --git a/examples/gatsby-starter-source-s3/gatsby-config.js b/examples/gatsby-starter-source-s3/gatsby-config.js index bc1dbbda..3c01ebfd 100644 --- a/examples/gatsby-starter-source-s3/gatsby-config.js +++ b/examples/gatsby-starter-source-s3/gatsby-config.js @@ -2,7 +2,7 @@ require("dotenv").config(); module.exports = { siteMetadata: { - title: `gatsby-starter-source-s3` + title: `gatsby-starter-source-s3`, }, plugins: [ { @@ -11,13 +11,17 @@ module.exports = { aws: { accessKeyId: process.env.AWS_ACCESS_KEY_ID, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, - region: process.env.AWS_REGION + region: process.env.AWS_REGION, }, - buckets: ["gatsby-source-s3-example", "gatsby-source-s3-example-2"] - } + buckets: [ + "gatsby-source-s3-example", + "gatsby-source-s3-example-2", + "gatsby-source-s3-continuation-token", + ], + }, }, // the sharp transformer and plugin are required to process images `gatsby-transformer-sharp`, - `gatsby-plugin-sharp` - ] + `gatsby-plugin-sharp`, + ], }; diff --git a/examples/gatsby-starter-source-s3/src/pages/index.js b/examples/gatsby-starter-source-s3/src/pages/index.js index 8944bcbc..9d010cd1 100644 --- a/examples/gatsby-starter-source-s3/src/pages/index.js +++ b/examples/gatsby-starter-source-s3/src/pages/index.js @@ -3,12 +3,26 @@ import { graphql } from "gatsby"; import Img from "gatsby-image"; const IndexPage = ({ data }) => ( - <> +

{data.site.siteMetadata.title}

- {data.allS3Object.nodes.map(image => ( - {image.Key} - ))} - +
+ {data.allS3Object.nodes.map((image) => ( +
+ {image.Key} +
+ Key: {image.Key} +
+ Bucket: {image.Bucket} +
+ ))} +
+
); export const IMAGES_QUERY = graphql` @@ -21,6 +35,7 @@ export const IMAGES_QUERY = graphql` allS3Object { nodes { Key + Bucket localFile { childImageSharp { fixed(width: 256) { diff --git a/package.json b/package.json index 14d53fe7..9a4183f8 100644 --- a/package.json +++ b/package.json @@ -34,8 +34,9 @@ "scripts": { "build": "tsc", "lint": "eslint '*/**/*.{ts,tsx}'", - "prestart": "yarn build && npm pack && (cd examples/gatsby-starter-source-s3 && yarn install && yarn add file:../../robinmetral-gatsby-source-s3-0.0.0-semantically-released.tgz)", + "prestart": "yarn build && npm pack && (cd examples/gatsby-starter-source-s3 && yarn install)", "start": "(cd examples/gatsby-starter-source-s3 && gatsby build && gatsby serve)", + "start:local": "yarn cache clean && (cd examples/gatsby-starter-source-s3 && rm -rf node_modules .cache public yarn.lock) && yarn start", "test": "cypress run", "e2e": "start-server-and-test http://localhost:9000" }, diff --git a/src/gatsby-node.ts b/src/gatsby-node.ts index bd410fd2..d5111207 100644 --- a/src/gatsby-node.ts +++ b/src/gatsby-node.ts @@ -1,6 +1,8 @@ import { createRemoteFileNode } from "gatsby-source-filesystem"; import AWS = require("aws-sdk"); +const s3 = new AWS.S3(); + const isImage = (key: string): boolean => /\.(jpe?g|png|webp|tiff?)$/i.test(key); @@ -26,8 +28,6 @@ export async function sourceNodes( AWS.config.update(awsConfig); // get objects - const s3 = new AWS.S3(); - const getS3ListObjects = async (params: { Bucket: string; ContinuationToken?: string; @@ -86,19 +86,11 @@ export async function sourceNodes( const objects = allBucketsObjects.reduce((acc, val) => acc.concat(val), []); // create file nodes - // todo touch nodes if they exist already objects?.forEach(async (object) => { - const { Key, Bucket } = object; - const { region } = awsConfig; - createNode({ ...object, - // construct url - Url: `https://s3.${ - region ? `${region}.` : "" - }amazonaws.com/${Bucket}/${Key}`, // node meta - id: createNodeId(`s3-object-${Key}`), + id: createNodeId(`s3-object-${object.Key}`), parent: null, children: [], internal: { @@ -123,9 +115,16 @@ export async function onCreateNode({ }) { if (node.internal.type === "S3Object" && node.Key && isImage(node.Key)) { try { + // get pre-signed URL + const url = s3.getSignedUrl("getObject", { + Bucket: node.Bucket, + Key: node.Key, + Expires: 60, + }); + // download image file and save as node const imageFile = await createRemoteFileNode({ - url: node.Url, + url, parentNodeId: node.id, store, cache, diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 00000000..6f43f3f7 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,22 @@ +# Testing gatsby-source-s3 + +The plugin is tested end-to-end using [Cypress](https://www.cypress.io/). + +The CI builds and serves the +[example Gatsby site](../examples/gatsby-starter-source-s3) in this repo, and +runs the [Cypress tests](e2e/main.spec.js) against it. + +The example site sources images from real S3 buckets, so the tests cover the +following cases: + +- [x] processing images with `sharp` and rendering with `gatsby-image` +- [x] sourcing from multiple buckets (added in #14) +- [x] sourcing from private buckets (added in #20, using a pre-signed URL) +- [x] sourcing from a bucket with more than 1000 objects (added in #43, using a + continuation token) + +In total, the buckets hold 1502 objects, and the e2e tests assert that all are +correctly sourced and rendered. + +Read more about the example site's AWS setup in the +[example site's README](../examples/gatsby-starter-source-s3). diff --git a/tests/main.spec.js b/tests/e2e/main.spec.js similarity index 55% rename from tests/main.spec.js rename to tests/e2e/main.spec.js index 8f741c5b..3d43d1dd 100644 --- a/tests/main.spec.js +++ b/tests/e2e/main.spec.js @@ -4,4 +4,8 @@ describe("e2e", () => { cy.get("img").should("exist"); }); + + it("should contain all images", () => { + cy.get(".images-grid").find(".s3-image").should("have.length", 1502); + }); });