SSG not storing to s3 for dynamic pages with fallback: true #798

arelaxend · 2020-11-14T20:58:52Z

Hi people,

Describe the bug

Say, we have dynamic pages /test/[id].js with some getStaticPaths(...fallback: true) and getStaticProps(...). When we request a new id, it renders the html correctly after a loading but it does not store anything on s3.

So, SSG is not working here!

We suspect that it does not deal multi-regions correctly, but we actually don't know. Please read what follows.

Actual behavior

Nothing is stored in s3 when fallback:true for pages /test/xxx not statistically generated at build time. But it does work in localhost.

Expected behavior

After a hit, subsequent requests just follow the RFC that is get from s3, hit, render.

Additional information

We tested this with the following in /test/[id].tsx

import { GetStaticPaths, GetStaticProps } from "next";
import Head from "next/head";
import { useRouter } from "next/router";
import { useRef } from "react";

export const getStaticPaths: GetStaticPaths = async () => {
  return {
    fallback: true,
    paths: [{ params: { id: "e2efdd70-177f-4bf8-907d-a953c702a304" } }],
  };
};

export const getStaticProps: GetStaticProps = async ({}: any) => {
  return {
    props: {
      id: "e2efdd70-177f-4bf8-907d-a953c702a304",
      upload: {
        title: "UNICEF Children, food and nutrition.",
      },
    },
  };
};

export default function Publication({ upload }: any) {
  const router = useRouter();

  if (router.isFallback) {
    return (
      <div>
        <h1>Loading&hellip;</h1>
      </div>
    );
  }

  const title = useRef((upload && upload.title) || "docduo");

  return (
    <Head>
      <title>{title.current}</title>
    </Head>
  );
}

The content of /static-pages/test/[id].html is the following:

The content of _next/static/QTXnzCYbWqOui3ql6FK75/_buildManifest.js:

The content of _next/static/QTXnzCYbWqOui3ql6FK75/_ssgManifest.js is:

When I test the results of this with the following url: https://e.docduo.fr/test/59407fcb-a848-4ab5-95f0-019f613ea0c1
I expect to have something store in s3 after the first hit.. but in the s3 folder there is just:

The lambda function (we only see one lambda, Default @ Lambda/Edge, no API) has the following permissions (we also test with administrator permissions, does not work).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Resource": "*",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ]
        },
        {
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::1k4854-p0yvzbh/*",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ]
        }
    ]
}

Strangely, there is no lambda invocation (on us-east-1, where the plugin creates the lambda)

The serverless includes a domain where we issue a certificate in us-east-1

# serverless.yml
nextamplified:
  component: "@sls-next/[email protected]"
  inputs: 
    domain: ["e", "docduo.fr"]

Also maybe this is relevant, the cloudwatch logs are in ue-west-1:

But the plugin setups one lambda in us-east-1.

In cloudfront, under _next/data* we have this setup (automatically by the plugin, it looks nice)

Under s3, there is the following perms:

{
    "Version": "2012-10-17",
    "Id": "PolicyForCloudFrontPrivateContent",
    "Statement": [
        {
            "Sid": " Grant a CloudFront Origin Identity access to support private content",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity E376221HJ5Y29H"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::1k4854-p0yvzbh/*"
        }
    ]
}

Versions

OS/Environment: mac os, chrome
@sls-next/serverless-component version: 1.19.0-alpha.7
Next.js version: 10

Checklist

A.

The text was updated successfully, but these errors were encountered:

dphang · 2020-11-15T06:03:42Z

Thanks, looks like the problem is in here:

serverless-next.js/packages/libs/lambda-at-edge/src/default-handler.ts

Lines 361 to 369 in 84a3f40

    
           } else if ( 
        
             pagePath === "pages/_error.js" || 
        
             !prerenderManifest.routes[normalisedDataRequestUri] 
        
           ) { 
        
             // Break to continue to SSR render in two cases: 
        
             // 1. URI routes to _error.js 
        
             // 2. URI is not unmatched, but it's not in prerendered routes, i.e this is an SSR data request, we need to SSR render the JSON 
        
             break S3Check; 
        
           }

. Instead of breaking to do SSR render of the data request in origin request handler for fallback data requests, we should try to return an s3 request to tell CloudFront fetch it from S3 (which will fail the first time, but the origin response handler will then fetch and store it in s3).

Not sure why it wasn't caught in unit tests, I'll fix it and update both the unit/e2e tests to catch this.

arelaxend · 2020-11-15T15:24:03Z

@dphang Got it! Well now it stores on s3🥇 . You are doing very great 👍

Just a comment, It may occur that if you just return (<Head></Head>) or null for example, the fallback loading appears at each re-render. Also it downloads the json file every times.

I may open a thread for this, but who cares.

dphang · 2020-11-15T17:11:18Z

Thanks for testing. I actually didn't publish the latest yet.

I did find that issue too, it is because fallback page gets cached after 1st hit. But if you bust the cache after hitting a non-prerendered route (add a random query parameter), with the new change it would pick up the page that was just stored in s3, so all props are populated.

To fix it, we need to make the fallback page never cached, so the lambda can return the newly generated page (and cache it) the next time the same route is hit. But we also need to version the pages properly as I realized we are not clearing them properly (we only have one set of pages under static-pages, so subsequent deploys may pick up an old version).

arelaxend · 2020-11-15T17:22:32Z

Thanks for testing. I actually didn't publish the latest yet.

@dphang we tested with the PR.

To fix it, we need to make the fallback page never cached, so the lambda can return the newly generated page (and cache it) the next time the same route is hit. But we also need to version the pages properly as I realized we are not clearing them properly (we only have one set of pages under static-pages, so subsequent deploys may pick up an old version).

Absolutely, it makes sense since in the end what users request is cached, still up-to-date versions at the edges. Keep up 💯

rahul3103 · 2020-11-23T11:27:54Z

@arelaxend Can you tell me how you are uploading your static generated files to s3.

Right now, what I am doing is after build I upload all the static files and folder on S3 and do yarn start after updating the asset prefix. But, I am stuck with this incremental static page generation and also with next-images which generate so many images. But, I am not able to upload these on S3. Can you guide me how you are uploading your build time and incremental static files on S3

arelaxend · 2020-11-23T16:22:04Z

@rahul3103 can you create a request / issue following the instructions and check that your question has no precedence. Thank you

danigrandem · 2021-02-25T13:33:15Z

Im having the same problem.
How can i solve this?

Thanks

dphang mentioned this issue Nov 15, 2020

fix(lambda-at-edge): fix fallback (non-prerendered) SSG requests not … #800

Merged

dphang closed this as completed in #800 Nov 15, 2020

This was referenced Nov 16, 2020

[RFC] SSG with incremental-static-regeneration and revalidate #804

Closed

unstable_revalidate supported ? #559

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SSG not storing to s3 for dynamic pages with fallback: true #798

SSG not storing to s3 for dynamic pages with fallback: true #798

arelaxend commented Nov 14, 2020 •

edited

Loading

dphang commented Nov 15, 2020 •

edited

Loading

arelaxend commented Nov 15, 2020 •

edited

Loading

dphang commented Nov 15, 2020 •

edited

Loading

arelaxend commented Nov 15, 2020 •

edited

Loading

rahul3103 commented Nov 23, 2020

arelaxend commented Nov 23, 2020

danigrandem commented Feb 25, 2021

SSG not storing to s3 for dynamic pages with fallback: true #798

SSG not storing to s3 for dynamic pages with fallback: true #798

Comments

arelaxend commented Nov 14, 2020 • edited Loading

Describe the bug

Actual behavior

Expected behavior

Additional information

Versions

Checklist

dphang commented Nov 15, 2020 • edited Loading

arelaxend commented Nov 15, 2020 • edited Loading

dphang commented Nov 15, 2020 • edited Loading

arelaxend commented Nov 15, 2020 • edited Loading

rahul3103 commented Nov 23, 2020

arelaxend commented Nov 23, 2020

danigrandem commented Feb 25, 2021

arelaxend commented Nov 14, 2020 •

edited

Loading

dphang commented Nov 15, 2020 •

edited

Loading

arelaxend commented Nov 15, 2020 •

edited

Loading

dphang commented Nov 15, 2020 •

edited

Loading

arelaxend commented Nov 15, 2020 •

edited

Loading