Improve gatsby-develop bootup time (query-on-demand)

[m-allanson]

Summary

When developing a Gatsby site there's no need to run all graphql queries before serving the site. Instead Gatsby could run queries for pages as they're requested. This would avoid having to wait for slower queries (like image processing) if you're editing an unrelated part of a site.

There's an old PR investigating this at #4775

[pieh]

some updates - this is my very WIP and very unclean dev branch - master...pieh:dev-query-on-demand
that make it somewhat work - https://www.youtube.com/watch?v=OrX_btT3Ikk

Few notes:

• static queries are running as they were before, so they will run during bootstrap and all of them will be re-running if they are marked as dirty
• it tracks pages that are active (needed to register path earlier so it triggers potential query run before we try to fetch data)
• right now image processing runs when query is being executed which is fine if we only run queries for active page, but it would be great to make it lazy, so register job for image processing and only run this job if dev server has request for specific img - this way we wouldn't need to run all responsive sizes for gatsby develop but we would need to track this between builds, so builds would know what image processing it needs to do
• it doesn't prerun queries same way as we prefetch resources - mostly because image processing is so resource intensive that it would make it really problematic to use - once point above is solved this would be fine to make it prerun queries for linked pages
• it doesn't invalidate client query results map, so if you visit page A, navigate to page B, make changes to data for page A, navigate to back page A it will show stale results for a moment before new results are emitted

[gatsbot[bot]]

Old issues will be closed after 30 days of inactivity. This issue has been quiet for 20 days and is being marked as stale. Reply here or add the label "not stale" to keep this issue open!

[asilgag]

Which is the current status of this PR? It seems to be abandoned, but it looks very promising as it solves one of the main problems when developing with Gatsby.

[m-allanson]

@asilgag this has been split out into smaller parts. The first part is to just run image processing on demand, as this is often takes a significant amount of time. There's a PR that's nearly ready to land over at #10964

[mrlubos]

Hey! What's the current status of this feature? Unless I am missing something, I believe this is still a pain as the site grows, correct? Having to process hundreds of images takes 2½ minutes even on a really fast hardware.

[gusliedke]

Mine "Generating image thumbnails" takes around 20mins in CircleCI (4454/4454 - 1130.272 s)

Can we opt to skip this process?

Is thumbnails here = all images?

It should regenerate only the modified ones.

[jackblackCH]

Update?

[agrawalaayushi]

Any update?

[Digital-E]

Update?

[muescha]

good that i find this issue, i was planning to creating an new issue "disable image generation and give us "fake thumbnails (just the original image) or generic placeholder thumbnails" because sometimes i don't care about the images while running in gatsby develop.

sad that the option lazyImageGeneration is turned off in #10964 refactor(gatsby-plugin-sharp): split single file into more maintainable chunks

[jlkiri]

I'll add that gatsbyjs.org itself takes ages to build and it would be nice to have some workaround in develop mode. First time I tried to build www it SEGFAULTed on my Mac with 16GB RAM. Trying again.

[coldice]

+1, I was also expecting a flag to at least disable the generation of the various thumbs for dev.

[alexfornuto]

y'all see their experimental incremental build flag? While not a direct solution to the root cause of image generation, it sure does ease the symptoms, especially when paired with a build/CI stack that caches .cache and public.

[jlkiri]

Btw if anyone working with gatsbyjs.org's code (like me) hasn't noticed - there is an environment variable GATSBY_SCREENSHOT_PLACEHOLDER which you can set to true in development and it will skip generating responsive versions of images which makes the build much faster. (It is literally in README)

[muescha]

@jlkiri GATSBY_SCREENSHOT_PLACEHOLDER is only disabling the screenshot function and not image generation in general

[sjelfull]

@alexfornuto One example of that: Every time you change the schema or data in a source used with gatsby-source-graphql, you need to clear the whole cache to get the updates.

It would be great if usual suspects like image generation was flag on the cache, so you could do it in a granular way if you want to.

[muescha]

also if you change some component / plugin code and need to see the changes you need to clear the pages cache to get pages rerendered with the new code.

[muescha]

it is mostly in the development phase where you not care about image quality, but you check complete other functionality

[muescha]

It would be great if usual suspects like image generation was flag on the cache, so you could do it in a granular way if you want to.

a granular cache management would be also ok, like:

• i need to clear page with slug xxxxx/ssss/
• i need to clear pages with slugs yyyy/abc-*

[josephmarkus]

Totally agree with @muescha. I work on a site that has a few thousand pages with lots of images, so when it comes to debugging or making changes to configuration to implement new functionality, the cache gets flushed and it takes a really long time for images to get resized and for pages to be built.

It's frustrating enough that we've gone and implemented a yarn command that uses environment variables in order to not build the biggest part of the site, so that local development cycles could be 70% faster. This sort of hack required the least code changes. Ideally, it would be nice to have page queries on demand when in development. Although, I appreciate that this would introduce quite a level of divergence from production cycles, so some issues wouldn't get caught.

However, I wonder if run page queries on demand could be run on a separate command that could be evolved over time to address any kind of discrepancies between the two.

[alexfornuto]

@josephmarkus I'd be interested in seeing your yarn command workaround hack.

[josephmarkus]

@alexfornuto here it is:

In my gatsby-config.js where I source all Markdown files I have this configuration for gatsby-source-filesystem:

// at the top of the file
const getGatsbySourceFileSystemPathsToExclude = () => {
  // By default we want to make sure that we don't exclude anything
  if (process.env.WITHOUT_HUB !== 'true') {
    return [];
  }

  return [`**/content/hub/**`];
};

// inside your plugin array
{
  resolve: 'gatsby-source-filesystem',
  options: {
    path: `${__dirname}/content`,
    name: 'content',
    ignore: getGatsbySourceFileSystemPathsToExclude(),
  },
},

// because those before mentioned Markdown files are handled
// by a custom Gatsby plugin, there's more config for another plugin
{
  resolve: 'gatsby-plugin-hub',
  options: {
    skip: process.env.WITHOUT_HUB,
  },
},

And then inside my gatsby-plugin-hub in gatsby-node.js file:

exports.createPages = async ({ graphql, actions }, pluginOptions) => {
  // 'skip' is the option that's defined in `gatsby-config.js` alongside the plugin
  if (pluginOptions.skip === 'true') {
    // eslint-disable-next-line no-console
    console.log(
      `This build will not contain Hub pages. Check WITHOUT_HUB flag usage.`,
    );

    return false;
  }

  const { createPage } = actions;

  return buildHubPages(createPage, graphql);
};

Finally, in my package.json I have:

"start:without-hub": "cross-env NODE_ENV=development WITHOUT_HUB=true gatsby develop",

The above yarn start:without-hub command should be the same as your yarn start command, it's just that in addition I pass WITHOUT_HUB=true here as well

In my case, the specified folder contains a few thousand Markdown files that I do not want to be processed when I'm looking to get the rest of the application up and running. With this, my build times in development are down from 10 min to 3 min when without cache.

Ask away if any of this is unclear 😅

[muescha]

instead of the skip you can also add per javascript dynamic your plugins

for example see dynamicPlugins in www/gatsby-config.js

[alexfornuto]

Ask away if any of this is unclear sweat_smile

I think I'm just out of my depth as a Gatsby user who isn't a JS dev. But thanks for sharing anyway! Hopefully it will help / inspire others.

[josephmarkus]

Thanks @muescha, that's an interesting approach!

@alexfornuto what's the setup like in your project? In my case, I can ignore a particular folder where most content lives. Do you have a similar problem?

[alexfornuto]

My project is the Pantheon Docs - the big killer is the image building on uncached builds (see #12857), which I think count as queries?

Assuming they do, then the folder I'd want for on-demand queries would be source/images. But again, I may be conflating two different processes.

[josephmarkus]

@alexfornuto I had a look at your project. You have some options here:

• you use my example to create an equivalent to exclude your documentation pages from development builds. Far from ideal, but if you're looking to play around with your configuration change or you are working on some component that for some reason needs caching to be cleared, it could be a good temporary space/environment
• you could move all of your images to an external service such as imgix or Cloudinary. That way your gatsby build would not need to create a version of each image for mobile, tablet and desktop. That heavy lifting could be done by a third party. Obviously, that doesn't come for free. If currently you have an image that is 3 MB in size, but in your gatsby build this is resized to 80 KB, part of your build process means that a plugin would have to parse 3 MB of data before applying some algorithms to resize and compress each file before it becomes 80 KB. And you do that every time the cache is flushed
• the third option is to wait for gatsby to allow users to build pages on demand in development. I suspect that this would significantly diverge how gatsby works under the surface - one setup for production vs that for development. Pretty tricky
• the fourth option is to wait for gatsby-image or gatsby-transformer-sharp to give developers an option to load images as is when in development. Perhaps under a flag

There's no one ideal option. Each of one these is a sort of compromise. You might have to pick the least worst option for your setup.

Btw, re:

the big killer is the image building on uncached builds (see #12857), which I think count as queries?

This is how I think it works:

• gatsby-source-filesystem - will get gatsby your images
• gatsby-transformer-sharp - will then parse your images and resize them

Both of these are separate processes, but parsing and resizing of images would be most expensive in terms of time and resource

[rori4]

The most annoying process is that when you change something in gatsby-ssr or gatsby-node, sometimes the Gatsby deletes the cache and needs to regenerate the images again!? WHY?
The change is not even remotely related to the images. At least keep them when the change is not relevant.
Honestly this is a very bad design. Its waste of resources and time. Please fix this.

[KyleAMathews]

Hey everyone! We've shipped an initial version of query on demand in 2.27.0 — upgrade & try it out!

GATSBY_EXPERIMENTAL_FAST_DEV=true gatsby develop

Read more about it + some other initiatives we're working on to speedup starting the dev server https://github.com/gatsbyjs/gatsby/blob/master/docs/reference/release-notes/v2.27/index.md

[muescha]

read more page can be found here: https://github.com/gatsbyjs/gatsby/blob/master/docs/docs/reference/release-notes/v2.27/index.md

[LekoArts]

This got shipped in 2.29

https://www.gatsbyjs.com/docs/reference/release-notes/v2.29