dev: run image processing on demand

[pieh]

Summary

Don't process all images in develop mode and only do it when it's needed (requested by browser).

Motivation

Booting up development server on large sites can take several minutes, because we wait for:

• graphql query results
• and image processing to finish (which usually is more time/cpu intensive task of the two).

Those are needed to produce complete production builds, but we don't need all the images for development. We only need images that user is looking at and we can lazy generate them when browser requests them.

Details

This can be done by adjusting gatsby-plugin-sharp alone:

• In development, instead of creating and running image processing job as soon as possible (during query running), push the job entry into some kind of in-memory store - we will need all the information needed to perform job and also way to find job that produces given file.
• We will need to persist lazy jobs data. Because image processing would be lazy we have no guarantee that all images would be produced and we need to keep track of them.
• Hook into dev express server and intercept requests for image files that are not yet produced -> trigger job processing to get that file -> respond with freshly produced image.

For reference: this is split from #7348 (running queries on demand).

/cc @wardpeet

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[wardpeet]

i've put it to blocked as lot of efforts is already done but we need some kind of internal invalidation api so we know which nodes are valid.

cc @pieh 😛

[wardpeet]

Going to close this in favor of #7348