Wishlist for the next major version of Gatsby (v3)

[m-allanson]

Summary

This is a wishlist issue for thinking about what changes might go into the next major version of Gatsby .

This is NOT a V3 umbrella issue. There will be separate issue/project with subset of this list once enough ideas are gathered and we do research on impact, migration paths, engineering/documentation efforts etc of each of the items

Note that we have not started work on the next major version of Gatsby, and there is no timeline for when we’ll start, or when it will be available.

The list

This issue is intended to collect any features that might need a breaking change to implement or shift conventions in a big way (this doesn't necessarily need breaking changes technically as often times we can support old way and new way, but major releases is good platform to shift convention / soft deprecate APIs). Nothing on here is guaranteed to happen! It's much more likely that only subset of those ideas will be used for next major version and rest will need to wait for another one.

Breaking changes:

Current gatsby behaviour

• set loose: false in babel-preset-gatsby
• page context shape in graphql
• normalise path handling across the board (trailing vs non-trailing slashes, develop vs production)
• error instead of warn on a11y eslint rules?
• hide component path in componentChunkName

Gatsby APIs

• inference default behaviour for createTypes
• reduce accepted date formats, use date-fns instead of moment?
• remove loki (it served it's purpose as experiment, but it's tedious to maintain multiple node stores implementations in the long run, especially as we will start looking into out-of-process data stores - then we would have 3+ implementantions) (loki has been removed)
• remove APIs deprecated in v2 (core):
  • gatsby-config:
    • __experimentalThemes ( code )
  • actions:
    • deleteNodes action ( code with comments / v1->v2 migration guide )
    • Calling deleteNode with a nodeId argument ( code with comments )
    • Calling createNodeField with fieldName/fieldValue params ( code )
  • gatsby-node context / helpers
    • boundActionCreators ( deprecation code / jsdoc / gatsby helpers docs
    • hasNodeChange ( jsdoc / gatsby helper docs )
  • gatsby-browser
    • getResourcesForPathnameSync and getResourcesForPathname args (code with comments - deprecated in April 2019)
    • replaceComponentRenderer hook ( deprecation mark in jsdocs - this is one iffy, because we didn't provide deprecation warnings in runtime and only had info in jsdoc / gatsby-browser docs)
  • page template props:
    • pathContext (it was renamed to pageContext in v2) ( deprecation code )
  • page template / static graphql queries:
    • support for using graphql function without importing it from gatsby( code )
• remove APIs deprecated in v2 (gatsby-link):
  • remove push, replace, navigateTo functions ( code with deprecation messages )

Dependencies

• use Webpack 5
• use babel 8 ( Babel 8 Release Plan )
• switch back from @reach/router to react router (after the projects have merged. Maybe this won't be a breaking change)
• React v16.7/8 min support (for suspense)
• bump sharp version (requires Node 10+)
• Replace react-hot-loader with react-refresh (it has some behaviour differences + seems to require react >= 16.10 according to this comment - react-refresh + ReactDOM: hot reloading only works when bundling React facebook/react#17626 (comment) )
• drop Node 8 support (not clear which category to put this in - this affect not only core, but also plugins that we maintain, and we are mostly bitten by inability to bump various dependencies) (chore: bump node min version to 10.13.0 #22400)
• css-loader bump (core in on v1, there is v3 available) - this might also need rethinking if we could somehow make it easier to replace builtin rules, particularly ones that are then used by other plugins (in here css-loader rule defined by core is then used in plugins for sass, less, etc)

Unclear if breaking changes:

• First class typescript support (we already have some typescript support via gatsby-plugin-typescript, it's unclear if for users of that plugin adding built in support would cause problems or not)
• First class Preact support
• First class Jest support
• built-in CSP support
• Perf budgets for page-data and js bundles (unclear because it depends how it's implemented, ideally final result would be failing builds if budgets are exceeded which would be breaking change, but we could start with warnings and change that to hard fails in next major)
• support gatsby-(config/node/browser/ssr) files in package subdirectories (allowing us to have dist/build directories in our plugins instead of placing them in root of the package, which makes any cleanup messy) - TBD if we should make use of main or some custom package.json option

Deprecations

• unify createNode and deleteNode function signature ( createNode(node) vs deleteNode({ node }) right now) - we can still support current deleteNode signature, doesn't have to be breaking change
• deprecate createJob, setJob, endJob (v1)
• deprecate schema customization actions in souceNodes API (only allow it in createSchemaCustomization API): createFieldExtension, createTypes, addThirdPartySchema)
• possibly start deprecating sync getNode APIs to prepare us to move to the out of memory data store in the future

Major new APis

This are meant as brand new APIs that will enable completely new workflows and features

• GraphQL renderable type
• Low level API to add more various dependencies to pages:
  • js modules (would be needed by GraphQL renderable type)
  • media (images, fonts, etc) - to preload critical media resources before changing the page - this would just preload them, but wouldn't make use of them directly - only to fill browser cache)
  • plain requests - for some resources (i.e. JSON objects that could be shared by subsets of pages - could act as escape hatch/work around Gatsby's inability to have "StaticQuery" with variables)
• Plugin options validation
• Better API to support content source supporting delta updates (as opposed to doing full sync everytime). Current handling is messy requiring plugins like gatsby-source-contentful to touch and delete nodes. We could have nicer higher level APIs for this.
• More control over cache invalidation (right now we blow away entire cache for everything, because it's safest thing to do). Complex issue as we have multiple kinds of caches - we have shared cache (redux state) for all nodes, but individual key-value stores available per plugin)
• Any APIs that would better support i18n (research in progress, so no details here yet)

Shifting conventions

Those doesn't strictly need breaking changes, those are meant as alternative APIs for what we already support (with goal of deprecating old ways if they same level APIs - adding shortcuts for common workflows won't result in deprecated lower level APIs it will rely on)

• Less APIs using magic exports Wishlist for the next major version of Gatsby (v3) #15277 (comment)
• Support for creating collection of pages (i.e. /src/pages/[MarkdownRemark].js)
• Built in support for pagination via some query directives
• Move reporter to gatsby core (or even to separate package, i.e. gatsby-reporter)
• Filters always returning an array, rather than null in certain cases (fix(gatsby): context.nodeModel.runQuery should return [] instead of null #25885)

Large refactors

(high chance of introducing behaviour changes, so might be best to schedule those for next major version)

• Run queries on demand in develop
• better code organisation - it's extremely difficult to follow what are the effects of actions in core codebase. Sometimes it will be just reducer. Sometimes there will be additional listeners set up for particular action. Sometimes there will be listeners listening for all action types (*). Unless developer already knows what will (or should) happen it's extremely difficult to follow effects that actions have and recognise implications of making changes.

Motivation

We try to avoid breaking changes where possible, but sometimes they are necessary in order to implement or improve features. We want to batch any breaking changes into a major version release. This is a place to keep track of what those changes might be.

[swyxio]

yay! am i allowed to suggest things? 🤗

• first class typescript support eg in gatsby config.
  • maybe think about codegenning types while writing queries, like https://github.com/dotansimha/graphql-code-generator
• compile selected pages to serverless functions?
• first class dynamic route api
• more visible/easier logging (of the redux stuff, i know its not used much and we dont have to use redux devtools, and it doesnt have to be too deep into the internals, just asking if we can opt into more verbosity for debugging). separate issue from profiling, which i've heard kyle tried before.

i know it looks a bit like i should just use nextjs with the first 3 ideas. i just think they're really good ideas. if thats not in line with your vision of course you should decline it.. just figured i woudl put it up

[sidharthachatterjee]

First class TypeScript support was pretty popular on https://twitter.com/chatsidhartha/status/1155820333095505920?s=20 as well

[wardpeet]

• maybe think about codegenning types while writing queries, like dotansimha/graphql-code-generator

We're probably going to rewrite how we expose our plugin api to get this working. We already have support for this, just super verbose with importing typedefs yourself.

• first class dynamic route api

Is something we can already do, without bumping major.

[wardpeet]

I'm adding dropping node 8 & perhaps node 10 support (depending on the timing).

Also opting to upgrade all possible dependencies to their latest like eslint 6...

[vladar]

Another thing mentioned was deprecating createTypes action in souceNodes API (only allow it in createSchemaCustomization API)

[pvdz]

I think this should be in by default but since it's a breaking change we'll postpone it to the next major bump.
#18791

[freiksenet]

Remove automatic __typename adddition to queries.

[pieh]

Less magical APIs (no more using magic exports):

(just examples, not the actual design):

• gatsby-(ssr/node/browser) files:

  -exports.sourceNodes = () => {}
  +const { API } = require(`gatsby/node`)
  +API.registerSourceNodes(() => {})

• gatsby-config:

  -module.exports = {}
  +const setConfig = require(`gatsby/config`)
  +setConfig({})

• plugin declaration:

  - {
  -    resolve: 'some-plugin'
  -    options: { <someOptions> }
  - }
  + require(`some-plugin/config`)(<someOptions>)

  this one can be implemented already in user land, but that's not the convention

• page queries:

  -export const unimportantName = graphql` <someQuery> `
  +export default withPageQuery(graphql` <someQuery> `, pageTemplate)

  (and/or using something like usePageQuery hook)

• graphql fragments:

  -export const unimportantName = graphql` <someFragment> `
  +import { registerFragment, graphql } from "gatsby"
  +registerFragment(graphql` <someFragment> `)

theme here is:

• hard to leverage existing js tooling ( for example type definitions for the first 3) with those magic exports and making them work more like regular JS (even if it might still be backed by "magic") will provide features in IDEs like autocompletion etc.
• page queries aren't the most intuitive right now - you have completely separate import and it somehow pushes data to component and there is no link trace of that in user code

also first time reading gatsby site code without going through tutorial or docs is very confusing (how do you know the exports.createPages is special if you never heard of it - it must be imported somewhere else in code, right?), making API design so it's easier to connect the dots (or at least have proper API names let users know what to even look for in docs)

Nice thing about it that we can support both (probably deprecate current way if we chose to make this happen)

[pieh]

Support for creating collection of pages (i.e. /src/pages/[MarkdownRemark].js)

(would require thinking through how page paths are determined)

[pieh]

Built in support for pagination via some query directives

[ascorbic]

My wishlist for feature additions:

• First class TypeScript support
• First class Preact support
• First class Jest support

[Barsonax]

Seconding the first class typescript support

[blainekasten]

FastRefresh hot loader as default (and then required is updating minimum react semver range)

[blainekasten]

FWIW, I don't think we should do all of this now. This list is more turning into, "what are all the breaking changes we want to do." Doing all of these for V3 could be quite a lot of churn.

[pieh]

FastRefresh hot loader as default (and then required is updating minimum react semver range)

Added with react bump needed note

FWIW, I don't think we should do all of this now. This list is more turning into, "what are all the breaking changes we want to do." Doing all of these for V3 could be quite a lot of churn.

There is note:

Nothing on here is guaranteed to happen! It's much more likely that only subset of those ideas will be used for next major version and rest will need to wait for another one.

But probably need more visibility (screaming red color! :) ) in the description

[blainekasten]

I would also like to standardize our deprecation approach. Both in the code and in process. Right now we just have littered console warnings in various places which doesn't feel maintainable. If we deprecate a feature, in order to go back and actually remove it in the next major would require some significant sleuthing.

[blainekasten]

If we update the minimum semver of React, then we can modernize our internal components too. That could also be a minor update after v3 is launched.

[pieh]

I would also like to standardize our deprecation approach. Both in the code and in process. Right now we just have littered console warnings in various places which doesn't feel maintainable. If we deprecate a feature, in order to go back and actually remove it in the next major would require some significant sleuthing.

I know ... I spend few hours browsing source code to generate list of deprecated things that we will possibly remove

If we update the minimum semver of React, then we can modernize our internal components too. That could also be a minor update after v3 is launched.

Yeah - I don't think we would update components immediately - maybe just some critical ones that are pretty difficult to reason about right now (thinking of you <EnsureResources> - instead it would just enable us to do it sometime in the v3 lifecycle

[kaushalyap]

Please also focus on 10890
Good security practices should be baked in

[wardpeet]

We should make the reporter part of gatsby and not gatsby-cli. (i'll create an issue with more details)

import { reporter } from 'gatsby'

The reporter will emit events so gatsby-cli, gatsby-desktop, gatsby-cloud can listen to them. This will remove our redux store merging and remove the complexity of our code.

2. Drop core-js for modern builds. Core-js is bloating the bundle and is hardly necessary in a modern browser context. People should add core-js themselves or use manual polyfilling.

[TylerBarnes]

@freiksenet does GQL renderable type enable a way to improve DX around using gatsby-image? If it does, that could be a great V3 feature since I know people get stuck on writing out queries for that.

[vladar]

We should start validating and sanitizing node.internal.type (related: #20981 and #20596).

Also worth considering sanitizing node field names. At the moment node field name can be anything (for example field%%%). In GraphQL schema we transform it to field___ but it would be way better to sanitize on node level and maybe warn when this happens.

[mbrandau]

reduce accepted date formats, use date-fns instead of moment?

Not including moment in my production bundle would save me ~18 KB.
IMHO date formatting support should be dropped completely and left up to the user. I'm not using it, but right now can't get rid of the 18 KB.

[KyleAMathews]

@mbrandau moment is used in the graphql layer btw — not for browser JS. We don't include any date libraries by default

[mbrandau]

@KyleAMathews well, sorry for wasting your time and cluttering up this issue. I searched for moment everywhere but not in gatsby-browser.js 🤦‍♂️ Problem solved, totally my fault.

[nandorojo]

Manual code splitting (see #18689) would be great! This would let a CMS dynamically pick content, and the front-end would only import certain components based on the CMS content.

[tomasp1189]

Definitely agree with @nandorojo, manual code-splitting would be a great foundational change for sites with different components not to bulk their javascript bundles (where Gatsby's current template/page based splitting approach may not be ideal for performance reasons). But with the open (draft) PR #24903, is there a reason why it would be a breaking/major change instead of a minor release? cc @pieh

[pieh]

is there a reason why it would be a breaking/major change instead of a minor release?

Core feature itself will not have breaking changes - just additional APIs. I hope that various plugins will be able to build abstractions on top of it, so breaking changes would lie in plugins most likely (if those would use those APIs, it forces users to upgrade gatsby core version)

[smashercosmo]

@pieh

Support for creating collection of pages (i.e. /src/pages/[MarkdownRemark].js)

Do I understand correctly, it will allow for multiple markdown collections, that will have different node types in graphql schema, like PostMarkdownRemark, ArticleMarkdownRemark etc.?

[pvdz]

Fix filtering API to always return an array, never null: #25885

[ascorbic]

Deprecate long-running async operations inside most lifecycle APIs. Currently there is no standardised place where source plugins create watchers, websockets etc for updating content. We should stop allowing them in most APIs, maintaining them for onCreatePagesStatefully and perhaps a new early bootstrap API specifically for this (createWatchers?).

[LekoArts]

Remove behavior that template- files in src/pages are ignored (see #27174)

[jstormail2]

I’d be remiss if I didn’t throw in my two cents.

For me, the strengths of Gatsby are its source plugins, graphql data layer, image optimizations, and SEO. The drags on my productivity are long build times and lack of delta updates.

Of course, the other popular framework I’ve used often is Next.js. Their strengths are its SEO and incremental static regeneration (delta updates / preview). Their drags on my productivity are images (RFC proposal in progress) and sourcing data.

I’ve read a bit of hearsay around the internet about how it “feels like Next are rapidly developing new feature while Gatsby are focused on repaying their investors” and “their effort seems to be put on improving their enterprise offer, as oppose to the project itself”. I don’t mean any disrespect, but it’s important to acknowledge the perceived risks of vendor lock in with Gatsby Cloud. I’ve found the only issue about hosting my own preview server is closed, there aren’t many references about the refresh endpoint, much less information about securing my custom server.

I think the next big feature set for gatsby involves incremental builds, updates, and previews. I hope those features are open and accessible, like Gatsby.js

What I would love to see in the next version of gatsby:

• Docs about schema customization and building data relationships in graphql

• Better API to support content source supporting delta updates (as opposed to doing full sync everytime). Current handling is messy requiring plugins like gatsby-source-contentful to touch and delete nodes. We could have nicer higher level APIs for this.

• More control over cache invalidation (right now we blow away entire cache for everything, because it's safest thing to do). Complex issue as we have multiple kinds of caches - we have shared cache (redux state) for all nodes, but individual key-value stores available per plugin)

• Run queries on demand in develop

• First class typescript support

• Less magical APIs (no more using magic exports):

• i18n / multilingual

Anyway, I 💜 gatsby, and I can’t wait to see what v3 holds!

[jlarmstrongiv]

The summary talks about “Perf budgets”, but I would also like to see this issue addressed:

Worse performance results with Lighthouse v6 (?) #24332

Performance is a never ending battle, but Gatsby makes it a breeze. Keep up the good work!

[ascorbic]

Hey @jlarmstrongiv. That's not an issue that needs to wait for a major release. You can see the progress in there, so it should be fixed sooner.

[KyleAMathews]

@jstormail2 docs for setting up a preview server are here — https://www.gatsbyjs.com/docs/running-a-gatsby-preview-server/ — let us know if you need any more help there.

Appreciate your feedback!

[aaronadamsCA]

• Unreserve NODE_ENV. By convention this is (a) set by the environment, and (b) defaults to (the same behaviour as) NODE_ENV=development. This is a big beginner "gotcha" in Gatsby.

• Stop recommending .env.${activeEnv} and start recommending .env, to align with the recommended use of dotenv:

  https://www.npmjs.com/package/dotenv#should-i-have-multiple-env-files

  Should I have multiple .env files?

  No. We strongly recommend against having a "main" .env file and an "environment" .env file like .env.test. Your config should vary between deploys, and you should not be sharing values between environments.

• Together, the above two changes would allow Gatsby to call dotenv.config() itself, as many other Node packages do; and maybe even ship a .env.example that defines NODE_ENV=development and instructs users to define NODE_ENV in their production environment.

• Replace the wrapRootExport magic exports from gatsby-browser and gatsby-ssr with a common explicit API for providers.

[aaronadamsCA]

Since this is a wish list, I know this will be controversial, but:

No more unnecessary backticks (template literals that could just be strings). They increase learning curve, they decrease readability.

I already use quotes: ["error", "double", { avoidEscape: true }] to adjust this in my projects, and anyone who really felt passionately about "backticks everywhere" could (and probably already does) use a similar ESLint rule in their own projects.

But if Gatsby wants to welcome as many people as possible, I think newcomers to ES6 looking at Gatsby v3 code for the first time would be better greeted by `strings` that actually look like "strings".

[andrewagain]

I think switching back to react router is super important because of issues like this one; reach/router#175

There is currently no way in gatsby to block a navigation change to tell the user "Hey are you sure you want to navigate away, you have unsaved changes?"

[andrewagain]

Also I think this article is spot on about GraphQL being overly complicated for the task: https://jaredpalmer.com/gatsby-vs-nextjs

I was a much bigger fan of Gatsby v1 before all the GraphQL stuff got added. I think it's worth considering moving to a more nextjs-like approach where components can just have a getStaticProps function - so simple and easy to understand.

[TylerBarnes]

It'd be great if there was a default limit set for node list queries. It's problematic that folks will query for all nodes of a type on every page query for nodes in another type. It's never desireable to load up all posts that exist on every page that exists. Folks don't notice this when they start on new projects because it isn't a problem until your site grows and you have a ton of content. Ideally this would be very low by default, maybe 10, and you could intentionally set the limit to null to query all nodes in that type.

[KyleAMathews]

APIs are run serially — which is often what we want but some APIs should be run in parallel e.g. sourceNodes — if you have multiple source plugins hitting different APIs, you're primarily bound by the remote API so it would be faster to run these operations in parallel vs. sequentially.

[KyleAMathews]

Move runtime code under a .cache/runtime directory. It's gotten messy in there.

[neelkarma]

Use the module/nomodule pattern to load modern javascript for modern browsers but falling back to older javascript when an older browser is detected.

[malisbad]

Built in support for i18n in a straightforward and consistent way would be fantastic. There is a lot of inconsistency in the way that people get it to work, and the blog posts aren't particularly helpful when they're split between a bunch of different libraries that don't really jazz.

The recommended plugin demands quite a bit of work for updating layouts with separate files for each supported language, and it definitely makes DRY codebases more difficult.