Support Local TypeScript Plugins (outside of /plugins/ folder)

[phil-lgr]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

It should be possible to have local plugins written in TypeScript in any folders.

Right now it only works if the plugin is located in /plugins/

Related:

• https://www.gatsbyjs.com/docs/how-to/custom-configuration/typescript/#current-limitations
• https://www.gatsbyjs.com/docs/creating-a-local-plugin/
• RFC: Ahead of Time Compilation for `gatsby-config`/`gatsby-node` (TypeScript Support) #34613
• RFC: Ahead of Time Compilation for `gatsby-config`/`gatsby-node` (TypeScript Support) #34613 (reply in thread)

Reproduction Link

https://github.com/newrade/gatsby-repros

Steps to Reproduce

1. git clone https://github.com/newrade/gatsby-repros && cd gatsby-repros
2. yarn
3. cd packages/gatsby-site
4. yarn start

Expected Result

The plugin gatsby-local-plugin located in packages/gatsby-local-plugin, works as expected.

Actual Result

Have the following error:

 ERROR 

There was a problem loading plugin "/Users/phil/Code/gatsby-repros/packages/gatsby-local-plugin". Perhaps you need to install its package?
Use --verbose to see actual error.


 ERROR 

Failed to resolve /Users/phil/Code/gatsby-repros/packages/gatsby-local-plugin unreachable



  Error: unreachable
  
  - resolve-plugin.ts:95 resolvePlugin
    [gatsby-repros]/[gatsby]/src/bootstrap/load-plugins/resolve-plugin.ts:95:11
  
  - index.js:37 resolveTheme
    [gatsby-repros]/[gatsby]/src/bootstrap/load-themes/index.js:37:29
  
  - index.js:115 
    [gatsby-repros]/[gatsby]/src/bootstrap/load-themes/index.js:115:30
  
  - util.js:16 tryCatcher
    [gatsby-repros]/[bluebird]/js/release/util.js:16:23
  
  - reduce.js:166 Object.gotValue
    [gatsby-repros]/[bluebird]/js/release/reduce.js:166:18
  
  - reduce.js:155 Object.gotAccum
    [gatsby-repros]/[bluebird]/js/release/reduce.js:155:25
  
  - util.js:16 Object.tryCatcher
    [gatsby-repros]/[bluebird]/js/release/util.js:16:23
  
  - promise.js:547 Promise._settlePromiseFromHandler
    [gatsby-repros]/[bluebird]/js/release/promise.js:547:31
  
  - promise.js:604 Promise._settlePromise
    [gatsby-repros]/[bluebird]/js/release/promise.js:604:18
  
  - promise.js:641 Promise._settlePromiseCtx
    [gatsby-repros]/[bluebird]/js/release/promise.js:641:10
  
  - async.js:97 _drainQueueStep
    [gatsby-repros]/[bluebird]/js/release/async.js:97:12
  
  - async.js:86 _drainQueue
    [gatsby-repros]/[bluebird]/js/release/async.js:86:9
  
  - async.js:102 Async._drainQueues
    [gatsby-repros]/[bluebird]/js/release/async.js:102:5
  
  - async.js:15 Immediate.Async.drainQueues [as _onImmediate]
    [gatsby-repros]/[bluebird]/js/release/async.js:15:14
  
  - timers:466 processImmediate
    node:internal/timers:466:21
  

not finished load gatsby config - 0.043s

Environment

System:
    OS: macOS 12.1
    CPU: (8) arm64 Apple M1
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 16.14.0 - /var/folders/z3/1b10ld5s49n70q3n77_6_qch0000gn/T/yarn--1646265333774-0.9440956859973659/node
    Yarn: 1.22.11 - /var/folders/z3/1b10ld5s49n70q3n77_6_qch0000gn/T/yarn--1646265333774-0.9440956859973659/yarn
    npm: 8.3.1 - ~/.nvm/versions/node/v16.14.0/bin/npm
  Languages:
    Python: 2.7.18 - /usr/bin/python
  Browsers:
    Chrome: 99.0.4844.51
    Safari: 15.2

Config Flags

None.

[phil-lgr]

In compile-gatsby-files.js

function constructParcel(siteRoot) {
  return new _core.Parcel({
    entries: [`${siteRoot}/${gatsbyFileRegex}`, `${siteRoot}/plugins/**/${gatsbyFileRegex}`],
    defaultConfig: require.resolve(`gatsby-parcel-config`, {
      paths: [siteRoot]
    }),
    mode: `production`,
    targets: {
      root: {
        outputFormat: `commonjs`,
        includeNodeModules: false,
        sourceMap: false,
        engines: {
          node: `>= 14.15.0`
        },
        distDir: `${siteRoot}/${COMPILED_CACHE_DIR}`
      }
    },
    cacheDir: `${siteRoot}/${PARCEL_CACHE_DIR}`
  });
}

it's pretty clear that Parcel won't compile anything outside of /plugins/

[karlhorky]

For the error with import, you need to instruct Parcel to transpile ESM to CommonJS:

#34613 (reply in thread)

More details about the ESM incompatibility with Gatsby here:
#31599

[marvinjude]

Hi @phil-lgr,

Yarn Workspaces aren't supported yet .

You can still use local plugins outside the plugins directory.

[LekoArts]

Hi, thanks for the issue!

Two things are at play here:

• We won't be supporting the compilation of a complete workspace. Instructions for workspaces (e.g. yarn workspaces) will be added/will be made easier as it is not a Gatsby concern and packages should be compiled outside of Gatsby
• I don't want to support local plugins outside of the plugins folder. It's a good convention that limits the scope that we have to cover and long-term convention over flexbility will be better here

So with those things in mind this is a "won't fix" and I'll close this. I'll follow this up though with some updates to our current documentation to make this clearer.

[phil-lgr]

On my phone right now

Well, I find it counter intuitive to introduce dozen of new deps around parcel and only support transpiling in the plugins folder.. but maybe I don’t have all the knowledge that you do

I guess I’ll setup transpiling with tsc or webpack and then point Gatsby to it, in that regard v4 removes a possibility that we had in v3 but it’s fine, I’ll find a way.

I’m also curious to know why this step is not being done with webpack (not sure about introducing many parcel deps for just this compile step, while we have already all the webpack deps used in gatsby).

So yea, I disagree with the move but ultimately you know what’s best for the project, thanks for looking into it

Hi @phil-lgr,

Yarn Workspaces aren't supported yet .

You can still use local plugins outside the plugins directory.

while I used yarn workspaces in the repro, I don’t think it has something to do with that issue, I’m using a relative path and not a symlink

so yea, the docs for local plugins will have to state that local plugins not in /plugins/ will need to be transpiled in commonjs as the parcel setup won’t do it outside of that particular folder